Orca: A New Open-Source Large Language Model Released

A new open-source large language model, named Orca, has been released. This model aims to bridge the gap between smaller, more accessible models and larger, more capable ones by incorporating insights from larger language models.

Training Methodology

Orca's development focused on mimicking the reasoning process of larger models. The model was trained on explanations generated by GPT-4, a proprietary large language model. This training involved teaching Orca to follow step-by-step thought processes and learn from detailed explanations, a method referred to as "explanation tuning." This approach allowed Orca to learn complex reasoning abilities by observing and imitating the outputs of more advanced models.

Performance and Capabilities

The research behind Orca indicates that it demonstrates significant improvements in complex reasoning tasks compared to other models of similar size. It has shown proficiency in tasks that require multi-step reasoning, such as those found in academic benchmarks like Big-Bench Hard and AGIEval. Orca's architecture and training enable it to tackle problems that typically challenge smaller models, achieving performance levels that approach those of much larger proprietary models.

In summary, Orca represents a novel approach to developing capable open-source language models. By utilizing explanation tuning and insights from advanced models like GPT-4, Orca has achieved enhanced reasoning abilities. This release offers a promising open-source alternative for tasks requiring sophisticated problem-solving.