Orca: Progressive Learning from Complex Explanation Traces of GPT-4

Orca, a 13-billion parameter model developed by Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi and Ahmed Awadallah, learns to imitate the reasoning process of large foundation models. It is trained on explanation traces and step-by-step thought processes from GPT-4, guided by teacher assistance from ChatGPT. The authors report that Orca surpasses Vicuna-13B by more than 100% on Big-Bench Hard and by 42% on AGIEval in zero-shot reasoning, reaches parity with ChatGPT on BBH, and trails GPT-4 on professional and academic exams such as the SAT, LSAT, GRE and GMAT.