GPT-Fathom: Evaluating LLMs under Aligned Settings

"Minecraft scenario: an engineer standing at the edge of a boat, holding a rope to fathom the depth of water."

(image generated by Midjourney V5.2)

---

By Shen Zheng^*, Yuyu Zhang^*, Yijie Zhu, Chenguang Xi, Pengyang Gao,

Xun Zhou, Kevin Chen-Chuan Chang

From ByteDance, University of Illinois at Urbana-Champaign

📃 Paper | 🐦 Twitter

Recently, with the rapid progress of large language models (LLMs), more and more LLMs (both open-source and closed-source) are making their debut. There's a burgeoning demand for building an open, systematic, rigorous and reproducible evaluation suite to assess and track their capabilities and limitations. Although there already exist many LLM leaderboards, some key limitations are yet to be addressed (discussed in details here). For example, existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may inadvertently encourage cherry-picking favored settings and prompts for better results.

To dispel the mist among LLM evaluations, we introduce GPT-Fathom, an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals. We systematically evaluate 10+ leading LLMs as well as OpenAI's legacy models on 20+ curated benchmarks across 7 capability categories, all under aligned settings. GPT-Fathom serves as a standard gauge to pinpoint the position of emerging LLMs, aiming to help the community measure and bridge the gap with leading LLMs. We also explore the impacts of model sensitivity on evaluation results with extensive experiments of various settings.

Our retrospective study on OpenAI's earlier models, including davinci (GPT-3), davinci-instruct-beta (InstructGPT), text-davinci-001, code-davinci-002, text-davinci-002 and text-davinci-003, offers valuable insights into the evolutionary path from GPT-3 to GPT-4. Our work is time-sensitive due to the scheduled deprecation of those legacy models announced by OpenAI.

Currently, the community is eager to know how GPT-3 progressively improves to GPT-4, including technical details like whether adding code data improves LLM's reasoning capability, which aspects of LLM capability can be improved by SFT and RLHF, how much is the alignment tax, etc. Our analysis sheds light on many of these questions, aiming to improve the transparency of advanced LLMs.

🚀 Updates

• [2024/03/31] We released model predictions on [Drive].

• [2023/12/18] We updated the evaluation results of the latest OpenAI models gpt-3.5-turbo-1106 and gpt-4-1106-preview (GPT-4 Turbo) in our main table. We observe that both models significantly regress on some benchmarks, such as DROP and BBH. Meanwhile, as revealed by our previous evaluations of the Web-based GPT-4, the new gpt-4-1106-preview (GPT-4 Turbo) model catches up on HumanEval with a striking 84.6 pass@1 score.

• [2023/11/20] We updated the evaluation results of the entire LLaMA and Llama 2 family, including models of various sizes. Refer to the table below for complete results.

• [2023/10/23] We updated the evaluation results of the latest OpenAI model gpt-3.5-turbo-instruct-0914, which slightly outperforms gpt-3.5-turbo-0613 on most benchmarks (except for some "Coding" and "Safety" tasks). [Twitter]

📖 Table of Contents

• 📊 Leaderboard
• 🔍 Analysis and Insights
• ☕ Quick Start
• 💡 FAQs
• 📍 Roadmap
• 🤝 Acknowledgements
• 📝 Citation

📊 Leaderboard

Main evaluation results of GPT-Fathom.

Note that GPT-Fathom supports various settings for evaluation. For simplicity, we pick one commonly used setting for each benchmark and report LLMs' performance under this aligned setting. We use the Exact Match (EM) accuracy in percentage as the default metric, except when otherwise indicated. For clarity, we also report the number of "shots" used in prompts and whether CoT prompting is used. For the AGIEval benchmark, we use the official few-shot (3-5 shots) setting. For PaLM 2-L, since its API access is not currently available yet, we instead cite the numbers from PaLM 2. Numbers that are not from our own experiments are shown in brackets. Numbers with * are obtained from optimized prompts.

According to the table above, we plot radar charts to visualize the capabilities of evaluated LLMs.

For Claude 2, we're still on the waitlist of API access, so we evaluate OpenAI's latest models (including Web-version GPT-3.5 and GPT-4, evaluated in 2023/09) under the same settings used by Claude 2.

The complete results of LLaMA and Llama 2 family are summarized below.

🔍 Analysis and Insights

History of OpenAI models

We evaluate 10+ OpenAI models, which have been actively evolving since GPT-3 was released in 2020. To tease out the relationships between these models, we illustrate OpenAI's evolutionary path from GPT-3 to GPT-4 in the following figure. We omit deprecated legacy models such as code-davinci-001 and only list the models evaluated in GPT-Fathom.

Caveat: As claimed in OpenAI's model index, their models generally used the best available datasets at the time of training. As a result, our analysis on OpenAI's models may not serve as a rigorous ablation study.

Impacts of SFT / RLHF on LLM capabilities

SFT - davinci (GPT-3) + SFT -> text-davinci-001: SFT significantly improves performance on most benchmarks. - code-davinci-002 + SFT -> text-davinci-002: SFT on stronger base model shows the opposite effect, which underperforms the base model on most benchmarks (except for coding tasks, which is further discussed below). - A similar trend is observed on open-source models: SFT boosts the performance of LLaMA-65B on MMLU (refer to reported performance of SFT model LLaMA-I in the original paper), however, all SFT models on top of Llama 2-70B in Open LLM Leaderboard show only marginal improvements on MMLU.

RLHF - text-davinci-002 + PPO -> text-davinci-003: RLHF model has comparable or slightly worse performance on most benchmarks compared to the base model code-davinci-002 (except for coding tasks). - A similar trend is observed on open-source models: the official RLHF version of Llama 2-70B performs slightly worse than its base model on MMLU, HellaSwag and ARC-c, as reported in Open LLM Leaderboard.

SFT / RLHF impacts on coding capabilities

• SFT and RLHF models excel in pass@1, compared to the base model, while slightly underperforms in pass@100.
• We interpret these intriguing results as follows:
• pass@100 gauges intrinsic ability to solve a coding problem, while pass@1 emphasizes the one-take bug-free coding capability.
• SFT and RLHF models still have to pay the alignment tax, showing a minor performance drop in pass@100, which aligns with their slightly worse performance across other tasks.
• SFT and RLHF can effectively distill the capability of pass@100 into pass@1, which transfers inherent problem-solving skills to one-take bug-free coding capability.
• As expected, smaller models like code-cushman-001 (Codex-12B) and gpt-3.5-turbo-0301 have limited intrinsic capability in terms of pass@100. However, their pass@1 can be dramatically improved by SFT and RLHF. This is good news for research on low-cost small-size LLMs.
• GPT-4 can be even more powerful! We observe that state-of-the-art LLMs like GPT-4 can inherently tackle complicated tasks, although sometimes succeed after many sampling trials (example). We anticipate that LLMs have yet to reach their full potential, since SFT and RLHF can consistently enhance LLMs by translating their intrinsic capabilities into one-take pass rates on reasoning-intensive tasks.

Impacts of pretraining with code data

• code-cushman-001 (Codex-12B) represents OpenAI's preliminary effort to train LLMs on code data, which shows decent coding performance despite its modest model size.
• code-davinci-002, which is pretrained on a mixture of text and code data, significantly outperforms the previous GPT-3 Series across various tasks. On some reasoning tasks such as LAMBADA and BBH, code-davinci-002 even beats the latest gpt-3.5-turbo-0613 model. This suggests that incorporating code data into LLM pretraining could universally elevate its potential, particularly in the capability of reasoning.

Seesaw phenomenon of LLM capabilities

• We observe a so-called "seesaw phenomenon" by comparing the OpenAI API models dated in 2023/03 and 2023/06, where certain capabilities exhibit improvement, while a few other capabilities clearly regress.
• For instance, compared to gpt-3.5-turbo-0301, the upgraded model gpt-3.5-turbo-0613 excels on HumanEval (53.9 -> 80.0) but dramatically degrades on MATH (32.0 -> 15.0). gpt-4-0613 outperforms gpt-4-0314 on DROP (78.7 -> 87.2) but plummets on MGSM (82.2 -> 68.7). OpenAI also admits that when they release a new model, while the majority of metrics have improved, there may be some tasks where the performance gets worse.
• The seesaw phenomenon presents a universal challenge that may obstruct LLM's path towards AGI, which necessitates a model that excels across all types of tasks. We strongly encourage the research community to pay more attention to this challenge.

OpenAI vs. non-OpenAI LLMs

• PaLM 2-L clearly outperforms gpt-3.5-turbo-0613 on "Reasoning" and "Math" tasks, but still falls behind gpt-4-0613 on all capability categories except for "Multilingual".
• Claude 2 stands as the leading non-OpenAI model, which achieves comparable or slightly worse performance on a variety of benchmarks, compared to latest OpenAI models.

Closed-source vs. open-source LLMs

• Llama 2-70B outperforms LLaMA-65B on most benchmarks, especially on "Reasoning" and "Comprehension" tasks.
• Llama 2-70B surpasses gpt-3.5-turbo-0613 on "Comprehension" tasks, but still underperforms on other capability dimensions, especially "Math", "Coding" and "Multilingual". We strongly encourage the open-source community to improve these capabilities of open-source LLMs.

OpenAI API-based vs. Web-version LLMs

• The dated API models gpt-3.5-turbo-0613 and gpt-4-0613 consistently perform slightly better than their front-end counterparts, i.e., Web-version GPT-3.5 (serving ChatGPT) and Web-version GPT-4.
• The latest GPT-4 Advanced Data Analysis (previously known as Code Interpreter) boosts the coding performance, which achieves a striking 85.2 pass@1 score on HumanEval.

Impacts of model sensitivity

Prompt sensitivity - Open-source models like LLaMA-65B and Llama 2-70B are prompt-sensitive. For instance, a minor tweak of the prompt template results in a significant drop of Llama 2-70B's score on TriviaQA (74.0 -> 55.5). - OpenAI's leading models are fairly robust against alteration of prompts (without changing the inherent meaning).

CoT prompting - The influence of CoT prompting varies across benchmarks. - On tasks that are knowledge-intensive, like MMLU, CoT has minimal or even slightly negative impact on performance. However, for reasoning-intensive tasks, such as BBH and GSM8K, CoT prompting markedly enhances LLM performance. For instance, on the GSM8K with 8-shot examples, gpt-4-0314 dramatically boosts its score when CoT prompting is employed (45.7 -> 92.1).

Number of "shots" - Benchmark performance generally improves with an increased number of "shots", however, the improvement rate quickly shrinks beyond 1-shot, particularly for stronger models. - For instance, gpt-4-0314 achieves 94.9 on ARC-c with 1-shot example, and only marginally increases to 95.6 with 25-shot examples. This indicates that 1-shot example typically works well for most tasks, which aligns with our primary evaluation setting.

Sampling variance - LLMs (especially base models) tend to underperform with a higher temperature $T$. On codi