ESM-bench Leaderboard

Can AI agents understand the physics behind Earth System Model code?

Generated: 2026-04-25 01:20  |  Mode: Direct prompting (single API call, no scaffold)

Track A: Repository Localization

Given a physics description, can the agent find the right file(s)?

File Localization F1 by Model

claude-opus-4-7 (noah_mp)
0.858
gpt-5.4-2026-03-05 (noah_mp)
0.826
qwen.qwen3-coder-480b-a35b-v1:...
0.804
claude-sonnet-4-6 (noah_mp)
0.779
gpt-4.1-2025-04-14 (noah_mp)
0.772
mistral-large-3-675b-instruct ...
0.745
qwen.qwen3-coder-480b-a35b-v1:...
0.740
deepseek.v3.2 (clm5)
0.723
claude-opus-4-7 (clm5)
0.714
qwen.qwen3-235b-a22b-2507-v1:0...
0.713
gpt-5.4-2026-03-05 (clm5)
0.698
deepseek.v3.2 (noah_mp)
0.687
# Model Repo Tasks File F1 iF1 score for file-level prediction. Did the model identify the correct source file(s) to modify? File Prec File Recall Exact Set iDid the model predict exactly the right set of files (no extras, no missing)?
1 claude-opus-4-7 noah_mp 39 0.858 0.872 0.866 77% review
2 gpt-5.4-2026-03-05 noah_mp 39 0.826 0.842 0.841 69% review
3 qwen.qwen3-coder-480b-a35b-v1:0 clm5 11 0.804 0.791 0.905 45% review
4 claude-sonnet-4-6 noah_mp 39 0.779 0.799 0.786 69% review
5 gpt-4.1-2025-04-14 noah_mp 39 0.772 0.795 0.777 67% review
6 mistral-large-3-675b-instruct noah_mp 39 0.745 0.756 0.751 64% review
7 qwen.qwen3-coder-480b-a35b-v1:0 noah_mp 39 0.740 0.718 0.828 54% review
8 deepseek.v3.2 clm5 11 0.723 0.758 0.799 45% review
9 claude-opus-4-7 clm5 11 0.714 0.670 0.886 45% review
10 qwen.qwen3-235b-a22b-2507-v1:0 noah_mp 39 0.713 0.693 0.802 51% review
11 gpt-5.4-2026-03-05 clm5 11 0.698 0.773 0.799 36% review
12 deepseek.v3.2 noah_mp 39 0.687 0.664 0.777 44% review
13 o4-mini-2025-04-16 noah_mp 39 0.685 0.692 0.696 59% review
14 o4-mini-2025-04-16 clm5 11 0.674 0.636 0.788 36% review
15 claude-sonnet-4-6 clm5 11 0.674 0.667 0.758 36% review
16 gpt-4.1-2025-04-14 clm5 11 0.556 0.591 0.606 36% review
17 mistral-large-3-675b-instruct clm5 11 0.556 0.518 0.750 27% review
18 qwen.qwen3-235b-a22b-2507-v1:0 clm5 11 0.545 0.503 0.724 27% review

Track B: Physics-Aware Patch Synthesis

Given the correct file(s), can the agent produce a physically valid code fix?

Best F1 by Model (Track B)

claude-opus-4-7
0.312
v4
claude-sonnet-4-6
0.175
v3
gpt-4.1-2025-04-14
0.100
v2
mistral-large-3-675b-inst...
0.017
v2
nova-pro-v1:0
0.013
v2
llama3-3-70b-instruct-v1:...
0.012
v2

Category Difficulty (v3 claude-opus-4-7)

PBF = bug fixes (easiest) → PSS = scheme switches (hardest)

PBF
0.302
PO
0.200
PRM
0.083
PSS
0.065
# Model Ver iTask version. v2=truncated source, v3=full source + physics hints, v4=full source + code-level oracle hints (L1/L2/L3). Task Group iSubset being evaluated, such as a codebase or a future nested tier like noah_mp/implementation. RevCov iTask-level review coverage. The fraction of unique tasks in this run that have at least one ingested human review. ReviewPass@k iTask-level review success: for each reviewed task, did at least one reviewed sample avoid a zero on every 0/1/2 rubric item? Rubric iMean normalized rubric score across reviewed submissions. 1.0 means all six rubric items scored 2. Parse iFraction of responses containing a valid unified diff. This is the cheapest objective validity check, not the final benchmark score. Exact iExact match after normalization. The strictest metric — did the model reproduce the developer's exact code change? F1 iToken-level F1 between predicted and GT diff lines. This is a structural proxy used in the current pilot, not a substitute for review-based benchmark scoring. Prec iPrecision: of the lines the model produced, what fraction appear in the GT? High precision = few false additions. Recall iRecall: of the GT lines, what fraction did the model produce? High recall = few missed changes. File iFile accuracy: did the model modify the correct file(s)? Basic sanity check — the model must at least find the right file. PBF iAverage F1 on Physics Bug Fix tasks (22 tasks). Bug fixes in existing physics code — the most solvable category. PO iAverage F1 on Parameter Optimization tasks (2 tasks). Adjusting physical constants to match observations. PRM iAverage F1 on Process Representation Modification tasks (4 tasks). Adding or modifying physics parameterizations — often requires new code. PSS iAverage F1 on Parameterization Scheme Switching tasks (5 tasks). Implementing alternative physics schemes — the hardest category, often requires writing new subroutines. Hint iPrompt hint level. v4: L1=baseline (description+file), L2=oracle localization (+subroutine/line), L3=oracle+expert (+deleted lines+Fortran idioms). v3: L0=minimal, L1=physics constraints, L2=commit msg.
1 claude-opus-4-7 v4 noah_mp — — — 100% 0% 0.312 0.558 0.248 100% 0.312 0.000 0.000 0.000 3 review
2 claude-opus-4-7 v3 noah_mp — — — 100% 0% 0.241 0.427 0.197 100% 0.302 0.200 0.083 0.065 0 review
3 claude-opus-4-7 v3 noah_mp — — — 98% 0% 0.223 0.367 0.182 98% 0.330 0.095 0.108 0.066 0 review
4 claude-opus-4-7 v4 noah_mp — — — 100% 0% 0.212 0.498 0.178 100% 0.212 0.000 0.000 0.000 2 review
5 claude-opus-4-7 v4 noah_mp — — — 100% 0% 0.209 0.512 0.163 100% 0.209 0.000 0.000 0.000 1 review
6 claude-opus-4-7 v3 noah_mp — — — 94% 0% 0.208 0.336 0.175 94% 0.322 0.095 0.070 0.047 1 review
7 claude-opus-4-7 v3 noah_mp — — — 94% 0% 0.194 0.326 0.164 94% 0.299 0.074 0.093 0.029 2 review
8 claude-opus-4-7 v2 clm5 — — — 98% 0% 0.188 0.294 0.173 98% 0.231 0.135 0.155 0.209 0
9 claude-sonnet-4-6 v3 noah_mp — — — 88% 0% 0.175 0.299 0.148 88% 0.244 0.145 0.102 0.058 0 review
10 claude-sonnet-4-6 v2 clm5 — — — 89% 0% 0.108 0.192 0.103 89% 0.105 0.126 0.088 0.136 0
11 gpt-4.1-2025-04-14 v2 clm5 — — — 71% 0% 0.100 0.107 0.129 71% 0.072 0.067 0.117 0.126 0
12 claude-opus-4-7 v2 mom6 — — — 100% 0% 0.098 0.322 0.072 100% 0.148 0.025 0.083 0.074 0
13 claude-opus-4-7 v3 clm5 — — — 99% 0% 0.094 0.338 0.069 99% 0.129 0.081 0.069 0.134 0 review
14 claude-opus-4-7 v2 noah_mp — — — 97% 0% 0.067 0.104 0.060 97% 0.094 0.000 0.007 0.024 0 review
15 claude-opus-4-7 v2 noah_mp — — — 97% 0% 0.057 0.083 0.052 97% 0.078 0.000 0.025 0.011 1 review
16 claude-opus-4-7 v2 noah_mp — — — 97% 0% 0.057 0.076 0.052 97% 0.078 0.000 0.018 0.016 2
17 claude-sonnet-4-6 v2 noah_mp — — — 97% 0% 0.050 0.061 0.048 97% 0.073 0.000 0.013 0.000 0
18 gpt-4.1-2025-04-14 v2 mom6 — — — 70% 0% 0.042 0.133 0.044 70% 0.050 0.009 0.051 0.017 0
19 gpt-4.1-2025-04-14 v2 noah_mp — — — 100% 0% 0.039 0.028 0.080 100% 0.055 0.000 0.009 0.010 0
20 mistral-large-3-675b-instruct v2 noah_mp — — — 100% 0% 0.017 0.025 0.020 100% 0.021 0.000 0.015 0.011 0
21 nova-pro-v1:0 v2 noah_mp — — — 100% 0% 0.013 0.020 0.010 88% 0.013 0.000 0.003 0.023 0
22 llama3-3-70b-instruct-v1:0 v2 noah_mp — — — 100% 0% 0.012 0.019 0.010 100% 0.012 0.000 0.006 0.017 0
23 gpt-4.1-2025-04-14 v3 noah_mp — — — 0% 0% 0.000 0.000 0.000 0% 0.000 0.000 0.000 0.000 0
Metrics shown: RevCov, ReviewPass@k, and Rubric come from ingested human review exports when available. Parse, Exact, F1, Precision, Recall are diff-structure proxies retained for the current pilot. Category breakdown (PBF/PO/PRM/PSS) shows difficulty gradient. Executable audits remain a future second tier. See paper §4 for the complete evaluation framework.