Research benchmark · Cycle 1 beta · October 13, 2026
Using Real-World Judicial Outcomes to Evaluate AI Legal Reasoning
LegalForecastBench is an effort to develop a new, challenging benchmark that uses the signal from judicial decisions to evaluate frontier models on high-level legal tasks against an objective real-world ground truth. Models receive the same briefs a federal judge received when ruling on a motion to dismiss, and are asked to estimate the probability that the judge dismissed each claim challenged by a defendant.
In partnership with LegalQuants.
Working on legal reasoning evaluations? I'd welcome feedback on the task design, calibration results, and next evaluation cycle. Get in touch →
The Cycle 1 cohort
91
Federal cases
387
Scored units
Claim × defendant forecasts
21
Model configurations
Ranked this cycle
73.6%
Dismissal base rate
Units fully dismissed
Results
Forecast quality by model
The beta includes 91 cases and 387 scored claim–defendant units. Of the 21 frontier models tested, GPT-6 Sol has the lowest micro-Brier loss (0.118) and the highest accuracy (84.5%).
Ranked by micro Brier. Lower is better. Click any column to re-sort.
- 1GPT-6 SolOpenAI$11.10◆ Frontier0.1176Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: April 20, 2026 (knowledge cutoff)
- 2Claude Fable 5.1Anthropic$83.180.1288Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: June 2026 (training-data cutoff)
- 3Grok 4.6SpaceXAI$15.720.1337Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: February 1, 2026 (knowledge cutoff)
- 4Claude Opus 5Anthropic$45.890.1399Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: May 2026 (training-data cutoff)
- 5GPT-5.6 SolOpenAI$24.910.1401Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: February 16, 2026 (knowledge cutoff)
- 6Kimi K3Moonshot$16.870.1413Micro BrierQualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published
- 7GPT-5.6 LunaOpenAI$1.48◆ Frontier0.1432Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: February 16, 2026 (knowledge cutoff)
- 8Claude Opus 5.5Anthropic$20.590.1454Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: June 2026 (training-data cutoff)
- 9GPT-6.1 SolOpenAI$10.85◆ Frontier0.1495Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: April 30, 2026 (knowledge cutoff)
- 10GPT-6 AstraOpenAI$59.740.1522Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: April 30, 2026 (knowledge cutoff)
- 11Gemini 3.8 FlashGoogle$35.610.1533Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: March 2026 (knowledge cutoff)
- 12Gemini 4 ArgonGoogle$25.020.1539Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: March 2026 (knowledge cutoff)
- 13Ultramarine 1 FlashPaper Instruments$1.35◆ Frontier0.1550Micro BrierQualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published
- 14Grok 4.7SpaceXAI$33.000.1583Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: May 2026 (knowledge cutoff)
- 15Claude Sonnet 5Anthropic$24.680.1585Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: January 2026 (training-data cutoff)
- 16Mistral Large 4Mistral$16.650.1596Micro BrierQualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published
- 17GPT-6 LunaOpenAI$0.63◆ Frontier0.1637Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: May 18, 2026 (knowledge cutoff)
- 18Muse WonderMeta$39.380.1654Micro BrierQualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published
- 19Claude Sonnet 5.5Anthropic$6.990.1730Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: June 2026 (training-data cutoff)
- 20Muse Spark 1.3Meta$8.780.1812Micro BrierQualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published
- 21Gemini 3.1 Pro PreviewGoogle$27.730.1834Micro BrierEligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: January 2025 (knowledge cutoff)
| Rank by micro Brier, the headline metric. Tied scores share a rank. | Model | Headline. Mean squared error of the dismissal probability across all 387 units; every unit weighs the same. Lower is better. | Sensitivity analysis. Averages each case's Brier first, so a case with many claims counts the same as a case with one. Lower is better. | Share of units where a probability of at least 50% matched dismissal. Ignores confidence. | Wrong predictions among those at least 90% confident either way. Scored units, not independent cases. | Estimated standard-rate API cost for the 91-case cohort, reconstructed from recorded token usage. Each model page states its cost basis. | Whether the training data/knowledge cutoff is before the decisions we used to test. Hover a badge for the model's cutoff. |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Sol OpenAIPareto frontier for: Micro Brier, Equal-case Brier, Unit accuracy. On the observed cost/quality frontier: no other model is both cheaper and at least as good, and this model strictly improves over others on at least one dimension. (Note that the improvement does not have to be statistically significant.) | 0.1176 | 0.1295 | 84.5% 327/387 | 8 of 169 | $11.10 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: April 20, 2026 (knowledge cutoff) |
| 2 | Claude Fable 5.1 Anthropic | 0.1288 | 0.1365 | 82.2% 318/387 | 0 of 81 | $83.18 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: June 2026 (training-data cutoff) |
| 3 | Grok 4.6 SpaceXAI | 0.1337 | 0.1348 | 80.6% 312/387 | 1 of 112 | $15.72 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: February 1, 2026 (knowledge cutoff) |
| 4 | Claude Opus 5 Anthropic | 0.1399 | 0.1355 | 80.1% 310/387 | 4 of 121 | $45.89 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: May 2026 (training-data cutoff) |
| 5 | GPT-5.6 Sol OpenAI | 0.1401 | 0.1332 | 80.9% 313/387 | 26 of 228 | $24.91 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: February 16, 2026 (knowledge cutoff) |
| 6 | Kimi K3 Moonshot | 0.1413 | 0.1537 | 78.3% 303/387 | 1 of 92 | $16.87 | Qualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published |
| 7 | GPT-5.6 Luna OpenAIPareto frontier for: Micro Brier, Equal-case Brier, Unit accuracy. On the observed cost/quality frontier: no other model is both cheaper and at least as good, and this model strictly improves over others on at least one dimension. (Note that the improvement does not have to be statistically significant.) | 0.1432 | 0.1508 | 80.9% 313/387 | 20 of 182 | $1.48 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: February 16, 2026 (knowledge cutoff) |
| 8 | Claude Opus 5.5 Anthropic | 0.1454 | 0.1476 | 77.3% 299/387 | 0 of 65 | $20.59 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: June 2026 (training-data cutoff) |
| 9 | GPT-6.1 Sol OpenAIPareto frontier for: Equal-case Brier. On the observed cost/quality frontier: no other model is both cheaper and at least as good, and this model strictly improves over others on at least one dimension. (Note that the improvement does not have to be statistically significant.) | 0.1495 | 0.1389 | 78.6% 304/387 | 18 of 199 | $10.85 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: April 30, 2026 (knowledge cutoff) |
| 10 | GPT-6 Astra OpenAI | 0.1522 | 0.1427 | 78.8% 305/387 | 21 of 207 | $59.74 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: April 30, 2026 (knowledge cutoff) |
| 11 | Gemini 3.8 Flash Google | 0.1533 | 0.1390 | 80.1% 310/387 | 35 of 276 | $35.61 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: March 2026 (knowledge cutoff) |
| 12 | Gemini 4 Argon Google | 0.1539 | 0.1482 | 77.0% 298/387 | 24 of 225 | $25.02 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: March 2026 (knowledge cutoff) |
| 13 | Ultramarine 1 Flash Paper InstrumentsPareto frontier for: Micro Brier, Equal-case Brier. On the observed cost/quality frontier: no other model is both cheaper and at least as good, and this model strictly improves over others on at least one dimension. (Note that the improvement does not have to be statistically significant.) | 0.1550 | 0.1721 | 75.5% 292/387 | 1 of 77 | $1.35 | Qualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published |
| 14 | Grok 4.7 SpaceXAI | 0.1583 | 0.1645 | 77.0% 298/387 | 4 of 98 | $33.00 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: May 2026 (knowledge cutoff) |
| 15 | Claude Sonnet 5 Anthropic | 0.1585 | 0.1596 | 75.7% 293/387 | 5 of 60 | $24.68 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: January 2026 (training-data cutoff) |
| 16 | Mistral Large 4 Mistral | 0.1596 | 0.1561 | 76.0% 294/387 | 7 of 128 | $16.65 | Qualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published |
| 17 | GPT-6 Luna OpenAIPareto frontier for: Micro Brier, Equal-case Brier, Unit accuracy. On the observed cost/quality frontier: no other model is both cheaper and at least as good, and this model strictly improves over others on at least one dimension. (Note that the improvement does not have to be statistically significant.) | 0.1637 | 0.1738 | 76.5% 296/387 | 19 of 183 | $0.63 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: May 18, 2026 (knowledge cutoff) |
| 18 | Muse Wonder Meta | 0.1654 | 0.1690 | 77.3% 299/387 | 10 of 118 | $39.38 | Qualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published |
| 19 | Claude Sonnet 5.5 Anthropic | 0.1730 | 0.1718 | 76.2% 295/387 | 5 of 63 | $6.99 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: June 2026 (training-data cutoff) |
| 20 | Muse Spark 1.3 Meta | 0.1812 | 0.1784 | 72.1% 279/387 | 15 of 135 | $8.78 | Qualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published |
| 21 | Gemini 3.1 Pro Preview Google | 0.1834 | 0.1808 | 78.0% 302/387 | 57 of 320 | $27.73 | Eligible: the provider-reported training data/knowledge cutoff is before the earliest decision we used to test; a month-only cutoff counts as the first day of that month.Cutoff: January 2025 (knowledge cutoff) |
Qualified comparisons
Cost evidence (JSON) →- Kimi K3: cutoff not published
- Muse Spark 1.3: cutoff not published
- Muse Wonder: cutoff not published
- Mistral Large 4: cutoff not published
- Ultramarine 1 Flash: cutoff not published
Rail tick: constant base-rate forecast (0.1941)# is the micro Brier rank, whatever the sortHover a column name for its definitionEligible: training data/knowledge cutoff is before the decisions we used to test
Cost and quality
What each point of accuracy costs
The dashed line is the Pareto frontier: no one model was both cheaper and more accurate.
Headline metric
Calibration at the extremes
When models are sure, are they right?
We noticed that some model families made sharper predictions, but proved to be poorly calibrated when they expressed high confidence. For example, of the predictions where GPT-6 Astra expressed at least 90% confidence, its average confidence was 95.5% but its actual accuracy was only 89.9%. That made Astra underperform GPT-6 Sol, where confidence and accuracy were much more closely matched.
This chart compares models' confidence (on the predictions with at least 90% confidence) with their accuracy. It illustrates one noteworthy trend between model families: OpenAI and Gemini models erred on the side of overconfidence, Anthropic and SpaceXAI on the side of underconfidence.
Share of high-confidence predictions that were rightMisses / predictions ≥90%
- Gemini 3.1 Pro PreviewGemini 3.1 Pro Preview: mean stated confidence 95.4%, realized accuracy 82.2% on 320 predictions57 / 320 (83% of units)
- GPT-5.6 SolGPT-5.6 Sol: mean stated confidence 97.0%, realized accuracy 88.6% on 228 predictions26 / 228 (59% of units)
- Gemini 3.8 FlashGemini 3.8 Flash: mean stated confidence 95.2%, realized accuracy 87.3% on 276 predictions35 / 276 (71% of units)
- GPT-5.6 LunaGPT-5.6 Luna: mean stated confidence 94.9%, realized accuracy 89.0% on 182 predictions20 / 182 (47% of units)
- GPT-6 LunaGPT-6 Luna: mean stated confidence 95.4%, realized accuracy 89.6% on 183 predictions19 / 183 (47% of units)
- Gemini 4 ArgonGemini 4 Argon: mean stated confidence 95.1%, realized accuracy 89.3% on 225 predictions24 / 225 (58% of units)
- GPT-6 AstraGPT-6 Astra: mean stated confidence 95.5%, realized accuracy 89.9% on 207 predictions21 / 207 (53% of units)
- GPT-6.1 SolGPT-6.1 Sol: mean stated confidence 95.7%, realized accuracy 91.0% on 199 predictions18 / 199 (51% of units)
- Muse Spark 1.3Muse Spark 1.3: mean stated confidence 93.5%, realized accuracy 88.9% on 135 predictions15 / 135 (35% of units)
- Muse WonderMuse Wonder: mean stated confidence 93.4%, realized accuracy 91.5% on 118 predictions10 / 118 (30% of units)
- Claude Sonnet 5Claude Sonnet 5: mean stated confidence 92.6%, realized accuracy 91.7% on 60 predictions5 / 60 (16% of units)
- GPT-6 SolGPT-6 Sol: mean stated confidence 95.6%, realized accuracy 95.3% on 169 predictions8 / 169 (44% of units)
- Claude Sonnet 5.5Claude Sonnet 5.5: mean stated confidence 91.6%, realized accuracy 92.1% on 63 predictions5 / 63 (16% of units)
- Mistral Large 4Mistral Large 4: mean stated confidence 92.6%, realized accuracy 94.5% on 128 predictions7 / 128 (33% of units)
- Grok 4.7Grok 4.7: mean stated confidence 92.9%, realized accuracy 95.9% on 98 predictions4 / 98 (25% of units)
- Claude Opus 5Claude Opus 5: mean stated confidence 93.6%, realized accuracy 96.7% on 121 predictions4 / 121 (31% of units)
- Kimi K3Kimi K3: mean stated confidence 92.8%, realized accuracy 98.9% on 92 predictions1 / 92 (24% of units)
- Grok 4.6Grok 4.6: mean stated confidence 93.0%, realized accuracy 99.1% on 112 predictions1 / 112 (29% of units)
- Ultramarine 1 FlashUltramarine 1 Flash: mean stated confidence 92.2%, realized accuracy 98.7% on 77 predictions1 / 77 (20% of units)
- Claude Fable 5.1Claude Fable 5.1: mean stated confidence 92.9%, realized accuracy 100.0% on 81 predictions0 / 81 (21% of units)
- Claude Opus 5.5Claude Opus 5.5: mean stated confidence 92.8%, realized accuracy 100.0% on 65 predictions0 / 65 (17% of units)
80%85%90%95%100%
Approach
Why judicial outcomes
“It is emphatically the province and duty of the Judicial Department to say what the law is.” Marbury v. Madison, 5 U.S. 137, 177 (1803). Every litigated judgment has the force of law as between the parties, and definitively resolves any disputes between the parties on the interpretation of the law or its application to a specific fact pattern. Moreover, one of the important, foundational principles of our common law heritage is that these judicial dispositions are public. Judges interpret, and make, law so their reasoned opinions are a public asset.
LegalForecastBench is an illustration of one specific use of the signal that lies in this data, using judges' rulings on motions to dismiss to evaluate the predictive abilities of frontier models. The benchmark focuses on district court level motions because lower courts are heavily constrained by precedent and largely decide garden-variety disputes in which the ability to correctly analyze and apply the law to the alleged facts will be the primary way to predict the outcome. Obviously, not every judicial disposition is accurate. But unless and until overturned on appeal, these dispositions have the force of law and are the main source of signal that real lawyers rely on to understand how the law will be applied to concrete fact patterns.
Feedback
This version of the paper is intended as a working draft, and feedback from researchers, lawyers, and others on how to improve this benchmark or make our findings more useful is welcome.