# Can AI forecast motions to dismiss? > An open research benchmark testing whether AI models can forecast federal motion-to-dismiss outcomes from the court record available before the decision. Canonical: https://www.legalforecastbench.org/ LegalForecastBench is an effort to develop a new, challenging benchmark that uses the signal from judicial decisions to evaluate frontier models on high-level legal tasks against an objective real-world ground truth. Models receive the same briefs a federal judge received when ruling on a motion to dismiss, and are asked to estimate the probability that the judge dismissed each claim challenged by a defendant. ## Results The beta includes 91 cases and 387 scored claim–defendant units. Of the 18 frontier models tested, GPT-6 Sol has the lowest micro-Brier loss (0.118) and the highest accuracy (84.5%). Lower micro Brier is better. Summary-pipeline conditions are a separate experiment and are not in this table. A constant forecast at the cohort's dismissal rate scores 0.194 micro Brier; always answering 50% scores 0.250. | Model | Micro Brier | Equal-case Brier | Accuracy | Eligibility | | --- | --- | --- | --- | --- | | [GPT-6 Sol](https://www.legalforecastbench.org/models/gpt-6-sol/) | 0.1176 | 0.1295 | 84.5% | eligible | | [Claude Fable 5.1](https://www.legalforecastbench.org/models/claude-fable-5-1/) | 0.1288 | 0.1365 | 82.2% | eligible | | [Grok 4.6](https://www.legalforecastbench.org/models/grok-4-6/) | 0.1337 | 0.1348 | 80.6% | eligible | | [Claude Opus 5](https://www.legalforecastbench.org/models/claude-opus-5/) | 0.1399 | 0.1355 | 80.1% | eligible | | [GPT-5.6 Sol](https://www.legalforecastbench.org/models/gpt-5-6-sol/) | 0.1401 | 0.1332 | 80.9% | eligible | | [Kimi K3](https://www.legalforecastbench.org/models/kimi-k3/) | 0.1413 | 0.1537 | 78.3% | qualified | | [GPT-5.6 Luna](https://www.legalforecastbench.org/models/gpt-5-6-luna/) | 0.1432 | 0.1508 | 80.9% | eligible | | [Claude Opus 5.5](https://www.legalforecastbench.org/models/claude-opus-5-5/) | 0.1454 | 0.1476 | 77.3% | eligible | | [GPT-6.1 Sol](https://www.legalforecastbench.org/models/gpt-6-1-sol/) | 0.1495 | 0.1389 | 78.6% | eligible | | [GPT-6 Astra](https://www.legalforecastbench.org/models/gpt-6-astra/) | 0.1522 | 0.1427 | 78.8% | eligible | | [Gemini 3.8 Flash](https://www.legalforecastbench.org/models/gemini-3-8-flash/) | 0.1533 | 0.1390 | 80.1% | eligible | | [Grok 4.7](https://www.legalforecastbench.org/models/grok-4-7/) | 0.1583 | 0.1645 | 77.0% | eligible | | [Claude Sonnet 5](https://www.legalforecastbench.org/models/claude-sonnet-5/) | 0.1585 | 0.1596 | 75.7% | eligible | | [GPT-6 Luna](https://www.legalforecastbench.org/models/gpt-6-luna/) | 0.1637 | 0.1738 | 76.5% | eligible | | [Muse Wonder](https://www.legalforecastbench.org/models/muse-wonder-1-0/) | 0.1654 | 0.1690 | 77.3% | qualified | | [Claude Sonnet 5.5](https://www.legalforecastbench.org/models/claude-sonnet-5-5/) | 0.1730 | 0.1718 | 76.2% | eligible | | [Muse Spark 1.3](https://www.legalforecastbench.org/models/muse-spark-1-3/) | 0.1812 | 0.1784 | 72.1% | qualified | | [Gemini 3.1 Pro Preview](https://www.legalforecastbench.org/models/gemini-3-1-pro-preview/) | 0.1834 | 0.1808 | 78.0% | eligible | ## Why judicial outcomes “It is emphatically the province and duty of the Judicial Department to say what the law is.” [*Marbury v. Madison*](https://www.archives.gov/milestone-documents/marbury-v-madison), 5 U.S. 137, 177 (1803). Every litigated judgment has the force of law as between the parties, and definitively resolves any disputes between the parties on the interpretation of the law or its application to a specific fact pattern. Moreover, one of the important, foundational principles of our common law heritage is that these judicial dispositions are public. Judges interpret, and make, law so their reasoned opinions are a public asset. LegalForecastBench is an illustration of one specific use of the signal that lies in this data, using judges' rulings on motions to dismiss to evaluate the predictive abilities of frontier models. The benchmark focuses on district court level motions because lower courts are heavily constrained by precedent and largely decide garden-variety disputes in which the ability to correctly analyze and apply the law to the alleged facts will be the primary way to predict the outcome. Obviously, not every judicial disposition is accurate. But unless and until overturned on appeal, these dispositions have the force of law and are the main source of signal that real lawyers rely on to understand how the law will be applied to concrete fact patterns. --- # Law as a Verifiable Domain: Using Real-World Judicial Outcomes to Evaluate AI Legal Reasoning > Working paper on using real-world judicial outcomes to evaluate AI legal reasoning. Canonical: https://www.legalforecastbench.org/paper/ Working paper by John J. Hughes, III. Manuscript date: 2026-10-13 ## Abstract Reinforcement learning (RL) techniques, in domains that can offer objectively verifiable rewards, have led to incredible and noteworthy recent gains in the capabilities of frontier AI models. But not all domains are equally verifiable. Many high-level legal judgment tasks are difficult to verify, and model capability gains on those tasks have not been nearly as noteworthy as model gains in domains that are susceptible to automated verification like math and coding. This paper argues that the heavy reliance of existing legal benchmarks on human-written rubrics and artificially constructed environments fails to take advantage of key properties of the common law judicial tradition, which gives precedential effect and the force of law to reasoned judicial decisions issued by judges in specific cases or controversies. For AI researchers, those judicial dispositions supply an external signal that allows for evaluation of some high-level legal judgment tasks against an objectively verifiable real-world ground truth. In the legal system, the opinions of legal experts or data providers as to what the law is have no authoritative or precedential value. The pronouncements of a tribunal, by contrast, carry the force of law (and often precedential effect). LegalForecastBench illustrates one mechanism by which the signal from judicial decisions can be used to objectively evaluate the legal reasoning capabilities of frontier models. Models receive the same briefs that a federal judge received when ruling on a motion to dismiss, and are asked to estimate the probability that the judge dismissed each claim challenged by a defendant. The beta includes 91 cases and 387 scored claim–defendant units. Of the 18 frontier models tested, GPT-6 Sol has the lowest micro-Brier loss (0.118) and the highest accuracy (84.5%). Notably, calibration had a large impact on the performance of models. Claude Fable 5.1 had no errors in its 81 high-confidence (at least 90%) forecasts, while for Gemini 3.1 Pro Preview, 57 out of 320 high-confidence predictions it made were inaccurate. Several newer model versions scored below their predecessors on this cohort, although none of these differences was statistically significant where it could be tested. PDF: https://www.legalforecastbench.org/paper/legalforecastbench-working.pdf --- # Download the forecast results > Download the results LegalForecastBench displays, reproduce the scoring, and see where the evidence comes from. Canonical: https://www.legalforecastbench.org/data/ Code is maintained in the LegalForecastBench repository; documents used in the benchmark evaluation, including the documents I purchased in order to complete the evaluation, have been made available to the public through CourtListener. I am also happy to provide copies of the litigation documents as they were shown to models to interested researchers. Forecasts by 18 ranked AI model configurations, plus GPT-4.1 as a historical reference for non-thinking models, of whether each challenged claim in 91 federal motions to dismiss would be fully dismissed, scored against the actual rulings (387 claim-defendant units). - Release: cycle-1-91-2026-09-08-luna-r5 - Main results JSON: https://www.legalforecastbench.org/data/current.json - Significance JSON: https://www.legalforecastbench.org/data/significance/comparison.json - Cost evidence: https://www.legalforecastbench.org/data/costs.json - Data license: https://creativecommons.org/licenses/by/4.0/ - Code license: Apache License 2.0 - Source: https://github.com/johnhughes3/LegalForecastBench --- # Claude Fable 5.1 > Claude Fable 5.1 on LegalForecastBench: micro Brier 0.1288 across 387 units. Canonical: https://www.legalforecastbench.org/models/claude-fable-5-1/ - Provider: Anthropic - Reasoning: Adaptive thinking (provider default) - Public release: 2026-09-01 - Training cutoff: June 2026 (training-data cutoff) - Eligibility: eligible. The reported training-data cutoff (June 2026) is in the month of the first scored decision (June 30, 2026). It is treated as preceding that decision, because a ruling is very unlikely to enter training data the day it issues. - Micro Brier: 0.1288 - Equal-case Brier: 0.1365 - Accuracy: 82.2% (318 of 387) - High-confidence predictions: 81, of which 0 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Grok 4.6 > Grok 4.6 on LegalForecastBench: micro Brier 0.1337 across 387 units. Canonical: https://www.legalforecastbench.org/models/grok-4-6/ - Provider: SpaceXAI - Reasoning: High - Public release: 2026-08-12 - Training cutoff: February 1, 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (February 1, 2026) predates the first scored decision on June 30, 2026. xAI's pages report January 2026 and February 1, 2026; the later date is used. - Micro Brier: 0.1337 - Equal-case Brier: 0.1348 - Accuracy: 80.6% (312 of 387) - High-confidence predictions: 112, of which 1 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Claude Opus 5 > Claude Opus 5 on LegalForecastBench: micro Brier 0.1399 across 387 units. Canonical: https://www.legalforecastbench.org/models/claude-opus-5/ - Provider: Anthropic - Reasoning: Adaptive thinking (provider default: high) - Public release: 2026-07-24 - Training cutoff: May 2026 (training-data cutoff) - Eligibility: eligible. The reported training-data cutoff (May 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1399 - Equal-case Brier: 0.1355 - Accuracy: 80.1% (310 of 387) - High-confidence predictions: 121, of which 4 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # GPT-5.6 Sol > GPT-5.6 Sol on LegalForecastBench: micro Brier 0.1401 across 387 units. Canonical: https://www.legalforecastbench.org/models/gpt-5-6-sol/ - Provider: OpenAI - Reasoning: High - Public release: 2026-06-26 - Training cutoff: February 16, 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (February 16, 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1401 - Equal-case Brier: 0.1332 - Accuracy: 80.9% (313 of 387) - High-confidence predictions: 228, of which 26 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Kimi K3 > Kimi K3 on LegalForecastBench: micro Brier 0.1413 across 387 units. Canonical: https://www.legalforecastbench.org/models/kimi-k3/ - Provider: Moonshot - Reasoning: High - Public release: 2026-07-17 - Training cutoff: Not disclosed - Eligibility: qualified. The provider has not published a training or knowledge cutoff. Moonshot AI has not published a cutoff. - Micro Brier: 0.1413 - Equal-case Brier: 0.1537 - Accuracy: 78.3% (303 of 387) - High-confidence predictions: 92, of which 1 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # GPT-5.6 Luna > GPT-5.6 Luna on LegalForecastBench: micro Brier 0.1432 across 387 units. Canonical: https://www.legalforecastbench.org/models/gpt-5-6-luna/ - Provider: OpenAI - Reasoning: High - Public release: 2026-06-26 - Training cutoff: February 16, 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (February 16, 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1432 - Equal-case Brier: 0.1508 - Accuracy: 80.9% (313 of 387) - High-confidence predictions: 182, of which 20 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # GPT-6 Astra > GPT-6 Astra on LegalForecastBench: micro Brier 0.1522 across 387 units. Canonical: https://www.legalforecastbench.org/models/gpt-6-astra/ - Provider: OpenAI - Reasoning: High - Public release: 2026-09-03 - Training cutoff: April 30, 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (April 30, 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1522 - Equal-case Brier: 0.1427 - Accuracy: 78.8% (305 of 387) - High-confidence predictions: 207, of which 21 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Gemini 3.8 Flash > Gemini 3.8 Flash on LegalForecastBench: micro Brier 0.1533 across 387 units. Canonical: https://www.legalforecastbench.org/models/gemini-3-8-flash/ - Provider: Google - Reasoning: High thinking - Public release: 2026-09-02 - Training cutoff: March 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (March 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1533 - Equal-case Brier: 0.1390 - Accuracy: 80.1% (310 of 387) - High-confidence predictions: 276, of which 35 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Claude Sonnet 5 > Claude Sonnet 5 on LegalForecastBench: micro Brier 0.1585 across 387 units. Canonical: https://www.legalforecastbench.org/models/claude-sonnet-5/ - Provider: Anthropic - Reasoning: Adaptive thinking (provider default) - Public release: 2026-06-30 - Training cutoff: January 2026 (training-data cutoff) - Eligibility: eligible. The reported training-data cutoff (January 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1585 - Equal-case Brier: 0.1596 - Accuracy: 75.7% (293 of 387) - High-confidence predictions: 60, of which 5 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Muse Spark 1.3 > Muse Spark 1.3 on LegalForecastBench: micro Brier 0.1812 across 387 units. Canonical: https://www.legalforecastbench.org/models/muse-spark-1-3/ - Provider: Meta - Reasoning: High - Public release: 2026-09-02 - Training cutoff: Not disclosed - Eligibility: qualified. The provider has not published a training or knowledge cutoff. Meta has not published a cutoff. - Micro Brier: 0.1812 - Equal-case Brier: 0.1784 - Accuracy: 72.1% (279 of 387) - High-confidence predictions: 135, of which 15 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # GPT-6 Sol > GPT-6 Sol on LegalForecastBench: micro Brier 0.1176 across 387 units. Canonical: https://www.legalforecastbench.org/models/gpt-6-sol/ - Provider: OpenAI - Reasoning: High - Public release: 2026-09-22 - Training cutoff: April 20, 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (April 20, 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1176 - Equal-case Brier: 0.1295 - Accuracy: 84.5% (327 of 387) - High-confidence predictions: 169, of which 8 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # GPT-6 Luna > GPT-6 Luna on LegalForecastBench: micro Brier 0.1637 across 387 units. Canonical: https://www.legalforecastbench.org/models/gpt-6-luna/ - Provider: OpenAI - Reasoning: High - Public release: 2026-09-22 - Training cutoff: May 18, 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (May 18, 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1637 - Equal-case Brier: 0.1738 - Accuracy: 76.5% (296 of 387) - High-confidence predictions: 183, of which 19 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Claude Opus 5.5 > Claude Opus 5.5 on LegalForecastBench: micro Brier 0.1454 across 387 units. Canonical: https://www.legalforecastbench.org/models/claude-opus-5-5/ - Provider: Anthropic - Reasoning: High - Public release: 2026-09-22 - Training cutoff: June 2026 (training-data cutoff) - Eligibility: eligible. The reported training-data cutoff (June 2026) is in the month of the first scored decision (June 30, 2026). It is treated as preceding that decision, because a ruling is very unlikely to enter training data the day it issues. - Micro Brier: 0.1454 - Equal-case Brier: 0.1476 - Accuracy: 77.3% (299 of 387) - High-confidence predictions: 65, of which 0 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Grok 4.7 > Grok 4.7 on LegalForecastBench: micro Brier 0.1583 across 387 units. Canonical: https://www.legalforecastbench.org/models/grok-4-7/ - Provider: SpaceXAI - Reasoning: High - Public release: 2026-09-21 - Training cutoff: May 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (May 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1583 - Equal-case Brier: 0.1645 - Accuracy: 77.0% (298 of 387) - High-confidence predictions: 98, of which 4 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # GPT-4.1 > GPT-4.1 on LegalForecastBench: micro Brier 0.6051 across 387 units. Canonical: https://www.legalforecastbench.org/models/gpt-4-1/ - Provider: OpenAI - Reasoning: None (non-reasoning model) - Public release: 2025-04-14 - Training cutoff: June 1, 2024 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (June 1, 2024) predates the first scored decision on June 30, 2026. - Micro Brier: 0.6051 - Equal-case Brier: 0.5752 - Accuracy: 39.0% (151 of 387) - High-confidence predictions: 357, of which 229 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Gemini 3.1 Pro Preview > Gemini 3.1 Pro Preview on LegalForecastBench: micro Brier 0.1834 across 387 units. Canonical: https://www.legalforecastbench.org/models/gemini-3-1-pro-preview/ - Provider: Google - Reasoning: High thinking - Public release: 2026-02-19 - Training cutoff: January 2025 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (January 2025) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1834 - Equal-case Brier: 0.1808 - Accuracy: 78.0% (302 of 387) - High-confidence predictions: 320, of which 57 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # GPT-6.1 Sol > GPT-6.1 Sol on LegalForecastBench: micro Brier 0.1495 across 387 units. Canonical: https://www.legalforecastbench.org/models/gpt-6-1-sol/ - Provider: OpenAI - Reasoning: High - Public release: 2026-09-29 - Training cutoff: April 30, 2026 (knowledge cutoff) - Eligibility: eligible. The reported knowledge cutoff (April 30, 2026) predates the first scored decision on June 30, 2026. - Micro Brier: 0.1495 - Equal-case Brier: 0.1389 - Accuracy: 78.6% (304 of 387) - High-confidence predictions: 199, of which 18 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Claude Sonnet 5.5 > Claude Sonnet 5.5 on LegalForecastBench: micro Brier 0.1730 across 387 units. Canonical: https://www.legalforecastbench.org/models/claude-sonnet-5-5/ - Provider: Anthropic - Reasoning: High - Public release: 2026-09-28 - Training cutoff: June 2026 (training-data cutoff) - Eligibility: eligible. The reported training-data cutoff (June 2026) is in the month of the first scored decision (June 30, 2026). It is treated as preceding that decision, because a ruling is very unlikely to enter training data the day it issues. - Micro Brier: 0.1730 - Equal-case Brier: 0.1718 - Accuracy: 76.2% (295 of 387) - High-confidence predictions: 63, of which 5 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard --- # Muse Wonder > Muse Wonder on LegalForecastBench: micro Brier 0.1654 across 387 units. Canonical: https://www.legalforecastbench.org/models/muse-wonder-1-0/ - Provider: Meta - Reasoning: High - Public release: 2026-10-12 - Training cutoff: Not disclosed - Eligibility: qualified. The provider has not published a training or knowledge cutoff. Meta's announcement, model card and developer documentation do not report a training-data or knowledge cutoff. - Micro Brier: 0.1654 - Equal-case Brier: 0.1690 - Accuracy: 77.3% (299 of 387) - High-confidence predictions: 118, of which 10 were wrong Ranked comparison: https://www.legalforecastbench.org/#leaderboard