Data & code

Data and code

Code is maintained in the LegalForecastBench repository; documents used in the benchmark evaluation, including the documents I purchased in order to complete the evaluation, have been made available to the public through CourtListener. I am also happy to provide copies of the litigation documents as they were shown to models to interested researchers.

Repository · CourtListener · Report an error

Downloads

File	Contents
Leaderboard	Each model's scores, training cutoff, eligibility and cost
Forecasts	One row per claim–defendant unit: case ID, unit ID, the model's probability of dismissal, and the outcome Claude Opus 5 Claude Opus 5.5 Claude Sonnet 5 Claude Sonnet 5.5 Gemini 3.1 Pro Preview GPT-4.1 GPT-5.6 Luna GPT-5.6 Sol GPT-6.1 Sol GPT-6 Luna GPT-6 Sol Grok 4.7 Muse Wonder
Significance	Paired bootstrap intervals for every pair of models above (inputs)
Full exports	Complete scored output for each run, with cost accounting and its GitHub Actions runs GPT-6 Sol GPT-6 Luna Claude Opus 5.5 Grok 4.7 GPT-4.1 Gemini 3.1 Pro Preview GPT-6.1 Sol Claude Sonnet 5.5 Claude Opus 5 Claude Sonnet 5 GPT-5.6 Sol Muse Wonder
Jev experiment	Forecasts from case summaries, kept separate from the leaderboard Jev (reasoning n/a (returns probabilities directly)) GPT-5.6 Luna (reasoning high) GPT-6 Luna (reasoning off) Jev · Grok shorter summaries (reasoning n/a (returns probabilities directly)) significance

Reproduce

License

The files above: CC BY 4.0. Code: Apache 2.0. Court filings are public records and are not licensed here.

How to cite

From the repository's CITATION.cff.

Working paper

First posted 2026-10-13. Cite this for the benchmark and its results.

BibTeX

@misc{hughes2026law,
  author = {Hughes, III, John J.},
  title = {{Law as a Verifiable Domain: Using Real-World Judicial Outcomes to Evaluate AI Legal Reasoning}},
  year = {2026},
  date = {2026-10-13},
  note = {Working paper},
  url = {https://www.legalforecastbench.org/paper/},
}

APA

Hughes, J. J., III. (2026). Law as a Verifiable Domain: Using Real-World Judicial Outcomes to Evaluate AI Legal Reasoning [Working paper]. https://www.legalforecastbench.org/paper/

Software

Release 0.1.0a3. Cite this to identify the code you ran.

BibTeX

@software{hughes2026legalforecastbenchforecastingfederalmotionstodismisslegalforecastmtd,
  author = {Hughes, III, John J.},
  title = {{LegalForecastBench: Forecasting Federal Motions to Dismiss (LegalForecast-MTD)}},
  version = {0.1.0a3},
  year = {2026},
  date = {2026-10-13},
  url = {https://www.legalforecastbench.org},
}

APA

Hughes, J. J., III. (2026). LegalForecastBench: Forecasting Federal Motions to Dismiss (LegalForecast-MTD) (Version 0.1.0a3) [Computer software]. https://www.legalforecastbench.org