Skip to content
LegalForecastBenchIn partnership with LegalQuants
← Leaderboard

Paper Instruments

Qualified: no published training data/knowledge cutoff, or the cutoff is not before the earliest decision we used to test.Cutoff: not published

Ultramarine 1 Flash

Micro BrierMicro Brier is the headline metric: squared error of each dismissal forecast, averaged over all claim-defendant units. Lower is better; always answering 50% scores 0.25.

0.1550

Rank 13 of 21 · On the observed cost/quality frontier: no other model is both cheaper and at least as good, and this model strictly improves over others on at least one dimension. (Note that the improvement does not have to be statistically significant.)

Equal-case Brier

0.1721

Rank 18 of 21 · On the observed cost/quality frontier: no other model is both cheaper and at least as good, and this model strictly improves over others on at least one dimension. (Note that the improvement does not have to be statistically significant.)

Unit accuracy

75.5%

Rank 20 of 21

Cost and quality

Quality metric
Headline metric
GPT-6 Sol: Micro Brier 0.1176, cost $11.10Claude Fable 5.1: Micro Brier 0.1288, cost $83.18Grok 4.6: Micro Brier 0.1337, cost $15.72Claude Opus 5: Micro Brier 0.1399, cost $45.89GPT-5.6 Sol: Micro Brier 0.1401, cost $24.91Kimi K3: Micro Brier 0.1413, cost $16.87GPT-5.6 Luna: Micro Brier 0.1432, cost $1.48Claude Opus 5.5: Micro Brier 0.1454, cost $20.59GPT-6.1 Sol: Micro Brier 0.1495, cost $10.85GPT-6 Astra: Micro Brier 0.1522, cost $59.74Gemini 3.8 Flash: Micro Brier 0.1533, cost $35.61Gemini 4 Argon: Micro Brier 0.1539, cost $25.02Ultramarine 1 Flash: Micro Brier 0.1550, cost $1.35Grok 4.7: Micro Brier 0.1583, cost $33.00Claude Sonnet 5: Micro Brier 0.1585, cost $24.68Mistral Large 4: Micro Brier 0.1596, cost $16.65GPT-6 Luna: Micro Brier 0.1637, cost $0.63Muse Wonder: Micro Brier 0.1654, cost $39.38Claude Sonnet 5.5: Micro Brier 0.1730, cost $6.99Muse Spark 1.3: Micro Brier 0.1812, cost $8.78Gemini 3.1 Pro Preview: Micro Brier 0.1834, cost $27.73
Observed cost/quality frontier (descriptive, not significance)Other models

High-confidence forecasts

77 of 387 predictions (19.9%) were at least 90% confident either way. 1 were wrong, an error rate of 1.3%. Mean stated confidence in that subset was 92.2%.

  • Gemini 3.1 Pro Preview
    Gemini 3.1 Pro Preview: mean stated confidence 95.4%, realized accuracy 82.2% on 320 predictions
    57 / 320 (83% of units)
  • GPT-5.6 Sol
    GPT-5.6 Sol: mean stated confidence 97.0%, realized accuracy 88.6% on 228 predictions
    26 / 228 (59% of units)
  • Gemini 3.8 Flash
    Gemini 3.8 Flash: mean stated confidence 95.2%, realized accuracy 87.3% on 276 predictions
    35 / 276 (71% of units)
  • GPT-5.6 Luna
    GPT-5.6 Luna: mean stated confidence 94.9%, realized accuracy 89.0% on 182 predictions
    20 / 182 (47% of units)
  • GPT-6 Luna
    GPT-6 Luna: mean stated confidence 95.4%, realized accuracy 89.6% on 183 predictions
    19 / 183 (47% of units)
  • Gemini 4 Argon
    Gemini 4 Argon: mean stated confidence 95.1%, realized accuracy 89.3% on 225 predictions
    24 / 225 (58% of units)
  • GPT-6 Astra
    GPT-6 Astra: mean stated confidence 95.5%, realized accuracy 89.9% on 207 predictions
    21 / 207 (53% of units)
  • GPT-6.1 Sol
    GPT-6.1 Sol: mean stated confidence 95.7%, realized accuracy 91.0% on 199 predictions
    18 / 199 (51% of units)
  • Muse Spark 1.3
    Muse Spark 1.3: mean stated confidence 93.5%, realized accuracy 88.9% on 135 predictions
    15 / 135 (35% of units)
  • Muse Wonder
    Muse Wonder: mean stated confidence 93.4%, realized accuracy 91.5% on 118 predictions
    10 / 118 (30% of units)
  • Claude Sonnet 5
    Claude Sonnet 5: mean stated confidence 92.6%, realized accuracy 91.7% on 60 predictions
    5 / 60 (16% of units)
  • GPT-6 Sol
    GPT-6 Sol: mean stated confidence 95.6%, realized accuracy 95.3% on 169 predictions
    8 / 169 (44% of units)
  • Claude Sonnet 5.5
    Claude Sonnet 5.5: mean stated confidence 91.6%, realized accuracy 92.1% on 63 predictions
    5 / 63 (16% of units)
  • Mistral Large 4
    Mistral Large 4: mean stated confidence 92.6%, realized accuracy 94.5% on 128 predictions
    7 / 128 (33% of units)
  • Grok 4.7
    Grok 4.7: mean stated confidence 92.9%, realized accuracy 95.9% on 98 predictions
    4 / 98 (25% of units)
  • Claude Opus 5
    Claude Opus 5: mean stated confidence 93.6%, realized accuracy 96.7% on 121 predictions
    4 / 121 (31% of units)
  • Kimi K3
    Kimi K3: mean stated confidence 92.8%, realized accuracy 98.9% on 92 predictions
    1 / 92 (24% of units)
  • Grok 4.6
    Grok 4.6: mean stated confidence 93.0%, realized accuracy 99.1% on 112 predictions
    1 / 112 (29% of units)
  • Ultramarine 1 Flash
    Ultramarine 1 Flash: mean stated confidence 92.2%, realized accuracy 98.7% on 77 predictions
    1 / 77 (20% of units)
  • Claude Fable 5.1
    Claude Fable 5.1: mean stated confidence 92.9%, realized accuracy 100.0% on 81 predictions
    0 / 81 (21% of units)
  • Claude Opus 5.5
    Claude Opus 5.5: mean stated confidence 92.8%, realized accuracy 100.0% on 65 predictions
    0 / 65 (17% of units)
80%85%90%95%100%
Mean stated confidenceRealized accuracySorted by the gap between them. Units, not independent cases: several misses can come from one case.

Statistical comparison coverage

Included in the paired significance analysis.

Analysis and reproducible inputs →

Statistically distinguishable differences

  • Better than GPT-4.1 on Micro Brier, Equal-case Brier, and Unit accuracy.
Comparison data →