Model Scorecard

Generated: 2026-09-14T05:00:37.968Z | Window: 2026-08-15 → 2026-09-14

Cost, quality, and speed per LLM, joined from runstate trails (GitHub), runs.jsonl (token/cost telemetry), and events.jsonl (dispatch tier, stall/escalation counts). Regenerate with npm run scorecard -- --days 30. See #566.

n is a confidence column, not a metric — a row with n=1 is one data point, not a trend. Read cost/delivered and delivery rate alongside n, never alone.

Per model × repo × tier

Totals per model (all repos/tiers)

One row per model, with the gateways it was served through as ↳ sub-rows whenever there is more than one — the fold that makes the model row readable would otherwise hide a gateway costing more or delivering less than the same weights elsewhere.

Billable tokens count uncached input + cache-creation + cache-read + output — cache reads are billed and, on this fleet, are the dominant term (issue #540: 262 uncached vs 13.4M cache-read). That is the same total batch-pilot-2-execution.md §13 publishes.

Of the 46 delivered issues with a cost figure, 17 were recovered from the dispatch’s own agent log because runs.jsonl recorded none — see Limitations.

Wall-clock per phase (all models)

Median seconds between a phase’s milestone and the previous one, from the trails’ own at= stamps. Wall-clock is the only per-phase measure available: cost AND API-minutes both come from one agent session that usually spans several phases, so neither can be attributed to a phase (see Limitations).

Reconciliation

This is a regenerated snapshot, not the first one — see git history for docs/reports/model-scorecard.md for prior windows. The first-snapshot reconciliation against batch-pilot-2-execution.md §13.3 and model-agnostic-fleet.md ran once, at #566.

Limitations

• A moving version tag folds onto its pin only where someone declared the mapping. glm-latest → glm-5.3 is declared (MODEL_ALIASES in cli/src/runstate-stats.ts, the mapping #566 states) and folds, along with every routed spelling of it. Which pin a -latest tag points at is a fact about the provider’s state, not about the string, so it cannot be derived here and it goes stale the moment the provider ships a new version under the same tag — when that happens, update the value in MODEL_ALIASES rather than reading the row as one version. Undeclared tags (kimi-latest, which has both kimi-k3 and kimi-k3-fast as plausible pins) keep their own row and are named in Data warnings — a guessed alias misattributes cost and quality silently, a missing one only splits a row.
• A context-window variant keeps its own row. claude-opus-5[1m] does not fold into claude-opus-5: the milestone protocol says the suffix should never have been written (gate records the bare model id), but 1M-context is billed differently, so folding it would blend two cost profiles to fix a formatting slip. Read the two rows together when judging quality, separately when judging cost.
• Cost comes from two sources, and the column says which. ~/.dossier/runs.jsonl is authoritative; where a dispatch predates the telemetry fix (#564) and left it null, the figure is recovered from that dispatch’s own agent log under ~/.dossier/sched/<slug>/runs/. Both are on-host only — a dispatch run from another machine has neither, and its row reads N/A because the data is elsewhere, not because it was free.
• Δ vs prev is snapshot-over-snapshot, not a strict 7-day delta. It compares this run against whatever sidecar is on the base branch — two overlapping 30-day windows when the weekly cron produced both, and a longer gap whenever a weekly PR went unmerged. A baseline older than the window raises its own data warning naming the age, so the column is never silently stale; read it as “since the last published snapshot”.
• The window selects ISSUES by last update, not runs by date. The trail read is gh issue list --search "updated:>=<start>", and a matched issue contributes its whole dispatch history — so an old run on an issue merely touched inside the window counts in full. Read the window as “every run on an issue active in the last N days”, not “every run started in the last N days”.
• A <unknown> tier is usually an artifact of that mismatch, not an untiered dispatch. Tier and agent CLI come from events.jsonl events inside the window, while the trail data behind the same row is not time-filtered — so a run whose dispatch event predates the window keeps its outcome columns and loses its tier. The per-model totals fold tiers away and are unaffected; only the per-tier rows split.
• Cost and API-minutes per phase are not separable. A dispatch is usually one continuous agent session covering several phases, so both are recorded per issue, not per phase — the per-phase section above reports wall-clock only, which the milestone at= stamps do carry. For a per-phase breakdown of a single run rather than a median across many, run ai-dossier runstate stats directly.
• Stall/escalation/unverified-exit counts are a per-host gap. They come from ~/.dossier/sched/<project>/events.jsonl, which only exists on the machine that ran the dispatch. A run dispatched from another host reports 0 for these columns here even if it really stalled — fleet-cli-audit.sh documents which hosts exist; this script does not collect across them.
• Cost/delivered averages only delivered issues. Work that was dispatched and then blocked, evicted, or abandoned is not in the denominator or the numerator — see the n vs Delivered columns for how much of a bucket that excludes.

Data warnings

• imboard-ai/ai-dossier: 33 run(s) recorded no model= — their row is bucketed as , and its outcome columns are not attributable to any model
• imboard-ai/ai-dossier: 26 of 166 run(s) are classifier dispatches (classify milestones only) — excluded from the by-model and by-class tables, since they record no model= and never ship
• imboard-ai/ai-dossier: 111 of 140 run(s) (79%) carry no classifier risk= verdict — they are bucketed as , so the class rows describe only the classified remainder
• imboard-ai/imboard-monorepo: issue imboard-ai/imboard-monorepo#3684 run r-3684-a0cf: gate milestone has an unusable at= value (’$(date’) — skipped, and the next phase’s duration is reported as unknown
• imboard-ai/imboard-monorepo: 51 run(s) recorded no model= — their row is bucketed as , and its outcome columns are not attributable to any model
• imboard-ai/imboard-monorepo: ‘kimi-latest’ is a moving version tag with no declared pin — its 1 run(s) sit in their own row, apart from whatever pinned version the tag resolves to; add it to MODEL_ALIASES to fold them
• imboard-ai/imboard-monorepo: 71 of 320 run(s) are classifier dispatches (classify milestones only) — excluded from the by-model and by-class tables, since they record no model= and never ship
• imboard-ai/imboard-monorepo: 203 of 249 run(s) (82%) carry no classifier risk= verdict — they are bucketed as , so the class rows describe only the classified remainder

Rendered from docs/reports/model-scorecard.md in the repository. Edit it there.