Can AI find the commit that broke prod?
A monitor pages. p99 latency is up 20x, or pods are getting OOMKilled. Forty commits landed in the last three hours and someone flipped a feature flag. The on-call engineer has to find the one commit that did it, without getting fooled by the innocent deploy that landed thirty seconds before the graph went vertical. RootCauseBench asks whether an LLM can do that. Every model gets the same data and the same shell. We measure the reasoning, not the tooling.
Model ranking
| # | Model | Graded reward (95% CI) | Pass rate | easy | medium | hard | adversarial | no-code-cause |
|---|---|---|---|---|---|---|---|---|
| 1 | kimi-k3 | 0.991 ± 0.018 | 99% | 100% | 100% | 100% | 97% | 100% |
| 2 | claude-opus-5 | 0.981 ± 0.026 | 98% | 100% | 100% | 100% | 94% | 100% |
| 3 | glm-5.2 | 0.981 ± 0.026 | 98% | 100% | 100% | 100% | 94% | 100% |
| 4 | claude-fable-5 | 0.968 ± 0.032 | 96% | 100% | 100% | 94% | 94% | 100% |
| 5 | gpt-5.6-sol | 0.967 ± 0.033 | 96% | 100% | 100% | 100% | 89% | 100% |
| 6 | grok-4.5 | 0.963 ± 0.036 | 96% | 100% | 100% | 100% | 89% | 100% |
| 7 | fugu-ultra | 0.952 ± 0.039 | 94% | 100% | 100% | 94% | 89% | 96% |
| 8 | claude-opus-4.8 | 0.944 ± 0.043 | 94% | 100% | 100% | 97% | 86% | 100% |
| 9 | deepseek-v4-flash | 0.942 ± 0.043 | 94% | 100% | 96% | 94% | 89% | 83% |
| 10 | gemini-3.5-flash | 0.935 ± 0.047 | 94% | 100% | 100% | 92% | 89% | 88% |
| 11 | gemini-3.1-pro-preview | 0.925 ± 0.048 | 92% | 100% | 100% | 92% | 83% | 88% |
| 12 | claude-sonnet-4.6 | 0.917 ± 0.052 | 92% | 100% | 100% | 92% | 83% | 88% |
| 13 | gpt-5.5 | 0.917 ± 0.052 | 92% | 100% | 100% | 92% | 83% | 92% |
| 14 | kimi-k2-thinking | 0.852 ± 0.066 | 84% | 100% | 89% | 81% | 81% | 62% |
| 15 | gpt-5.4 | 0.845 ± 0.067 | 83% | 100% | 100% | 94% | 56% | 92% |
| 16 | kimi-k2.5 | 0.815 ± 0.074 | 81% | 100% | 96% | 78% | 69% | 58% |
| 17 | gpt-5.4-mini | 0.739 ± 0.076 | 69% | 78% | 89% | 78% | 42% | 88% |
| 18 | qwen3-235b-a22b-2507 | 0.643 ± 0.084 | 59% | 89% | 74% | 72% | 28% | 79% |
| 19 | gemini-3.1-flash-lite | 0.518 ± 0.091 | 49% | 100% | 59% | 50% | 28% | 25% |
| 20 | gpt-oss-120b | 0.465 ± 0.088 | 42% | 100% | 56% | 33% | 25% | 38% |
| 21 | claude-haiku-4.5 | 0.446 ± 0.089 | 41% | 78% | 41% | 44% | 28% | 29% |
| 22 | qwen3-32b | 0.356 ± 0.085 | 31% | 33% | 52% | 36% | 11% | 38% |
| 23 | gpt-oss-20b | 0.302 ± 0.075 | 22% | 56% | 30% | 22% | 8% | 54% |
How graded reward is scored: 1.0 for naming the correct culprit commit; 0 for blaming an innocent deploy that just happened to land near the incident (the worst possible answer, because it sends the response team the wrong way); otherwise up to 0.5 partial credit when the commit is wrong but the diagnosis (failing service, blast radius, remediation) is right. Averaged over 108 trials with a 95% confidence interval; overlapping intervals mean the models are effectively tied.
Key finding
The v3 adversarial tier resolves the five-way tie at a perfect 1.000: kimi-k3 now leads at 0.991 graded, with claude-opus-5 and glm-5.2 at 0.981. The twelve new scenarios (guilty-looking decoys exonerable only by reading code semantics, beyond-context data volumes, degraded telemetry with clock skew and 1% trace sampling, and an abstention trap) land in the discriminating range by design: per-scenario pass rates across the field span 39% to 80%, so nothing is saturated and nothing is unsolvable. The no-code-cause column still measures confabulation resistance, and the tail still confabulates: gemini-3.1-flash-lite and claude-haiku-4.5 convict an innocent commit on most no-cause incidents.
Scenarios
| Scenario | Type | What it tests |
|---|---|---|
| checkout-latency-n-plus-one | Real culprit | An added per-item catalog query inside the order loop spikes checkout p99 (N+1). |
| payment-nil-deref-panic | Real culprit | A missing nil-check on an optional 3DS field panics every charge. |
| inventory-connection-pool-exhaustion | Real culprit | Inventory exhausts its pool, but the culprit is the shared DB client library. |
| recommendation-memory-leak | Real culprit | A package-level slice grows unbounded and OOMKills long after the deploy. |
| auth-jwt-validation-regression | Real culprit | Every service rejects tokens at once. The culprit is the shared JWT verify library. |
| cache-ttl-stampede | Real culprit | A TTL change stampedes productdb minutes after the deploy. |
| frontend-race-condition-5xx | Real culprit | A removed mutex in the rate limiter races under load and throws 5xx. |
| grpc-deadline-too-tight | Real culprit | A tightened gRPC deadline on a catalog call starts failing slow requests. |
| unbounded-query-delayed-onset | Real culprit | A deleted LIMIT clause makes a search query degrade as data grows. |
| logging-debug-disk-fill | Real culprit | A config flip to debug logging at full sample rate fills the disk. |
| search-mapping-query-break | Real culprit | A renamed index field (title to name) breaks search queries. |
| dashboard-db-schema-missing-table | Real culprit | Code queries a favorites table whose migration never shipped. |
| ai-agent-registration-missing | Real culprit | A bootstrap refactor deletes the agent-registration call at startup. |
| metric-ingestor-metadata-deser | Real culprit | A shared schema field rename (meta to metadata) breaks deserialization. |
| olapdb-tso-cas-retry-budget | Real culprit | A shared TSO client drops its CAS retry budget, causing timeouts. |
| dynamodb-write-capacity-breach | Real culprit | A shared persistence-lib change drives writes past DynamoDB capacity. |
| transformer-dependency-startup-crash | Real culprit | A protobuf-runtime version clash crashes the transformer into CrashLoopBackOff. |
| bad-data-poison-record | No code cause | An external partner feed sends one poison record. No commit. Answer: none. |
| upstream-payment-provider-outage | No code cause | An external Stripe outage, visible in the status feed. Answer: none. |
| cloud-region-impairment | No code cause | An AWS S3 us-east-1 regional impairment. Answer: none. |
| dns-resolver-degradation | No code cause | Cluster DNS and upstream resolver degradation, not a commit. Answer: none. |
| tls-cert-expiry | No code cause | The payment service leaf certificate expired. Answer: none. |
| traffic-surge-flash-sale | No code cause | A flash sale drives a 6x organic surge, not a regression. Answer: none. |
| noisy-neighbor-node-saturation | No code cause | A batch pod lands on the node and starves its neighbors. Answer: none. |
| shared-config-pool-drain | adversarial | A guilty-looking decoy sits on the loud service; the real culprit is a de-fingerprinted shared-library pool refactor, provable only from derived evidence. |
| orders-index-migration-drift | adversarial | An index-consolidation migration detonates roughly 40 hours after deploy; every recent deploy looks more guilty than the real one. |
| webhook-keepalive-default-flip | adversarial | The culprit is a dependency bump that flips a keep-alive default; the failing service's own deploys are all innocent. |
| ingester-flush-interval-oom | adversarial | An adaptive flush-scheduling change OOMs the ingester while the log patterns point somewhere else entirely. |
| notification-buffer-oom-silent | adversarial | The failing service goes silent at onset: its own logs stop, and the comment-free culprit diff must be convicted on secondhand evidence. |
| session-cache-clockskew | adversarial | Clock skew scrambles cross-service timestamps; ordering events correctly is the prerequisite to finding the eviction-change culprit. |
| config-fanout-sampled-traces | adversarial | Traces are 1%-sampled; the per-record config-resolution culprit must be inferred from what sampling left behind. |
| catalog-cache-key-cardinality | adversarial | Beyond-context: 307 commits and ~24MB of telemetry; only cache-key cardinality separates the culprit from cache-adjacent decoys. |
| ingest-partition-skew | adversarial | Beyond-context: 267 commits and >8MB of telemetry hiding a partition-skew culprit. |
| orders-fanout-nplusone | adversarial | Beyond-context: 286 commits and >20MB of telemetry hiding an N+1 fanout. |
| report-scheduler-lock-removal | adversarial | Beyond-context: 259 commits and >8MB of telemetry; a removed scheduler lock. |
| payment-refund-poison-batch | adversarial | The abstention trap: a malformed partner refund record is the trigger, not any commit. Saying "none" amid guilty-looking deploys is the only right answer. |
Model-scenario matrix
Every model against every scenario. Toggle between pass rate, average cost, and average time per scenario to see which models handle specific failure patterns well, even when their overall score is lower. Hatched cells are scenarios a model never solved.
Cost efficiency
Average API cost per scenario against pass rate. The dashed line is the Pareto frontier: the most cost-efficient models for a given level of accuracy. 2,484 runs cost $600.42 in total.
Speed vs quality
Average time per scenario against pass rate. The frontier shows the models that balance solution quality against how long they take to reason.
What we measure
Given a frozen incident, the model writes a single JSON answer: the root-cause commit, the first failing service, the blast radius, and a remediation. Only one field gates pass or fail.
- →Graded reward (what the leaderboard ranks on): 1.0 for the right commit, 0 for blaming a decoy deploy, partial credit up to 0.5 for a correct diagnosis with the wrong commit.
- →Primary, binary: root_cause_commit must exactly match the ground-truth culprit SHA (a correct short prefix is accepted).
- →Commit messages are neutralized, so the model must reason from the diff, not the description.
- →Secondary, never fatal: first-failing-service, blast-radius overlap, remediation match, and whether the model fell for the innocent-deploy decoy.
- →Eight incidents have no code cause at all, including the adversarial tier's abstention trap. The correct answer is "none", which measures confabulation resistance.
How scenarios are built
Scenarios are fault injections on a real microservices app (a fork of Online Boutique) or reconstructions of representative production incident classes on a fictional platform. Each is a frozen window of alert, logs, metrics, traces, patterns, and full change context (commits, deploys, flags).
- →Author one regression commit (N+1 query, nil deref, pool exhaustion, memory leak) and surround it with dozens of innocent commits.
- →Plant an innocent deploy near onset to punish the "blame the latest change" heuristic.
- →Allow delayed onset, so the bad deploy can detonate minutes later.
- →Hand-label the ground truth and keep it out of the agent container.
Run it yourself
Requires Harbor, Docker, and an OpenRouter key. Re-score any published trajectory yourself, no API key needed.
git clone https://github.com/edgedelta/root-cause-bench.git
cd root-cause-bench
# put OPENROUTER_API_KEY=... in .env, then:
source .env && uv run harbor run -c configs/all-models-docker.yaml
# summarize a run into a per-model table:
uv run scripts/process_results.py jobs/<timestamp>Put a real AI Teammate on call
Edge Delta's AI Teammates triage, investigate, and find root cause in your stack.