Can a model you host yourself diagnose incidents like a frontier one?

How we tested

We built edgedelta-32B by fine-tuning the base 32B on 256 golden threads, the investigation transcripts our AI Teammates leave behind. This is plain supervised fine-tuning with QLoRA: the model learns only from the agent's own tool calls and final answer, never the incident text it reads, and there is no reward model or reinforcement learning on top. A deterministic hash splits threads into train and held-out sets, so the 30 Edge Delta incidents scored here share no thread with anything the model trained on. The 20 generic scenarios go further, since they are the stock sev1-bench set and never touched Edge Delta data at all.

Every model runs through the same agent, the smolagents implementation from sev1-bench, so the only variable is the model itself.

The result

The headline metric is the hybrid judge with remediation excluded. Golden threads capture diagnosis, not remediation, so remediation ground truth is weak; dropping it isolates root-cause quality. The population is the 30 real Edge Delta incidents. Bars show mean score, with pass rate at the 0.7 threshold.

On Edge Delta incidents, edgedelta-32B beats base 32B by 42% (0.95 vs 0.67), passes 28 of 30, and closes most of the gap to the frontier reference (GPT-5.5 at 1.00) with 32 billion parameters. It holds up off home turf too: on the generic SRE set it stays ahead of its base (0.71 vs 0.61), with no catastrophic forgetting. Size alone isn't the answer either: the raw base 235B reaches 0.90 on our incidents, and the 32B edgedelta model still edges it. The 235B pulls ahead only on generic scenarios (0.79 vs 0.71).

By population

Each run is scored two ways. The deterministic scorer checks the answer for exact keyword hits and scans for forbidden actions; it is objective but strict, so it undercounts correct answers phrased differently. The hybrid scorer keeps that forbidden-action check but hands root cause and remediation to an LLM judge (claude-opus-4.6) that reads the answer and decides whether it names the real cause. Hybrid is the metric we trust; deterministic is a floor. Within each, w/rem uses the standard weights (root cause 0.5, remediation 0.3, no forbidden actions 0.2) and no-rem drops remediation and renormalizes, since golden threads capture diagnosis, not fixes. Cells show mean score with pass rate in parentheses.

Per-scenario: Edge Delta incidents

Hybrid judge, remediation excluded. 1.00 means the correct root cause; 0.29 means missed. Base 32B fails a cluster of crashloop, secret, and escalation incidents that edgedelta-32B solves. The raw 235B recovers a different set and even lands byconity-karpenter-eviction, a hard multi-signal case the 32B models miss, but it drops others edgedelta-32B gets (k8s-diskpressure-karpenter-capacity, slack-429). byconity-sqs-backlog-scheduling is the hard case only GPT-5.5 solves.

Where edgedelta-32B still misses

On our 30 incidents it misses just two, and they rhyme. In both, edgedelta-32B finds the failing ByConity pod and the downstream cascade, then blames an application-level cause instead of the Kubernetes trigger. This is the hard infra-trigger class, the one place the larger base still does better.

On the generic set, the misses cluster into three modes: huge-context overflows (a 500KB log and a large join exceed even the 131K serving window), noise rejection (over-diagnosing a scheduled ETL job or an acknowledged upstream Stripe outage), and multi-incident triage (collapsing concurrent incidents into one false cascade).

The fine print

edgedelta-32B

Our 32B fine-tune, served self-hosted on vLLM. Incident data never leaves your environment.

base 32B

The same compact model, un-tuned. The head-to-head.

base 235B

A 235B model. The scale reference.

GPT-5.5

The frontier model we benchmarked against (Apr 2026). Parameter count undisclosed; ~4.7T is an independent estimate.

The 30 Edge Delta incidents come from held-out golden threads; the 20 generic cases are the standard sev1-bench set. We recomputed every number from the raw score files, and our scorer matches the sev1-bench repo's on shared scenarios.

golden-threads · sev1-bench · scored 2026-07-17 · hybrid judge claude-opus-4.6 · 4 models × 50 scenarios