ORCA-bench: How Ready Are Language Model Agents for Oncall?

Getting started

Stage the snapshot's /app/ into a host-side cache keyed by image id, then bind-mount it read-only into every trial. Requires docker and the harbor CLI.

# Stage the snapshot's /app/ to a host-side cache keyed by image id.
SNAPSHOT_IMAGE=orcabench/sre-otel-snapshot:data-0418-harbor-template
docker pull "$SNAPSHOT_IMAGE"
CACHE_DIR="$HOME/.cache/sre-snapshot-cache/$(docker image inspect "$SNAPSHOT_IMAGE" -f '{{.Id}}' | cut -d: -f2 | cut -c1-12)"
[ -z "$(ls -A "$CACHE_DIR" 2>/dev/null)" ] && { mkdir -p "$CACHE_DIR"; CID=$(docker create "$SNAPSHOT_IMAGE"); docker cp "$CID:/app/." "$CACHE_DIR/"; docker rm "$CID"; }

# Run the Harbor trials with the cache (read-only) bind-mounted
SNAPSHOT_CACHE_HOST_DIR="$CACHE_DIR" harbor run \
    --mounts-json "[{\"type\":\"bind\",\"source\":\"$CACHE_DIR\",\"target\":\"$CACHE_DIR\",\"read_only\":true}]" \
    -d orca-bench/ORCA-bench

TL;DR

We pair a live OpenTelemetry-instrumented microservice system — six days of metrics, logs, and traces plus full source-code access — with 1,079 root-cause-analysis tasks.
Across five frontier agents, the best RCA accuracy is just 25.3% on realistic Medium-difficulty tasks and 10.0% on Hard.
The weakest model hallucinates an implausible root cause in roughly 40% of incident reports.
Removing source-code access degrades every metric — code is not optional context.

The problem

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began.

ORCA-bench puts general-purpose coding agents in a production-fidelity oncall setting. It pairs a live, OpenTelemetry-instrumented microservice system — exposing six days of telemetry through real interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access — with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's κ_w = 0.88).

1,079

RCA tasks

6 days

of live telemetry

50 GB

curated testbed

0.88

judge–human κ_w

Headline results

RCA accuracy and the depth of investigation across five frontier models, with access to both code and telemetry (incident tasks, N=884). Each model is paired with the Terminus-2 agent (reasoning effort = medium, temperature = 1, max tokens = 16,384, max timeout = 1 hr). Bars show mean ± 1 SE.

RCA Accuracy vs. RCA Depth

Mean ± 1 SE in percent. Depth near 48% means the top agents typically confirm symptoms and make partial progress (a per-rubric score around 1.4 of 3) — but Accuracy under a third shows they rarely pin down every plausible root cause.

Accuracy falls off a cliff as tasks get harder

RCA accuracy stratified by prompt difficulty. Because Accuracy credits an agent only when it names every plausible fault, it collapses on Hard tasks (avg. 4.4 concurrent ground-truth events): the best agent drops from ~59% on Easy to 10% on Hard.

Hallucination: implausible root causes

Share of non-empty reports naming a root cause that matches none of the plausible faults. It doesn't track capability: DeepSeek-V4-Pro scores low on accuracy yet hallucinates least (7%), while GLM-5 both scores low and hallucinates most (40%). Lower is better.

Watch the agents work

Replay all five agents on the same Hard task (orca-bench/9bdcf561ef5292ea). The reported symptom is only “shoppers cannot use their cart” (ground truth: the productCatalogFailure fault flag). Claude Opus 4.7 and GPT-5.5 trace failed checkouts through ProductCatalogService to the enabled flag and nail it (GPT-5.4 judge 3/3); the other three miss the active root cause — e.g. GLM-5 latches onto a persistent background issue.

Source code is not optional

Removing codebase access — leaving agents with telemetry only — degrades every metric for every model.

Full results table

Incident tasks (N=884), code + telemetry access. All values mean ± 1 SE in percent; judge is GPT-5.4. Best value in each column highlighted. Click any column header to sort.

Model↕	RCA Depth↕	RCA Accuracy↕	Hallucination↕	Mechanism↕	Inc. Time↕	Metrics (any)↕	Logs (any)↕	Traces (any)↕

How the benchmark works

📡

Real telemetry

Agents query live Prometheus, Jaeger, and OpenSearch through Grafana — the same interfaces a human oncall engineer uses.

🧩

Varied difficulty

Tasks systematically vary report specificity, time-to-detection, and co-occurring faults across Easy, Medium, and Hard.

✅

Expert ground truth

Symptoms are curated and signed off by expert SREs; the LLM judge is validated against human scores at κ_w=0.88.

These results come from a curated 50 GB / six-day testbed with tasks investigated in isolation, on a system whose code and instrumentation are public. Real production systems are orders of magnitude larger, more dynamic, and more idiosyncratic — so the gap reported here is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability.

Citation

@inproceedings{orcabench2026,
  title     = {ORCA-bench: How Ready Are Language Model Agents for Oncall?},
  author    = {Gong, Albert and Choi, Kyuseong and Agarwal, Abhineet and Schechner, Jason and Huang, Ryan and Agrawal, Raj and Agarwal, Anish and Dwivedi, Raaz},
  booktitle = {Under review},
  year      = {2026},
  url       = {https://hub.harborframework.com/datasets/orca-bench/ORCA-bench}
}

@article{gong2026orca,
  title   = {ORCA-bench: How Ready Are Language Model Agents for Oncall?},
  author  = {Gong, Albert and Choi, Kyuseong and Agarwal, Abhineet and Schechner, Jason and Huang, Ryan and Agrawal, Raj and Agarwal, Anish and Dwivedi, Raaz},
  journal = {arXiv preprint arXiv:2607.28545},
  year    = {2026}
}