Traceway AI Benchmarks
We believe you should bring your own AI. Traceway isn't here to sell you an agent. It lets the model you already trust reach your production data, on your terms. These benchmarks show how well different models do on our eval suite, and help us make it better.
Pass rate, model by model.
Every model ran the same 172 runs. A run passes when its answer is right, its evidence is grounded and its writes are safe. Tool-call counts are reported but never gate.
Pass rate by model, all 172 runs
Traceway agent evals, run 2026-10-21. Performance (35 cases), Debug (14 cases) and Write (13 cases) on 5 setups, 1 run per case, graded by deterministic code.
Tracewayanthropic/claude-haiku-5.5
- Right answer with evidence
- 126/172
- Agent cost
- $1.90
- Cost per run
- $0.011
deepseek/deepseek-v4.1-flash-1017
- Right answer with evidence
- 149/172
- Agent cost
- $2.17
- Cost per run
- $0.013
anthropic/claude-opus-5.5
- Right answer with evidence
- 161/172
- Agent cost
- $51.81
- Cost per run
- $0.301
Every benchmark, side by side.
Pass rates per suite, setup and case group, then what the runs cost. The best result in each row is in bold, and a shaded cell means that model led every number in the row.
| Claude Opus 5.5Anthropic | DeepSeek V4.1 Flashopen weights | Claude Haiku 5.5Anthropic | |
|---|---|---|---|
| Performance diagnosisperf · 35 cases, CLI | 94.3%with the skill85.7%without the skill | 77.1%with the skill77.1%without the skill | 80.0%with the skill57.1%without the skill |
| Performance diagnosis over MCPperf · 35 cases, MCP server | 91.4% | 88.6% | 77.1% |
| Root-cause debuggingdebug · 13 cases, CLI | 84.6%with the skill92.3%without the skill | 84.6%with the skill84.6%without the skill | 53.8%with the skill30.8%without the skill |
| Root-cause debugging over MCPdebug · 13 cases, MCP server | 84.6% | 92.3% | 69.2% |
| Safe write actionswrite · 13 cases | 100.0%write CLI + skill100.0%write MCP server | 92.3%write CLI + skill100.0%write MCP server | 76.9%write CLI + skill92.3%write MCP server |
| Controls and trapsperf-ctl-* · 6 cases, 18 runs | 100.0% | 61.1% | 50.0% |
| Databaseperf-db-* · 8 cases, 24 runs | 95.8% | 95.8% | 91.7% |
| App codeperf-app-* · 5 cases, 15 runs | 93.3% | 86.7% | 73.3% |
| Network and upstreamsperf-net-* · 6 cases, 18 runs | 94.4% | 88.9% | 83.3% |
| Cachingperf-cache-* · 3 cases, 9 runs | 66.7% | 55.6% | 77.8% |
| Infrastructureperf-infra-* · 4 cases, 12 runs | 75.0% | 66.7% | 41.7% |
| Cross-stack and mobileperf-xs-* · 3 cases, 9 runs | 88.9% | 100.0% | 66.7% |
| Backend fixesdebug-backend-* · 9 cases, 27 runs | 85.2% | 96.3% | 37.0% |
| Mobile crashesdebug-mobile-* · 3 cases, 9 runs | 100.0% | 88.9% | 100.0% |
| Debug controls and policydebug-ctl-* · 2 cases, 5 runs | 60.0% | 0.0% | 20.0% |
| Overall pass rateall 172 runs | 90.7% | 83.7% | 68.0% |
| Right answer and evidencetrajectory check not counted | 93.6% | 86.6% | 73.3% |
| Agent spendall 172 runs, lower is better | $51.81 | $2.17 | $1.90 |
| Cost per passing runagent spend ÷ passed runs, lower is better | $0.332 | $0.015 | $0.016 |
One run per case and setup, so a single run moves the smallest groups (5 runs) by 20 points. Open a PR or email dusan@tracewayapp.com if you'd like to see your model featured.
What the runs say, and what they don't.
01 · Judgment
Opus never false-alarms.
The control cases are traps: a healthy endpoint, endpoints marked slow on purpose, a misleading p95 metric, a pasted alert. Opus passed 18 of 18 control runs. DeepSeek passed 11 and Haiku passed 9. The gap between the models is in judgment, not in reading telemetry.
02 · Value
DeepSeek is the value pick.
84% of its runs passed for $2.17 of agent spend, $0.015 per passing run. Haiku is next at $0.016 per passing run: 68% passed for $1.90, 27 fewer passing runs. Opus passed 12 more again, for 24× DeepSeek's spend. On MCP perf, DeepSeek is 2pp behind Opus.
03 · The skill
The skill helps the weakest model most.
On the perf suite, CLI + skill against the same CLI without it: Haiku +23pp, Opus +9pp and DeepSeek ±0pp. On the debug suite: Haiku +23pp, Opus −8pp and DeepSeek ±0pp. DeepSeek is the only model that does better on MCP perf (89%) than on CLI + skill (77%).
04 · Our bugs
Some failures are ours.
Four problems behind these failures are in our own cases and skill, not in the models. They account for 7 of Haiku's 55 failures, 5 of DeepSeek's 28 and 6 of Opus's 16. We still count them as failures. Here they are.
05 · Fixing code
Opus fixes the most bugs.
On the debug suite the agent has to find the culprit from the telemetry and patch it, and hidden tests check the fix. Opus passed 35 of 41, DeepSeek passed 34 of 41 and Haiku passed 20 of 41. Backend fixes checked by hidden tests: Haiku 10/27, DeepSeek 26/27 and Opus 23/27; mobile crashes checked statically: Haiku 9/9, DeepSeek 8/9 and Opus 9/9. No model handled the cleanup request with a write credential cleanly on both setups.
perf-infra-disk-ioThe skill's CPU recipe counts iowait as busy CPU
On the disk case the database host's idle CPU falls while iowait rises. The skill defines busy CPU as 1 minus idle, so runs that followed it called the disk bottleneck CPU saturation, which is the case's distractor.
Runs hit: Haiku 2 · DeepSeek 1 · Opus 0
perf-cache-stampedeThe cache stampede case insists on a logs query
The case requires a logs.query call, but agents now find the burst of identical slow rebuilds through the newer endpoint request listing. These runs had the right answer and grounded evidence.
Runs hit: Haiku 2 · DeepSeek 1 · Opus 3
perf-infra-memory-gcThe memory case rejects a fair label
These runs found the memo that never evicts, but filed it under unbounded in-memory work instead of memory pressure and GC pauses. The grader treats that label as the distractor, though an unbounded memo is honestly both.
Runs hit: Haiku 1 · DeepSeek 1 · Opus 2
perf-net-chatty-fanoutThe fan-out case fails grounding for every model
Every model fails this case's grounding check on most setups: the case wants the two fanned-out routes, GET /stock/{sku} and GET /prices/{sku}, in the output of a cited call, and the models cite the aggregate view that shows the fan-out without those route names.
Runs hit: Haiku 2 · DeepSeek 2 · Opus 1
Pass rate by suite and setup.
Performance and Debug cases run with the CLI and skill, the CLI alone as a control group, and the MCP server. Write cases, and the debug case that hands the agent a write credential, run with the write CLI and skill, and the write MCP server.
| Setup | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |
|---|---|---|---|
| Performance · 35 cases | |||
| CLI + skillcli+skill | 28/3580% 29/35 right answer + evidence | 27/3577% 27/35 right answer + evidence | 33/3594% 34/35 right answer + evidence |
| CLI, no skillcli-noskill | 20/3557% 23/35 right answer + evidence | 27/3577% 29/35 right answer + evidence | 30/3586% 32/35 right answer + evidence |
| MCP servermcp | 27/3577% 27/35 right answer + evidence | 31/3589% 32/35 right answer + evidence | 32/3591% 34/35 right answer + evidence |
| Debug · 14 cases | |||
| CLI + skillcli+skill | 7/1354% 9/13 right answer + evidence | 11/1385% 11/13 right answer + evidence | 11/1385% 11/13 right answer + evidence |
| CLI, no skillcli-noskill | 4/1331% 6/13 right answer + evidence | 11/1385% 12/13 right answer + evidence | 12/1392% 12/13 right answer + evidence |
| MCP servermcp | 9/1369% 9/13 right answer + evidence | 12/1392% 12/13 right answer + evidence | 11/1385% 11/13 right answer + evidence |
| Write CLI + skillcli-write+skill | 0/10% 0/1 right answer + evidence | 0/10% 1/1 right answer + evidence | 1/1100% 1/1 right answer + evidence |
| Write MCP servermcp-write | 0/10% 0/1 right answer + evidence | 0/10% 0/1 right answer + evidence | 0/10% 0/1 right answer + evidence |
| Write · 13 cases | |||
| Write CLI + skillcli-write+skill | 10/1377% 11/13 right answer + evidence | 12/1392% 12/13 right answer + evidence | 13/13100% 13/13 right answer + evidence |
| Write MCP servermcp-write | 12/1392% 12/13 right answer + evidence | 13/13100% 13/13 right answer + evidence | 13/13100% 13/13 right answer + evidence |
| All runs172 per model | 117/17268% 126/172 right answer + evidence | 144/17284% 149/172 right answer + evidence | 156/17291% 161/172 right answer + evidence |
| Differences | |||
| Skill uplift, perfCLI + skill minus CLI without the skill | +23pp80% vs 57% | ±0pp77% vs 77% | +9pp94% vs 86% |
| Skill uplift, debugCLI + skill minus CLI without the skill | +23pp54% vs 31% | ±0pp85% vs 85% | −8pp85% vs 92% |
| CLI vs MCP, perfCLI + skill minus MCP server | +3pp80% vs 77% | −11pp77% vs 89% | +3pp94% vs 91% |
| CLI vs MCP, debugCLI + skill minus MCP server | −15pp54% vs 69% | −8pp85% vs 92% | ±0pp85% vs 85% |
| CLI vs MCP, writeWrite CLI + skill minus Write MCP server | −15pp77% vs 92% | −8pp92% vs 100% | ±0pp100% vs 100% |
“Right answer + evidence” counts runs whose answer and grounding passed, whatever happened to the trajectory check. The differences are in percentage points, computed before rounding.
Where each model wins and loses.
Passes over every setup a case ran on. Groups follow the case names: perf-db-n-plus-1 is in Database, perf-ctl-healthy in Controls and traps, and the debug groups follow the stack the fix is in.
| Group | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |
|---|---|---|---|
| App codeperf-app-*Slow code in the request path: serialization, quadratic loops, large allocations, debug logging, work that belongs in the background.5 cases · 15 runs per model | 11/1573% | 13/1587% | 14/1593% |
| Cachingperf-cache-*Cold starts after a release, a missing cache, a cache stampede.3 cases · 9 runs per model | 7/978% | 5/956% | 6/967% |
| Controls and trapsperf-ctl-*A healthy endpoint, endpoints marked slow on purpose, a misleading metric, a pasted alert and an onset to pin down. Some have nothing to fix.6 cases · 18 runs per model | 9/1850% | 11/1861% | 18/18100% |
| Databaseperf-db-*N+1 queries, a missing index, pool exhaustion, lock contention, over-fetching, unbounded results.8 cases · 24 runs per model | 22/2492% | 23/2496% | 23/2496% |
| Infrastructureperf-infra-*CPU saturation, disk I/O, memory pressure, a noisy neighbor.4 cases · 12 runs per model | 5/1242% | 8/1267% | 9/1275% |
| Network and upstreamsperf-net-*Retries, chatty fan-out, sequential calls, large payloads, connection reuse, a slow third party.6 cases · 18 runs per model | 15/1883% | 16/1889% | 17/1894% |
| Cross-stack and mobileperf-xs-*A browser or mobile app fails, and the agent has to follow it into the backend.3 cases · 9 runs per model | 6/967% | 9/9100% | 8/989% |
| Backend fixesdebug, Go / Python / NodeA panic, a lowered timeout, a race, a renamed field, a config error at startup, two bugs under one issue hash. The patch has to pass hidden tests.9 cases · 27 runs per model | 10/2737% | 26/2796% | 23/2785% |
| Mobile crashesdebug, Android / iOS / FlutterA Kotlin, Swift or Dart crash: find the culprit from the stack trace and patch it. The patch is checked statically.3 cases · 9 runs per model | 9/9100% | 8/989% | 9/9100% |
| Debug controls and policydebug controlsA rare 500 the telemetry cannot explain, and a cleanup request made with a write credential.2 cases · 5 runs per model | 1/520% | 0/50% | 3/560% |
| Write actionswrite-*Acknowledge or resolve pages, snooze or disable alerts, run a monitor, archive issues, write post-mortems, refuse what a token may not do.13 cases · 26 runs per model | 22/2685% | 25/2696% | 26/26100% |
The group's best count is set in bold. Groups hold 2 to 13 cases, so a single run moves a small group a lot. Caching, Infrastructure and Network and upstreams include the case and skill problems listed under Key findings.
What a diagnosis costs.
Agent spend is the model cost the harness recorded for each run, through OpenRouter. The judge's spend is separate. Opus cost 27× as much as Haiku for the same 172 runs.
Cost per run against pass rate
Up is better, left is cheaper. Hover or focus a dot for its numbers; every value is also in the tables on this page.
| All 172 runs | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |
|---|---|---|---|
| Agent spend | $1.90 | $2.17 | $51.81 |
| Average cost per run | $0.011 | $0.013 | $0.301 |
| Cost per passing run | $0.016 | $0.015 | $0.332 |
| Average tool calls per run | 14.4 | 22.2 | 13.3 |
| Average wall time per run | 56.1s | 68.7s | 79.0s |
| Cache hit rate (input tokens) | 85% | 91% | 87% |
| Average judge score (0–1) | 0.78 | 0.89 | 0.93 |
| Judge spend | $0.183 | $0.236 | $0.174 |
Runs went three at a time on one machine, in three batches (perf and write on 2026-10-21; debug on 2026-10-22; the debug CLI + skill runs again on 2026-10-22), so wall time includes some contention. The judge scored 171 of Haiku's 172 runs.
The full matrix: every case, setup and model.
One cell per run. Hover a cell for the first reason the graders gave, or click a row to read every failure of that case.
Performance · 35 cases × 3 setups × 3 models
| Case | CLI + skill | CLI, no skill | MCP server | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Haiku | DeepSeek | Opus | Haiku | DeepSeek | Opus | Haiku | DeepSeek | Opus | |
| App code | |||||||||
| Caching | |||||||||
| Controls and traps | |||||||||
| Database | |||||||||
| Infrastructure | |||||||||
| Network and upstreams | |||||||||
| Cross-stack and mobile | |||||||||
| Passed | 28/35 | 27/35 | 33/35 | 20/35 | 27/35 | 30/35 | 27/35 | 31/35 | 32/35 |
Debug · 14 cases × 5 setups × 3 models
| Case | CLI + skill | CLI, no skill | MCP server | Write CLI + skill | Write MCP server | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Haiku | DeepSeek | Opus | Haiku | DeepSeek | Opus | Haiku | DeepSeek | Opus | Haiku | DeepSeek | Opus | Haiku | DeepSeek | Opus | |
| Backend fixes | |||||||||||||||
| Mobile crashes | |||||||||||||||
| Debug controls and policy | |||||||||||||||
| Passed | 7/13 | 11/13 | 11/13 | 4/13 | 11/13 | 12/13 | 9/13 | 12/13 | 11/13 | 0/1 | 0/1 | 1/1 | 0/1 | 0/1 | 0/1 |
Write · 13 cases × 2 setups × 3 models
| Case | Write CLI + skill | Write MCP server | ||||
|---|---|---|---|---|---|---|
| Haiku | DeepSeek | Opus | Haiku | DeepSeek | Opus | |
| Passed | 10/13 | 12/13 | 13/13 | 12/13 | 13/13 | 13/13 |
How we measure.
Each task is a fresh Traceway backend seeded with weeks of telemetry from a system broken in one way, plus a planted red herring. The model works with the real CLI or MCP server, and deterministic graders check its answer, its evidence and anything it changed. Perf and write ran on 2026-10-21 at commit 3b00b083; Debug ran on 2026-10-22 at commit 21bbe026; The debug CLI + skill runs ran again on 2026-10-22, with the skill fix on top of commit 21bbe026, 1 run per case. Every run was scored by the graders as they stood on 2026-10-22: after a grader fix the saved transcripts are re-scored, so no number here compares two graders.
Give your agent production context.
Install the skill or connect the MCP server, and let your agent take the next slow endpoint or the next crash.