Measuring whether repository-knowledge tools improve coding-agent outcomes across correctness, time, tokens, and tool calls.
Leaderboard
Hover to inspect a point. Select a point or table row to pin it, then press Escape or select the chart background to clear.
Bars show overall average correctness. Whiskers show the minimum–maximum of the three repetition averages; each repetition covers all three issues. Native Codex is the baseline: Codex without a repository-knowledge tool.
How to read the metrics
Average across 9 runs per tool (arithmetic mean).
- Behavioral correctness
- Protected outcome score: 80% requested behavior and 20% regression safety. Higher is better.
- Solve time
- Wall-clock time for the Codex implementation process only. Installation, setup, indexing, and verification are excluded.
- Weighted tokens
- Uncached input + 10% of cached input + output, including reasoning. Cached input is downweighted because it is reused rather than entirely new input. This models workload, not price.
- Tool calls
- Calls started by the coding agent during the solve. Fewer calls can mean a shorter interaction, but not necessarily a better result.
Codebase Knowledge Bench is a controlled study of the same coding agent solving the same tasks with different repository-knowledge systems. The published run compares seven setups over three original engineering issues and three repetitions.
- Controlled comparison: every workflow uses gpt-5.6-sol with high reasoning effort against the same task set.
- Multiple outcomes: correctness is shown alongside elapsed time, modeled weighted tokens, and tool-call volume.
- Complete execution: all 63 benchmark runs completed.
- Measured, not universal: three issue clusters limit generalization beyond this benchmark.
jCodeMunch and Sverklo shared the highest observed correctness at 91.7%. No knowledge workflow reached the declared support threshold for a correctness improvement, so the point estimates should be read with the benchmark's stated uncertainty.
Benchmark setup
Inspect the run behind the ranking.
- Model
- gpt-5.6-sol
- Reasoning
- high
- Design
- 3 issues × 3 repetitions × 7 setups
- Primary outcome
- Behavioral correctness