Do codebase knowledge tools help Codex produce better results, or achieve similar quality with lower cost or less time?
The benchmark runs the same coding tasks with Codex alone and with each tool, then compares the result, model cost, and coding time.
Get benchmark updatesSummary
Some tools produced the same result as Codex alone, but none produced a better result.
Serena, Sverklo and Prethink produced results similar to Codex alone, but with different trade-offs:
- Serena: Model cost was 4.5% lower, but coding time was 11.2% longer.
- Sverklo: Coding time was 3.0% shorter, but model cost was 6.9% higher.
- Prethink: Coding time was 1.7% shorter, but model cost was 20.7% higher.
The other four tools either had a worse result or a similar result without reducing cost or coding time.
Results
Similar results
Worse results
Model cost and coding time are averages per run.
How to read these results
No row selected.
Why this benchmark exists
Codebase knowledge tools are often benchmarked with prepared search queries: can the tool find the expected file, symbol, or code block? That is useful, but it does not tell you whether a coding agent will actually use the tool, whether the final implementation will be better, or whether its influence on model cost or coding time will make it worth installing.
This benchmark tests the complete coding workflow. Codex solves the same issue alone and with each tool. The model, task, starting code, and tests are the same every time.
It differs from retrieval-only benchmarks in five ways:
- Complete coding tasks: Codex investigates the issue, changes the code, runs tests, and fixes mistakes. The final implementation is scored, not the search result.
- Strong baseline: Codex alone can still use the shell, ripgrep (rg), Git, read files, and run tests.
- Actual tool use: The benchmark records whether Codex used the tool and whether the returned context was relevant. It reports the overall result separately from evidence that the tool itself helped.
- Independent and open: None of the evaluated tool vendors created the benchmark. The benchmark code and data are public.
- Reusable for your codebase: You can run the benchmark on your own repository using solved issues and trusted tests.
The current results cover one Java codebase, three issues, and four repetitions. Different codebases and tasks can produce different results. Run the benchmark on your own codebase to see what helps there.
Tasks behind these results
These results use three resolved issues from Symphony for Trello, a Java application built with Quarkus. Each issue has a known-good implementation and tests that check the required behavior. Together, the issues cover command-line parsing, configuration, and runtime behavior.
Configuration for these results
- Model
- gpt-5.6-sol
- Reasoning level
- High
- Codex CLI
- 0.146.0
- Codebase
- Symphony for Trello
- Language and framework
- Java and Quarkus
- Tools
- 7
- Baseline
- Codex alone
- Issues
- 3
- Repetitions
- 4
- Runs
- 96
- Total model cost at API prices
- $358.63
Get benchmark updates
Get notified when I publish results for new tools, codebases, programming languages, or tasks. I also share practical advice about software quality and coding with AI.
About one email per week. Unsubscribe anytime.