AI Tool Atlas

Hands-on

Coding agents on the same task

We give Claude Code, OpenAI Codex and Gemini CLI an identical small repository and an identical prompt, then record what each one cost and what came out. Every number is read from the tool's own usage report or from the test runner. Runs we have not performed yet say so.

Method: How we review · Protocol and fixtures are published in the project repository.

Runs completed: 1Runs pending: 21of 3 runs
  • Runs completed 1
  • Runs pending 2

Fix a bug, add a feature and a CLI flag in a small Python package

Starting from a 40-line Python package with 5 failing tests: fix CSV date parsing and a trailing blank line, add totals_by_customer(), and add a --top N CLI flag. Standard library only, tests must not be edited.

Opening prompt, identical for every agent
Read README.md and make all tests in tests/ pass. Do not edit the tests. Run the tests yourself before you finish.

What we measure

  • Tests passed out of 5, read from pytest
  • Wall time from sending the prompt to the agent saying it is done
  • Tokens and USD cost as reported by the tool itself (/cost in Claude Code, session summary in Codex)
  • Extra user turns and questions the agent asked
  • Size of the diff (files, lines) from git

Results

Claude Code

5/5tests

Time
00:48
Cost
$1.22
Output tokens
1,820
Extra turns
0
OpenAI Codex

—not yet run

Gemini CLI

—not yet run

Claude Code · cost
$1.22
Claude Code · time
00:48
AgentStatusTestsWall timeTokens in / outCostExtra turnsDiffModel · plan
Claude Code Run Oct 2, 2026 5/5 00:48 162 / 1,820
250,959 cached		 $1.22
reported by the tool		 0 2 files, +22 / −8 claude-fable-5-1 · Headless run (claude -p) on the reviewer's account; cost shown at API list price (costBasis: list), subscription plan not applied · v2.1.284
OpenAI Codex Not yet run
Gemini CLI Not yet run
Claude Code: raw usage output and notes 
{
  "duration_ms": 48154,
  "duration_api_ms": 46010,
  "num_turns": 8,
  "total_cost_usd": 1.2184397500000002,
  "usage": {
    "input_tokens": 162,
    "output_tokens": 1820,
    "cache_read_input_tokens": 250959,
    "cache_creation_input_tokens": 53154
  },
  "modelUsage": {
    "claude-fable-5-1": {
      "inputTokens": 162,
      "outputTokens": 1820,
      "cacheReadInputTokens": 250959,
      "cacheCreationInputTokens": 53154,
      "webSearchRequests": 0,
      "costUSD": 1.2184397500000002,
      "contextWindow": 1000000,
      "maxOutputTokens": 64000,
      "thinkingTokens": 201,
      "canonicalModel": "claude-fable-5-1",
      "provider": "firstParty",
      "costBasis": "list"
    }
  }
}

Opinion: clean argparse implementation, all 5 tests green on the first self-check. Caveats: (1) run via the non-interactive `claude -p` command with a tool allowlist (Read/Edit/Write/Bash for python and git), not the interactive TUI; (2) the reviewer's global Claude Code settings and skills were active, which a fresh install would not have; (3) the agent also removed the old "Total:" output line that no test covered.

How a run works

  1. A fresh copy of the fixture repository is created and committed, so the diff is measurable.
  2. The agent gets the opening prompt above, verbatim, with its default model for the plan named in the results.
  3. No hints. If the agent asks, the only answer is “Use your judgement.” Every extra message is counted.
  4. The run stops when the agent says it is done or after 20 minutes.
  5. Tests are run by us, the diff is measured with git, and tokens and cost are copied from the tool's own usage output.

Limits: one task, one run per agent, defaults only. This is a reproducible spot check, not a leaderboard.