agentmark — Benchmark AI Coding Agents on Your Codebase

Objective

Build an open-source Python CLI that lets developers benchmark and compare AI coding agents (Claude Code, Aider, Codex, etc.) on their own codebase with their own tasks. Produce a beautiful terminal report showing pass/fail, time, cost, token usage, and LLM call counts.

Goal: Become the de facto standard for answering "which AI agent works best for MY code?" — the Hyperfine of AI coding agents.

Viral thesis: Benchmark comparison results are inherently shareable. Developers will screenshot the results table and post it on Twitter/HN/Reddit. Tribal loyalty (Claude fans vs GPT fans) drives organic engagement. Every comparison post creates more users.

Research Foundation

Based on: "ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory" — Zhong et al., 2025 (https://arxiv.org/abs/2602.20502)

Key Paper Findings That Motivate This Tool

Reactive vs programmatic agents differ by 11.8x in cost. Reactive agents (observe → LLM call → act → repeat) require O(N) LLM calls. Programmatic agents (plan once → execute) require O(1). The paper measured 11.8x cost difference and 2x latency improvement for the same tasks. Different coding agents have different strategies — agentmark quantifies which are wasteful.
Token usage varies 5.67x across approaches. For identical tasks, input tokens ranged from 8.1K to 62.3K depending on the agent architecture. This is invisible without measurement.
LLM calls vary 5.67x. Average 1.8 calls (programmatic) vs 10.2 calls (reactive) per task. Some coding agents re-read the same file repeatedly. agentmark exposes this.
Model choice causes 20-30% failure rate differences. With weaker planning models (GPT-4o), 20-30% of executions need validation and hot-patching. Stronger models (Claude Sonnet) rarely do. agentmark measures pass rates per agent.
Task ambiguity causes 6% of failures. The paper found that 6 of 106 task failures came from ambiguous specs, not agent limitations. agentmark's structured task format (prompt + validation command) eliminates this variable.

Paper Concept → agentmark Mapping

Paper Concept	agentmark Feature
O(N) vs O(1) LLM calls	Measure LLM call count per agent per task
11.8x cost difference	Track cost per agent per task
Task disambiguation	Structured bench.yaml with clear prompts + validation
Validator with hot-patching	Validation commands (test, typecheck, lint)
State machine isolation	Git worktree per run = clean reproducible state

Competitive Landscape (as of Feb 2025)

What Exists (NOT competitors — they solve different problems)

Tool	What It Does	Why It's Different
SWE-bench	Fixed GitHub issues from fixed repos	Academic; not YOUR codebase, not YOUR tasks
THUDM/AgentBench	Evaluates LLMs in 8 generic environments	Academic framework; not coding-focused, not personal
jadbox/codeagentbench	Predefined coding test cases	Fixed test cases, not your repo; 4 stars, dormant
Terminal-Bench	Tests AI in terminal environments	Academic; general terminal tasks, not code changes
Codex Arena Mode	Compare 2 models side-by-side	Built into Codex only; not cross-tool
Aider leaderboard	Aider + different LLMs	One tool, different models — not different agents
Blog comparisons	Manual side-by-side reviews	Not a tool; not reproducible; opinion-based

The Gap We Fill

No tool exists where you:

Define YOUR tasks on YOUR codebase
Run DIFFERENT agent CLIs (not just different models)
Get isolated runs (git worktrees)
Get structured cost/time/token/success comparison
Get a shareable report

Architecture

Core Loop (simple)

for each task in bench.yaml:
    for each agent in config:
        1. Create git worktree (clean isolated copy)
        2. Start timer + metrics collection
        3. Run agent CLI with task prompt (subprocess)
        4. Parse agent stdout/stderr for token/cost data
        5. Run validation command (test, typecheck, lint)
        6. Record: pass/fail, wall time, tokens, cost, LLM calls
        7. Remove git worktree (cleanup)

Produce comparison table + optional JSON/CSV export

Project Structure

agentmark/
├── pyproject.toml              # Project config, dependencies, entry point
├── README.md                   # Hero demo GIF + quick start
├── src/
│   └── agentmark/
│       ├── __init__.py
│       ├── cli.py              # CLI entry point (click or typer)
│       ├── config.py           # Parse bench.yaml
│       ├── sandbox.py          # Git worktree create/cleanup
│       ├── runner.py           # Core benchmark loop
│       ├── agents/             # Agent adapters (one per tool)
│       │   ├── __init__.py
│       │   ├── base.py         # Abstract agent interface
│       │   ├── claude_code.py  # claude --print adapter
│       │   ├── aider.py        # aider --message adapter
│       │   └── codex.py        # codex adapter
│       ├── metrics.py          # Token/cost/time collection + parsing
│       ├── validator.py        # Run validation commands, check exit codes
│       └── reporter.py         # Rich terminal output (tables, summary)
├── tests/
│   ├── test_config.py
│   ├── test_sandbox.py
│   ├── test_runner.py
│   └── test_metrics.py
└── examples/
    └── bench.yaml              # Example task file

Key Design Decisions

Agents are black boxes. We only interact via their CLI. No SDK integration. This keeps the tool agent-agnostic and easy to extend. Users add new agents by specifying a shell command.
Git worktrees for isolation. Each run gets a clean copy of the repo at the current commit. No cross-contamination between agents or tasks. Cheap to create, automatic cleanup.
Validation is user-defined. The user provides a shell command (e.g., bun test, pytest, tsc --noEmit). Pass = exit code 0. This keeps agentmark domain-agnostic.
Metrics from stdout parsing. Each agent adapter knows how to extract token counts, cost, and LLM calls from that agent's output. Fallback: just measure wall time and pass/fail.
No LLM calls in agentmark itself. The tool is pure orchestration and measurement. Zero AI cost to run the benchmarker.

Task Definition Format

bench.yaml — lives in user's repo root:

# Optional metadata
project: my-app
base: HEAD           # Git ref to benchmark against (default: HEAD)
timeout: 120         # Default timeout per task in seconds

# Tasks to benchmark
tasks:
  - name: fix-type-error
    prompt: "Fix the TypeScript type error in src/auth/login.ts"
    validate: "npx tsc --noEmit"

  - name: add-pagination
    prompt: "Add offset/limit pagination to GET /api/users endpoint"
    validate: "bun test test/api/users.test.ts"
    timeout: 180

  - name: write-tests
    prompt: "Write unit tests for src/payments/charge.ts achieving >80% coverage"
    validate: "bun test test/payments/"

# Agents to compare
agents:
  - name: claude-code
    command: "claude --print --max-turns 10 '{prompt}'"

  - name: aider
    command: "aider --message '{prompt}' --yes-always --no-git"

  - name: codex
    command: "codex --quiet '{prompt}'"

Users can also pass agents via CLI flags:

agentmark run --agent "claude-code:claude --print '{prompt}'"
agentmark run --agent "aider:aider --message '{prompt}' --yes-always"

CLI Interface

# Initialize a bench.yaml in current repo
agentmark init

# Run all tasks against all agents
agentmark run

# Run specific task(s)
agentmark run --task fix-type-error

# Run specific agent(s)
agentmark run --agent claude-code --agent aider

# Output formats
agentmark run --json           # JSON report
agentmark run --csv            # CSV for spreadsheets
agentmark run --md             # Markdown table

# Compare two previous runs
agentmark compare run-001 run-002

# List past runs
agentmark history

Terminal Output (target UX)

$ agentmark run

 agentmark v0.1.0 — benchmarking 3 agents on 3 tasks
 repo: my-app (commit a1b2c3d)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

 Task 1/3: fix-type-error
 "Fix the TypeScript type error in src/auth/login.ts"
 validate: npx tsc --noEmit

   claude-code ····· PASS   15s   $0.08   4.2K tokens   2 LLM calls
   aider ··········· PASS   23s   $0.14   8.7K tokens   5 LLM calls
   codex ··········· PASS   31s   $0.21  12.1K tokens   8 LLM calls

 Task 2/3: add-pagination
 "Add offset/limit pagination to GET /api/users endpoint"
 validate: bun test test/api/users.test.ts

   claude-code ····· PASS   54s   $0.31  18.3K tokens   3 LLM calls
   aider ··········· FAIL   89s   $0.52  34.1K tokens  12 LLM calls
   codex ··········· PASS   72s   $0.44  28.9K tokens   9 LLM calls

 Task 3/3: write-tests
 "Write unit tests for src/payments/charge.ts"
 validate: bun test test/payments/

   claude-code ····· PASS   67s   $0.22  12.1K tokens   2 LLM calls
   aider ··········· PASS   94s   $0.38  22.4K tokens   7 LLM calls
   codex ··········· FAIL  110s   $0.61  41.2K tokens  14 LLM calls

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 RESULTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

                Pass Rate   Avg Time   Avg Cost   Total Tokens   Efficiency
 claude-code    3/3 100%      45s       $0.20       34.6K        *****
 aider          2/3  67%      69s       $0.35       65.2K        ***
 codex          2/3  67%      71s       $0.42       82.2K        **

 Winner: claude-code (highest pass rate, lowest cost, fewest tokens)

 Full report saved to .agentmark/runs/2025-02-25-001.json

Tech Stack

Component	Choice	Reason
Language	Python 3.11+	AI/ML audience lives in Python; PyPI reach
CLI framework	`click`	Mature, composable, well-documented
Terminal UI	`rich`	Beautiful tables, progress bars, panels
Config	`pyyaml`	Standard for YAML parsing
Isolation	`git worktree` via subprocess	Clean, fast, built into git
Package manager	`uv`	Fast installs, modern Python tooling
Testing	`pytest`	Standard Python testing
Build/publish	`hatch` or `flit`	Simple pyproject.toml → PyPI

Agent Adapters — How Each Works

Claude Code

command: "claude --print --max-turns 10 '{prompt}'"
# Token/cost parsing: claude outputs "Total tokens: X" and "Total cost: $Y"
# in its stderr/summary output

Aider

command: "aider --message '{prompt}' --yes-always --no-git"
# Token/cost parsing: aider outputs token counts and cost per session
# in its final summary. Also check ~/.aider/usage.json

Codex

command: "codex --quiet '{prompt}'"
# Token/cost parsing: codex reports usage in its output

Custom agents

Users add any agent by providing:

A shell command with {prompt} placeholder
Optionally, a regex pattern to extract tokens/cost from output

Build Plan (MVP in 4 weeks)

Week 1: Core Framework

Project scaffolding (pyproject.toml, src layout, CI)
bench.yaml parser + validation (pydantic models)
Git worktree sandbox (create, cleanup, error handling)
Core runner loop (sequential: task × agent matrix)
Validation runner (execute check command, capture exit code)
Basic wall-time metrics

Week 2: Agent Adapters + Metrics

Base agent adapter interface
Claude Code adapter (command + output parsing)
Aider adapter (command + output parsing)
Codex adapter (command + output parsing)
Metrics collection: time, tokens, cost, LLM calls, pass/fail
Run history storage (.agentmark/runs/*.json)

Week 3: Terminal Output + Polish

Rich reporter: per-task results table
Rich reporter: summary table with winner
JSON/CSV/Markdown export
agentmark init command (generate sample bench.yaml)
agentmark history command
Error handling: agent timeout, crash recovery, partial results
Parallel task execution (optional: run agents concurrently)

Week 4: Launch Prep

README with hero GIF (asciinema/vhs recording)
Example bench.yaml files for common stacks (Node, Python, Go)
Blog post: "I Benchmarked 3 AI Agents on My Codebase — Here's What I Found"
PyPI publish (pip install agentmark)
Homebrew formula (stretch goal)
Submit to: Hacker News, r/programming, r/MachineLearning, Twitter

Viral Launch Strategy

Content that drives stars

Hero screenshot/GIF — The terminal comparison table IS the marketing. Must look beautiful. Record with vhs (charm.sh/vhs).
Blog post with real data — Run agentmark on a real project (your own qmd repo works). Show actual cost/time/pass-rate differences. Real numbers >> hypothetical examples.
Twitter thread — "I built a tool that benchmarks AI coding agents on your own codebase. Here's what I found when I ran Claude Code vs Aider vs Codex on my project:" + screenshot of results table.
Hacker News — "Show HN: agentmark — benchmark AI coding agents on your codebase" + link to repo with great README.

Growth flywheel

Developer finds agentmark
    → runs it on their codebase
    → screenshots results
    → posts on Twitter/Reddit ("Claude beat GPT on my code!")
    → followers see it, try it themselves
    → more screenshots, more posts
    → repeat

Name candidates (finalize before launch)

agentmark — benchmark + agent. Clean, memorable.
pitbench — agents competing in a pit
codepit — code arena
benchit — action-oriented

Check PyPI + GitHub availability before committing.

Future Roadmap (post-MVP)

v0.2 — Richer Metrics

Agent strategy classification (reactive vs programmatic)
File I/O tracking (which files did the agent read/write?)
Diff quality scoring (how clean is the generated code?)
Retry/backtrack detection (did the agent undo its own changes?)

v0.3 — Community Features

agentmark share — upload results to a public leaderboard
Leaderboard website showing aggregate results across projects
Compare by language, framework, task type

v0.4 — Advanced Execution

Parallel agent runs (run all agents concurrently)
Statistical significance (run N times, report mean + stddev)
Cost budget limits ("stop if agent exceeds $1")
Custom scoring functions (weighted: 60% pass, 20% speed, 20% cost)

v0.5 — Ecosystem

GitHub Action: run agentmark in CI on every PR
VS Code extension: run benchmarks from editor
Agent plugin system: agents self-register with agentmark
MCP integration: expose benchmark results as MCP resources

Success Metrics

Metric	Target (3 months)
GitHub stars	1,000+
PyPI installs	5,000+
Supported agents	5+ (claude-code, aider, codex, cline, copilot-cli)
Blog/tweet mentions	50+
Contributors	10+

References

ActionEngine paper: https://arxiv.org/abs/2602.20502
SWE-bench: https://www.swebench.com/
THUDM/AgentBench: https://github.com/THUDM/AgentBench
jadbox/codeagentbench: https://github.com/jadbox/codeagentbench
Terminal-Bench: https://www.tbench.ai/
Codex Arena Mode: https://openai.com/index/introducing-codex/
Aider leaderboard: https://aider.chat/docs/leaderboards/
Rich library: https://github.com/Textualize/rich
VHS (terminal GIF recorder): https://github.com/charmbracelet/vhs

agentmark — Benchmark AI Coding Agents on Your Codebase

agentmark — Benchmark AI Coding Agents on Your Codebase

Objective

Research Foundation

Key Paper Findings That Motivate This Tool

Paper Concept → agentmark Mapping

Competitive Landscape (as of Feb 2025)

What Exists (NOT competitors — they solve different problems)

The Gap We Fill

Architecture

Core Loop (simple)

Project Structure

Key Design Decisions

Task Definition Format

CLI Interface

Terminal Output (target UX)

Tech Stack

Agent Adapters — How Each Works

Claude Code

Aider

Codex

Custom agents

Build Plan (MVP in 4 weeks)

Week 1: Core Framework

Week 2: Agent Adapters + Metrics

Week 3: Terminal Output + Polish

Week 4: Launch Prep

Viral Launch Strategy

Content that drives stars

Growth flywheel

Name candidates (finalize before launch)

Future Roadmap (post-MVP)

v0.2 — Richer Metrics

v0.3 — Community Features

v0.4 — Advanced Execution

v0.5 — Ecosystem

Success Metrics

References

Related Documents

OABench: Benchmarking Large Language Models on the Brazilian Bar Examination

Ruby 2.7

BENCHMARKS