LongCLI-Bench: A Benchmark for Long-Horizon Agentic Programming in the CLI (2026)
FreeBenchmarks terminal-based coding agents on long-horizon programming tasks that require sustained planning, repo navigation, debugging, and recovery over many steps instead of single-fix patches
About LongCLI-Bench: A Benchmark for Long-Horizon Agentic Programming in the CLI (2026)
LongCLI-Bench is a benchmark designed to evaluate AI agents on long-horizon programming tasks in command-line interfaces. It addresses limitations of existing benchmarks such as short task horizons, data contamination from GitHub scraping, and lack of fine-grained evaluation metrics. The benchmark includes 20 high-quality tasks curated from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. It employs a dual-set testing protocol that measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), along with step-level scoring to pinpoint execution failures. Experiments show that even state-of-the-art agents achieve pass rates below 20%, with most tasks stalling at less than 30% completion. Human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements than self-correction alone.
Key Features
Pros & Cons
- Addresses limitations of existing benchmarks (short horizons, data contamination, lack of fine-grained metrics)
- Provides step-level scoring for detailed failure analysis
- Includes diverse task categories covering full development lifecycle
- Reveals critical early-stage failures in agent performance
- Demonstrates significant improvements from human-agent collaboration
- Only 20 tasks, which may limit generalizability
- Tasks are from academic assignments and curated workflows, may not represent all real-world scenarios
- Current state-of-the-art agents achieve pass rates below 20%
- Majority of tasks stall at less than 30% completion
- Self-correction offers only marginal gains