End-to-end TeamCity framework to run AI agents on SWE-Bench Lite. Spin up isolated Docker images per task, extract patches, score with the official harness, and aggregate success rates. As an example, we'll look at Junie and Google Gemini CLI

SWE-Bench AI Agent Testing with TeamCity

TeamCity gives you a reproducible CI/CD backbone for AI agent evaluation: orchestrate parallel, isolated (Docker) runs across benchmark tasks (e.g., SWE-Bench), validate patches automatically, and capture metrics, logs, and artifacts to track performance and costs at scale via Kotlin-DSL pipelines.

This TeamCity configuration provides a complete framework for testing AI agents against the SWE-Bench Lite dataset, which contains 300+ software engineering tasks from popular Python repositories.

Overview

The system evaluates AI agents by:

1. Preparing isolated Docker environments for each SWE-Bench task
2. Running AI agents against specific coding problems
3. Evaluating solutions using the official SWE-Bench evaluation harness
4. Collecting performance metrics and success rates

Architecture

Projects Structure

• JetBrains Junie AI Agent (JetBrain_Junie_AI_Agent.kt)

  • Downloads Junie CLI from GitHub releases and IntelliJ IDEA
  • Creates task subsets for progressive testing
  • Individual task execution builds for all 300+ SWE-Bench tasks

• Google Gemini CLI AI Agent (Google_Gemini_CLI_AI_Agent.kt)

  • Builds from the official Google Gemini CLI repository (https://github.com/google-gemini/gemini-cli.git)
  • Uses Node.js execution environment with npm build process
  • Creates task subsets for progressive testing

• SWE-Bench Lite (SWE_Bench_Lite.kt): Core dataset and e