Can a 4B model replace Cursor? SmallCode hits 87% benchmark accuracy — here's what the number means

SmallCode, a free open-source terminal agent, achieves 87% accuracy on single-file tasks with a 4B model, offering a compelling zero-cost alternative

This article was originally published on aicoderscope.com

TL;DR: SmallCode, a free open-source terminal agent, hits 87% on a custom 100-task single-file benchmark running Gemma 4 E4B (~4B active parameters) locally — beating larger 14B models on OpenCode and Pi Agent. On multi-file tasks it drops to 46%. Cursor Pro at $20/month still wins for production codebases, but SmallCode is a legitimate zero-cost option for focused single-file sessions if you have 8GB+ VRAM.

Honest take: SmallCode's 87% is real but scoped to single-file tasks — it's the best result for under-8B active parameters, not a Cursor replacement. Use it for isolated scripts and bug fixes; keep Cursor or Claude Code for anything spanning 3+ files.

A developer posted to r/LocalLLaMA earlier this week: "I built a coding agent that gets 87% on benchmarks with a 4B parameter model." Within 48 hours it crossed 100 points on Hacker News. The project is SmallCode, and the claim holds up — with important context the threads largely glossed over.

Here is what the number actually means, what hardware you need to run it, and whether it does anything to your $20/month Cursor subscription.

The model: Gemma 4 E4B is not what "4B" usually implies

The "4B" in SmallCode's headline refers to active parameters, not total model size. The model it ships with is Gemma 4 E4B — Google's smallest Mixture-of-Experts model in the Gemma 4 family. Total weights: roughly 8B parameters. At inference time, the MoE router activates approximately 4B of them per forward pass.

This distinction matters practically:

• A dense 4B model (Phi-3 Mini, earlier Qwen variants) runs in 4GB VRAM at Q4 quantization
• Gemma 4 E4B needs 6–8GB VRAM at Q4 because all 8B weights must be loaded for the router to function, even though only half fire per token
• The output quality is closer to a well-trained 8B dense model, not a 4B one — the MoE routing gives it an efficiency edge

That's the good news. Gemma 4 E4B punches above its weight class. The important caveat is that E4B was designed as an edge model — for devices, not for driving agentic coding loops with tool calls and multi-turn context. SmallCode's harness engineering is what makes it competitive.

For reference, the larger Gemma 4 26B A4B model uses 26B total parameters but only 3.8B active, scores 78.5% on HumanEval, and requires 14–18GB VRAM at Q4. SmallCode supports both variants but the 87% headline number was measured on E4B.

What the 87% benchmark actually tests

SmallCode's number comes from a custom internal benchmark — not SWE-bench Verified, not LiveCodeBench. This is the most important thing to understand before sharing the headline.

The benchmark is 100 single-file coding tasks, 10 tasks per language category:

OpenCode and Pi Agent in this comparison were running Qwen2.5-Coder-14B and Devstral Small (~14B) — models with 3–4× more active parameters than SmallCode's E4B. SmallCode beats them both with a fraction of the compute.

The "single-file" constraint is not a minor asterisk. These tasks do not require reading an entire codebase, tracking import graphs, or coordinating changes across multiple modules simultaneously. They are the kind of tasks you'd give a junior developer for a focused 30-minute session: write a Python class implementing an interface, fix a specific bug in an isolated function, implement a data structure from a description. Meaningful work — just not architecture work.

On multi-file tasks, SmallCode's score drops to 46% (or roughly 60%+ with its BoneScript codebase-awareness feature enabled). That gap is where Cursor Agent Mode, Claude Code, and well-configured Aider setups pull decisively ahead.

For context on how SWE-bench Verified scores compare: Cursor running Claude Opus 4.7 reaches 87.6% on SWE-bench Verified's 500 real GitHub issues with actual test suites. The task difficulty gap between SmallCode's benchmark and SWE-bench is substantial — SWE-bench tasks involve multi-file reasoning by default.

How SmallCode beats 14B models on single-file tasks

The performance gap between SmallCode + Gemma 4 E4B and OpenCode + Qwen 14B isn't accidental — it is harness engineering. SmallCode compensates for the model's limits through four specific techniques:

Context budgeting. Rather than appending full tool outputs to the conversation, SmallCode maintains a 4k-character hard cap on tool results and evicts history mid-turn when context grows too large. Frontier models handle context overflow gracefully; small models degrade fast. SmallCode adapts rather than assumes.

2-stage tool routing. A standard agent harness sends the full tool schema — every available tool — to the model on every turn. For a model with limited context bandwidth, that schema overhead competes directly with task-useful tokens. SmallCode routes in two steps: the model first picks a high-level category (read / write / search / run / plan), then receives only the schemas for that category. Schema overhead roughly halves.

Forgiving JSON parsing. Frontier models produce clean, syntactically valid JSON tool calls. Small models produce "almost JSON" — missing brackets, YAML-like structure, plain text descriptions of intent. SmallCode's parser accepts all of it and attempts recovery, turning near-misses into successful tool calls instead of hard failures. Without this, SmallCode estimates the 87% benchmark result would fall below 70%.

Improvement loops with auto-validation. When a task produces failing output, SmallCode detects the failure pattern (repeated identical edits, infinite patch spiral, tool call returning empty results) and retries at a different temperature with adjusted prompting. An auto-validation step with test runner discovery catches bugs the first pass misses.

The harness is the product here, not the model. Any agent built on small LLMs that ignores these four problems will underperform against well-resourced frontier setups by a much wider margin than raw benchmark numbers suggest.

Hardware requirements

For Gemma 4 E4B (SmallCode's tested configuration):

• Minimum: 8GB VRAM GPU — RTX 3060 12GB, RTX 4060 Ti, or Apple Silicon with 16GB unified memory
• Comfortable: 12GB VRAM or more
• Backend: Ollama 0.15+, LM Studio, or any OpenAI-compatible local inference server

For Gemma 4 26B A4B (higher quality, same ~4B active parameters):

• Minimum: 16GB VRAM — RTX 3090 (24GB) runs it fine, RTX 4080 (16GB) at Q4 is tight but workable
• Comfortable: RTX 4090 (24GB) at Q4, or Mac with 32GB unified memory

If you are deciding which GPU tier to invest in for local AI coding, the runaihome.com guide on [best local AI models by VRAM](https://runaihome.com/blog/best-local-ai-models-