Composio, an AI tooling company, put DeepSeek V4 Flash through its paces across four agent frameworks and found that no single option wins every category. The benchmark, published on Aug 6, 2026, tested Claude Code, Codex, OpenCode, and Oh My Pi using 30 tasks that relied on real-world tools like Gmail, GitHub, Slack, and Notion. The results show that the software wrapper around an AI model has a major impact on cost, and the choice of framework significantly affects both speed and price.
Oh My Pi Leads in Success but Lags in Speed
Oh My Pi achieved the highest success rate of 17/30 tasks, edging out its rivals. But that performance came at a cost in time. The framework was the slowest of the four, averaging 272 seconds per task. That is a notable slowdown compared to the competition, and it highlights a trade-off that teams will need to weigh carefully. Overall success rates were close across frameworks, with Oh My Pi only slightly ahead of the pack.
Claude Code Wins on Speed but Packs a Price
Claude Code turned out to be the fastest framework, completing tasks in an average of 122 seconds per task. That speed, however, does not come cheap. Claude Code was also the most expensive option at $0.195 per successful task. The higher cost is striking because Claude Code used the fewest tool calls and generated the least output tokens among all frameworks tested. Matthias Bastian, author at The Decoder, noted that Claude Code is the fastest agent framework but costs nearly three times more than the cheapest rival.
OpenCode Offers the Best Deal
For teams watching their budget, OpenCode stands out as the cheapest framework at $0.073 per successful task. That price point is nearly three times lower than Claude Code's cost, making it an attractive option for cost-conscious developers. OpenCode did trail slightly in success rate, however, finishing with 14/30 tasks completed. That is a small gap compared to Oh My Pi's 17/30, but it could matter for projects where reliability is the top priority.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
Cost and Speed Vary Widely Across Frameworks
The benchmark revealed dramatic differences in both cost and speed. The price difference between the cheapest and most expensive frameworks was nearly 3x, while the speed difference between the fastest and slowest was 2.2x. Those gaps are significant for teams planning to run AI agents at scale. The source analysis found that success rates were similar across frameworks, but cost and speed varied widely, suggesting that the choice of framework is less about capability and more about operational priorities.
Seven Tasks Hinge on Framework Choice
A curious detail emerged from the testing: seven tasks passed or failed based solely on which framework ran them. That means the same task could succeed with one framework and fail with another, with no change to the underlying model. This finding underscores how much the surrounding software matters. The Decoder's coverage, written by Matthias Bastian, emphasized that Claude Code's higher cost came despite its efficient use of tool calls and output tokens, pointing to the wrapper itself as the driver of expense.
The benchmark was shared via X, formerly Twitter, and included an image credit to Composio via X. For teams evaluating agent frameworks, the takeaway is clear: there is no universal winner, and the right choice depends on whether speed, cost, or success rate matters most.