Neura Market
prompt logo

prompt

Free

Metrics-first, sandbox-verified prompt optimization

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Company
ai-boost
Visit Website GitHub (40 stars)

About prompt

The Sandboxed Prompt Engineer is a prompt optimization system inspired by the SPEAR paper (arXiv 2605.26275, May 2026). It automates prompt improvement using a code-as-action loop with four tools: evaluate, python, set_prompt, and finish. The system employs a metrics-first philosophy, enforcing monotone improvement via automatic rollback on metric regression and optional guard metric floors. It uses a Python sandbox for structural error analysis—confusion matrices, error clustering, subgroup breakdowns—rather than intuition-based rewrites. Reported performance includes κ 0.857 on tool-selection, 0.815 F1-macro on filter-relevance, and 0.938 accuracy on BBH-7, outperforming baselines GEPA and TextGrad. The prompt is open source and hosted in the ai-boost/awesome-prompts GitHub repository.

Key Features

Four-tool code-as-action loop: evaluate, python, set_prompt, finish
Automatic rollback on metric regression with guard floor support
Python sandbox for structural error analysis (confusion matrices, error clustering, subgroup breakdowns)
Metrics-first philosophy: prompt changes justified only by measured deltas
High performance: κ 0.857 on tool-selection, 0.815 F1-macro on filter-relevance, 0.938 accuracy on BBH-7
Open source and freely available on GitHub

Pros & Cons

Pros
  • Eliminates guesswork by using quantitative metrics to guide prompt changes
  • Automatic rollback prevents performance regression
  • Sandboxed Python analysis enables deep understanding of failure modes
  • Proven performance improvements over baselines like GEPA and TextGrad
  • Open source and free to use
Cons
  • Requires a fixed evaluation set and metric definition, limiting applicability to ad-hoc tasks
  • Setup complexity may be non-trivial for users unfamiliar with the code-as-action paradigm
  • Only optimizes prompts within the defined constraint of four tools—no external integrations

Best For

Automatic prompt engineering for LLMs on specific tasksBenchmarking and improving model performance on classification, reasoning, or generative tasksResearch in prompt optimization and code-as-action systemsIterative refinement of prompts with quantitative validation

FAQ

What is the Sandboxed Prompt Engineer?
It is an automated prompt optimizer that uses a code-as-action loop with four tools (evaluate, python, set_prompt, finish) to iteratively improve prompts based on quantitative metrics.
How does it ensure prompt improvements?
It enforces monotone improvement by automatically rolling back to the previous best checkpoint if the aggregate metric drops after a set_prompt call. A guard floor can also prevent finishing with a prompt that violates certain metrics.
What performance does it achieve?
Reported results include κ 0.857 on tool-selection, 0.815 F1-macro on filter-relevance, and 0.938 accuracy on BBH-7, outperforming GEPA (0.628 accuracy) and TextGrad (0.484 accuracy).
Is the Sandboxed Prompt Engineer free to use?
Yes, it is open source and hosted on GitHub under the ai-boost/awesome-prompts repository, available for anyone to use or modify.