Meval Grok Rules

# meval: New-Model Triage and Slot Recommendation

Date: 2026-07-18
Status: Approved design, pending implementation plan

## 1. Goal

When a new LLM drops, run one command and get a grounded recommendation for where (if anywhere) it fits in the owner's workflow. Public benchmarks answer capability; this repo answers taste and character, which no leaderboard measures.

- Command: `meval run <model-id>` => deterministic character profile + public-signal snapshot + blind taste gallery => `reports/<model>/index.html` with a per-slot verdict, linked from the main `reports/index.html`.
- Budget per evaluation: ~$1-4 in candidate tokens (OpenRouter), $0 judging (omp headless on existing subscriptions), ~15 minutes of human time.
- Principle: never re-test intelligence privately. Public signals are the shortlist filter; private probes measure behavior and taste only.

## 2. Workflow slots

| Slot | Purpose | Incumbent baseline |
|---|---|---|
| exec | Main coding executor: lint fixes, minor refactors, exploration. Fast + correct. | grok 4.5 |
| plan | Planning/solutioning: deep analysis, brainstorming, vague-goal scoping. | fable 5 |
| learn | Learning/teaching: explanations pitched just right. | opus 4.8 |
| review | Review/advisory: alternative viewpoint, honest critique. | gpt 5.6 sol |
| assist | VPS personal assistant (named "hermes"): intent, tool use, summarization. | grok 4.5 |
| frontend | Frontend design: single-file HTML builds, illustration, animation, mockups. | None. Bootstrapped: first run generates outputs from 2-3 current models, owner votes blind, winner becomes baseline. |

## 3. Architecture

```
meval run model-x
  |-- Layer 1: signals    (AA API + OpenRouter API -> json cache; manual yml for scrape-only boards)
  |-- Layer 2: probes     (~30 character tasks via pinned route, avg@3, content-hash cache)
  |     |-- code graders  (deterministic: diff inspection, metrics, schema validation)
  |     |-- judge graders (judge+advisor duo via omp headless, pairwise vs baseline, position-swapped, family-swap)
  |-- Layer 3: gallery    (frontend/illustration builds -> headless screenshots -> blind A/B votes -> Bradley-Terry)
  `-- report              (reports/<model>/index.html: per-slot keep / trial / replace + evidence)
```

Layers 1 and 2 run unattended. Layer 3 requires one human session (`meval review`). The report regenerates deterministically from cached data at any time.

### Runs and re-evaluation (drift check)

Every evaluation is an immutable **run**: `runId = <model>/<UTC timestamp + random suffix>` (ULID-style; date is display metadata, never identity). All responses, verdicts, and votes are stored under their runId; runs are append-only and never overwritten. Each run records a status (`in-progress` / `complete` / `aborted`); a retry after partial failure resumes the same in-progress run or starts a new one, but never reuses a completed runId.

- Each run stores an **execution manifest**: candidate model ID, route, pinned upstream, params, and per-probe revision hashes. `meval run <model>` reuses the latest completed run only when its manifest matches the current config (free regrade/re-report); on mismatch it creates a **derived run** that records the parent runId and references the parent's response artifacts for every probe whose (revision, route, upstream, params) entry is unchanged, buying fresh responses only for changed probes. Rotating 3 probes therefore costs 3 probes, not a full run. Responses remain stored once, under the run that bought them; derived runs hold references. Never silently reports stale outputs after a probe or route change (`recheck` remains the all-fresh path).
- `meval recheck <model>` always creates a NEW run: same probes, same params, same pinned upstream, but fresh API calls (cache is per-run, so old responses are kept, not reused). Use when a model feels dumber than at launch. Same-day rechecks are fine; runIds cannot collide.
- Drift analysis compares runs of the same model where probes, params, upstream, and `harnessVersion` all match: deltas beyond the avg@3 run-to-run spread indicate provider-side change (silent requant, serving change) rather than noise. If the pinned upstream differs between runs, the comparison is flagged, since the upstream itself is the prime suspect.
- Each recheck also fires a small **control run**: a few probes against a presumed-stable reference model in the same session, preferably an open-weight model on a pinned provider (public static weights make silent updates less likely and requants detectable). If the control drifts too, suspect the harness, route, or a global cause, not the rechecked model.
- Recheck optionally adds self-pairs to the gallery session: model-today vs model-at-launch, blind, so taste drift gets human eyes too.
- Rechecks cost candidate tokens again (that is the point); judge stays $0 via omp.

## 4. Layer 1: public signals

Decision (option B): automate only the two real APIs; hand-fill the rest.

- **Artificial Analysis Data API** (free personal key): composite indices (Intelligence, Coding, Agentic), individual benchmark scores (GPQA, HLE, LiveCodeBench, tau2-bench, Terminal-Bench, IFBench, ...), pricing, median tok/s, TTFT. Primary ingestion backbone.
- **OpenRouter API**: live per-provider pricing, latency, adoption rankings.
- **`data/signals/<model>.manual.yml`**: optional hand-filled entries for scrape-only boards (LMArena style-control Elo, SWE-bench Pro, Aider polyglot, ARC-AGI-2, BFCL, GAIA, OSWorld). Filled from announcement-day reading; all fields optional.

Raw responses cached with date stamps in `data/signals/`. Each slot defines capability floors and quality/cost/latency weights in config (exec weights tok/s heavily; plan ignores speed; assist weights tool-use scores + price). Signals gate "worth probing?" and feed the Pareto section of the report.

Caveats encoded in the report template: contamination, saturation, harness variance (10-20 pt swings), arena style bias. Public numbers are directional; the private layers decide.

## 5. Layer 2: character probes

~30 small fixed tasks, TS-defined, one file per probe (`probes/`), tagged by slot and grader type. They measure behavior, not capability: cheap tokens, high taste signal.

| Slot | Probes | Grader |
|---|---|---|
| exec | Scope-creep trap (fix one lint error in a file with tempting unrelated flaws; diff must touch only the target). Plan adherence (explicit 5-step plan with "do NOT" constraints; violation checklist). Verbosity metrics (prose chars, comment density, diff-to-task size ratio vs incumbent). Style match in an opinionated snippet. | Code (deterministic), small judge assist for style match |
| plan | Vague-goal probe (scopes and asks vs hallucinates requirements). Planted-tradeoff probe. | Judge, pairwise vs baseline |
| learn | 3 fixed topics at stated levels. | Judge pairwise + owner blind text A/B (folded into gallery session) |
| review | Planted-flaw plan + user-pressure follow-up ("I really like this approach"); flags the flaw or folds. Alternative-viewpoint probe. | Judge, binary |
| assist | Intent parsing to fixed JSON schema. Tool-call selection. Summarization faithfulness against source. | Code (schema validation) + judge pass/fail for faithfulness |
| frontend | Handled in Layer 3. | - |

Grading doctrine:

- Code graders first; judge only where the property is not verifiable.
- Judge verdicts are binary (pass/fail or A/B/tie) plus a one-line critique. No 1-5 scales.
- avg@3 runs at temperature 0 (temp 0 reduces variance but does not guarantee determinism). The spread across runs is the empirical noise floor: a "win" must clear it; borderline judge results escalate AUTOMATICALLY to avg@5 (one escalation pass: a derived run references the parent's 3 attempts and buys ONLY the 2 additional attempts for the borderline probes, then regrades; run immutability preserved -- new attempts live in the derived run, never appended to the completed parent). Report pass rate and consistency, not bare means.
- Baseline (incumbent) outputs are themselves runId'd runs of the incumbent, identified by `(incumbent, route, wrapper)` with the incumbent's OWN pinned upstream recorded per run (upstream is per-model identity, never a cross-model match criterion: a Kimi candidate on openrouter/moonshot correctly pairs with a grok baseline on openrouter/xai). For each pairwise comparison the runner selects the baseline run matching the candidate's route + wrapper, generating it on first need (omp-route baselines cost $0; openrouter ones are one-time). Only when no matching baseline can be produced (incumbent unreachable on that route) does the comparison proceed mixed-route, flagged per the route-honesty policy in section 8, never silently. Every comparison records the baseline runId. Baselines are NOT immortal: an incumbent recheck (or drift suspicion) regenerates them; comparisons against a superseded baseline are marked in the report.

## 6. Layer 3: taste gallery

- Builds: landing page, admin dashboard, one rotating archetype, SVG illustration, CSS animation, product mockup. Single-file HTML/SVG only, no frameworks, no build step.
- Runner renders via headless Chromium: static screenshot per build; animation gets a short capture plus a link to open the live file.
- `meval review` serves a local page: anonymized side-by-side pairs, position-randomized, owner votes A / B / tie / both-bad (~10-15 picks), identities revealed after voting.
- Votes persist to `data/votes.jsonl`; Bradley-Terry ratings recomputed on each session (order-independent, preferred over Elo for a fixed offline pool). BT ratings are reported with confidence intervals; with the small per-session vote counts, gallery evidence alone can support at most a **trial** verdict, never **replace**.
- After reveal, the gallery doubles as a permanent browsable taste archive per model, gated per model: a model's outputs are browsable only once it has no pending votes, so future blind sessions are not spoiled.

## 7. Judging

- Judging is a **judge + advisor duo**; the judge never works alone. Fixed pair: `anthropic/claude-fable-5:medium` and `openai-codex/gpt-5.6-sol:high` (Sol always runs at high thinking, in either role).
- **Family-swap rule**, resolved PER COMPARISON (candidate family + baseline family), not per candidate:
  1. If exactly one duo member is cross-family from BOTH sides, it judges; the other advises. (Kimi vs fable baseline => judge sol, advisor fable. Kimi vs sol baseline => judge fable, advisor sol.)
  2. If both are cross-family from both sides (e.g. Kimi vs grok baseline), default: judge fable, advisor sol.
  3. If neither is fully cross-family (e.g. OpenAI candidate vs fable baseline), the judge is the member sharing family with the BASELINE, never the candidate: residual self-preference then biases toward "keep incumbent", a conservative and disclosed direction. Tagged `judge-baseline-family` in the report.
  - Non-pairwise judged probes (binary pass/fail with no baseline output in context) use candidate family only: Anthropic candidate => judge sol; otherwise judge fable.
- **Same-family-advisor guard:** whenever the assigned advisor shares a family with the candidate (e.g. sol advising on an OpenAI candidate), reconciliation is skipped entirely, so the advisor's critique never reaches the judge and cannot flip a verdict; on disagreement the judge's original verdict is recorded for display and the probe is marked contested, which scores as tie.
- Flow per judged probe: judge and advisor each score the outputs against the rubric **independently and blind to each other** (binary verdict + one-line critique). Verdicts are then compared: agreement => final. Disagreement => one reconciliation pass where each sees the other's critique and may revise (skipped under the same-family-advisor guard); still unresolved => verdict marked **contested**: counts as tie in scoring and is surfaced in the report and queued into the human gallery session.
- Both roles run through the `omp` route (headless), backed by existing subscriptions: $0 marginal cost. Estimated volume roughly doubles vs single-judge (~180 short calls per evaluation), still trivial against subscription limits.
- Judge + advisor models and both prompts are frozen and part of `harnessVersion`.
- Pairwise judge calls run both A/B orders (position swap); inconsistent verdicts count as tie.
- One-time calibration: owner hand-labels ~30-50 judge-domain examples; measure agreement (Cohen's kappa) and TPR/TNR; adjust judge prompt until aligned; re-check only when the judge duo changes.

## 8. Provider abstraction

One interface; every route is an adapter. Models are registered under a stable CANONICAL key (used for runs, reports, slots) with PER-ROUTE bindings, because request IDs differ by route (omp `anthropic/claude-opus-4-8` vs OpenRouter `anthropic/claude-opus-4.8`): `{ family, route (default), routes: { omp?: { id, thinking? }, openrouter?: { id, pin } } }`. A baseline on route R requires the incumbent to carry a `routes[R]` binding; if absent, the comparison falls back to the incumbent's default route and is flagged mixed-route per the honesty policy below.

```ts
interface ModelRoute {
  id: string; // "openrouter" | "omp" | "direct"
  call(req: ProbeRequest): Promise<ProbeResponse>;
  // ProbeResponse always carries: text, usage, latencyMs,
  // meta.route, meta.upstream (pinned provider or CLI version),
  // meta.wrapper ("none" | "omp-system-prompt" | "omp-profiled")
}
```

| Adapter | Backing | Use |
|---|---|---|
| `openrouter` | Metered API, **upstream provider pinned** (`provider.order`, `allow_fallbacks: false`) | Default for candidates; new models day-one |
| `omp` | omp headless (`omp --model X -p ...`), OAuth subscriptions (Claude Max, ChatGPT+, Super Grok, opencode-go) | Judge (always); candidates when burning subscription quota is preferred |
| `direct` | Raw baseURL + key | BYOK or not-on-OpenRouter edge cases (interface defined in v1, adapter deferred) |

Upstream pinning rule: prefer the first-party provider (moonshot for Kimi, openai for GPT, anthropic for Claude, xai for Grok); otherwise the highest-precision endpoint; pinned once per model-route binding, recorded in every result.

Route honesty (baked in, not hidden):

- Every cached response, vote, and report row carries its `route` tag.
- The report warns on mixed-route comparisons (omp wraps probes in its system prompt; openrouter is bare). Same-slot comparisons should share a route when possible; mixed comparisons are flagged, never silently blended.
- omp CLI version pinned in result metadata.

## 9. Report

HTML, regenerated deterministically from cached data. `reports/index.html` lists every evaluated model and links to `reports/<model>/index.html`.

- Per-model page, per slot: verdict (**keep incumbent / trial / replace / pending / no evidence**) + evidence rows: signal floors met, character metrics vs incumbent, BT rating, price/speed Pareto position for that slot's weights.
- **Verdict decision rule (deterministic, thresholds in config):** a slot verdict is computed, never hand-waved. Let `winRate` = fraction of that slot's probe comparisons won beyond the noise floor (contested and within-noise results count as ties, weighted 0.5). Sanity/gate checks (build validity, schema validity) are prerequisites, NEVER comparisons: they cannot produce wins. **replace** requires: all signal floors met AND `winRate >= 0.65` AND price/latency not worse than the incumbent under the slot's weights AND (for frontend/learn) BT superiority whose CI excludes 0. **trial** requires: floors met AND `winRate >= 0.5` (or positive BT evidence). Gallery-evidenced slots with NO votes yet render **pending** (human comparison outstanding), and slots with no probes run render **no evidence** -- neither gets keep/trial/replace. Otherwise **keep**. Gallery-only evidence caps at trial.
- Per-model run timeline when multiple runs exist: probe-level deltas vs the first run, drift verdict (stable / drifted / upstream-changed), and links to each run's artifacts. `reports/index.html` shows latest run date and drift status per model.
- Evidence links to artifacts: raw probe transcripts, diffs from code graders, judge critiques, gallery screenshots, and live single-file HTML builds; the taste gallery pages are linked from (and shared with) the report.
- Auto-drafted character summary: verbose or terse, obedient or freelancing, sycophantic or honest, taste level.
- Route/wrapper metadata, `harnessVersion` per row, and any comparability warnings.
- Static files only, no server needed to browse; `meval review` serves the same tree locally for the voting flow.

## 10. Repo shape

Bun + TypeScript. Private repository.

```
meval/
  src/            runner, adapters (openrouter, omp), graders, gallery server, report generator, BT fit
  probes/         one file per probe: prompt + grader + slot tags
  data/           runs/<model>/<runId>/ (responses, verdicts, status), signals, votes.jsonl, BT ratings (git-tracked)
  reports/        generated static HTML (index + per-model pages, linked artifacts)
  meval.config.ts slots, incumbents, floors, weights, model routes, judge rules
```

## 11. Hygiene

- Probe texts never pasted publicly; repo stays private.
- 2-3 canary probe variants to detect memorization of leaked probes.
- Rotate a few probes quarterly; rotated probes archived and marked non-comparable across rotation boundaries.
- Two-tier cache, scoped per run: raw responses keyed on (runId, probe id + revision, model, route, upstream, params) and never invalidated by grading changes (they cost tokens); graded verdicts and derived metrics additionally keyed on `harnessVersion` = hash of (judge duo models, both prompts, grader revision). Regrading or re-reporting reuses the same run's responses; only `recheck` buys fresh ones. Every report row carries its `harnessVersion`; rows from different harness versions are marked non-comparable.

## 12. v1 scope

In: runner + openrouter/omp adapters + per-run cache; `run` and `recheck` commands with drift comparison; exec, frontend, review probes (highest signal); code graders; judge duo flow with family swap; gallery + BT; report with verdict decision rule; AA + OpenRouter fetchers; manual signals yml.

Out (later): `direct` adapter implementation, plan/learn/assist probe sets beyond stubs, computer-use probes for assist, judge calibration tooling, subscription-quota accounting.

## 13. Open risks

- omp headless output capture: adapter must extract the final answer cleanly; contained inside the adapter, verified during implementation.
- AA free-tier rate limits and schema drift: cache aggressively, tolerate missing fields.
- OpenRouter upstream availability varies; the `direct` adapter slot in the route interface is the escape hatch (implemented on first need).