Testing

← Back to docs

Your vault is your second brain. You don't hand it to software you can't trust.

3,292 defined tests | 185 test files | 64,400+ lines of test code

• Test Philosophy
• CI Pipeline
• Performance Benchmarks
• Concurrency & Stress
• Battle-Hardening (Fuzzing)
• Security Testing
• Cold Start & Edge Cases
• Read-Side Testing
• What Isn't Tested
• Retrieval Benchmark (HotpotQA)
  • Results (500 hard questions, 4,960 documents)
  • How Flywheel compares
  • CI Regression Gate
• Retrieval Benchmark (LoCoMo)
  • Unit-Level Results (1,531 scored questions, 272 session notes)
  • Vault Mode Comparison
  • How the E2E Benchmark Works
  • End-to-End Results (695 questions, balanced, Claude Sonnet + Flywheel MCP)
  • How Flywheel compares to other memory systems
• Graph Quality (266 tests, 31 files)
  • Baselines
  • Multi-Generation Stress Test
• Live AI Testing
  • Test Suites
  • Why This Matters
  • Design Decisions
  • Tool Adoption Results
    • Bundle Adoption
• Per-Bundle Results
  • corrections
  • diagnostics
  • graph
  • memory
  • note-ops
  • read
  • schema
  • search
  • tasks
  • temporal
  • wikilinks
  • write
• Adoption Summary
• Overall
  • Per-Tool Coverage
• Results by Category
  • search (3/3)
  • read (3/3)
  • write (7/7)
  • graph (8/10)
  • schema (6/7)
  • wikilinks (7/7)
  • corrections (4/4)
  • tasks (3/3)
  • memory (1/2)
  • note-ops (3/4)
  • temporal (4/4)
  • diagnostics (14/14)
• Summary
  • Tools Never Adopted
  • Per-Category Summary
  • Demo Beat Coverage
• Per-Beat Results
• Running the Tests

---

Test Philosophy

Three principles guide every test in this project:

1. Prove it at scale. Not with 5 notes in a toy vault -- with 100,000-line files and 2,500-entity indexes. If it works at scale, it works everywhere. Small test vaults mask O(n) regressions. A 5-note vault won't reveal that entity scoring degrades above 1,000 entities, or that FTS5 queries slow down with 100k lines. The performance thresholds exist because we measured where things actually break.

2. Break it before users do. The test suite is adversarial by design: property-based fuzzing with randomized inputs, injection attacks against every input surface, race conditions under concurrent load. A vault is user data. Unlike a web app where a bug shows an error page, a vault bug silently corrupts markdown. The adversarial stance exists because silent corruption is the worst failure mode -- the user doesn't know their notes changed.

3. Document with tests. README examples run against demo vaults. Tool counts, config keys, and category mappings are cross-checked against source in CI. The readme-examples.test.ts and demo-vault-assertions.test.ts files exist specifically so that every claim in the README is backed by a test that would fail if the claim became false. Coverage is strongest for read-side MCP flows, write-side logic, security boundaries, and concurrency.

---

CI Pipeline

12 focused test jobs, each a separate GitHub Actions check:

Job	What it gates	Why separate
Lint	Type check	Catches type errors before test runtime
Build	Package compilation	Ensures the published artifact compiles
Test: core	vault-core unit tests	Core library must pass independently
Test: read tools	Read-side MCP handlers	Search, graph, schema tools
Test: write core	Writer, git, validator	Mutation engine correctness
Test: write tools & workflows	Write-side MCP handlers	Tool integration
Test: write security	Injection, path traversal, permissions	Attack surface coverage
Test: write stress & battle-hardening	Concurrency, fuzzing	Load and chaos resistance
Test: platform & publish	WSL paths, package startup	Cross-platform correctness
Test: graph quality	Precision/recall, ablation, regression gate	Algorithm quality
Test: bench	vault-core benchmark suite	Scale and performance
Test: full matrix	Node 22 + 24 x Ubuntu + Windows	Cross-platform + Node version matrix

Each job surfaces as its own pass/fail check in GitHub. A PR cannot merge if any job fails.

The full matrix job runs the entire test suite on Node 22 and 24 (Ubuntu + Windows). This catches Node version regressions (especially native module compatibility with better-sqlite3 v12) and Windows-specific path handling issues (WSL path translation, case sensitivity, file watching).

---

Performance Benchmarks

Measured thresholds are generous to absorb CI variability. Local runs are typically faster.

Operation	Threshold	Typical
1,000-line file mutation	<100ms	~15ms
10,000-line file mutation	<500ms	--
100,000-line file mutation	<2s	--
100 consecutive mutations	<5x degradation	~1.2x
Heading extraction (1,000 lines)	<10ms	--
Entity scoring (1,000 entities)	<50ms	--
Entity scoring (5,000 entities)	<200ms	--
Wikilink suggestion (1,000 chars)	<25ms	~2ms

Memory: verified that a 2,500-entity index initializes and scores correctly without excessive allocation.

Source: packages/mcp-server/test/write/performance/benchmarks.test.ts

---

Concurrency & Stress

A vault tool that corrupts files under concurrent access is worse than no tool at all. The stress suite doesn't just test parallel writes -- it verifies that the result is deterministic. Same inputs, same order, same output, every time.

• 100 parallel mutations to different files -- zero corruption, all entries verified.
• 100 sequential mutations to the same file -- all entries preserved, ordering maintained.
• Sustained load: 4 batches of 25 mutations, performance ratio stays below 10x first-batch time.
• Mixed operations: 5 rounds of parallel mutations across 20 files interleaved with sequential writes to a single file.
• Frontmatter integrity: complex YAML (nested objects, arrays, tags) survives 10 consecutive mutations unchanged.
• Unicode/emoji preservation: special characters (Japanese, Greek, emoji) round-trip through write cycles without loss.

Source: packages/mcp-server/test/write/stress/concurrency.test.ts

---

Battle-Hardening (Fuzzing)

Vault mutations accept arbitrary markdown -- a finite set of example inputs can't cover the input space. We use property-based testing via fast-check because it generates inputs we'd never think to write by hand. Each property runs 50 randomized iterations by default (configurable).

Properties tested:

• Mutation safety -- edit_section never corrupts file structure, regardless of generated input.
• Section heading preservation -- all original headings survive any mutation.
• Frontmatter round-trip -- YAML keys and values survive read-write cycles.
• Unicode/emoji preservation -- CJK, mathematical symbols, arrows, and emoji persist through mutations.
• Large content non-truncation -- files with 100-500 lines are never silently truncated.
• Idempotency -- applying the same mutation to two identical files produces identical results.
• Ordering -- prepend inserts before existing content, append inserts after. Order is deterministic.
• Markdown special characters -- headings, bold, italic, code, links, blockquotes, tables, and horizontal rules inside content do not corrupt document structure.
• Nested list preservation -- 10-level deep indented lists survive mutation and round-trip intact.

Source: packages/mcp-server/test/write/battle-hardening/fuzzing.test.ts

---

Security Testing

Every input surface that accepts user-provided strings is a potential injection vector. The security tests are organized by attack class (injection, path encoding, permission bypass, boundaries) rather than by feature, because attackers don't respect feature boundaries. Six dedicated security test files cover the attack surface:

• injection.test.ts -- YAML injection in frontmatter fields, shell command injection via crafted git commit messages, template injection in policy YAML definitions.
• path-encoding.test.ts -- URL-encoded path traversal (%2e%2e%2f), double encoding, null byte injection, Windows backslash sequences, Unicode normalization attacks.
• permission-bypass.test.ts -- attempts to bypass permission boundaries (e.g., writing outside vault root).
• boundaries.test.ts -- Unicode normalization collisions, case sensitivity collisions (macOS HFS+), extreme content sizes (memory exhaustion resistance), deep heading nesting (stack exhaustion resistance).
• platform.test.ts -- cross-platform path handling and behavioral differences.
• sensitive-files.test.ts -- prevention of access to sensitive files (.env, credentials, private keys).

Source: packages/mcp-server/test/write/security/

---

Cold Start & Edge Cases

Tests for first-run and degraded environments:

• Empty vault: first note creation succeeds, entity index initializes from nothing.
• Missing directories: auto-creation when writing to nested paths that don't exist yet.
• Non-git vault: mutations succeed normally; git operations (commit, undo) fail gracefully with clear errors.
• Read-only vault: appropriate error handling when the filesystem denies writes.

Source: packages/mcp-server/test/write/coldstart/

---

Read-Side Testing

The read path is where users interact most. These tests verify that the index, search, graph, and watcher work correctly under realistic conditions.

• FTS5 search -- Full-text search queries across vault content, frontmatter, and tags. Tests cover ranking, phrase matching, prefix search, and edge cases like empty queries and special characters. Source: fts5.test.ts
• Entity search -- Entity index queries, category filtering, alias resolution. Source: entity-search.test.ts
• Graph operations -- Backlinks, forward links, graph analysis modes (hubs, bridges, clusters), connection strength, link paths. Source: graph.test.ts
• Embeddings & semantic -- Embedding generation, similarity search, hybrid ranking (BM25 + semantic via RRF). Source: embeddings.test.ts
• File watcher -- Change detection, rename detection (delete+upsert pairs within 5s), batch processing, graceful recovery from watcher errors. WSL-specific watcher tests for Windows path translation. Sources: watch.test.ts, wsl-watcher.test.ts
• Multi-vault isolation -- Per-request vault scoping via AsyncLocalStorage. Interleaved operations across two vaults never cross-contaminate. Source: multi-vault-isolation.test.ts
• Co-occurrence & edge weights -- NPMI scoring, retrieval co-occurrence, edge weight accumulation and staleness gating. Sources: cooccurrence.test.ts, edgeWeights.test.ts
• Recency -- Time-bucketed entity recency, watcher step integration. Source: recency.test.ts
• Read tool integration -- MCP tool handlers for search, primitives, graph analysis, schema, notes, wikilinks, temporal, and health. Each tool tested with realistic vault fixtures. Source: packages/mcp-server/test/read/tools/

---

What Isn't Tested

Honesty about coverage gaps:

• No automated Obsidian plugin testing. Flywheel Crank (the Obsidian plugin) is tested manually. Browser/plugin automation is not part of CI.
• Live AI tests are not part of CI. They require Claude API credits and are run on-demand before releases. Results are committed as markdown reports.
• No load testing above 100 concurrent operations. Flywheel is a single-user vault tool. 100 parallel writes is well beyond any realistic usage pattern.
• Edge weight and retrieval co-occurrence are newer. These layers (added in schema v22 and v30) have less test coverage than older layers like FTS5 and entity scoring.

---

Retrieval Benchmark (HotpotQA)

End-to-end retrieval quality measured on HotpotQA -- a standard multi-hop question answering benchmark from CMU/Stanford. Real Claude + Flywheel via claude -p, no pre-processing, no cherry-picking.

Results (500 hard questions, 4,960 documents)

Metric	Score
Document Recall	90.0% (90/100 supporting docs found)
Full Recall (both docs found)	80.0% (40/50)
Partial Recall (≥1 doc found)	100.0% (50/50)
Cost	$0.083/question

How Flywheel compares

HotpotQA is primarily used as a QA benchmark (answer extraction, measured by EM/F1). The retrieval stage -- finding the right documents -- is what we measure. Most academic systems are purpose-built ML retrieval models trained specifically on HotpotQA; Flywheel is a general-purpose vault tool.

System	Type	Retrieval Recall	Approach	Notes
Flywheel	MCP vault tool	90.0%	BM25 + entity search + 2-hop backfill + query expansion	General-purpose, zero training, 50 questions end-to-end via Claude
BM25 baseline	IR baseline	~70-75%	TF-IDF keyword matching	Standard academic baseline
TF-IDF + Entity	IR baseline	~80%	TF-IDF with named entity overlap	Original HotpotQA paper baseline
MDR	Trained retriever	~88%	Multi-hop dense retrieval (iterative)	Facebook, 2021. Trained on HotpotQA. Two-hop BERT encoder
Baleen	Trained retriever	~85%	Condensed retrieval with hop-aware filtering	Stanford, 2021. Trained on HotpotQA
ColBERTv2	Trained retriever	~90%+	Late interaction dense retrieval	Stanford, 2022. Fine-tuned on MS MARCO
Beam Retrieval	Trained retriever	~93%	Beam search over retrieval paths	2023. Trained end-to-end for multi-hop

Not apples-to-apples -- read this before comparing:

• Training data. MDR, Baleen, and Beam Retrieval are neural models fine-tuned on HotpotQA training data. They learned query-document relationships from thousands of labeled examples. Flywheel has seen zero HotpotQA training data.
• Test setting. Standard HotpotQA "distractor" gives each query only 10 documents (2 relevant + 8 distractors). "Fullwiki" searches 5M+ documents. Flywheel pools all 4,960 documents from 500 questions into one vault, so each query searches ~5,000 docs. This is harder than distractor but far easier than fullwiki. The numbers are not directly comparable to either setting.
• Sample size. Flywheel: 50 questions. Academic baselines typically use the full dev set (7,405 questions). Our confidence intervals are wider than a 500-question run.
• What's real. Flywheel's 90.0% beats the standard BM25 baseline (~75%) by +15pp, attributable to 2-hop backfill, query expansion, and FTS5 OR-mode with BM25 ranking. Run-to-run variance of ~1-2pp is expected due to LLM non-determinism.

Source: demos/hotpotqa/ | packages/mcp-server/test/retrieval-bench/

CI Regression Gate

The CI test (hotpotqa-bench.test.ts) runs a 200-question subset (seed 42) with conservative thresholds: recall_at_5 >= 0.3 and mrr >= 0.2. These are designed to catch catastrophic retrieval regressions (e.g., a broken index or missing search path), not to enforce the headline 90.0% recall. The 200-question sample has wider confidence intervals, and CI thresholds are set low enough to avoid false failures from normal variance.

The published 90.0% comes from the latest 50-question benchmark run documented above. A full 500-question run is available from March 2026 (92.4%).

---

Retrieval Benchmark (LoCoMo)

Retrieval quality measured on LoCoMo (Long-Term Conversational Memory) -- a benchmark from Snap Research (ACL 2024) for evaluating memory over extended multi-session conversations. 10 conversations, each spanning 19-32 sessions over weeks/months, with 1,986 QA pairs across 5 categories.

LoCoMo tests a different, more natural fit for Flywheel than HotpotQA: each conversation session becomes a vault note (like daily notes or meeting logs), and questions test whether the system can retrieve the right sessions when asked about events, facts, or temporal relationships.

Unit-Level Results (1,531 scored questions, 272 session notes)

FTS5 retrieval quality on the full LoCoMo-10 dataset. Each conversation session is stored as a markdown note with frontmatter dates, speaker names, and dialog content.

Category	Questions	Recall@5	Recall@10	MRR
Overall	1,531	84.8%	90.4%	0.767
Commonsense	841	95.4%	98.3%	0.835
Single-hop	320	88.1%	91.7%	0.737
Multi-hop	281	58.1%	72.7%	0.674
Temporal	89	56.9%	67.4%	0.528

446 adversarial questions are excluded from retrieval scoring (they test the ability to say "I don't know", which is an LLM capability, not a retrieval one).

Vault Mode Comparison

LoCoMo provides three representations of the same conversations. Flywheel builds each as a separate vault and measures retrieval quality:

Vault Mode	Recall@5	Recall@10	Notes
Dialog (raw turns)	84.8%	90.4%	Best -- most keyword-rich
Summary	82.7%	89.2%	Close second -- concise but retains key facts
Observation	76.9%	84.5%	Shorter, less keyword overlap

How the E2E Benchmark Works

The benchmark measures whether a real Claude agent with Flywheel tools can answer questions about a vault of conversation transcripts. Here's exactly what happens:

1. Vault build (demos/locomo/build-vault.js):

• Downloads the LoCoMo-10 dataset (10 conversations, 272 sessions)
• Each session becomes a markdown note with frontmatter (date, speakers, session number)
• People get stub notes with type: person frontmatter in a people/ folder
• Output: 290 clean markdown files, zero wikilinks, no .flywheel/ state

2. Pre-warm (one Claude Haiku session with full,memory preset):

• doctor(action: health) → confirms FTS5 index built, entities scanned
• vault_init mode='enrich' → auto-links all notes with wikilinks (same as production usage)
• refresh_index → re-indexes with new wikilinks
• init_semantic → builds semantic embeddings for hybrid search
• doctor(action: health) → confirms everything ready

This matches how a real vault works: Flywheel indexes it, auto-links it, and builds embeddings. The vault state after pre-warm is representative of production usage.

3. Questions (695 individual Claude Sonnet sessions with default preset):

• Each question gets its own claude -p session with a fresh MCP server instance
• The server loads from the warm .flywheel/state.db (entities, embeddings, co-occurrence cached)
• Claude sees a minimal prompt: "Answer this question about conversations in the vault"
• Non-MCP tools (Bash, Grep, Read, etc.) are stripped via --disallowedTools + --strict-mcp-config
• JSONL output captures every tool call and the final answer

4. Evaluation (demos/locomo/analyze-benchmark.py):

• Evidence recall: did the agent's tool calls access the correct source notes? (path matching)
• Answer accuracy: LLM-as-judge (Claude Haiku) scores each answer as CORRECT/WRONG against ground truth
• Wilson 95% confidence intervals for all accuracy numbers

What's in the vault state when questions run:

• FTS5 content index (BM25 keyword search)
• Entity index with categories (people detected from people/ folder + frontmatter)
• Semantic embeddings (hybrid BM25 + cosine similarity via RRF)
• Auto-inserted wikilinks (from the enrich step -- same as production auto-linking)
• Co-occurrence and edge weight data (from the index rebuild after enrichment)

This is deliberately not a cold-start test. It tests the system as it runs in production: indexed, linked, and embedding-warm.

End-to-End Results (695 questions, balanced, Claude Sonnet + Flywheel MCP)

695 questions. Stratified sampling across all 10 conversations (seed 42). Matches the Mem0 competition paper sample size.

Category	Questions	Evidence Recall	Accuracy (Judge)	95% CI
Overall	695	81.9%	54.0% (27/50)	[40.4%, 67.0%]

Cost: $0.112/question. Answer accuracy is LLM-as-judge (Claude Haiku) -- the primary answer quality metric. Token F1 (0.431 after extraction, 0.202 raw) is a diagnostic metric reported alongside. Both metrics are reported automatically on every benchmark run.

How Flywheel compares to other memory systems

Competitor numbers sourced from the Mem0 paper. All systems use 695 questions from the same dataset.

System	Type	Evidence Recall	Single-hop Recall	Multi-hop Recall	Questions	Infrastructure
Flywheel	MCP vault tool	81.9%	97.4%	73.7%	695	Local (SQLite + markdown)
Mem0	Cloud memory	--	--	--	695	Redis + Qdrant
Zep	Cloud memory	--	--	--	695	Cloud service
LangMem	Memory framework	--	--	--	695	Varies
MemGPT/Letta	Agent memory	--	--	--	695	Cloud/local

Competitors report answer accuracy via GPT-4o judge but do not report evidence recall. Flywheel reports evidence recall, LLM-as-judge accuracy (54.0%, Claude Haiku), and token F1 (diagnostic).

Methodology differences -- read this before comparing:

• Metrics differ. Flywheel reports evidence recall (84.3%) and LLM-as-judge accuracy (58.7%, Claude Haiku). Competitors report answer accuracy via GPT-4o judge. Judge methodology is comparable; judge model differs.
• Vault mode. Flywheel uses dialog mode (raw conversation turns) -- the most keyword-rich representation. Summary mode scores ~1-2pp lower on retrieval. Competitors may use different representations.
• Vault enrichment. HotpotQA notes have minimal frontmatter (heuristic-inferred type: only). LoCoMo notes include temporal metadata (date, time), speaker arrays, session numbers, and entity stubs -- closer to a real vault. Both are generated by the harness build-vault.js, not present in the source datasets. HotpotQA's 92.4% recall with near-zero metadata is closer to pure search engine performance.
• What each benchmark measures. HotpotQA measures retrieval only (did the agent's tool calls access the right documents?). LoCoMo measures retrieval and answer quality. Each LoCoMo question requires a Sonnet answer session, and conversational questions typically need more tool calls to piece together answers from multi-session dialog. This is why LoCoMo costs roughly double per question ($0.12 vs $0.058).
• Prompt. Claude is told the vault structure (notes are conversation sessions) but is not given a retrieval strategy.

What the numbers suggest:

• Single-hop recall: 97.4% -- Flywheel finds the right note almost every time for direct questions. 77.0% answer accuracy.
• Multi-hop recall: 73.7% -- harder, requires chaining searches. 38.9% answer accuracy -- the ceiling here is context assembly, not retrieval.
• Temporal recall: 69.2% -- requires precise date reasoning across sessions. 96 questions. 53.1% answer accuracy.
• Commonsense recall: 96.4% -- strongest category. 78.4% answer accuracy -- high evidence recall translates well to correct answers.
• Adversarial recall: 98.9% -- system reliably finds relevant context even for trick questions. 47.8% accuracy on correctly refusing unanswerable questions.
• Infrastructure. Flywheel runs locally on markdown files with SQLite. Mem0 requires Redis + Qdrant. Zep requires a cloud service.

Source: demos/locomo/ | packages/mcp-server/test/retrieval-bench/locomo-bench.test.ts

---

Graph Quality (266 tests, 31 files)

The graph quality suite validates that the wikilink suggestion engine works correctly across every scenario that matters: precision/recall, scoring layers, archetypes, feedback loops, temporal evolution, and regression gates.

Precision & Scoring: precision/recall, 13-layer ablation, parameter sweep (+ deep), golden set, strictness differentiation, baselines

Stability & Evolution: multi-generation (50 gen), temporal evolution, vault lifecycle, learning curve, flywheel pipeline

Topology & Resilience: 6 archetypes, topology resilience, health metrics, health snapshot

Feedback & Recovery: regression gate, feedback integration, sparse feedback, agent feedback, cross-vault learning, recovery

Robustness: chaos, chaos mutations, alias collision, property-based invariants (7 properties × 100 runs)

Observability: pipeline tracing, observability APIs, score breakdowns

Baselines

Locked in baselines.json (2026-02-26). CI fails if any metric regresses >5pp. These are regression baselines, not the same thing as the latest generated proof report in QUALITY_REPORT.md.

Mode	Precision	Recall	F1	MRR
Conservative	51.2%	71.7%	59.7%	0.742
Balanced	27.5%	76.7%	40.5%	0.742
Aggressive	26.1%	76.7%	39.0%	0.742

Measured against a 96-note/61-entity ground truth vault. Links stripped, engine must rediscover them.

The latest generated proof report checked into this repo is QUALITY_REPORT.md, generated on March 27, 2026 21:45 UTC. Its current balanced-mode headline is 40.2% precision / 71.7% recall / 51.5% F1.

Multi-Generation Stress Test

50 generations of suggest -> feedback (85/15 noise) -> mutate vault -> rebuild index. 85/15 noise is deliberately hostile -- real users don't reject 15% of suggestions. But if the algorithm survives 15% noise without F1 collapse, it will handle the 1-5% noise of real usage comfortably. Proves the Beta-Binomial suppression model prevents F1 death spiral under sustained noisy feedback.

6 CI assertions:

1. F1 stays within 20pp of baseline across all generations
2. No single generation drops F1 by more than 15pp
3. Trend slope ≥ -0.002 after generation 10
4. At least 3 entity categories maintain F1 > 0
5. Vault note count grows (mutations accumulate)
6. Suppression rate stays below 50%

Source: packages/mcp-server/test/graph-quality/multi-generation.test.ts

For full per-category, per-tier, per-archetype breakdown, see QUALITY_REPORT.md.

Sources: packages/mcp-server/test/graph-quality/

To regenerate baselines: npx tsx packages/mcp-server/test/graph-quality/generate-baselines.ts

---

Live AI Testing

Unit tests prove handlers work. Live AI tests prove the product works -- that an AI agent given a natural-language question and a vault of notes will discover the right tools, call them in the right order, and get the answer.

This is a fundamentally different claim than "the handler returns valid JSON." It tests:

• Tool descriptions -- is the description clear enough that Claude picks the right tool?
• Response shapes -- does the AI get enough information to answer without extra calls?
• Tool composition -- does Claude chain multiple tools correctly in a multi-step workflow?
• Regression detection -- if a code change makes Claude stop using a tool or pick a worse one, the test catches it

Every test is a real claude -p session against a demo vault. Claude gets --strict-mcp-config (no filesystem, no web -- vault tools only). The output is captured as stream-json JSONL, and Python analyzers extract every tool_use event to compute adoption rates, tool sequences, and category breakdowns. Clean state between runs: StateDb deleted, write operations git-restored. Nothing is mocked.

This kind of test is valuable because it measures actual tool selection and composition, not just handler correctness.

Test Suites

Test Suite	What it proves	Sessions	Result	Script
Per-tool coverage	Claude discovers and uses each tool in the current full preset surface	65	See latest results	run-coverage-test.sh
Bundle adoption	Claude finds the right tools for each of 12 bundles	36 (12 × 3 runs)	11/12 at 100%	run-bundle-test.sh
Sequential workflow	9-beat workflow (retrieval + learning loop + operational) where each beat builds on previous vault state	9 beats	--	run-demo-test.sh
HotpotQA benchmark	End-to-end retrieval quality on HotpotQA multi-hop questions	500 questions	92.4% recall	hotpotqa/run-benchmark.sh
LoCoMo benchmark	Retrieval + answer accuracy on long-term conversational memory (5 categories)	695 scored questions	84.3% evidence recall, 58.7% accuracy	locomo/run-benchmark.sh

Why This Matters

A tool might work perfectly in isolation but fail in practice because its description is ambiguous, its response is too large, or it overlaps confusingly with another tool. Live AI testing catches these issues:

• If Claude reaches for search instead of graph when asked about connections, the tool description needs work
• If Claude makes 5 follow-up calls after a search, the response shape needs more information
• If a write tool consistently gets skipped, the AI doesn't understand when to use it

These are product quality signals that no amount of handler unit testing can detect.

Design Decisions

• --strict-mcp-config -- prevents Claude from bypassing vault tools with raw filesystem access. If a tool can't answer the question, the test reveals it.
• --no-session-persistence -- each run starts fresh. No cached tool schemas from prior sessions.
• --permission-mode bypassPermissions -- avoids interactive approval for write operations.
• Seeded RNG -- HotpotQA benchmark uses SEED=42 and a ground-truth file for reproducible comparisons across runs.
• Python analyzers -- analyze-bundle-test.py, analyze-coverage-test.py, analyze-demo-test.py parse JSONL and generate markdown reports with per-tool adoption rates, tool sequences, and category breakdowns.

Tool Adoption Results

Results from live testing (2026-03-22). Claude discovers and uses flywheel tools when enabled.

Current tool surface reference from packages/mcp-server/src/config.ts:

• Total registered tools: see packages/mcp-server/src/config.ts / generated preset docs
• full preset surface: full static category surface at startup
• agent preset surface: focused default preset

The embedded reports below are historical artifacts from older tool surfaces and older tool names. Keep them as dated evidence only; use the current config and generated docs for the live product surface.

Bundle Adoption

Each of 12 tool bundles tested with a targeted prompt against carter-strategy vault.

<!-- BEGIN BUNDLE TEST RESULTS -->

Bundle Adoption Test Report

Date: 2026-03-20 16:40
Runs/bundle: 3
Bundles tested: 12
Total runs: 36
Results dir: bundle-20260320T151406

Per-Bundle Results

corrections

Category: corrections (4 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	vault_record_correction	builtin:1	Yes	ToolSearch -> vault_record_correction
2	vault_record_correction	builtin:1	Yes	ToolSearch -> vault_record_correction
3	vault_record_correction	builtin:1	Yes	ToolSearch -> vault_record_correction

diagnostics

Category: diagnostics (20 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	get_vault_stats, health_check	builtin:1	Yes	ToolSearch -> health_check -> get_vault_stats
2	get_vault_stats, health_check	builtin:1	Yes	ToolSearch -> health_check -> get_vault_stats
3	get_vault_stats, health_check	builtin:1	Yes	ToolSearch -> health_check -> get_vault_stats

graph

Category: graph (9 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	get_connection_strength, get_link_path	builtin:1, search:2	Yes	ToolSearch -> search -> search -> get_connection_strength -> get_link_path
2	get_connection_strength, get_link_path	builtin:1, search:2	Yes	ToolSearch -> search -> search -> get_connection_strength -> get_link_path
3	get_connection_strength, get_link_path	builtin:1, search:2	Yes	ToolSearch -> search -> search -> get_link_path -> get_connection_strength

memory

Category: memory (3 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	brief	builtin:2, search:1, write:1	Yes	ToolSearch -> brief -> ToolSearch -> search -> policy
2	brief	builtin:3, search:2, tasks:1	Yes	ToolSearch -> brief -> ToolSearch -> search -> ToolSearch -> tasks -> search
3	brief	builtin:2, search:3, tasks:1	Yes	ToolSearch -> brief -> ToolSearch -> search -> search -> search -> tasks

note-ops

Category: note-ops (4 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	vault_rename_note	builtin:1, search:1	Yes	ToolSearch -> search -> vault_rename_note
2	vault_rename_note	builtin:1, search:1	Yes	ToolSearch -> search -> vault_rename_note
3	vault_rename_note	builtin:2, search:1	Yes	ToolSearch -> search -> ToolSearch -> vault_rename_note

read

Category: read (3 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	get_note_structure	builtin:3	Yes	ToolSearch -> get_note_structure -> ToolSearch -> Read
2	get_note_structure	builtin:3	Yes	ToolSearch -> get_note_structure -> ToolSearch -> Read
3	get_note_structure	builtin:3	Yes	ToolSearch -> get_note_structure -> ToolSearch -> Read

schema

Category: schema (7 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	schema_conventions, vault_schema	builtin:1, search:2	Yes	ToolSearch -> vault_schema -> vault_schema -> schema_conventions -> schema_conventions -> search -> search
2	schema_conventions, vault_schema	builtin:1, search:1	Yes	ToolSearch -> vault_schema -> schema_conventions -> vault_schema -> schema_conventions -> search
3	schema_conventions, vault_schema	builtin:1, search:1	Yes	ToolSearch -> vault_schema -> schema_conventions -> vault_schema -> schema_conventions -> search

search

Category: search (3 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	search	builtin:1	Yes	ToolSearch -> search
2	search	builtin:1	Yes	ToolSearch -> search
3	search	builtin:1	Yes	ToolSearch -> search

tasks

Category: tasks (3 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	tasks	builtin:1	Yes	ToolSearch -> tasks
2	tasks	builtin:1	Yes	ToolSearch -> tasks
3	tasks	builtin:1	Yes	ToolSearch -> tasks

temporal

Category: temporal (4 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	predict_stale_notes, temporal_summary	builtin:1	Yes	ToolSearch -> temporal_summary -> predict_stale_notes
2	get_context_around_date, predict_stale_notes, temporal_summary	builtin:1	Yes	ToolSearch -> temporal_summary -> predict_stale_notes -> predict_stale_notes -> get_context_around_date
3	predict_stale_notes, temporal_summary	builtin:1	Yes	ToolSearch -> temporal_summary -> predict_stale_notes -> predict_stale_notes -> predict_stale_notes

wikilinks

Category: wikilinks (7 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	discover_cooccurrence_gaps, unlinked_mentions_report	builtin:1	Yes	ToolSearch -> unlinked_mentions_report -> discover_cooccurrence_gaps
2	unlinked_mentions_report	builtin:1	Yes	ToolSearch -> unlinked_mentions_report
3	unlinked_mentions_report	builtin:1	Yes	ToolSearch -> unlinked_mentions_report

write

Category: write (7 tools)

Run	Target Tools Used	Other Tools	Hit?	Sequence
1	vault_add_to_section	builtin:1	Yes	ToolSearch -> vault_add_to_section
2	vault_add_to_section	builtin:2, search:1	Yes	ToolSearch -> search -> ToolSearch -> vault_add_to_section
3	vault_add_to_section	builtin:6, read:3, search:1	Yes	ToolSearch -> search -> ToolSearch -> get_section_content -> ToolSearch -> get_note_structure -> get_note_structure -> ToolSearch (+3)

Adoption Summary

Bundle	Tools Available	Adoption Rate	Distinct Tools Hit	Tools Never Used
corrections	4	3/3 (100%)	1/4	absorb_as_alias, vault_list_corrections, vault_resolve_correction
diagnostics	14	3/3 (100%)	2/14	dismiss_merge_suggestion, flywheel_config, flywheel_doctor, get_all_entities, get_folder_structure, get_unlinked_mentions, refresh_index, server_log, suggest_entity_merges, vault_activity, vault_growth, vault_init
graph	9	3/3 (100%)	2/9	get_backlinks, get_common_neighbors, get_forward_links, get_strong_connections, graph_analysis, list_entities, semantic_analysis
memory	2	3/3 (100%)	1/2	memory
note-ops	4	3/3 (100%)	1/4	merge_entities, vault_delete_note, vault_move_note
read	3	3/3 (100%)	1/3	find_sections, get_section_content
schema	7	3/3 (100%)	2/7	migrate_field_values, note_intelligence, rename_field, rename_tag, schema_validate
search	3	3/3 (100%)	1/3	find_similar, init_semantic
tasks	3	3/3 (100%)	1/3	vault_add_task, vault_toggle_task
temporal	4	3/3 (100%)	3/4	track_concept_evolution
wikilinks	7	3/3 (100%)	2/7	discover_stub_candidates, suggest_entity_aliases, suggest_wikilinks, validate_links, wikilink_feedback
write	7	3/3 (100%)	1/7	policy, vault_create_note, vault_remove_from_section, vault_replace_in_section, vault_undo_last_mutation, vault_update_frontmatter

Overall

• Bundles adopted: 12/12
• Distinct flywheel tools used: 20/65

<!-- END BUNDLE TEST RESULTS -->

Per-Tool Coverage

Each tool tested with a targeted prompt against the carter-strategy vault. The embedded report below is a historical full-surface run from before preset rationalization and merged tool names; rerun demos/run-coverage-test.sh if you need a fresh report for the current surface.

<!-- BEGIN COVERAGE TEST RESULTS -->

Tool Coverage Test Report

Date: 2026-03-20 16:40
Runs/tool: 1
Tools tested: 69
Total runs: 69
Results dir: coverage-20260320T153941

Results by Category

search (3/3)

Tool	Hit Rate	Status	Tools Called
find_similar	1/1	PASS	find_similar, search
init_semantic	1/1	PASS	init_semantic
search	1/1	PASS	search

read (3/3)

Tool	Hit Rate	Status	Tools Called
find_sections	1/1	PASS	find_sections
get_note_structure	1/1	PASS	get_note_structure, search
get_section_content	1/1	PASS	get_note_structure, get_section_content, search

write (7/7)

Tool	Hit Rate	Status	Tools Called
policy	1/1	PASS	get_note_structure, policy, search
vault_add_to_section	1/1	PASS	search, vault_add_to_section
vault_create_note	1/1	PASS	policy, vault_create_note
vault_remove_from_section	1/1	PASS	get_note_structure, get_section_content, search, vault_remove_from_section
vault_replace_in_section	1/1	PASS	get_note_structure, get_section_content, search, vault_replace_in_section
vault_undo_last_mutation	1/1	PASS	search, vault_add_to_section, vault_undo_last_mutation
vault_update_frontmatter	1/1	PASS	search, vault_update_frontmatter

graph (8/10)

Tool	Hit Rate	Status	Tools Called
get_backlinks	1/1	PASS	get_backlinks
get_common_neighbors	1/1	PASS	get_common_neighbors, search
get_connection_strength	1/1	PASS	get_connection_strength
get_forward_links	1/1	PASS	get_forward_links
get_link_path	0/1	FAIL	list_entities, search
get_strong_connections	0/1	FAIL	get_connection_strength, graph_analysis
graph_analysis	1/1	PASS	graph_analysis, semantic_analysis
list_entities	1/1	PASS	list_entities
semantic_analysis	1/1	PASS	semantic_analysis

schema (6/7)

Tool	Hit Rate	Status	Tools Called
migrate_field_values	1/1	PASS	migrate_field_values
note_intelligence	1/1	PASS	get_note_structure, note_intelligence, search
rename_field	1/1	PASS	rename_field
rename_tag	1/1	PASS	get_note_structure, rename_tag, search
schema_conventions	0/1	FAIL	vault_schema
schema_validate	1/1	PASS	schema_conventions, schema_validate, search
vault_schema	1/1	PASS	vault_schema

wikilinks (7/7)

Tool	Hit Rate	Status	Tools Called
discover_cooccurrence_gaps	1/1	PASS	discover_cooccurrence_gaps, search
discover_stub_candidates	1/1	PASS	discover_stub_candidates, unlinked_mentions_report
suggest_entity_aliases	1/1	PASS	suggest_entity_aliases
suggest_wikilinks	1/1	PASS	get_note_structure, search, suggest_wikilinks
unlinked_mentions_report	1/1	PASS	unlinked_mentions_report
validate_links	1/1	PASS	validate_links
wikilink_feedback	1/1	PASS	wikilink_feedback

corrections (4/4)

Tool	Hit Rate	Status	Tools Called
absorb_as_alias	1/1	PASS	absorb_as_alias, search
vault_list_corrections	1/1	PASS	vault_list_corrections
vault_record_correction	1/1	PASS	vault_record_correction
vault_resolve_correction	1/1	PASS	vault_record_correction, vault_resolve_correction

tasks (3/3)

Tool	Hit Rate	Status	Tools Called
tasks	1/1	PASS	tasks
vault_add_task	1/1	PASS	get_note_structure, search, vault_add_task
vault_toggle_task	1/1	PASS	get_note_structure, search, tasks, vault_toggle_task

memory (1/2)

Tool	Hit Rate	Status	Tools Called
brief	1/1	PASS	brief, get_note_structure, search, tasks
memory	0/1	FAIL	(none)

note-ops (3/4)

Tool	Hit Rate	Status	Tools Called
merge_entities	0/1	FAIL	search
vault_delete_note	1/1	PASS	search, vault_delete_note
vault_move_note	1/1	PASS	search, vault_move_note
vault_rename_note	1/1	PASS	search, vault_rename_note

temporal (4/4)

Tool	Hit Rate	Status	Tools Called
get_context_around_date	1/1	PASS	get_context_around_date, search, temporal_summary
predict_stale_notes	1/1	PASS	predict_stale_notes
temporal_summary	1/1	PASS	temporal_summary
track_concept_evolution	1/1	PASS	get_note_structure, search, track_concept_evolution

diagnostics (14/14)

Tool	Hit Rate	Status	Tools Called
dismiss_merge_suggestion	1/1	PASS	dismiss_merge_suggestion, suggest_entity_merges
flywheel_config	1/1	PASS	flywheel_config
flywheel_doctor	1/1	PASS	flywheel_doctor
get_all_entities	1/1	PASS	get_all_entities, search
get_folder_structure	1/1	PASS	get_folder_structure
get_unlinked_mentions	1/1	PASS	get_all_entities, get_unlinked_mentions
get_vault_stats	1/1	PASS	get_vault_stats
health_check	1/1	PASS	health_check
refresh_index	1/1	PASS	refresh_index
server_log	1/1	PASS	server_log
suggest_entity_merges	1/1	PASS	suggest_entity_merges
vault_activity	1/1	PASS	vault_activity, vault_growth
vault_growth	1/1	PASS	vault_growth
vault_init	1/1	PASS	get_vault_stats, policy, vault_init

Summary

Historical coverage: 64/69 tools adopted (92%) -- from a dated pre-rationalization run. The current default-visible surface is 18 agent tools; rerun to get a current coverage number for today's presets.

Tools Never Adopted

• get_link_path (graph)
• memory (memory)
• merge_entities (note-ops)
• schema_conventions (schema)

Per-Category Summary

Category	Tools	Adopted	Rate
search	3	3	100%
read	3	3	100%
write	7	7	100%
graph	10	8	80%
schema	7	6	85%
wikilinks	7	7	100%
corrections	4	4	100%
tasks	3	3	100%
memory	3	2	66%
note-ops	4	3	75%
temporal	4	4	100%
diagnostics	14	14	100%

<!-- END COVERAGE TEST RESULTS -->

Demo Beat Coverage

9-beat sequential demo across retrieval, learning loop, and operational beats.

<!-- BEGIN DEMO TEST RESULTS -->

Demo Beat Analysis

Date: 2026-03-20 16:40
Results dir: demo-20260320T151406

Per-Beat Results

Beat	Status	Expected	Tools Used	Categories
beat1-brief	PASS	brief	ToolSearch, ToolSearch, brief, Read	builtin:3, memory:1
beat2-billing	PASS	search	ToolSearch, search, tasks	builtin:1, search:1, tasks:1
beat3-tasks	PASS	policy, vault_add_task	ToolSearch, ToolSearch, search, search, vault_add_task, vault_add_task, ToolSearch, policy, policy, vault_add_task, vault_add_task	builtin:3, search:2, tasks:4, write:2
beat4-showstopper	PASS	vault_add_to_section	ToolSearch, search, search, search, search, search, search, search, search, get_note_structure, get_note_structure, get_note_structure, TodoWrite, get_section_content, get_note_structure, get_note_structure, ToolSearch, search, get_note_structure, TodoWrite, get_section_content, get_section_content, ToolSearch, Read, Read, Read, vault_add_to_section, vault_add_to_section, vault_add_to_section, policy, refresh_index, vault_add_to_section, get_folder_structure, policy, vault_add_to_section, vault_update_frontmatter, vault_update_frontmatter, TodoWrite, refresh_index, vault_update_frontmatter, search, search, search, vault_create_note, vault_create_note, vault_add_to_section, vault_add_to_section, vault_add_to_section, vault_update_frontmatter, vault_update_frontmatter, TodoWrite	builtin:10, diagnostics:3, read:9, search:12, write:17
beat5-assign	PASS	vault_update_frontmatter	ToolSearch, search, search, search, search, search, search, get_note_structure, vault_update_frontmatter, vault_create_note, vault_create_note, vault_add_task, ToolSearch, vault_create_note, vault_create_note	builtin:2, read:1, search:6, tasks:1, write:5
beat6-meeting	PASS	vault_create_note	ToolSearch, policy, search, search, search, get_folder_structure, vault_create_note	builtin:1, diagnostics:1, search:3, write:2
beat7-pipeline	PASS	policy, search	ToolSearch, search, search, search, search, search, search	builtin:1, search:6

7/7 beats passed. 12 distinct flywheel tools used across all beats.

<!-- END DEMO TEST RESULTS -->

Source: demos/

---

Running the Tests

git clone https://github.com/velvetmonkey/flywheel-memory.git
cd flywheel-memory
npm install
npm test