Evaluation and Observability

Sources: Huyen (AI Engineering, ch. 3–4, 10), Brousseau & Sharp (LLMs in Production), Pydantic Evals documentation analysis, RAGAS framework documentation analysis, 2025–2026 production patterns

Covers: evaluation methodology, LLM-as-judge, RAG evaluation, golden datasets, production monitoring, tracing, feedback loops.

Why Evaluation Is the Hard Part

LLM applications fail in ways traditional software does not:

• Correct code produces wrong answers
• The same prompt produces different outputs at different times
• Quality degrades when prompts are updated upstream
• Failures are semantic, not syntactic — no stack trace

Without evaluation infrastructure, you are shipping blind. Every prompt change is a regression risk you cannot detect.

Evaluation Methodology

Three-Layer Framework

Layer	Approach	When to Use
Functional correctness	Deterministic code checks	Schema validation, format compliance
Similarity metrics	BLEU, ROUGE, semantic similarity	Translation, summarization with reference
Semantic quality	LLM-as-judge	Open-ended tasks; when correctness is subjective

Apply layers cheapest-first. Run schema validation before LLM-as-judge. A response that fails schema validation does not need an expensive semantic quality check.

Evaluation Granularity

Evaluate at three levels:

System level:    End-to-end quality (does the app do its job?)
Component level: Each pipeline stage (retrieval, generation, routing)
Prompt level:    Each template variant (A/B test prompt changes)

Only system-level evaluation reveals emergent failures. Only component-level evaluation reveals where to fix them.

Human vs Automated Evaluation

Approach	Scalability	Accuracy	Use For
Human annotation	Low (expensive)	High (ground truth)	Calibrating judges; golden dataset creation
Rule-based automated	High	Medium (brittle to edge cases)	Schema, format, length checks
LLM-as-judge	High	Medium-High	Semantic quality, open-ended tasks
Specialized model	High	High for specific task	Code correctness, factual accuracy

Use human evaluation to create the golden dataset. Use automated evaluation for ongoing regression testing.

LLM-as-Judge

Use a strong LLM to evaluate another LLM's output. Effective for tasks where correctness is context-dependent or subjective.

Judge Prompt Pattern

System: You are an evaluator assessing response quality.
        Score the response on the criterion below.
        Return JSON: {"score": 1-5, "reasoning": "one sentence"}
        Do not let response length influence your score.

User:
    Criterion: Faithfulness — does the response contain only
               information present in the provided context?

    Context: {retrieved_chunks}
    Question: {user_query}
    Response: {llm_response}

    Score 1-5 where:
    1 = Contains significant hallucinations
    3 = Mostly faithful, minor unsubstantiated claims
    5 = Fully grounded in the provided context

Judge Configuration Rules

Rule	Rationale
Request reasoning alongside score	Enables debugging failed cases
Evaluate one criterion per call	Multi-criterion prompts produce inconsistent scores
Use stronger model than the one being evaluated	Judge must exceed the model's capability
Align judge with human labels before deploying	Misaligned judge produces misleading metrics
Randomize response position in comparative eval	Position bias: models favor first/last response

Judge Alignment Process

Before using LLM-as-judge in production:

1. Collect 100 labeled examples (human annotated)
2. Run judge on the same examples
3. Compute judge–human agreement (Cohen's kappa or accuracy)
4. Identify disagreement patterns (where does judge fail?)
5. Refine judge prompt to address disagreements
6. Repeat until agreement > 0.7 (substantial) or 0.8 (strong)

Target 80%+ agreement between judge and human labels before trusting the judge for regression testing.

RAG-Specific Evaluation

Evaluate RAG in two stages: retrieval quality and generation quality. A perfect generator cannot compensate for poor retrieval.

Retrieval Metrics

Metric	Definition	Compute Without Golden Labels?
Context Relevance	Fraction of retrieved chunks relevant to the query	Yes — LLM judge per chunk
Context Recall	Fraction of needed information that was retrieved	No — requires golden context
Context Precision	Fraction of retrieved chunks that were actually used in the answer	Partial

Generation Metrics

Metric	Definition	Compute Without Golden Labels?
Faithfulness	Answer uses only retrieved context (no hallucination)	Yes — LLM checks each claim
Answer Relevance	Answer addresses the actual question asked	Yes — LLM judge
Answer Correctness	Answer matches known-correct answer	No — requires golden answer

Minimal viable eval set: Start with faithfulness + answer relevance. Both use LLM-as-judge and require no pre-labeled data. Add answer correctness when building a curated golden set.

Faithfulness Check Pattern

for each claim in llm_response:
    prompt = "Is this claim supported by the provided context?
              Context: {retrieved_chunks}
              Claim: {claim}
              Answer: yes / no / partially"
    result = judge.evaluate(prompt)

faithfulness_score = count(yes) / total_claims

Golden Datasets

Construction

A golden dataset is a curated set of (input, expected_output) pairs used for regression testing.

Golden dataset structure:
{
    "id":           unique identifier,
    "input":        {user_query, context, conversation_history},
    "expected":     {answer, key_points, must_include, must_exclude},
    "difficulty":   "easy" | "medium" | "hard",
    "category":     "factual" | "reasoning" | "multi-hop" | "refusal",
    "source":       "user_feedback" | "expert_annotation" | "adversarial"
}

Dataset Sources (Priority Order)

Source	Quality	Volume	Process
Failed production cases	High	Low	Review user complaints → curate
Expert annotation	Highest	Low	Expensive; use for calibration
User feedback (thumbs down)	High	Medium	Convert ratings → labeled cases
AI-generated + human reviewed	Medium	High	Generate diverse, review all
Adversarial (red team)	High	Low	Test boundary/refusal behavior

Target Dataset Size

Purpose	Minimum	Target
Initial quality baseline	50	200
Regression testing	100	500
Fine-tuning evaluation	500	2000+

Prioritize quality over quantity. Fifty well-labeled examples beat 500 poorly labeled ones.

Production Monitoring

Monitoring catches regressions that evaluation misses. Evaluation runs before deployment; monitoring runs after.

What to Monitor

Signal	Metric	Alert Threshold
Response quality	Average judge score	Drop > 5% from baseline
Latency	P50, P95, P99 per pipeline stage	P99 > 2× P50
Error rate	Failed requests / total	> 1%
Cost	Tokens per request, $ per request	> 20% increase
Cache hit rate	Hits / total	Drop > 10%
Retrieval quality	Context relevance score	Drop > 10%

Trace Structure

Log one trace per request. Each trace spans the full pipeline.

Trace:
    trace_id:        UUID
    user_id:         anonymized
    timestamp:       ISO 8601
    total_latency_ms: end-to-end time

    Spans:
        - name: "retrieval"
          latency_ms: 80
          retrieved_chunks: 5
          context_relevance: 0.82
        - name: "generation"
          latency_ms: 1200
          model: "claude-3-5-sonnet"
          input_tokens: 2400
          output_tokens: 350
          cost_usd: 0.0094
        - name: "judge_evaluation"
          latency_ms: 450
          faithfulness: 0.95
          answer_relevance: 0.88

Store traces in a searchable backend (LangSmith, Langfuse, Honeycomb, or custom). Sample 100% in development; 10–20% in high-volume production.

User Feedback Loops

Structured user feedback is the highest-quality signal. Convert every negative signal into an evaluation case.

Feedback Signal Types

Signal	Quality	Collection Method
Thumbs down / explicit reject	High	Explicit UI action
Edited response	High	Detect post-generation edits
Retry / regenerate	Medium	Detected from session behavior
Session abandonment	Low	Inferred from session end
Explicit rating (1–5)	Highest	Rare; adds UI friction

Feedback → Test Case Pipeline

User submits thumbs down
    → Log: {session_id, message_id, user_id, timestamp}
    → Async job: retrieve trace for message_id
    → Queue for human review
    → Human labels: {is_failure: bool, failure_category, correct_response}
    → If is_failure: add to golden_dataset

Even 10 labeled failures per week compound quickly. 50 weeks = 500 test cases without a formal annotation project.

Evaluation Pipeline Design

Pre-Deployment (Offline)

Pull latest golden dataset
→ Run candidate model/prompt against all examples
→ Compute metrics (faithfulness, answer relevance, correctness)
→ Compare to baseline (previous passing version)
→ Gate: merge only if no metric degrades > 5%

Post-Deployment (Online)

Production traffic (sampled 10%)
→ LLM judge evaluates each sampled response
→ Aggregate into daily metric report
→ Alert if rolling 7-day average drops below threshold
→ Failed cases queued for golden dataset review

Evaluation Cadence

Event	Evaluation Action
Prompt change	Full offline eval against golden dataset
Model version change	Full offline eval + 24h online monitoring
Weekly	Review online metrics; review failed cases
Monthly	Add new cases to golden dataset; re-baseline

Observability Stack Options

Tool	Type	Best For
LangSmith	Managed (LangChain)	LangChain-based apps; trace visualization
Langfuse	Open source / managed	Multi-framework; cost tracking; self-hostable
Phoenix (Arize)	Open source	Model debugging; embedding visualization
Weights & Biases (Weave)	Managed	Teams already using W&B
Custom (OpenTelemetry)	DIY	Full control; no vendor lock-in

Instrument with OpenTelemetry-compatible spans regardless of backend. Swap backends without code changes.