Deutsche Version

Glossary

Terms and concepts used in the experiment documents. Grows with each phase.

---

Evaluation Dataset (Golden Dataset)

The evaluation dataset (also "golden dataset" or "ground truth") is the foundation of all retrieval metrics. It defines which documents are considered "relevant" for a given query. All Precision, Recall, MRR, and nDCG values are only as meaningful as the dataset they are measured against.

Structure

An evaluation dataset consists of query-document pairs:

• Query: A natural-language question or search query
• Expected Sources: List of documents considered relevant
• Must-Contain Keywords: Keywords that must appear in the results
• Reference Chunk Text: Reference text for semantic comparison

Quality Criteria

Criterion	Description	Risk if Violated
Coverage	Queries cover various topics and difficulty levels	Blind spots for certain document types
Completeness	All relevant documents per query are annotated	Recall is measured too high — unannotated relevant docs are ignored
Correctness	Only actually relevant documents are marked	Artificially low Precision (false positives)
Size	Enough queries for statistically reliable results	Random fluctuations dominate the metrics
Negative Tests	Queries for which no relevant documents exist	System robustness is not tested

Our Dataset

data/evaluation/aws_cert_eval_v1.json: 53 queries (50 positive, 3 negative) covering AWS Certification Docs. Validation uses three levels: Source Matching (OR) Keyword Matching (OR) Semantic Similarity.

Limitation: 53 queries are sufficient for relative comparisons between configurations, but too few for statistically significant statements about absolute quality. Small changes to a few queries can shift metrics by 1-2pp.

---

Retrieval Metrics

Precision@k

Proportion of relevant documents among the top-k results.

Formula: Precision@k = |relevant documents in top-k| / k

Example: At k=10, 10 documents are returned. 8 of them are relevant. Precision@10 = 8/10 = 0.80

Interpretation: High Precision = little noise in the results. Important when every returned document serves as context for the LLM and irrelevant documents can degrade answer quality.

Recall@k

Proportion of found relevant documents out of all relevant documents in the index.

Formula: Recall@k = |relevant documents in top-k| / |all relevant documents|

Example: There are 12 relevant documents in the index. At k=10, 9 of them are found. Recall@10 = 9/12 = 0.75

Interpretation: High Recall = few relevant documents are missed. In RAG systems particularly critical: what is not found cannot be compensated downstream. Precision can be improved through reranking, Recall cannot.

MRR (Mean Reciprocal Rank)

Average of the reciprocal rank of the first relevant result across all queries.

Formula: MRR = (1/N) * sum(1/rank_i) where rank_i is the rank of the first relevant result for query i.

Example:

• Query 1: First relevant result at rank 1 → 1/1 = 1.0
• Query 2: First relevant result at rank 3 → 1/3 = 0.33
• Query 3: First relevant result at rank 1 → 1/1 = 1.0
• MRR = (1.0 + 0.33 + 1.0) / 3 = 0.78

Interpretation: MRR close to 1.0 means the first relevant result almost always appears at rank 1. Particularly important for user-facing systems, where the user expects the first result to be relevant.

nDCG@k (Normalized Discounted Cumulative Gain)

Measures ranking quality: Are relevant documents ranked higher than irrelevant ones?

Idea: A relevant document at rank 1 is more valuable than at rank 10. The "gain" is logarithmically discounted with increasing position. Normalized to [0, 1] by dividing by the ideal ranking.

Formula:

DCG@k = sum(rel_i / log2(i + 1))  for i = 1..k
nDCG@k = DCG@k / IDCG@k           (IDCG = DCG of perfect ranking)

Example: At k=5 with relevance labels [1, 0, 1, 1, 0]:

• DCG = 1/log2(2) + 0/log2(3) + 1/log2(4) + 1/log2(5) + 0/log2(6) = 1.0 + 0 + 0.5 + 0.43 + 0 = 1.93
• Ideal ranking [1, 1, 1, 0, 0]: IDCG = 1.0 + 0.63 + 0.5 + 0 + 0 = 2.13
• nDCG = 1.93 / 2.13 = 0.91

Interpretation: nDCG@k close to 1.0 means relevant documents are optimally placed. Combines Precision (how many are relevant?) with ranking quality (where are they placed?). Considered the best single metric for retrieval quality.

Metrics Comparison

Metric	Measures	Strength	Weakness
Precision@k	Proportion of relevant results	Simple, intuitive	Ignores ranking order
Recall@k	Coverage of all relevant docs	Important for RAG completeness	Ignores noise
MRR	Position of the first hit	Good for "need only 1 result"	Ignores all other hits
nDCG@k	Overall ranking quality	Best overall metric	More complex to interpret

---

Chunking

Recursive Chunking

Splits text along natural boundaries: first paragraphs, then sentences, then words. Attempts to preserve semantically coherent units.

Advantages: Respects document structure, fewer context breaks. Disadvantages: Uneven chunk sizes.

Fixed Chunking

Splits text into pieces of fixed length (in tokens), regardless of text structure.

Advantages: Even chunk sizes, more consistent embeddings. Disadvantages: Can split in the middle of a sentence or paragraph.

Overlap

Number of tokens that overlap between consecutive chunks. Prevents relevant information from being lost at chunk boundaries.

Typical values: 10-20% of chunk size (e.g., 50 for chunk size 512).

---

Embeddings

Embedding Dimension

Length of the vector that the embedding model produces per text chunk.

• 768d (e.g., BGE-base): Standard BERT size. Good for shorter texts.
• 1024d (e.g., BGE-large, E5-large): More capacity for semantic nuances. Benefits especially with longer chunks (1024+ tokens), as more context needs to be encoded.

Trade-offs: Higher dimensions improve retrieval quality but also increase embedding latency (larger model), memory consumption in the vector index (each vector occupies more bytes), and search latency (more dimensions per distance computation). At 100k chunks: 768d ~ 300 MB, 1024d ~ 400 MB (float32).

BGE (BAAI General Embedding)

Embedding models from BAAI (Beijing Academy of Artificial Intelligence). Variants: bge-base (768d), bge-large (1024d). Type prefix: "bge".

E5 (EmbEddings from bidirEctional Encoder rEpresentations)

Embedding models from Microsoft Research. E5-large-v2 has 1024 dimensions. Type prefix: "e5". Uses "[query]"/"[passage]" prefixes for asymmetric search.

---

Vector Search

HNSW (Hierarchical Navigable Small World)

Graph-based algorithm for Approximate Nearest Neighbor (ANN) search. Standard index in ChromaDB, Qdrant, Weaviate.

Key parameters:

• ef_construction: Accuracy during index construction (higher = more accurate, slower)
• M: Number of connections per node (higher = more accurate, more memory)

Cosine Similarity

Similarity measure between two vectors. Measures the angle, not the magnitude. Values from -1 (opposite) to 1 (identical). Standard metric for embedding search.

Dot Product (Scalar Product)

Mathematical operation between two vectors: a . b = sum(a_i * b_i). For sparse vectors, only the shared non-zero positions are multiplied and summed. In the context of BM25: the dot product between query sparse vector and document sparse vector yields the BM25 score.

Difference from Cosine Similarity: Cosine normalizes by vector length (measures direction), Dot Product also considers length (measures direction + magnitude).

---

Search Strategies

Dense Vector Search

Semantic search over embedding vectors. Each text is converted by an embedding model into a dense vector (fully-populated vector, all dimensions have values). Similar texts have similar vectors (measured via Cosine Similarity).

Strengths: Captures semantic similarity ("car" also finds "vehicle"). Weaknesses: Can miss exact keyword matches, especially with technical terms or proper nouns not seen during training.

Dense Vector

Fully-populated vector of fixed length (e.g., 1024 dimensions), where every position has a float value. Produced by embedding models like E5-large-v2 or BGE. Encodes the semantic meaning of the entire text.

Storage: 1024d * 4 bytes (float32) = 4 KB per vector. At 150k chunks: ~600 MB.

Sparse Vector

Sparsely-populated vector with high dimensionality (vocabulary size, e.g., 50,000+), but only a few non-zero entries (typically 20-100 per document). Stored as pairs of indices (which positions) and values (which weights).

Example: A document with the terms "Lambda", "serverless", "function" has a sparse vector with 3 non-zero entries at the positions of these terms in the vocabulary.

Storage: Only the non-zero entries are stored. At 50 terms per chunk: 50 * (4 + 4) bytes = 400 bytes per vector. At 150k chunks: ~60 MB.

BM25 (Best Match 25)

Classic keyword ranking algorithm (Robertson et al., 1994). Scores documents based on overlap with query terms, considering:

1. Term Frequency (TF): How often a query term appears in the document. BM25 uses saturated TF — the benefit of each additional occurrence diminishes:

   tf_component = tf * (k1 + 1) / (tf + k1 * (1 - b + b * dl / avgdl))
   

   • k1 (default 1.5): Controls TF saturation. Higher = more weight for frequent terms. k1=0 ignores TF completely.
   • b (default 0.75): Controls length normalization. b=1 normalizes fully, b=0 ignores document length.
   • dl: Document length (number of tokens)
   • avgdl: Average document length in the corpus

2. Inverse Document Frequency (IDF): Weighting of rare terms. The rarer a term in the entire corpus, the more informative it is:

   IDF(t) = log((N - df(t) + 0.5) / (df(t) + 0.5) + 1)
   

   • N: Total number of documents in the corpus
   • df(t): Number of documents containing term t

3. BM25 Score for a document d and query q:

   BM25(d, q) = sum(IDF(t) * tf_component(t, d))  for all t in q
   

Strengths: Excellent for exact keyword matches, technical terms, proper nouns. Weaknesses: No semantic understanding ("car" does not find "vehicle").

In our system: BM25 weights are stored as sparse vectors in Qdrant. Document vectors contain the TF component (without IDF), query vectors contain the IDF values. The dot product yields the BM25 score.

Named Vectors (Qdrant)

Qdrant feature (since v1.7) that allows multiple vector spaces per collection. Each vector space has a name and its own parameters (dimension, distance metric).

Example of our hybrid collection:

Collection: "recursive_1024_100__e5-large-v2__hybrid"
├── "dense": 1024-dim, Cosine Similarity (E5-large-v2 embeddings)
└── "bm25":  Sparse Vector (BM25 TF weights)

Each document has both vectors. Queries can target a specific vector space (using="dense" or using="bm25") or combine both.

Hybrid Search

Combination of Dense (semantic) and Sparse (keyword-based) search. Goal: Combine the strengths of both approaches — semantic understanding (Dense) plus exact keyword matches (Sparse).

Process in our system:

1. Send query to dense index → top-2k results with scores
2. Send query to sparse index → top-2k results with scores
3. Normalize scores (Min-Max to [0,1])
4. Weighted combination: combined = α * dense + (1-α) * sparse
5. Sort by combined score, return top-k

Alpha (Hybrid Weight)

Weighting parameter for Hybrid Search. Determines the ratio between Dense (semantic) and Sparse (keyword) components.

Alpha	Dense Share	Sparse Share	Description
1.0	100%	0%	Pure Dense (= pure semantic search)
0.7	70%	30%	Semantics-dominant
0.5	50%	50%	Balanced
0.3	30%	70%	Keyword-dominant
0.0	0%	100%	Pure Sparse (= pure BM25)

Intuition: For specialized domains with specific terminology (like AWS certifications), a keyword component can help find exact service names and technical terms more reliably, while the semantic component covers conceptual similarities.

Score Fusion

Merging results from two (or more) search methods into a unified ranking list.

Alpha Blending (score-based): Normalizes the scores of both methods to [0,1] and weights them: α * score_A + (1-α) * score_B. Requires score normalization, since Dense (0-1) and Sparse (0-30+) have different scales.

RRF (Reciprocal Rank Fusion): Rank-based alternative: score = sum(1 / (k + rank_i)). No alpha parameter, treats all sources equally. Independent of score scales, but no weighting possible.

Our choice: Alpha Blending, because we want to experimentally control and compare the influence of Dense vs. Sparse.

Min-Max Normalization

Scaling values to the range [0, 1]:

normalized = (x - min) / (max - min)

Necessary for Score Fusion, since raw scores from different search methods have different scales:

• Dense (Cosine Similarity): typically 0.3 - 0.95
• Sparse (BM25 Dot Product): typically 0 - 30+

After normalization, both are on [0, 1] and can be combined with weights.

Edge case: If all scores are equal (max = min), all are set to 1.0.

---

Reranking

Reranking (Second-Stage Scoring)

Post-processing of initial retrieval results with a more precise model. Retrieval (Stage 1) quickly delivers a candidate list (e.g., top-100 from 150k), the reranker (Stage 2) scores these candidates more accurately and re-sorts them.

Impact on metrics:

• Improves Precision (irrelevant results are pushed down)
• Improves nDCG and MRR (better ranking of relevant results)
• Does not improve Recall — what was not found in retrieval cannot be added by the reranker. Therefore, over-retrieval (fetching more candidates than finally needed) is critical.

Bi-Encoder

Model that encodes query and document independently — one separate embedding each. Similarity is computed via vector operations (e.g., Cosine Similarity).

Advantages: Document embeddings can be precomputed and stored in a vector index. Search over 150k chunks in milliseconds (ANN algorithms).

Disadvantages: The independent encoding cannot capture fine interactions between query and document terms.

In our system: E5-large-v2 (Phase 2) is a Bi-Encoder for retrieval.

Cross-Encoder

Model that encodes query and document jointly — both are concatenated as a pair and passed through a transformer. The model can directly model interactions between query and document terms.

Advantages: Significantly more accurate than Bi-Encoders, since the model can see which query terms appear in the document and in what context.

Disadvantages: O(n) expensive — each (query, document) pair requires a separate forward pass. With 100 candidates = 100 inference calls per query. Therefore only usable as a reranker for pre-selected candidates, not for full-index search.

Typical architecture: AutoModelForSequenceClassification (e.g., BGE-reranker-v2-m3). Input: [CLS] query [SEP] document [SEP] → Output: relevance score.

Bi-Encoder vs. Cross-Encoder

The standard pattern in modern RAG systems:

Full Index (150k Chunks)
    |
    v  Bi-Encoder (fast, O(1) via ANN)
Top-100 Candidates
    |
    v  Cross-Encoder (accurate, O(n))
Top-5 Results
    |
    v  LLM (Synthesis)
Answer

The Bi-Encoder quickly filters out the majority of irrelevant documents. The Cross-Encoder optimizes the ranking of the remaining candidates. This two-stage approach combines the speed of the Bi-Encoder with the accuracy of the Cross-Encoder.

Generative Reranker

Reranker based on a language model (e.g., Qwen2, Qwen3). Scores relevance through token generation instead of sequence classification.

Typical approach: The model receives query + document as a chat prompt and decides whether the document is relevant. The score is the probability of the "yes" token relative to the "no" token.

Difference from Cross-Encoder: Cross-Encoders have a dedicated classification head (linear layer). Generative rerankers use the language modeling head and extract scores from token probabilities.

Examples: Mixedbread mxbai-rerank-v2 (Qwen2-based), Qwen3-Reranker.

Over-Retrieval (retrieve_k)

Strategy of fetching more candidates from the index than finally needed, to give the reranker a broader selection.

Example: For k=5 final results, retrieve_k=100 candidates are fetched. The reranker scores all 100 and returns the best 5.

Typical ratios: 3-20x the final k. Too few candidates limit the reranker, too many increase latency (especially with LLM-based rerankers).

In our system: --retrieve-k 100 (CLI parameter in run_eval.py).

---

Response Synthesis

Response Synthesis

The LLM generates a natural-language answer from retrieved chunks. Different modes determine how chunks are presented to the LLM. The mode affects quality, latency, and token consumption.

Response Mode: tree_summarize

Hierarchical summarization: Chunks are summarized pairwise, the summaries summarized again → final answer. Requires multiple LLM calls (O(log n)), but delivers the best quality with many chunks, since every chunk is considered.

Response Mode: refine

Iterative refinement: First chunk → first answer. Each subsequent chunk refines the answer. N LLM calls (N = number of chunks). Good when the order of chunks matters.

Response Mode: compact

Fits as many chunks as possible into one prompt (up to context window limit). Typically 1-2 LLM calls. Faster and cheaper, but less thorough with many chunks.

Temperature

Sampling parameter of the LLM. Controls how deterministic or creative the answers are.

Value	Effect	Recommendation
0.0	Deterministic (always the most likely token)	Factual answers
0.3	Slight variation	Good compromise
0.7+	Creative/random	Not recommended for RAG

For RAG systems with factual answers: 0.0-0.3 recommended.

---

Synthesis Metrics

Faithfulness (LLM-as-Judge)

Measures whether the answer is based exclusively on the source chunks (no fabrications). LLM-as-Judge: An evaluator LLM checks every claim in the answer against the source texts.

Values: 0.0 (all claims fabricated) to 1.0 (all claims verifiable).

Status: Implemented in LLMMetrics.faithfulness(). Not invoked in Phase 6 (embedding-based metrics instead). Activatable from Phase 7 via --enable-faithfulness flag. Uses Claude Haiku 4.5 as evaluator.

LLM-as-Judge

Evaluation approach where one LLM assesses the quality of another LLM's output. Scalable and automatable — no manual annotations needed.

Risk: Self-evaluation bias — a model rates itself more favorably. Therefore: Use a different model as evaluator (e.g., Claude Haiku 4.5 evaluates GPT-4o-mini outputs).

In Phase 6: LLM-as-Judge (Faithfulness) was planned but deferred in favor of deterministic embedding-based metrics.

Answer Relevance (Embedding-Based)

Semantic similarity between query and answer, measured as embedding Cosine Similarity. A high value means the answer actually addresses the question, not just contains related material.

Method: Query embedding vs. Response embedding via E5-large-v2. Type: Deterministic, no LLM call.

Hallucination Score (Embedding-Based)

Per-sentence embedding comparison of the answer against all source chunks. Each sentence of the answer is encoded as an embedding and compared against the source chunk embeddings. Sentences with low similarity to all sources are potentially hallucinated.

Values: 0.0 (strong hallucination) to 1.0 (no hallucination detected). Type: Deterministic, no LLM call. Complementary to Faithfulness — measures the same aspect (source fidelity), but embedding-based instead of LLM-based.

Keyword Coverage (Rule-Based, Custom)

Proportion of expected keywords (must_contain_keywords from the eval dataset) in the generated answer. Custom metric for this project, since the eval dataset has no reference answers but defines keywords per query.

Formula: keyword_coverage = |found keywords| / |expected keywords| Method: Case-insensitive substring match.

Example: Keywords = ["SageMaker", "machine learning", "managed service"]. Answer contains "SageMaker" and "machine learning" → Coverage = 2/3 = 0.67.

Context: More practical than purely semantic metrics, because concrete facts are checked. At the same time limited: Synonyms and paraphrases are not recognized ("fully managed" does not match "managed service").

Source Attribution (Embedding-Based)

Measures which source chunks actually contributed to the answer. Embedding similarity between the answer and each source chunk.

Values: attribution_rate from 0.0 (no sources used) to 1.0 (all sources contributed). A high value means the LLM effectively uses the provided sources rather than relying on its own knowledge. Type: Deterministic, no LLM call.

Latency Percentiles (p50, p95, p99)

Statistical distribution measures for response times across all queries:

Percentile	Meaning
p50 (Median)	50% of queries are faster
p95	95% of queries are faster (tail latency)
p99	99% of queries are faster (worst case)

Why not just avg? The average obscures outliers. p95/p99 show how the system performs under load. For user-facing systems, p95 is often the more relevant metric than avg.

Quality Composite Score

Weighted score from all synthesis metrics. Normalized to [0, 1].

Weights (with Faithfulness):

• Answer Relevance: 35%
• Keyword Coverage: 25%
• Source Attribution: 20%
• Hallucination Penalty (1 - halluc_score): 10%
• Faithfulness: 10%

Without Faithfulness: Relevance 40%, Keywords 25%, Attribution 20%, Halluc 15%.

Cost Efficiency

Ratio of quality to cost: quality_score / total_cost_usd. Higher = better price-performance ratio.

---

LLM Hosting

vLLM

Open-source LLM inference engine. Hosts open-source models on your own GPU infrastructure with an OpenAI-compatible API. Instead of token-based billing, you only pay for GPU compute time.

Key Features:

• PagedAttention for efficient KV-Cache management
• Continuous Batching for high throughput
• Tensor Parallelism for large models across multiple GPUs
• OpenAI-compatible API (/v1/chat/completions)

In our system: vLLM runs as a Kubernetes Deployment on GPU nodes. The model is configured via --served-model-name and accessible through an internal service (vllm-svc.ml-models.svc.cluster.local:8000). Important: The service is named vllm-svc (not vllm), because Kubernetes for a service named vllm automatically injects VLLM_PORT=tcp://..., which conflicts with vLLM's own VLLM_PORT env var.

GPU-Time Pricing

Cost model for self-hosted LLMs. Instead of token-based billing (like OpenAI), the GPU usage duration is charged:

cost = latency_ms / 1000 / 3600 * gpu_hourly_rate

Example: 500ms inference on g6.xlarge ($0.98/h): 0.5 / 3600 * 0.98 = $0.000136 per query

Advantage: Costs scale with inference time, not token count. Large prompts are not more expensive than small ones (at the same inference time).

Tensor Parallelism (TP)

Distribution of a model across multiple GPUs by splitting weight matrices along columns/rows. Each GPU holds only a part of the model.

When needed: When the model does not fit on a single GPU.

• 70B-Q4 (~35GB) does not fit on 1x A10G (24GB), but on 4x A10G (96GB) with TP=4
• 27B/32B BF16 fits on 1x L40S (48GB) without TP

Overhead: TP requires inter-GPU communication at every transformer layer. 1x L40S without TP is therefore often faster than 4x A10G with TP=4 for models that fit on a single GPU.

Quantization (AWQ, GPTQ)

Reduction of weight precision from BF16/FP16 (16 bit) to INT4 (4 bit). Halves VRAM requirements with minimal quality loss.

AWQ (Activation-aware Weight Quantization): Quantizes weights asymmetrically, keeps important weights (based on activation distribution) at higher precision. Natively supported by vLLM.

GPTQ (GPT Quantization): Older method, quantizes weights per layer with error correction. Also natively supported by vLLM.

TorchAO: PyTorch-native quantization format. Requires torchao>=0.10.0 — not pre-installed in vLLM v0.14. Models with TorchAO quantization (e.g., pytorch/gemma-3-27b-it-AWQ-INT4) do not work out-of-the-box.

Practical experience from Phase 7:

Model	BF16 VRAM	Quantized	VRAM (est.)	GPU
Phi-4 (14B)	~28GB	AWQ-INT4	~8GB	L4 (24GB)
Gemma-3-27B	~54GB	GPTQ-INT4	~14GB	L40S (48GB)
Qwen3-32B	~64GB	AWQ	~16GB	L40S (48GB)

Without quantization, mid-tier models (27B, 32B) do not fit on single GPUs. Quantization is not optional in practice, but a prerequisite.

Thinking Mode (Qwen3)

Qwen3 models generate <think>...</think> reasoning tokens by default before the actual answer. This leads to massively increased latency (~21s/query for 8B on L4 instead of expected ~2-3s).

Impact on quality: Despite high latency, Qwen3 delivers the best Keyword Coverage (0.929 for 32B-AWQ) — the thinking tokens appear to actually improve answer quality.

--override-generation-config '{"enable_thinking": false}' did NOT fix the issue in our tests. Latency remained unchanged.

---

End-to-End Evaluation (Phase 8)

Naive Baseline

The intentionally unoptimized reference pipeline, serving as the starting point. All components at default/beginner settings:

• Chunking: fixed_256_25 (fixed 256-token chunks, 25 token overlap)
• Embedding: bge-base-en-v1.5 (768 dimensions)
• VectorDB: ChromaDB (in-process)
• Search: Dense
• Reranking: None

Per-Layer Impact Ranking

Method for quantifying each optimization layer's contribution. Compares retrieval metrics (MRR, nDCG, Precision, Recall) before and after each phase optimization. Sorted by absolute delta of the primary metric (e.g., MRR).

Cost-at-Volume Modelling

Cost projection for different query volumes (100/1k/10k/100k per month):

• API costs: cost_per_query * volume (from Phase 7 eval results)
• GPU costs: (avg_ms / 3_600_000) * gpu_hourly_rate * volume

Insight from Phase 8: GPT-4o-mini ($0.00025/query) is cheaper per query than self-hosted Qwen3-8B on L4 ($0.006/query at 21s latency due to Thinking Mode). The GPU advantage lies in data privacy and quality, not in cost.