golden dataset
technique judged 97% likely a named technique or architecture
See how it has spread across media tiers
Mentions over time
Table view
| Week | Articles | Sources |
|---|---|---|
| 2026-W09 | 2 | 1 |
| 2026-W11 | 7 | 1 |
| 2026-W40 | 1 | 1 |
In their words
Latency (p50, p95), cost per query, refusal rate Build a golden dataset Start with 50 to 200 realistic questions, each paired with the source passages that answer it and, ideally, a reference answer.
Online Evaluation - Uses golden datasets with known answers - Fast iteration — run in CI on every commit - Catches prompt regressions immediately - Can measure RAGAS metrics automatically - Dataset may not reflect real user queries - Does not capture user satisfaction signals - Requires maintaining and updating datasets - Reflects actual user intent and behavior - Captures satisfaction through fee
First, offline evaluation: run a golden dataset through the classifier and measure misclassification rate — if more than 10% of complex queries get routed to Tier 1, the classifier needs tuning.
Eval tests score actual LLM outputs against golden datasets.
A/B Testing Prompts” A/B testing provides the only rigorous way to measure how a prompt change affects real users — offline evals tell you the golden dataset score, A/B tests tell you the business impact.
The evaluation process for chunking strategy selection: Step 1: Build a golden dataset.
For most RAG systems, this means running the A/B test on your full golden dataset at least once.
bi-encoders, when reranking is worth the latency cost - Agent patterns: ReAct, Plan-and-Execute, multi-agent orchestration - Caching strategies: Semantic caching, exact match caching, cache invalidation - Observability: Structured logging, distributed tracing, LLM-specific metrics You should be experimenting with: - Fine-tuning for specific use cases - Quantization and model compression - Self-hos