EdTech Tracker

← Coding lens

human evaluation

technique judged 90% likely a named technique or architecture

First seen
2024-06-26
Independent sources
4
Articles
10
Last 4 weeks vs prior
0 vs 1

See how it has spread across media tiers

Mentions over time

Articles per week (by publish date) using the term, in any spelling.
0362024-W262026-W092026-W41
Table view
WeekArticlesSources
2024-W2611
2025-W4011
2026-W0911
2026-W1161
2026-W4111

In their words

How to Evaluate LLMs: A Complete Guide to LLM Evaluation From automated metrics to human evaluation — build a robust evaluation pipeline for your LLM applications.

This approach scales better than human evaluation while correlating well with human judgments when the criteria are well-defined.

Monitoring Checklist Section titled “Monitoring Checklist” - Routing accuracy: track how often the supervisor picks the correct agent (use human evaluation samples) - Sub-agent success rate: percentage of sub-agent calls that produce usable outputs - End-to-end latency: p50 and p95 across the full graph execution - Token consumption: total tokens per request — compare against single-agent baseline

(2020), foundational work on few-shot prompting Evaluation: - LMSYS Chatbot Arena — Human evaluation benchmarks and model comparisons - promptfoo — Open-source CLI for prompt testing and comparison Related Section titled “Related” - GenAI Engineer Roadmap — Complete career progression guide for GenAI engineers - LLM Evaluation Guide — Metrics and frameworks for measuring LLM system quality - RAG A

Beyond RAGAS: Custom Metrics, Human Evaluation, and A/B Testing Section titled “Beyond RAGAS: Custom Metrics, Human Evaluation, and A/B Testing” RAGAS covers the core failure modes, but production systems often need additional evaluation layers.

This is not a perfect substitute for human evaluation, but it enables continuous monitoring and regression detection.

Know which constraint is least flexible for your use case: - Customer-facing chat: Latency is king (users abandon after 3 seconds) - Batch document processing: Cost matters most (processing millions of documents) - Medical/legal advice: Quality dominates (errors have serious consequences) Common Failure Modes Section titled “Common Failure Modes” Retrieval Failures Section titled “Retrieval Failur

- Human evaluation — sample 100 responses per week, have domain experts label hallucinated claims.