EdTech Tracker

← Coding lens

llm judge

technique judged 98% likely a named technique or architecture

First seen
2024-03-19
Independent sources
13
Articles
44
Last 4 weeks vs prior
0 vs 17

See how it has spread across media tiers

Mentions over time

Articles per week (by publish date) using the term, in any spelling.
06122024-W122026-W302026-W41
Table view
WeekArticlesSources
2024-W1211
2024-W2611
2024-W2711
2024-W4411
2025-W3111
2026-W0921
2026-W11111
2026-W1411
2026-W1711
2026-W3011
2026-W3111
2026-W3211
2026-W3311
2026-W3711
2026-W3822
2026-W3911
2026-W40123
2026-W4141

In their words

Your response:""" ## Part 4: Prompt Optimization ### Automated Prompt Optimization python prompting/optimization.py from dataclasses import dataclass from typing import List, Callable import asyncio @dataclass class PromptCandidate: prompt: str score: float = 0.0 metadata: dict = None class PromptOptimizer: def init ( self, llm, evaluator: Callable[[str, str], float], # (prompt, expected) -> score

items ()} def evaluate_generation ( self , test_cases : List [ RAGTestCase ]) -> dict : """ Measure answer quality using LLM judge.

With evaluation, you: Catch regressions before users do Compare models objectively (GPT-4o vs Claude vs Llama) Optimize prompts with data, not vibes Meet compliance/quality requirements Evaluation Levels Level What It Tests Speed Cost Reliability Unit Tests Deterministic functions Fast $0 High Automated Metrics BLEU, ROUGE, BERTScore Fast $0 Medium LLM-as-Judge Quality, correctness, style Medium L

A cross-encoder returns a calibrated score per query-chunk pair, which is what the LLM judge was being prompted to imitate.

No LLM judge, no vibes.

The setup I ended up with is simple to state: the LLM judges, and my code does everything that can be computed.

Score generation with care LLM-as-judge is the practical way to grade faithfulness and relevance at scale.

It needs to orchestrate a multi-turn conversation between your agent and another model (or the same one with different prompts), and then pass the transcript to an LLM-as-judge to evaluate.