EdTech Tracker

← Coding lens

held out set

technique judged 96% likely a named technique or architecture

First seen
2026-03-05
Independent sources
4
Articles
6
Last 4 weeks vs prior
0 vs 4

See how it has spread across media tiers

Mentions over time

Articles per week (by publish date) using the term, in any spelling.
01.532026-W092026-W402026-W41
Table view
WeekArticlesSources
2026-W0911
2026-W3311
2026-W4032
2026-W4111

In their words

On the repository's held-out set: 78 of 78 attacks caught, 0 of 85 benign texts flagged.

Keep a held-out set that prompt authors never tune against, and rerun it whenever the model, tool schema, or retrieval configuration changes.

Use a fresh held-out set for the next qualification decision rather than treating a tuned passing score as independent evidence.

The held-out set was run once and is now consumed; CI uses development queries.

Evaluate with a held-out set of questions and rubric-based scoring.

In practice, preference learning requires: - A structured representation of the action that makes deltas computable (not just free text) - A minimum sample threshold before incorporating preferences (20–50 examples per reviewer per action type is a reasonable starting point) - A validation step where the preference-updated agent is evaluated against a held-out set of prior decisions 7.