held out set
technique judged 96% likely a named technique or architecture
See how it has spread across media tiers
Mentions over time
Table view
| Week | Articles | Sources |
|---|---|---|
| 2026-W09 | 1 | 1 |
| 2026-W33 | 1 | 1 |
| 2026-W40 | 3 | 2 |
| 2026-W41 | 1 | 1 |
In their words
On the repository's held-out set: 78 of 78 attacks caught, 0 of 85 benign texts flagged.
Keep a held-out set that prompt authors never tune against, and rerun it whenever the model, tool schema, or retrieval configuration changes.
Use a fresh held-out set for the next qualification decision rather than treating a tuned passing score as independent evidence.
The held-out set was run once and is now consumed; CI uses development queries.
Evaluate with a held-out set of questions and rubric-based scoring.
In practice, preference learning requires: - A structured representation of the action that makes deltas computable (not just free text) - A minimum sample threshold before incorporating preferences (20–50 examples per reviewer per action type is a reasonable starting point) - A validation step where the preference-updated agent is evaluated against a held-out set of prior decisions 7.