noun · AI practice
Evals
also called evaluations
A test paper for the AI. You write the questions and the grading, then run it again whenever something changes.
Repeatable tests that score a model or AI application on a set of examples, used to measure quality and catch regressions.
A set of example inputs with a way to score the outputs, such as exact checks, rules, or a second model acting as judge. You run it on a schedule and after changes, and track the score so you can see whether a new prompt or model helped.
Real-life example
You tweak your support bot's instructions and a few answers feel better. You are not sure if the change helps overall. You run a saved set of 200 real questions through both versions and compare the scores. Some topics improved and one got worse, which you would not have noticed by hand.
Where the analogy breaks
-
Evals are not unit tests with one exact answer. Outputs vary and many checks are fuzzy, so you look at rates and trends instead of single pass or fail.
-
A narrow set of examples misleads. If the test cases do not look like real use, the score can rise while real quality does not.
-
A model used as judge has biases too. Check its grades against human ones now and then.