The eval set is your contract with your own system.

Once your system grows. a 3-step chain, a router with 4 routes, RAG with honest generate. you can't test it mentally anymore. You need a set of test cases that demonstrates the system does what you promise.

That set is the eval set. And the skill of writing a good one is what separates the LLM engineer from the demo-builder.

What makes an eval set good

An eval set isn't a handful of random cases. It's a curated selection of inputs covering:

1. Control cases. Obvious cases any decent system should pass. If they fail, the system is broken at the basics.
2. Edge cases. Cases on the boundaries. empty inputs, very long inputs, inputs in another language, inputs ambiguous between two categories.
3. Adversarial cases. Cases that try to break the system: typos, jargon, injected instructions, messages that SOUND like one category but are another.

Each case carries a comment: what it's there for. Without comments, when a case fails 6 months from now, you won't know if it was a control or an edge case you consciously decided not to support.

The task

Echo asks for an eval set for the step 02 classifier (4 categories: safety, maintenance, social, other).

At least 10 cases, distributed:

• 4+ with each category represented.
• 2+ ambiguous (where a mediocre classifier would get confused).
• 2+ adversarial (jargon, typos, injection, false positives).
• All with comment explaining what each case evaluates.

How it's evaluated

6 LLM-judge criteria on the set's quality:

1. At least 10 cases with the correct shape.
2. Covers the 4 categories.
3. Has labeled ambiguous cases.
4. Has labeled adversarial cases.
5. Non-trivial comments that justify each case.
6. Not just easy cases.

The mental test: if I, reading only your eval set, could reconstruct what the classifier does and where it struggles. you passed. If it looks like "10 random inputs with labels", you didn't.