Key takeaways
- Use public rankings to choose candidates, not to decide. They measure defined scenarios or user preferences, not your task.
- Write down the expected result for every case before you look at any model output.
- Run each case several times on each candidate, because outputs vary between runs, and keep every result, including the failures.
- Count critical failures separately from the average score. One invented payment detail can matter more than ten good drafts.
01 / 08
What do public benchmarks measure?
Public benchmarks measure different things, and both main types are useful within their limits. Chatbot Arena collects preferences: users compare two anonymous answers, vote for the one they prefer, and see the model names only after voting. The result shows which model people prefer across many general questions. HELM from Stanford evaluates language models with several metrics across many scenarios and states its own gaps in coverage; its authors write that their object of evaluation is the language model, not a scenario-specific system.
Neither measures how a model performs on your documents, in your language variety, under your approval rules and with your tools. That gap is the reason for your own test. Use public results to pick two or three candidates worth testing.
02 / 08
How do you choose the cases?
Start with twenty cases taken from real work. Anthropic suggests that 20 to 50 simple tasks drawn from real failures are a good start. Twenty is a starting sample to find obvious problems, not proof of reliability.
Mix the case types on purpose:
| Case type | Why include it |
|---|---|
| Ordinary cases | The bulk of the work. The model must handle these well. |
| Missing information | The correct result is a question back, not a guess. |
| Conflicting sources | Two documents disagree. The model must notice it, not pick one silently. |
| Cases to escalate | A complaint, a legal question or an unclear identity must go to a person. |
| A severe-risk case | A case where a wrong answer would cause real damage, such as wrong bank details. |
Keep the language, structure and difficulty of the originals. If the cases contain personal data, remember that replacing names is not enough: the UK regulator states that pseudonymised information is still personal data. Check whether a person could still be identified, and keep test data in access-controlled storage.
Put a few additional cases aside and do not use them while you adjust prompts. Test them at the end to see whether the result holds on cases you did not tune for.
03 / 08
How do you define a correct result?
A person who knows the work writes the expected result for each case before the test: required facts and fields, acceptable alternative wording, forbidden actions and the correct escalation. For fields such as an order number, exact comparison works. For a drafted reply, describe what must be in it and what must not.
OpenAI's evaluation guidance recommends task-specific evaluations and calibrating any automated scoring against people. It also names typical biases of AI judges: position bias, which favours the answer shown first, and verbosity bias, which favours longer answers. If you use an AI to pre-score, check its scores against human ones.
04 / 08
How do you make the test repeatable?
Write down everything that could change the result before the first run:
- the exact model identifier where the provider shows one, otherwise the visible model name and the plan;
- the date and time, the prompt version and the version of the source documents;
- tools, search or retrieval switched on or off, and whether memory was on;
- settings such as temperature or reasoning effort where they are exposed, and the region of the deployment.
A pinned model identifier helps, but it does not freeze everything. Anthropic's documentation states that each Claude model ID identifies a pinned version and that the serving infrastructure around the model can still change over time. Mark settings that a chat product does not show as "not exposed".
Run each case three times on each candidate, starting a fresh conversation every time. Model outputs vary between runs, which is why Anthropic recommends repeated trials. With 20 cases, three candidates and three runs, that is 180 results to score. Keep failed and retried runs; do not keep only the best answer.
Got a workflow like this?
Tell me about it. Honest take and a rough price, usually within a day.
05 / 08
How do you score fairly?
Hide which model produced which answer and shuffle the order before scoring, as Chatbot Arena does with its anonymous comparisons. Score each answer against the expected result:
| Score | Meaning |
|---|---|
| 2 | Usable after a routine check |
| 1 | Usable only after a substantive correction |
| 0 | Not usable |
Record pass or fail separately for required fields and forbidden actions, because an answer can read well and still break a rule. Then classify each failure: a minor presentation defect, a substantive error, or a critical failure such as invented payment details, disclosure of another customer's data or an unapproved send. Report critical failures on their own line. Decide before the test whether a single critical failure rules a candidate out. When two reviewers disagree, a second person who knows the work decides.
06 / 08
How do you compare cost and time?
Per run, record the billed units the provider shows (input, output, cached and reasoning tokens), tool charges and retries. Add the share of platform fees and the time a person spends checking and correcting. Keep model response time and human working time apart.
A useful figure is the cost per accepted result: total cost of all runs, including failed ones, divided by the number of results that met the expected outcome. If a candidate produced no accepted result, report that instead of a number. For subscriptions, state how you allocated the seat price; it is not a price per call. Our comparison of ChatGPT, Claude and Gemini lists the plans and their billing.
07 / 08
Take the worksheet
Worksheet
Download the benchmark worksheet
| case_id | case_type | expected_outcome | forbidden_actions |
|---|---|---|---|
| K01 | Ordinary enquiry (invented example) | Category "service"; reply names the next free slot from the calendar | Promising a fixed price |
| K02 | Missing information (invented example) | Asks for the order number; does not guess | Inventing an order status |
| K03 | Must escalate (invented example) | Hands over to a person because of a complaint about an injury | Sending any reply without a person |
Columns to fill in, in the file: candidate, run, pass_fail, score_0_to_2, severity, failure_reason, review_minutes.
Download the benchmark worksheet (CSV). The three example cases are invented and show the format. Replace them with your own cases, add one row per run and candidate, and keep the column that identifies the candidate out of the reviewers' copy.
08 / 08
What do you report at the end?
Report the number of passes with the number of runs behind them, the critical failures, the median response time, the review time and the cost per accepted result, per candidate. Note which cases were excluded and why. Then repeat the test whenever you change the model, the prompt, the tools or the source documents. Our acceptance tests for knowledge assistants and for document processing apply the same idea to specific tasks.
If you want to run such a test with your own cases, our page on choosing AI models describes how we do it in a trial run.
