fixture-exact
Returns the reference JSON object for the ticket.
Fixture output
Sample handlers, kept off the leaderboard
Extract name, email, order id, and issue from a short support ticket. Predictions must be one JSON object. The table is one run of that task, written by scripts/export_sample_results.py at 2026-10-14 13:29:35.421 UTC. The measured models are on the leaderboard.
| Target | Exact | Schema | Judge | p50 latencylocalhost | $ / 1killustrative | Errors |
|---|---|---|---|---|---|---|
| fixture-exact openai-compatible | 8/8 | 8/8 | 1.000 8 scored | 10.6 ms 8 live | $0.025763 | 0 |
| fixture-mutate openai-compatible | 0/8 | 4/8 | 0.750 8 scored | 8.1 ms 8 live | $0.023062 | 0 |
| fixture-prose openai-compatible | 0/8 | 0/8 | 0.000 8 scored | 7.4 ms 8 live | $0.018937 | 0 |
fixture-mutate
Even positions in dataset order (0, 2, 4, ...) lowercase order_id. Odd positions omit email. Every other field is copied from the reference.
fixture-prose
Returns one sentence and no JSON object.
fixture-judge
Returns the fraction of reference fields whose values are unchanged in the parsed prediction. A non-JSON prediction scores 0. This is not an LLM.
Per-sample diff
Expected JSON against each fixture. Removed lines are the reference. Added lines are the prediction.
Customer Jane Doe (jane@example.com) says order #A-1042 arrived damaged.
fixture-exact
- Exact
- 1.000
- Schema
- 1.000
- Judge
- 1.000
- Latency
- 21.8 ms
- Tokens
- 65 in / 25 out
- Cost
- $2.47e-5
| Field | Expected | Predicted | Match |
|---|---|---|---|
| name | Jane Doe | Jane Doe | match |
| jane@example.com | jane@example.com | match | |
| order_id | A-1042 | A-1042 | match |
| issue | arrived damaged | arrived damaged | match |
Judge: field overlap 1.000
{"name": "Jane Doe","email": "jane@example.com","order_id": "A-1042","issue": "arrived damaged"}
fixture-mutate
- Exact
- 0.000
- Schema
- 1.000
- Judge
- 0.750
- Latency
- 8.1 ms
- Tokens
- 65 in / 25 out
- Cost
- $2.47e-5
| Field | Expected | Predicted | Match |
|---|---|---|---|
| name | Jane Doe | Jane Doe | match |
| jane@example.com | jane@example.com | match | |
| order_id | A-1042 | a-1042 | differs |
| issue | arrived damaged | arrived damaged | match |
Judge: field overlap 0.750
{"name": "Jane Doe","email": "jane@example.com",- "order_id": "A-1042",+ "order_id": "a-1042","issue": "arrived damaged"}
fixture-prose
- Exact
- 0.000
- Schema
- 0.000
- Judge
- 0.000
- Latency
- 7.7 ms
- Tokens
- 65 in / 15 out
- Cost
- $1.87e-5
| Field | Expected | Predicted | Match |
|---|---|---|---|
| name | Jane Doe | — | no JSON |
| jane@example.com | — | no JSON | |
| order_id | A-1042 | — | no JSON |
| issue | arrived damaged | — | no JSON |
- output is not valid JSON
Judge: field overlap 0.000
- {- "name": "Jane Doe",- "email": "jane@example.com",- "order_id": "A-1042",- "issue": "arrived damaged"- }+ I extracted a ticket but this fixture does not emit JSON.
What the columns measure
Exact match
1 when the parsed JSON equals the reference, including every field. Key order does not matter. Prose scores 0.
JSON schema
1 when the output parses and a Pydantic model built from the task schema accepts it. Extra keys fail when additionalProperties is false.
Judge
A second chat completion that must return a score from 0 to 1. In this file the judge is the fixture overlap grader, not a model.
Latency
Wall time of the successful HTTP call. Cache hits are left blank in the mean so a warm cache does not look like inference speed.
Cost
Reported token counts times a price. Unknown models stay blank. The prices in this file are illustrative rates on fixture usage.
Each fixture price is an illustrative rate equal to the OpenAI gpt-4o-mini standard list price observed at https://developers.openai.com/api/docs/pricing on 2026-10-14 ($0.15 / 1M input, $0.60 / 1M output). It is applied to token counts the fixture itself put in the usage field (about 4 characters per token). It is not a bill from any provider.