Measured by evalkit against local Ollama on this machine. Each score is the harness metric on every row of the task file. The judge was not called: the task files point it at gpt-4o-mini, and this run did not use a paid API or a substitute grader. Cost is zero because these calls had no provider bill. Quantization and parameter size come from the Ollama show API for the tag that was requested. The CPU line comes from /proc. The inference library is the ggml shared object mapped into the llama-server process that served the calls.
Local CPU run
Ollama 0.40.5 served qwen2.5:0.5b, smollm2:360m, llama3.2:1b through its OpenAI-compatible API. Each table is the full task file. The fixture dashboard stays on the sample page.
CPU
Intel(R) Xeon(R) Processor
family 6 · model 207 · stepping 2 · 4 logical CPUs
Memory
15.63 GiB
3.45 GiB available when this file was written
Server
Ollama 0.40.5
llamacpp · libggml-cpu-sapphirerapids.so
Recorded
2026-10-16 05:09:30.501 UTC
Linux 6.12.94+ x86_64
Each model was stopped before the task. Samples ran one at a time, all rows of one model before the next model. The first sample of a model can include weight load in its latency. A fresh response cache was used, so none of these rows are cache hits.
Extract name, email, order id, and issue from a short support ticket. Predictions must be one JSON object.
2026-10-16 05:09:30.774 UTC · n=8
| Model | Quant | n | Exact | Schema | Judge | p50 latencywall clock | $ / 1kno provider bill | Errors |
|---|---|---|---|---|---|---|---|---|
| llama3.2:1b 1.2B · llama · ctx 4096 | Q8_0 | 8 | 3/8 | 8/8 | not run | 1123.0 ms 8 live | $0 | 0 |
| qwen2.5:0.5b 494.03M · qwen2 · ctx 4096 | Q4_K_M | 8 | 1/8 | 8/8 | not run | 716.4 ms 8 live | $0 | 0 |
| smollm2:360m 361.82M · llama · ctx 4096 | F16 | 8 | 0/8 | 7/8 | not run | 1149.9 ms 8 live | $0 | 0 |
Expected JSON against each model. Removed lines are the reference. Added lines are the prediction.
Customer Jane Doe (jane@example.com) says order #A-1042 arrived damaged.
| Field | Expected | Predicted | Match |
|---|---|---|---|
| name | Jane Doe | Jane Doe | match |
| jane@example.com | jane@example.com | match | |
| order_id | A-1042 | A-1042 | match |
| issue | arrived damaged | Order arrived damaged | differs |
{"name": "Jane Doe","email": "jane@example.com","order_id": "A-1042",- "issue": "arrived damaged"+ "issue": "Order arrived damaged"}
| Field | Expected | Predicted | Match |
|---|---|---|---|
| name | Jane Doe | — | no JSON |
| jane@example.com | — | no JSON | |
| order_id | A-1042 | — | no JSON |
| issue | arrived damaged | — | no JSON |
{"name": "Jane Doe","email": "jane@example.com","order_id": "A-1042",- "issue": "arrived damaged"- }+ "issue": "order #A-1042 arrived damaged."
| Field | Expected | Predicted | Match |
|---|---|---|---|
| name | Jane Doe | Jane Doe | match |
| jane@example.com | jane@example.com | match | |
| order_id | A-1042 | A-1042 | match |
| issue | arrived damaged | damaged | differs |
{"name": "Jane Doe","email": "jane@example.com","order_id": "A-1042",- "issue": "arrived damaged"+ "issue": "damaged"}
Classify a short support ticket as billing, shipping, product, or account. Predictions must be one JSON object with a label field.
2026-10-16 05:09:59.639 UTC · n=8
| Model | Quant | n | Exact | Schema | Judge | p50 latencywall clock | $ / 1kno provider bill | Errors |
|---|---|---|---|---|---|---|---|---|
| qwen2.5:0.5b 494.03M · qwen2 · ctx 4096 | Q4_K_M | 8 | 4/8 | 8/8 | not run | 170.0 ms 8 live | $0 | 0 |
| llama3.2:1b 1.2B · llama · ctx 4096 | Q8_0 | 8 | 2/8 | 4/8 | not run | 432.4 ms 8 live | $0 | 0 |
| smollm2:360m 361.82M · llama · ctx 4096 | F16 | 8 | 0/8 | 0/8 | not run | 677.1 ms 8 live | $0 | 0 |
Expected JSON against each model. Removed lines are the reference. Added lines are the prediction.
I was charged twice on my card for the same order.
| Field | Expected | Predicted | Match |
|---|---|---|---|
| label | billing | account | differs |
{- "label": "billing"+ "label": "account"}
| Field | Expected | Predicted | Match |
|---|---|---|---|
| label | billing | billing | match |
| status | null | pending | differs |
| message | null | You were charged twice on your card for the same order. Please try again later. | differs |
{- "label": "billing"+ "label": "billing",+ "status": "pending",+ "message": "You were charged twice on your card for the same order. Please try again later."}
| Field | Expected | Predicted | Match |
|---|---|---|---|
| label | billing | billing | match |
{"label": "billing"}
1 when the parsed JSON equals the reference. Key order does not matter. A wrong string in any field scores 0.
1 when the output parses and a Pydantic model built from the task schema accepts it. Extra keys fail when additionalProperties is false.
Not called. The task files point this metric at gpt-4o-mini. This run did not use a paid API or a substitute grader, so the column says not run.
Wall time of the successful call. Models were stopped before each task, so the first row can include weight load. Cache hits are left blank; this file has none.
Reported token counts times the price stored on the run. These tags are priced at $0, source “local Ollama; no provider bill”.