Capture the decision
Create one record per evaluated model and task. Identify the provider, exact model name, and variant separately. Do not shorten an identifier until two records can still be distinguished. The models documentation (opens in a new tab) covers model selection, while the providers documentation (opens in a new tab) is the reference for provider-specific access and configuration.
Use the same task, inputs, and rubric for every candidate. Store the prompt or a stable input reference, the rubric version, and the evaluation date. State the accepted result in observable terms, such as “valid JSON with all five fictional rows classified and no invented fields.” A model is accepted only against that stated criterion; a plausible answer is not automatically an accepted answer.
Record evidence, not guesses
For each attempt, record the outcome, elapsed time, and cost when those values are available from a trustworthy measurement. Keep a count of attempts, including retries. Mark each field verified, unavailable, or not applicable. Never turn a blank dashboard value into $0, an unknown latency into 0 ms, or an unobserved pass into “accepted.” Preserve the raw evidence or its stable reference so another reviewer can distinguish a measured value from a task instruction.
Synthetic example:
Task: classify 5 fictional invoices; rubric v1
Provider: Example Cloud
Model: example-text-2
Variant: standard
Accepted result: verified; 5/5 labels, valid JSON
Attempts: 2 (verified)
Time: unavailable
Cost: unavailable
Availability: verified at evaluation time
If the same model is unavailable for a second evaluator, add a separate observation: availability unavailable, with the date and reason if known. Do not copy the first run's time or cost into it. The record is a comparison aid, not a claim that one model always wins. Recheck names and access before relying on it. Verified 2026-09-06.