Arena · repeat and verify

Can you reproduce that result?

Run a small task more than once, keep the evidence and see what your checks actually show.

See what a record can show

Fictional arithmetic example, not an agent result: Two practice runs pass 4 of 5 and 5 of 5 checks. That is 9 of 10 checks across two assessed runs. If a third run was not assessed, record it as unassessed; it is not another zero out of five.

If one run's cost is unknown, an average cost for all runs is unknown. Even when numbers match, inspect the artifacts and the checks; a count is not proof of quality, safety or a universal winner.

Try the seed-data experiment first · Download a blank worksheet

Run, check, record

  1. Choose one task, input and independent rubric. The seed-data kata supplies fictional input and five checks; you can use a task from another field instead.
  2. Record the actual harness, model, versions, instructions, skills, tool permissions and date for each setup. Change one factor where possible; if several change, label it a full-stack comparison.
  3. Repeat the same task in fresh runs. Check each output yourself. Include failures and unassessed runs rather than silently dropping them. Record actual cost and timing only if available.
  4. Use the form below or the blank worksheet. Keep the task, input, checks and settings with your evidence; equal numbers of checks do not prove that the checks were equivalent.

Use a phone app, browser assistant or trusted CLI harness. Provider accounts, access and costs vary. Never paste credentials, private customer data or other information you cannot share into an assistant.

JavaScript is needed for the on-page calculation. Download the blank worksheet and calculate the checks by hand.

One line per run: checks passed, checks attempted, cost, seconds. Use NA,NA for unassessed checks. Leave cost or seconds blank if unknown; write 0 only when measured as zero.

Format examples only: 4,5,, means 4 of 5 checks with cost and time unknown; NA,NA,, means checks were not assessed. These are not measured results.

Setup A
Setup B

The Dojo does not save or send these inputs. The record remains in this page until you copy it or leave. The blank worksheet has space for source links, individual artifacts, failures and reviewer notes.

Read the result carefully

Counts pool the checks you report across assessed runs. Unassessed runs are listed separately and excluded from the passed-check fraction. This tool requires the same number of checks per assessed run, but cannot verify that the checks themselves, the data or the conditions matched.

It cannot verify your model versions, compare field-specific quality, establish statistical significance, price API usage or assess professional safety. Record unobserved values as unknown, repeat meaningful cases, inspect your artifacts and ask a qualified reviewer when the field requires one.

NIST: document test sets, metrics and measurement limits · Inspect a historical published experiment · Return to the Arena