Reproduce a Marina benchmark

One command per published setup. It boots isolated Marina servers, runs the official evaluator for each benchmark, files every run into a benchmark ledger, and prints a pooled, paired comparison between arms. This page covers mechanics only. Run it yourself to see the numbers on your own models.

See what you have, and the exact fix for what you don’t:

bun run repro doctor

List setups, arms, smoke and full sizes:

bun run repro list

Print the plan and estimated spend, with no model calls:

bun run repro hle-verified --dry-run

Run it at smoke size, 2 replicates per arm, with a $10 budget:

bun run repro hle-verified

Setups

setup	arms	official scoring	prerequisites
hle-verified	single (one model through Marina), verify (a verification crew)	HLE-Verified Gold answers, judged by a model through Marina	models
swebench-verified	single (one model patches), verify (patch plus reviewer)	the unmodified swebench harness in containers	models, podman or docker, Python swebench
tau2	single (one model as the agent), verify (marina/verify:<model>[+<checker>]); obligations (marina/obligations:<model>), argcheck (marina/argcheck:<model>) and obligations+argcheck (both), each only when named with --arm	the unmodified τ²-bench CLI, user simulator and evaluator	models, a τ²-bench checkout
futurex-backtest	cheap, verify forecast variants	resolved FutureX questions, forecast with date-bounded research only	models
arena-backtest	baseline, nowcast	MIT Social Simulation Arena rounds, scored in lock order	none (keyless)
automation	bare (x-marina-stack: none), on (the identified-caller default stack, x-marina-stack: lessons,obligations,argcheck,learn); full (x-marina-stack: full) only when named with --arm	the unmodified AutomationBench auto-bench agent and its assertions; a fixed list of 24 tasks over six domains (smoke 6, one per domain)	models, AUTOMATIONBENCH_BIN
ifbench	same arms as automation	one chat completion per prompt, graded strict by IFBench’s own run_eval (smoke 20, full 150)	models, IFBENCH_HOME, IFBENCH_INPUT

Each arm is compared with the first arm on the same items, pooled over replicates (a two-stage bootstrap over runs, then items). A single run is labelled “not replicated”.

Flags

flag	meaning
--arm a,b	which arms to run (default: all except opt-in arms such as τ²’s obligations)
--replicates N	independent runs per arm (default 2); a crew arm gets a fresh server per replicate
--limit N	items per arm and replicate (default: the smoke size); the full sizes are in repro list
--budget-usd X	refuse to start when the estimate exceeds X (default 10); X is also a hard total, split evenly across the run’s servers or SWE-bench runs
--model, --checker, --judge	the answer model, the reviewer or checker, and the judge or user simulator
--domain	τ²-bench domain (default airline)
--split	τ²-bench task split (test or base); runs the whole split unless --limit is given. base is the leaderboard split: every task, so --limit is refused with it. A re-run with the same --run-dir resumes an interrupted τ² run (--auto-resume)
--effort, --user-effort	τ²-bench agent and user-simulator reasoning effort (default high and low), sent in extra_body. automation: auto-bench --reasoning-effort (default medium)
--retrieval-config	τ³ banking_knowledge only: τ²’s knowledge-base retrieval configuration (default alltools, the board’s reference configuration; it is shown on the board and is part of the run’s configuration tag). Refused for other domains
--task-ids a,b,c	τ² only: run exactly these task ids (τ²’s --task-ids), for a pre-registered subset. Not with --limit or the board split base; the ids are part of the configuration tag
--env-image	SWE-bench: run the agent’s tests inside each instance’s environment image (full agent rather than agentless)
--server-env KEY=VALUE	Marina feature settings for every server the kit starts (repeatable; feature variables only — MARINA_OBLIGATIONS_*, MARINA_ARGCHECK*, MARINA_LESSONS*, MARINA_DECISION*, MARINA_ANTHROPIC_AUTO_CACHE, … — never a credential). The servers’ effective feature settings (your environment plus these) are part of the τ² configuration tag and are filed with every result (benchmark-import --server-features), so runs with different instruments never resume, pool or promote together
--review off|observe|auto|on	τ²: the pre-write review mode the agent’s requests choose (x-marina-review), capped by the server’s MARINA_OBLIGATIONS_REVIEW. The agent’s requests always carry x-marina-eval: benchmark=tau2-<domain>; mode=measure, so lessons learned on the board are never served back to it
--lessons	τ²: recall judged lessons on the agent’s requests (marina/lessons: on every arm); lessons learned from the board itself are still excluded, because the requests are tagged a measurement. Part of the configuration tag
--run-dir, --ledger	where runs, servers and the ledger live (default under ~/.local/share/marina-repro/, on disk)
--dry-run	print the plan and the estimate, then stop

Model tiers

The doctor reports which intelligence is reachable. Every setup runs at every tier and labels what changed, so a single small model can still run everything:

tier	how it is detected	what changes
frontier	OPENROUTER_API_KEY, or keys for two or more vendors	the published default models
single provider	one vendor key	every arm uses that provider; verification is the model checking itself, and the judge is the same model (not independent)
single local model	OLLAMA_BASE_URL or LLAMA_BASE_URL	the same, on marina/default or --model ollama/<id> / llama/<id>; the spend estimate is 0

Prerequisites and their fixes

bun run repro doctor [setup] checks each of these and prints the fix:

• Models: any key above, or a local model runtime.
• Optional data keys: TAVILY_API_KEY, ODDS_API_KEY, FRED_API_KEY, HF_TOKEN. Results differ without them, so note which you had when comparing.
• Container runtime (SWE-bench): podman needs its API socket (systemctl --user enable --now podman.socket).
• Podman network: on kernels without bridge networking, set netns = "pasta" in ~/.config/containers/containers.conf.
• Sub-UID range: a few SWE-bench images own files above 65,536 UIDs. Widen /etc/subuid and /etc/subgid (sudo usermod --add-subuids 100000-399999 --add-subgids 100000-399999 $USER && podman system migrate), or those instances are excluded.
• Scratch space: runs keep scratch under the run directory on disk. /tmp is often a RAM-backed tmpfs, too small for images and databases.
• SWE-bench harness: a Python with the swebench package, named by SWEBENCH_PYTHON.
• τ²-bench: a checkout with its virtualenv, named by TAU2_HOME.
• τ²-bench evaluator: OPENAI_API_KEY in .env. τ²’s NL-assertion judge calls OpenAI’s gpt-4.1 with keys from the environment; without one, those simulations end as infrastructure errors.
• AutomationBench: AUTOMATIONBENCH_BIN names an executable auto-bench (from a virtualenv with AutomationBench installed).
• IFBench: IFBENCH_HOME names an IFBench checkout (with run_eval.py; its .venv/bin/python is used when present, else python3), and IFBENCH_INPUT a prompt JSONL (key, prompt, instruction_id_list, kwargs), for example $IFBENCH_HOME/data/IFBench_test.jsonl. The kit answers its first --limit lines.
• τ³ knowledge shell (--domain banking_knowledge with alltools or terminal_use): τ² runs the agent’s knowledge-base shell under Anthropic’s sandbox-runtime, so srt (@anthropic-ai/sandbox-runtime@0.0.23, the version τ² pins), rg, bwrap and socat must be on PATH, and the τ² virtualenv needs τ²’s knowledge extra (pip install -e ".[knowledge]"). alltools also embeds with OpenAI (OPENAI_API_KEY); those embedding calls are τ²’s own and are not in Marina’s spend ledger.

What the kit does for you

These are the mechanics that otherwise trip a reproduction:

• Server keys: each server gets a fresh random MODEL_API_KEYS key, and the benchmark uses that same key. A shared “open” key is refused once keys are set, so nothing could file into the ledger. The key is never printed.
• Judging through Marina: graded answers are judged through a Marina server, not by calling a vendor directly.
• Time limits: crews get a long per-item limit, and the server’s request timeout is set above it.
• τ²-bench: the agent and user simulator go through Marina, and τ²’s own evaluator runs as shipped. The τ² process gets the provider keys from .env (never printed) and no OPENAI_BASE_URL override. Reasoning effort travels in extra_body, because τ² sets LiteLLM’s drop_params, which strips a top-level reasoning_effort for Marina-routed ids. The agent effort, user simulator and its effort appear in the plan’s labels. A run with any infrastructure error is invalid: tau2 convert --require-clean reports the count, writes no scores, and the run never reaches the ledger.
• Instrument-stack arms (automation, ifbench): each arm gets its own server, whose key is bound to an entity (MODEL_API_KEYS=<key>:BenchArm), so the harness is an identified caller that every stack member can serve. The arms differ only in the x-marina-stack header each request carries; on lists the default members explicitly, so a member that cannot run is refused rather than silently skipped. Every request also carries x-marina-eval: benchmark=<automationbench|ifbench>; mode=measure. AutomationBench sends both through auto-bench --headers, and the IFBench runner (scripts/ifbench.ts answer) sets them itself, with its key taken from the environment and never from argv. Each AutomationBench task runs in its own auto-bench process. scripts/automationbench.ts convert and scripts/ifbench.ts convert file only task names or input keys, with pass/fail (passed, or strict follow_all_instructions) and the fraction of assertions or instructions met. They never file transcripts, prompts or responses. With --require-clean, any item without a result (missing export, aborted task, failed request, ungraded prompt) refuses the whole run.
• Spend: the budget is a true total. Each server’s daily spend cap, and each SWE-bench run’s --max-usd, is an even share of --budget-usd, rounded down, with no per-server minimum, so together they never exceed it.
• The ledger: results are imported into one ledger, grouped by arm, so benchmark compare, leaderboard and replicates work on them afterwards.

Reading the output

Compare hle-verified-verify (A) vs hle-verified-single (B) on <n> shared items
  A <acc> over 2 run(s)   B <acc> over 2 run(s)
  A − B <delta>  95% [<low>, <high>]  p=<p>

The interval and p come from resampling both runs and items, so run-to-run noise is included. Inspect further from any Marina pointed at the ledger:

• benchmark compare <runA> <runB>;
• benchmark replicates <run>;
• benchmark frontier <benchmark> (accuracy against cost per item).

Nothing is submitted to any leaderboard. Each benchmark’s own guide describes its submission process: SWE-bench, τ²-bench, FutureX, Arena, ForecastBench, Metaculus.