FutureX

FutureX is a live benchmark for agents that predict future events. Each week it publishes a batch of open questions on Hugging Face; entrants answer before a deadline and are scored once the events resolve. Marina enters with its ordinary forecasting engine — the same typed forecasts any user gets. The adapter in benchmarks/futurex/ and scripts/futurex.ts only translates the batch into general forecast calls and writes the submission file. Nothing in Marina’s core knows about FutureX.

Commands

bun run futurex fetch                 # the current batch, at its Hugging Face commit sha
bun run futurex fetch --past          # the resolved-questions dataset (for backtests)
bun run futurex run                   # forecast every row with the earned live configuration
bun run futurex run --variant cheap   # a fixed configuration instead (the override for experiments)
bun run futurex run --variant cheap --variant frontier   # several variants, one file each
bun run futurex backtest --limit 40   # resolved rows: forecast with an early cutoff, score, record
bun run futurex backtest --clean --isolation closed-book --variant cheap --limit 160 --replicates 2 --lessons on
                                      # a non-leaking backtest (see Clean backtests)
bun run futurex watch --once --run earned   # poll for a new batch; on one, fetch and run
bun run futurex watch --once --run earned --daily --learn
                                      # …and re-forecast open rows daily (see Standing answers)
bun run futurex watch --once --run earned --daily --daily-rows stale --near-end-hours 48
                                      # …daily runs touch only stale rows (see Standing answers)
bun run futurex watch --once --run earned --answer late --final-lead-hours 6
                                      # …answer each row once, close to its cutoff (see Answering late)
bun run futurex schedule --run earned  # PRINT a systemd service + hourly timer for the late watcher
bun run futurex learn                 # resolved rows of filed batches → outcome lessons
bun run futurex status                # every submission recorded so far

Files go to data/futurex/ (ignored by git; --dir to change). Benchmark questions are never committed. DB_PATH chooses the ledger (default marina.db). The forecaster’s own settings (MARINA_FORECAST_*, forecasting) apply; OPENROUTER_API_KEY is required and TAVILY_API_KEY enables date-filtered search.

How a row becomes a forecast

benchmarks/futurex/map.ts reads the answer shape each row asks for:

The row	The forecast
lettered options (`A. …`), or boxed alternatives in older rows	`choice` — or `multi` for a level-2 bundle of independent outcomes (nominations, who qualifies, cumulative thresholds such as “at least …”); a count with bucketed options (“how many …”) is always one `choice`
“how many …”, a unit (“Report the value in …”, “…, in knots, …”), a measured value, or an official statistic titled by its series and period (“Job openings — July 2026”)	`number` (an integer for counts; the unit is passed on)
one top entry (“which club will be first …”, “rank No. 1”)	`text`
an ordered or “which N” list, “ranked from X to Y”, “which ” (“which identifiers …”), “winners of the N …”	`ranking`, sized when the row says how many
anything else (an exact title)	`text`

Each rule is checked against the shape of every resolved row’s truth. test/forecast-shape-scale-selection.test.ts holds synthetic cases. With FUTUREX_SHAPE_AUDIT_BATCH=<past batch json> set, it also checks, by row id, the rows a failure analysis found mistyped.

The full prompt goes along as the forecast’s context (it carries the settlement rules), and the row’s end time becomes the evidence cutoff. The prompt’s own format line (IMPORTANT: End with … \boxed{…}) is stripped by default. --format-line keep (or "formatLine": "keep" in a variant) keeps it, so the general answer contract can read it when MARINA_FORECAST_CONTRACT is observe or on. The default is unchanged, so the line’s effect can be measured as one switch; the choice is part of the variant, so a run’s journal and ledger configuration record it.

Freezing and filing

Every answer uses nothing published after the earlier of now and its question’s end time. A row whose end time has already passed is answered with its cutoff at that end time and flagged late.
Every row gets an answer: a missing prediction scores zero, so when no run produced a usable answer, the runs’ best partial answer (or, for a choice, the first substantive option) is filed and flagged fallback.
The submission file is named org-<org>-agent-<agent>-model-<model>.json and holds [{"id", "prediction"}]. Marina files as its own agent, Marina, under the organization h2o.ai (--org, --agent); the model segment names the variant’s model (--model overrides it for a single variant).
Nothing is sent. run prints the file path and the email fields (recipient, subject, dataset commit, model, framework, organization). Submitting is an operator’s act, or an approved connector’s.

Standing answers

FutureX questions stay open for days, so a filed answer is a standing answer per row and variant (out/<sha>/<variant>/standing.json). watch --daily re-forecasts the rows still open:

Cadence: once per UTC day at or after --daily-hour (default 06:00), plus one final run --final-lead-hours (default 4) before the batch’s deadline. Nothing runs after the deadline, and only the batch that is still the latest is re-forecast: a newer batch ends the older one’s re-forecasts.
Deadline: the first Wednesday 16:00 UTC after the batch was published (the Hugging Face commit time of its sha, recorded as publishedAt when the batch is fetched), never later than its last row’s end time. It never depends on when this machine fetched the batch, so a batch first seen after its deadline gets no daily runs. A batch saved before publishedAt was recorded falls back to its first fetch time (benchmarks/futurex/schedule.ts).
Rows: the first run of a new batch forecasts every row. A daily run re-forecasts open rows only: with --daily-rows all (the default) every open row; with --daily-rows stale only open rows whose standing answer (per variant) is missing or a fallback, or whose end time is within --near-end-hours (default 48) of now. The final run re-forecasts every open row.
Schedule: each run is written to out/<sha>/schedule.json when it starts (status: "running", with the planned rows per variant) and marked done or failed when it ends. A check while a run is in progress starts nothing. A run that never recorded its end is treated as in progress for --in-progress-hours (default 6); after that it is abandoned and counts as that day’s run.
Revision rule (src/forecast/revision.ts): a new answer replaces the standing one only on a material change:
- its confidence rises by at least 0.1; or
- its evidence is materially new (word overlap below 0.6) at about the same confidence.
An unchanged answer (ignoring case, spacing and list order), a number within 2 %, a fallback, or a different answer with nothing behind it keeps the standing answer.
Ledger: every decision, kept or revised, with its reason, is appended to revisions.jsonl. A run that revises something writes a new file, which is a new external_submissions row, and prints the email fields again. A run that revises nothing reproduces the same file, which is already recorded, and prints nothing to send.

Answering late and scheduling

watch --answer late does not forecast a batch when it arrives. Each open row is answered once, inside --final-lead-hours of its own cutoff. The cutoff is the batch deadline, or the row’s end time when that comes first. So the answer sees as much news as possible and costs one forecast per row (lateRows in benchmarks/futurex/schedule.ts). A row whose answer is missing or a fallback is retried on the next check inside its window. Nothing runs at or after the deadline, --daily is ignored, and the in-progress rule and schedule.json work as above. watch itself still defaults to --answer early.

Nothing schedules the watcher for you. bun run futurex schedule prints a systemd user service and an hourly timer that run watch --once, answering late by default, with a 6 h lead. Add `–out

` to write the two files instead of printing them. It installs nothing, enables nothing and never calls `systemctl`. The operator decides whether to activate it:

bun run futurex schedule --run earned --dir ~/futurex-data --env-file .env --scope-cap 25 --out /tmp/units
cp /tmp/units/marina-futurex-late.* ~/.config/systemd/user/ && systemctl --user daemon-reload
systemctl --user enable --now marina-futurex-late.timer     # only when you mean it

Options: --answer early|late, --final-lead-hours N (default 6 here), --run <variants> (default earned), --variants <file.json>, --dir, --learn, --repo (working directory, default the current one), --bun (default the running Bun), --env-file, --db-path (sets DB_PATH) and --name (unit name, default marina-futurex-late). Every model call is charged to the unit’s own spend scope (MARINA_SPEND_SCOPE=<name>), and --scope-cap (USD, default 25) must be above 0, because an uncapped timer is refused. The timer is Persistent=true, so a check missed while the machine slept runs on wake.

--learn (or bun run futurex learn) hands resolved weeks to the outcome-lesson loop on every watch tick, whether or not a re-forecast is due. Each standing answer whose row has a ground truth in the past dataset is scored and recorded once per variant and row (learned.json), so a repeated pass adds nothing; the past dataset is re-read only when its commit changed. The question, truth and answer are seen only by the lesson writer as private context, never stored. A row’s outcome counts as known one day after its end time (SETTLEMENT_MARGIN_MS in benchmarks/futurex/lessons.ts), never at the end time itself: the rows of a week share one end time, and a replay’s cutoff is that end time, so a sibling’s lesson can never reach it.

Records

Each submission is appended to external_submissions (migration 152): batch sha, variant, identity, file name and hash, rows answered, cost. The same file is recorded once.
A backtest — and a live batch once it resolves — is a scored run in the benchmark ledger, so benchmark compare, frontier, leaderboard and replicates rank it with every other run Marina has made.

Variants

Built-ins are:

earned (the live default for run, watch --run earned and schedule): no fixed configuration. Each row goes through Marina’s forecast surface (typedForecastFor) on board futurex: the operator’s env, then the earned forecast-config:futurex slot, then the family slot, then the installation’s built-in (resolveDefault), plus card routing and the lesson pool read as a live run. FutureX’s own pages stay barred from research. The run log prints where each field came from. A slot changes only through earned promotion (fileSlotPromotion, e.g. rescore-selection.ts --file).
cheap: one strong low-cost model, three runs. The backtest default: backtests stay fixed controls and never take earned.
verify: cheap runs with every draft checked by a different-vendor verifier.
frontier: three vendors’ current models, three runs, a frontier critic.
frontier-2607: a frontier mix released before the latest resolved rows, so it can be backtested cleanly.
crew: a Marina crew as the analyst, via marina:<crew>.

Model ids are pinned, never floating aliases. --variants <file.json> supplies others: an array of { label, model, analysts, planner?, critic?, verifier?, verify?, runs?, researchRounds?, critique?, selection?, budgetMs?, groundingFollowUp?, formatLine? }. selection: "confidence" takes each row’s most self-confident run (see Forecasting). budgetMs is a per-row time budget: the answer is combined from what has finished, and is labelled budget_forced in the ledger. groundingFollowUp bounds the forecaster’s follow-up when no run is grounded in verified evidence. With "all" (the default) one more research round runs, and every run is redone if that round verified something new. With "one" a single fresh run reads the fuller dossier and replaces the least-grounded run before the runs combine. With "off" the round is skipped. When the trigger fires, the answer records the mode, whether the round ran and how many runs were redone in its groundingFollowUp field.

bun benchmarks/futurex/rescore-selection.ts --dataset <past batch json> --runs <answers.json>,… re-scores saved runs under each selection mode, with no model calls. Rows split into selection and holdout halves by a stable hash. It reports whether confidence clears the promotion margin over agreement on the holdout half; pass --tried with the number of candidates already examined. Its own verdict is advisory. To change the default, add --file --variant <name> (the configuration the saved answers ran with; --db defaults to DB_PATH). This records one ledger run per answers file per mode and files the challenger through the one promotion path on forecast-config:futurex, the slot live FutureX reads. An empty slot is first seeded with the incumbent, so a challenger never takes an empty slot uncontested. The printed verdict is the promotion path’s: replicate minimum, holdout interval, fishing margin, and an append-only history row.

Live runs recall from the one judged lesson pool (forecast and arena domains, the same forecastLessonsFor every forecasting surface uses; MARINA_LESSONS on / observe / off); --lessons off disables that.

Lessons kept by the earlier, unjudged store (subject forecast-lesson in the forecast-lessons space of a Forecaster world account) are copied into the pool once — by futurex run and futurex learn, or explicitly with bun run futurex migrate-lessons — as unverified lessons with their original resolution time and a provenance record (store, account, space, id, version). Each copy is keyed by the record’s id and version, so a re-run never writes it twice. The originals are never changed; when every record has been copied and the account was created by the script, it is renamed to marina:legacy-forecast-lessons, a server-owned name no login can produce.

Clean backtests

bun run futurex backtest --clean measures skill on resolved rows without letting outcomes in. Each of the three ways an outcome can leak has a guard (benchmarks/futurex/clean.ts):

Model weights. Only rows whose cutoff (the end time minus --horizon-days, default 7) is more than 3 days (KNOWLEDGE_MARGIN_DAYS) after every model’s public release are used — by default, rows that end more than 10 days after it. This is the one rule configuration selection uses too (benchmarks/forecasting/knowledge.ts: one release table, one margin, floating aliases refused everywhere). Release dates come from the provider catalogue (MODEL_RELEASES) and are an upper bound on knowledge cutoffs. A model with no known release, a floating alias, or a crew (whose agents have their own tools) is refused unless --after <YYYY-MM-DD> is given. Every variant in one invocation runs on the same rows.
Retrieval. --isolation date-filtered | post-filtered | closed-book (with --retriever <spec> for the first two). An unfiltered engine is never used.
Memory. --lessons on writes a lesson after each scored row and recalls only lessons visible at each forecast’s cutoff. Rows run in order of resolution. Each run recalls only its own lessons, so --lessons on against off is an honest ablation, and every lesson is also kept in the shared space for live runs.

Leak audit. After scoring, each row is audited. A row is suspicious if its reasoning names a day after the event, it quotes the exact numeric outcome, or its kept evidence includes a page published after the cutoff. An evidence line dated on or before the cutoff counts as history and schedule, so past-tense wording and future dates in it are not flags. Undated evidence is suspicious if it names a day after the cutoff or reports a result. Headline scores are given with suspicious rows in and out.

Upper bracket. --allow-contaminated --isolation contaminated runs an unfiltered engine on past cutoffs. It can see outcomes, so it is filed under futurex-past-contaminated as an upper bound, never as a clean score. Between it and closed-book lies the range a live run can fall in.

Judged scoring. --judge <model> also grades string and list answers with a model judge, as the official weekly scoring does (mechanical string matching under-credits paraphrases). Options and numbers stay mechanical, and both overalls are reported.

Results. Each run gets the overall and per-level scores with a bootstrap interval, plus a score per batch week compared with --reference <file.json> (week → top, median, h2o, entered by hand; the website is never scraped). --replicates N repeats a run. Every run is filed into the benchmark ledger under a replicate group, futurex-clean:<variant>:<isolation>:lessons-<on|off>. A run in which more than MARINA_BENCHMARK_MAX_FALLBACK_RATE (default 25 %) of the rows got a fallback instead of an answer is recorded invalid and drops out of pooling and comparison; retire or restore a run by hand with benchmark invalidate|revalidate <run> reason:<text> or bun run benchmark:import --invalidate <run> --reason "<why>".

--first-replicate N numbers new replicates from N, so a later run adds replicates without reusing an earlier run’s name or lesson space.

Spend cap. Every futurex command that calls a model records its spend in the world database’s daily ledger (DB_PATH), shared with the server and every other run on that database, so an hourly timer does not get a fresh cap each run. Give each job its own budget inside the world cap with MARINA_SPEND_SCOPE=<name> and MARINA_SPEND_SCOPE_CAP_USD=<usd>; a run is refused at either cap. A run stops starting new rows when its spend plus a reserve (1.5 × concurrency × the average row cost, at least $2) would reach the tighter of the two caps. A run stopped this way is not filed. Rows that fall back because the cap or a provider refused them write no lessons.

Partial runs and --resume. Every finished row is appended to rows.jsonl in the run’s output directory as soon as it lands. A run stopped by a cap keeps those rows: a clean backtest writes partial.json beside them (reason, rows finished, cost) and files nothing to the ledger. Rerun the same command with --resume to forecast only the rows still missing; fallback rows run again. A journal written under another configuration (variant, rows, horizon, batch, isolation, retriever, lessons, replicate) is refused; run without --resume to start over. metaculus select and forecastbench select keep the same journals per candidate and replicate (<selection>-runs/); a resumed selection files each run once and counts only new spend against the selection budget.

Backtests without `--clean` are smoke tests

The resolved dataset’s outcomes are public. A backtest moves each cutoff --horizon-days before the question’s end time (default 7) and searches with date filters where the engine supports them, but web search without a date filter can still surface the answer. Treat a backtest as a check that the pipeline works end to end, never as an estimate of live skill. The local scorer (benchmarks/futurex/score.ts) follows the published metric definitions — exact match or F1 for options, a squared error scaled by 5 % of the true value for numbers, exact or partial overlap for lists, level weights 10/20/30/40 % — and matches strings mechanically, where the official scoring uses a model judge.