Prophet Arena

Prophet Arena scores forecasting agents live on real prediction-market events. Its Python SDK (prophet-arena, MIT) is used to register and run agents. benchmarks/prophet/marina_agent.py is a thin agent for that SDK. It sends each event to a Marina server’s POST /v1/forecast as one typed question, and returns the answer’s per-outcome probabilities.

How events are asked

Event	Asked as	Returned
mutually exclusive outcomes	a choice with probabilities	probabilities that sum to 1
otherwise	a multi-select with each outcome’s own probability	marginals, never renormalized
a single outcome	a yes/no choice	the probability of yes
more than 64 outcomes	balanced parts of at most 64 outcomes, one question each	the parts’ probabilities (an exclusive event’s are normalized to sum to 1)

The question carries this context: the event’s rules, the crowd prices, and the event’s frozen context sources.

When an event’s window has already closed (a dataset replay with prophet eval), the evidence is frozen at the window’s close. A live window is forecast as of now.

The rationale carries the runs’ reasons, plus the formation and models that produced it.

The agent registers on the agentic track, because Marina does its own research. The track cannot be changed after registration.

Your identity, not a default. No entrant name ships. The agent registers under the name you choose: set PROPHET_AGENT_NAME (the permanent slug) and, optionally, PROPHET_AGENT_DISPLAY_NAME and PROPHET_AGENT_DESCRIPTION, or replace the placeholders in benchmarks/prophet/arena.toml (your-entrant, Your Org Marina) — the environment wins. With the SDK installed, the agent refuses to load while the name is unset or still a placeholder.

Credentials

Variable	Where it comes from
ARENA_API_KEY	A pa_live_… key created at prophetarena.co/profile/api-keys (or stored with prophet login).
MARINA_URL	The Marina server (default http://localhost:3300).
MARINA_API_KEY	One of that server’s MODEL_API_KEYS.
PROPHET_AGENT_NAME, PROPHET_AGENT_DISPLAY_NAME	Your entrant slug and display name (no default; see above).

The server’s own environment chooses the models and formation, never the caller:

• MARINA_FORECAST_ANALYSTS, _PLANNER, _CRITIC, _VERIFY, _RUNS, _LOOKUPS;
• MARINA_FORECAST_FORMATION (ensemble, delphi or tournament).

To use a configuration chosen by backtest, run bun run forecastbench select --sources kalshi,polymarket (Prophet events are Kalshi markets), then set those variables to the pick’s models and formation. See ForecastBench.

Running

Install the Prophet client and run the offline mapping checks:

pip install prophet-arena
cd benchmarks/prophet
python3 -m unittest test_marina_agent.py

Score locally, at leaderboard grade:

prophet eval marina_agent.py --dataset prophet-arena-subset-100

See what it would submit:

prophet run marina_agent.py --dry-run --once

Run live (an operator act, under your name):

prophet submit your-entrant && prophet run marina_agent.py

A typed forecast takes minutes. The agent waits up to 15 minutes per event.

Write-back: filing and resolving answers

Prophet answers teach Marina only once they are filed on Marina’s outcome path and resolved there. The agent appends each answer it gets to a journal: data/answers.jsonl under its working directory, or the path in PROPHET_ANSWER_JOURNAL (off disables it). A wide event asked in parts gets one line per part. The journal is local, and the agent never fails a forecast because of it.

File the journal’s answers on the world database (each answer once):

bun run benchmarks/prophet/writeback.ts file --journal benchmarks/prophet/data/answers.jsonl --db marina.db

When events resolve, score every filed answer of each resolved event:

bun run benchmarks/prophet/writeback.ts resolve --resolutions resolved.json --db marina.db

--db defaults to DB_PATH, then marina.db. Point it at the database of the server the agent calls.

• Filing. Each answer is filed under the owner marina:prophet, with source prophet-arena and board id prophet:<event> (or prophet:<event>#<part>). A live window files with eval mode live: the answer teaches lessons and writes forecast history, but it never earns standing. A dataset replay (a past window, sent as measure) files with eval mode measure: it teaches other boards but never Prophet’s own measurements, and it never writes history.
• Resolving. resolved.json is a JSON array, or JSONL, of { "event": "<id>", "outcomes": ["<names that happened>"], "resolvedAt": "<iso>" }. A single "outcome": "<name>" also works, and resolvedAt defaults to now. Each filed answer is scored through the one outcome path, which records one outcome. The server’s delivery pass then turns that outcome into a lesson and a history record. A yes/no event resolves on whether its outcome happened. A part with none of its outcomes resolved stays open, because a multi-select needs at least one true option.