BrowseComp-Plus with Marina

BrowseComp-Plus (MIT, paper) is a deep-research benchmark. It has 830 hard questions over a fixed corpus of about 100,000 web documents, with no live web. An LLM judge grades each answer, and human-verified evidence documents allow retrieval to be measured too.

Marina runs it with two general pieces and one thin adapter:

• Local corpora: an offline BM25 index that any agent or tool can search. See Search → Local corpora.
• The model API: every model call goes through a Marina /v1, so passthru, marina/verify: and marina:<crew> are all targets, with spend, traces and the daily cap applied.
• The adapter: benchmarks/browsecomp-plus/ plus bun run browsecomp-plus. It runs the official agent prompt and tools, judges with the official grader, writes the official run and summary files, and files ledger runs.

Data

The dataset is published on Hugging Face as Tevatron/browsecomp-plus (queries) and Tevatron/browsecomp-plus-corpus (documents), and the qrels are in the GitHub repository. Keep everything outside the repository, on a disk with several GB free:

1. Download the parquet shards of both datasets (any Hugging Face client, or plain HTTPS from the dataset’s resolve/main/data/ URLs) into $DATA/hf/corpus/ and $DATA/hf/queries/.
2. Download the GitHub repository’s topics-qrels/ directory.
3. Export both datasets to JSONL (the exporter needs pyarrow):

DATA=~/.local/share/marina-browsecomp
python3 -m venv $DATA/venv && $DATA/venv/bin/pip install pyarrow
$DATA/venv/bin/python benchmarks/browsecomp-plus/export.py corpus  $DATA/hf/corpus  $DATA/data/corpus.jsonl
$DATA/venv/bin/python benchmarks/browsecomp-plus/export.py queries $DATA/hf/queries $DATA/data/queries.jsonl

The queries are published obfuscated. export.py decrypts them with the benchmark’s own scheme, and only into your private data directory. Never commit or publish the decrypted queries or answers. The benchmark asks that they not appear as plain text online.

Build the index once. It takes about 20 minutes and several GB, under MARINA_CORPUS_DIR (default ~/.local/share/marina/corpora):

SQLITE_TMPDIR=$DATA/tmp bun run corpus build browsecomp-plus $DATA/data/corpus.jsonl \
  --source "Tevatron/browsecomp-plus-corpus (MIT)"
bun run corpus search browsecomp-plus "a query"      # sanity check

SQLITE_TMPDIR keeps SQLite’s sort files off a small /tmp.

Run

Start a Marina server with a MODEL_API_KEYS entry. Then:

export MARINA_BENCH_API_KEY=<that key>
bun run browsecomp-plus run --queries $DATA/data/queries.jsonl \
  --qrels $DATA/topics-qrels/qrel_evidence.txt \
  --model openrouter/openai/gpt-6-luna --limit 100 --seed 1 --replicates 2 \
  --out $DATA/runs/luna --file-to http://localhost:3300

--model is any id the server serves:

Arm	--model	What answers
Single model	openrouter/<vendor>/<model>	One model in the official tool loop (search and get_document).
Verification formation	marina/verify:<proposer>[+<checker>]	Each step is drafted, reviewed by a checker and revised once on a flag (see τ²-bench).
Crew	marina:<crew>	A Marina crew answers in text. Its agents search the same corpus in-world with web search engines:corpus:browsecomp-plus <q> and web fetch corpus://browsecomp-plus/<docid>.

Spend cap and resuming

--max-usd <N> is a hard spend stop for the whole invocation: every replicate, every agent and the judge.

• Each call’s cost comes from Marina’s x-marina-cost-usd header. The running total is checked before every call, so the overshoot is at most the calls already in flight.
• A 429 spend_cap_reached from the server’s own daily cap (MARINA_DAILY_SPEND_CAP_USD) stops the run the same way, with or without --max-usd.
• A query the stop cuts off is not run. No run or eval file is written for it, it is never judged and it is never counted wrong. A turn-cap incomplete is different: it is a real outcome and counts as wrong, as in the official evaluator.
• A stopped replicate is reported incomplete, with the count of queries not run in summary.json (Marina.stopped_by, Marina.not_run). It is not filed to the ledger, and no later replicate starts.

--resume reuses every query already answered and judged in the output directory and runs only the rest. It continues a stopped or interrupted replicate without paying again for finished queries.

• An errored query (transport, HTTP or timeout) or a failed judge call is never reused: it runs again, so an outage can be repaired. An incomplete answer (turn cap, empty reply) is the target’s own outcome and is kept. summary.json counts re-run errors in Marina.retried_errors.
• Each replicate directory records its configuration in config.json: model, formation, lead model and turns, judge model, corpus, --k, snippet and document sizes, turn and token caps, seed, offset, limit and a hash of the sampled query ids. --resume refuses a directory recorded under a different configuration, naming the fields. A directory from before this record is adopted with a warning.
• A replicate already filed to the ledger (filed.json) is not run or filed again on resume. New replicates file into the group the arm started with (group.json); a contradicting --group is refused.

Price an arm on hard queries before setting its cap. Easy queries answer in a few turns, and hard ones run to the turn cap with a growing context, so a cost per query measured on easy ones can be several times too low.

Research formations

--formation turns one agent into a research team. Every member uses the same tools through the same Marina server:

--formation	Team
single (default)	The official loop: one agent.
ensemble:N	N independent researchers. A lead then weighs their reports and answers.
mapreduce:N	The lead plans N complementary angles over the question’s clues (map). N researchers pursue one angle each, and the lead reduces their reports.
sharding:N	N researchers, each searching one hash shard of the corpus, so together they read N times deeper into the ranking. get_document reads any document. A lead answers.
blackboard:NxR	N researchers over R rounds. Between rounds, each sees the team’s posted answers and evidence. A lead answers.
read-swarm	Marina’s read swarm (src/retrieval/read-swarm.ts). The question is split into clues; each clue is searched; reader models read the best documents in full (in chunks, not snippets) and report the clues each satisfies with verbatim quotes, which are checked mechanically. One lead gets the ranked candidate table and keeps researching: its search results are read by the same readers before it sees them, and get_document stays the official tool.

The lead.

• It answers in the official format.
• It may use the tools to check a report before answering (--lead-turns, default 12).
• --lead-model puts a different model in the lead.

The read swarm.

• --reader-model (default --model) reads; --lead-model (default --model) answers. One model can play both (the single-LLM tier).
• --swarm-docs N is the per-query read budget in documents (default 60; the character budget is 2,000,000). Once it is spent, search returns snippets again.
• --swarm-open N is the number of documents read before the lead’s first turn (default 16): each clue’s two best first, then the reranked order.
• --swarm-no-decompose reads against the question alone (no clue split).
• The lead runs --max-turns; combine with --final-answer so the cap ends in an answer.
• The run record adds a swarm_read item per document read (docid, clues, counts — no text) and swarm_stats in the metadata. Documents the readers read count toward recall only when a search showed them to the lead.

Composing with verification. --model is the researchers’ model. Making it a marina/verify:<proposer>[+<checker>] id puts each researcher’s steps under review.

Counting. A team’s run file holds every member’s steps, each tagged with an agent field:

• Recall counts every document any member’s search returned.
• Search calls count the whole team.
• Cost is the team’s.

Searches run in parallel. Searches run in a pool of --workers processes (default 6), each with its own read-only index handle, so one agent’s search never waits on another’s.

The tool loop

The tool loop follows the official harness:

• search returns the top --k hits (default 5), each with its docid, BM25 score and the document’s first --snippet-chars characters (default 2000, about the official 512-token snippet).
• get_document returns a document, capped at --doc-chars (default 20,000).
• A run that reaches --max-turns model turns (default 30) without a text answer is incomplete, and counts as wrong.
• --final-answer turns on budget-terminal answering, a Marina addition that is off by default so the official protocol stays comparable:
  • From about 75 % of --max-turns, the latest tool result carries a note saying how many turns are left and asking the agent to converge.
  • At the cap, the agent gets one more call with tool_choice: "none", asking for its final answer from what it has found.
  • A forced answer is labelled: budget_forced in the run’s metadata and in the ledger item, and budget_forced (a count) in summary.json.
  • In a formation, a lead that still has no answer falls back to the researchers’ most-agreed report (source: member-plurality).
  • The arm is its own configuration. Its label ends in +final-answer and its ledger target carries finalAnswer, so its runs are never pooled with official-protocol runs.
• Against a crew (marina:<crew>), each request carries the client’s timeout as x-marina-deadline-ms. The crew answers with its best draft before that deadline, and the answer is labelled x-marina-budget-forced.

Marina harness options. Off by default, so a plain run is the official harness. Each one used is recorded in the run’s metadata; report such a run as a Marina harness, not the official one.

• --snippet matched: each hit shows the window of the document that best matches the query, not its opening characters.
• --doc-paging: get_document takes an offset and returns total_chars / next_offset, so the agent can read past --doc-chars.
• --search-paging: search takes a page, served from the cached ranking.
• --first-move <model>: before the first turn, the question is split into clues by <model>, each clue is searched 50 deep, the rankings are fused and reranked by the same model, and the top --k documents are shown as opening context (recorded as a first_move_search call, so recall counts them). One model does the decomposition and the reranking, so a single-LLM setup can use it. Each clue search uses whatever ranking the corpus search uses (BM25, rescored BM25 or hybrid).
• --first-move-judge: the configured decision backend (MARINA_DECISIONS, e.g. the Jev family) scores each fused document’s relevance and replaces the model’s listwise rerank. Only the order is used, so an uncalibrated chat-classifier backend is fine. If the backend fails, the fused order stands. The run’s metadata records the backend as first_move_judge.
• MARINA_CORPUS_BM25_K1 / MARINA_CORPUS_BM25_B: rescore the corpus ranking with these BM25 parameters (see Search); the run’s metadata records them. Unlike in-world corpus search, which rescores by default, the harness keeps FTS5’s plain BM25 unless both are set.
• MARINA_CORPUS_EMBEDDINGS (+ vectors in the corpus): hybrid BM25 + dense search (see Search); the run’s metadata records the model and weight as hybrid. The official Qwen3-Embedding document vectors can be imported with bun run corpus vectors import instead of embedding the corpus; queries are then embedded by the configured endpoint with the official query instruction (--query-prefix).

The judge. It is the official grader prompt, with the official sampling: temperature 0.7, top-p 0.8, top-k 20, 4,096 tokens, thinking off. It runs through the same Marina server under --judge-model (default openrouter/qwen/qwen3-32b, the official Qwen3-32B). Every grader call sends x-marina-stack: none and none of the agent’s opt-in headers, so the passthru instrument stack (which an identified caller gets by default) never changes the grader prompt in any arm. As in the official evaluator, an incomplete run is never judged.

Metrics. These match evaluate_run.py:

• Accuracy: judged correct over all queries.
• Recall: the share of each query’s evidence documents that any search returned, averaged over queries.
• Search calls: the average per query.
• Calibration error: binned at β = 100, computed only with ≥ 100 parsed confidences.

For a crew, the adapter cannot see the agents’ searches, so recall is measured on the docids the answer cites. That is a lower bound, and the run file records it as metadata.recall_source: "cited".

Output

Each replicate writes to <out>/rep<i>/ (or straight to <out> for one replicate):

File	What
runs/run_<query_id>.json	The official run format: query_id, status, tool_call_counts, retrieved_docids, result[]. The official evaluate_run.py accepts this directory as-is.
evals/run_<query_id>_eval.json	The judge’s reply and verdict, citations, retrieval recall, cost and trace ids.
summary.json	The leaderboard fields: LLM, Retriever, Accuracy (%), Recall (%), Search Calls, Calibration Error (%), Link, Evaluation Date and per_query_metrics.
result.json	The ledger result: item ids, outcomes, cost and judge verdicts. It holds no query or answer text.

--file-to <url> files each replicate into that server’s benchmark ledger (POST /v1/benchmarks/runs) under one replicate group (--group, or a generated one). benchmark compare and benchmark replicates then pool the runs. Locally, compare two arms on their common queries:

bun run browsecomp-plus compare $DATA/runs/verify $DATA/runs/luna

This prints pooled accuracy per arm and A − B, with a two-stage bootstrap interval and p, and the range of single-pair McNemar p.

Submitting

Nothing is submitted by the adapter. The leaderboard takes the summary.json fields for all 830 queries, sent to the maintainers by email or through AgentBeats; see the repository README.

Choose the configuration on a held-out split, never on the scored set. --limit N --seed S --offset O takes N queries at offset O of one fixed shuffle, so different offsets give disjoint splits. A typical sequence:

1. Compare candidates at offset 0.
2. Confirm the chosen one at offset N.
3. Only then run all 830.

Disclose which queries were used to select the configuration, and the configuration’s cost per query.

Before submitting:

• Set --llm to the model’s public name.
• Keep the Retriever label honest. Marina’s index is BM25 over SQLite FTS5 (Porter stemming, default k1/b), not the official Pyserini BM25.
• Describe a marina/verify:, formation or crew arm as the system it is: agents with their own control flow, and how many of them.