Operator runbook

Start with readiness, then readiness providers if model-backed work is failing. In the dashboard, open the header Health badge or Admin → Ops. Record the running revision, affected capability, timestamps and request/trace IDs before changing settings. See deployment for installation and recovery for backups.

Readiness and health

Signal	Meaning	Action
ok / green	The capability’s configured checks pass. Provider configuration alone does not prove upstream connectivity.	Use readiness providers for a live provider probe.
degraded / amber	The capability is partially available or has an unmet dependency.	Follow that check’s remediation; inspect provider errors, agent status and spending limits.
off / dim	A capability is disabled or not configured. This can be intentional in minimal deployments.	Enable it only if the deployment needs it.
Red aggregate badge	No checks are healthy; inspect the individual checks.	Do not infer data corruption from the color alone.
/health returns 503 while stopping	Admission is draining.	Let outstanding work finish before restarting.

GET /health is a process/world liveness check, not a test of every provider or query. GET /api/readiness provides the detailed capability report through the dashboard’s normal authorization boundary. Inspect error rates and latency alongside both probes.

Tick and command latency

Slow room onTick(s): room=123ms identifies a room’s synchronous handler exceeding 100 ms. Tick budget exceeded means the overall synchronous room phase crossed 200 ms and deferred remaining rooms. An individual blocking handler cannot be preempted by this cooperative budget. Async handlers are tracked separately, never overlap themselves, and are drained during shutdown.

1. Correlate the named room with recent room code changes and tick-job status.
2. Remove unbounded synchronous loops and full-table work from onTick; bound each unit of work. Move slow I/O out of the synchronous phase and use supported sandboxes.
3. Validate the repaired room with build validate <room> and apply it with build reload <room> under the existing permissions. Observe several ticks.
4. If the event loop is unresponsive, use the process supervisor’s graceful stop; restarting unchanged blocking code only repeats the failure.

MCP serializes actions within each session. A slow action delays that session’s next action; separate participants can interleave. Cancellation before execution removes the action from execution, while a started mutation retains its FIFO slot until it finishes. A disconnected client must check the operation receipt/state before retrying a write. A timeout is not proof that a write did not commit.

Bounded admission and context reuse

World input admission counts waiting commands and commands moved into per-entity promise chains against the 5,000-command limit. A rejected command emits command_overloaded, retryable: true, executed: false; it is never silently dropped. MCP additionally admits at most 256 active/queued calls per engine and 16 per session. A queued call older than 30 seconds is rejected before starting. mcp_overloaded includes retryAfterMs: 1000 and executed: false. Retry with jitter; do not retry an already-started mutation solely because the client disconnected. These resource bounds apply to local deployments too, independently of token rate-limit bypass.

The MCP listener’s GET /health exposes aggregate admission (pending, high-water, rejected, expired, limits), world commands and contextCache counters. No identities, queries or memory contents are exposed. Growing rejection counts mean clients should reduce concurrency; high pending counts that never drain indicate a stuck handler. Admission does not interrupt a running SQLite statement or a started write.

Unified context uses at most 128 entries per database and a maximum five-second age. Every reuse checks connection-local revisions for canonical memory, identity and ranking changes plus SQLite data_version for external commits. Entries also expire at the next credential, evidence or delegation time boundary. Transactions and retrieval failures are not cached; concurrent withdrawals trigger a fresh retrieval. If the evidence keeps changing through three attempts, optional context fails explicitly and the preceding tool result remains intact. Provenance and recall-credit policy are unchanged. High mutation rates can legitimately produce few cache hits.

Dashboard command discovery coalesces concurrent requests on the same resident, connection, room, rank and mode; focus refreshes debounce for 150 ms. Inline assistance reuses validated public metadata locally for up to five seconds. Observed registry revisions, room/rank/modal changes, socket changes and logout invalidate reuse. Older servers and changes not yet observed fall back to the five-second bound. Expired entries revalidate with capability_key so unchanged catalogs are not retransmitted. Execution always checks current server permissions. Memory previews and writes are never coalesced in the browser, and changing identity invalidates pending responses.

To measure a deployment-shaped local workload without model calls:

bun run qualify:participation:load --directory /tmp/marina-participation-load --participants 16 --records 64 --operations 30
bun run qualify:memory:load --directory /tmp/marina-memory-load --tenants 16 --operations 40

Use a new temporary directory for each run. The first exercises real MCP HTTP clients, FULL SQLite durability, automatic private context, bounded overflow and recovery. It intentionally bypasses token rate throttles to measure admission capacity. The second exercises memory HTTP writes, throttling, cancellation and same-key retry. Both emit latency percentiles and fail on correctness violations. These are reproducible local qualifications, not universal throughput or power-loss guarantees. Measure on representative storage and corpus sizes before choosing production concurrency.

Coding workers and growing prompts

Check the last tool result and verification receipt before increasing token limits. A failed check includes a bounded tail of command output and the full output artifact ID; a completed background receipt includes that result. not_run means the environment or test selection prevented checks from running, not that the code passed. For a recipe that truly uses only local source and runtime builtins, dependencies:none skips preparation explicitly; other recipes need their dependencies in the execution environment.

After changing source or tests, rerun candidate verification before submitting. Stale or failed-check feedback includes a retry command with the previous receipt’s dependency, scope, typecheck and time-budget settings. Review those settings before running it; the hint never executes a command or installs dependencies automatically.

Compare per-turn input usage (including cache reads/writes) with the effective context window. Lifetime input totals add every request and do not represent the current prompt size. A harness request-byte ceiling is a separate bound, not evidence of provider context exhaustion. Silent turns indicate successful model responses without tool calls; transport errors are counted separately. Neither signal alone proves the model needs a larger output allowance.

Narrow source searches with path: and prefer surgical edits. Internal workers retain one full copy of unchanged repository instruction excerpts, restore it if compaction removes it, and archive originals before reducing context. Provider usage anchors are recalculated after transcript changes. Keep budget ceilings in place while diagnosing loops; tests and independent candidate review remain the completion evidence.

An unpriced provider/model can report zero recorded dollars despite consuming tokens. Keep call/token limits as well as spend caps; inspect the resolved model’s pricing and provider usage before treating the dollar counter as complete. The live qualification tool separately reserves a conservative upper bound before each approved upstream attempt.

Published panel freshness and actions

GET /api/panel-resources lists the deployed source adapters. Catalog sources call the original read APIs with the viewer’s credential, including their existing rate limits and permissions. Check those endpoint responses when a binding is unavailable. Personal Coding desks have no Canvas node: /panel desk or Work opens a view; /panel publish explicitly creates a shared one. The target repository remains a property of the canonical coding session, never of the layout.

Published panels reuse the dashboard and Canvas WebSockets plus existing authorized resource reads. If a Coding desk or participant resource appears stale, check that the resident is signed in, the socket has reconnected with that credential, and the corresponding resource request succeeds. Content-free change notices trigger rereads; reconnect also refreshes the snapshot. Visible views have a five-second polling fallback. Hidden views suspend reads until reopened, so lack of hidden polling is expected. Repeated dashboard views share reads while retaining separate drafts.

Access failures hide cached private content and show an error. Fix the credential or source access; opening a public panel does not authorize private resources. A missing or deleted publication is not an empty successful result. Browser layout presets restore targets and geometry, not unsent drafts.

Operational buttons show a review before submission. After a command timeout or lost response, inspect canonical activity before sending again: a started command may have committed. Participant message retries preserve the captured delivery ID and payload; a delivery receipt is not proof of completed work. Closing a panel only closes its view. See Published panels for authoring limits and Participant routing for delivery semantics.

SQLite pressure and WAL

Marina uses one writer connection and one read-only connection per MarinaDB. It does not have a general connection pool. More writers would still contend on SQLite’s single-writer lock; do not run multiple world processes against one database. The entry point’s database lease enforces that ownership.

Current connection settings are explicit in src/persistence/database.ts:

Setting	Current value / behavior
Journal	WAL
Durability	World and standalone service default to FULL; direct new MarinaDB() defaults to NORMAL for explicit callers/tests
Writer busy timeout	5,000 ms
Page cache	64,000 KiB configured on each connection
Memory mapping	Up to 256 MiB on each connection
Temporary tables	Memory on the writer
Checkpoint	checkpoint() uses PASSIVE; startup and close use TRUNCATE
Auto-checkpoint	SQLite default; Marina does not override wal_autocheckpoint

These are connection settings, not environment knobs. Running a PRAGMA in a separate CLI does not retune the server connection. See SQLite’s WAL documentation for checkpoints and long-lived readers. Use local persistent storage; WAL’s shared-memory coordination is not a multi-host network-filesystem architecture.

When latency or WAL size grows, check disk space/I/O, long read transactions, SQLITE_BUSY, active import/index jobs, and request volume. Reduce admission/concurrent optional jobs first; pause the affected agents or clients while keeping inspection available. Use narrow retrieval queries and bounded context budgets. Repeated overload requires measured capacity limits or separate worlds/services, not extra connections to the same writer. The service’s storage quotas fail writes explicitly; it does not silently drop writes or switch to an in-memory store.

Never delete live WAL/SHM files. Use the verified snapshot/restore workflow, not a raw copy of only the .db file. FULL durability remains the production default; switching to NORMAL trades crash durability for write latency and is not an automatic overload response. After disk-full or I/O failures, resolve storage pressure and inspect the failed receipt before retrying; use stable idempotency keys where the API supports them.

Compare checkpoint thresholds on representative storage

SQLite already checkpoints automatically, normally at 1,000 WAL pages. Smaller thresholds trade more frequent checkpoint work for a smaller WAL; larger ones can improve write throughput while increasing read cost and disk use. A long-lived reader can prevent a checkpoint from finishing at any threshold. See SQLite checkpoint behavior and the connection-local wal_autocheckpoint setting.

The participation qualifier accepts --wal-pages only for its disposable database. Run the same workload on the target storage with fresh directories:

bun run qualify:participation:load --directory /tmp/marina-wal-100 --wal-pages 100 --participants 16 --records 64 --operations 30
bun run qualify:participation:load --directory /tmp/marina-wal-1000 --wal-pages 1000 --participants 16 --records 64 --operations 30
bun run qualify:participation:load --directory /tmp/marina-wal-4000 --wal-pages 4000 --participants 16 --records 64 --operations 30

Each report records SQLite version, journal/durability/page settings, write and MCP latency percentiles, WAL bytes before a passive checkpoint, and its busy, log, checkpointed result. A gap between logged and checkpointed pages after load calls for investigating reader lifetimes. WAL file size alone is not an uncheckpointed-page count: SQLite can reuse the file. Repeat runs and compare tail latency, not just averages. All variants retain FULL durability and check admission recovery and memory isolation.

Keep the production default unless repeated representative runs justify a reviewed change on the owning writer connection in database.ts. A CLI PRAGMA on another connection does not tune the server. Change one setting at a time, repeat recovery checks, record the old value, and revert if latency or WAL pressure worsens. This is measurement guidance, not an automatic tuning algorithm or a power-loss simulation.

Reload or restart

Change / symptom	Preferred action
Supported room or dynamic-command source update	Validate, then the specific build ... reload command
Role definition update	role reload <name> under existing gates and self-modification restrictions
Binary, dependency, world module, plugin list, ports or environment change	Graceful restart with the updated configuration
Optional provider outage	Repair provider configuration or wait for recovery; inspect readiness before restarting
Database errors or unresponsive event loop	Diagnose storage/blocked work first; graceful restart after remediation if needed

SIGTERM/SIGINT stops admission, drains requests and command work, flushes agent state, then closes persistence. Allow at least the server’s 30-second shutdown deadline in the supervisor. Do not turn shutdown errors into successful health responses.

DB log failed: Cannot use a closed database means a producer outlived persistence. In tests, await adapter shutdown and command drains before db.close(). In production, retain the surrounding shutdown logs and treat recurrence as a lifecycle defect rather than hiding the warning. The shutdown integration test verifies persisted work and rejects this warning during a real SIGTERM sequence.