total-recall

Fast compaction and log mining of agent session rollouts, as an MCP server and a CLI. Reads the session store a CLI coding agent already writes on your machine, and turns it into a queryable long-term memory: compaction summaries through Inception Mercury 2.5, a diffusion LLM running at 1,100 tokens/sec.

Two surfaces, one binary. total-recall mcp is an MCP stdio server, so an agent calls the tools directly. total-recall <subcommand> is a CLI over the same adapters, for scripting and for reading a store without an agent in the loop.

Why

Every CLI coding agent (Claude Code, Codex CLI, OpenCode, Mistral Vibe) hits context limits on long sessions. Their built-in compaction is slow and uses expensive frontier models. Mercury 2.5 does the same job at 1,100 tok/s for $0.04/M input and $0.15/M output (launch pricing). Augment Code moved compaction to Mercury and cut latency 82% (150s to 27s) and costs 90%.

Log mining needs no LLM at all. Every index, profile, extract and search tool works in a vendor-free build with no API key: see Configuration.

Supported harnesses

A session store holds the transcripts. The shadow index root is a disposable folder of full-text indexes this tool builds beside the store, never inside it:

See index_sessions / do_android_dream_of_electric_sheep.

What the tools do

Reading a store is read-only

total-recall opens the store read-only — the OpenCode adapter connects with SQLITE_OPEN_READ_ONLY. It never writes to a session store, and the only files it creates live in the shadow root beside it. she_said_he_said_action pushes its matching into the query engine as a custom scalar function, so the rows stream out in one pass without the database ever being written.

Error contract

A payload that cannot be resolved or read is an error, never an empty session. Every adapter read path returns Result<T, String>: the MCP tool answers with an error result, the CLI prints the reason and exits 2. Damage is visible instead of a payload silently appearing empty. A payload that exists and is genuinely empty is not damage and still reads as an empty session. A JSONL line that fails to parse is skipped with a WARN carrying its line number, so a short read is diagnosable.

list_sessions is the one exception by design: a session whose payload cannot be read is still listed, with counts from whatever could be read, and carries a read_error field so the damage is visible in the index. index_sessions propagates the same error rather than building an empty index.

Where to go next

• Installation — release binary via mise, or build from source.
• MCP setup — register the server with Mistral Vibe or OpenCode.
• Configuration — storage-root overrides, the sandbox guard, API keys, vendor features.
• Troubleshooting — the failure modes worth knowing by heart.
• Operations — the opencode store-vacuum runbook.

Installation

Three routes, in order of how much they assume: a release binary fetched by a version manager, a release binary you download by hand, or a build from source. All three produce the same executable, total-recall, at the archive root.

mise

mise use github:prompt-cult/total-recall

mise selects the archive for your OS and CPU by reading the triple out of the asset name, so the same line installs the right binary on macOS arm64, macOS x86_64, Linux x86_64 and Linux arm64. It also fetches the release’s SHA256SUMS and verifies the download against it.

The older mise use ubi:prompt-cult/total-recall spelling still resolves, but mise has deprecated the ubi backend in favour of the github: backend.

aqua

aqua resolves assets through a registry entry rather than by inspecting release names, so it needs a registry.yaml mapping this project’s assets. The asset layout below is what such an entry targets:

packages:
  - type: github_release
    repo_owner: prompt-cult
    repo_name: total-recall
    asset: 'total-recall_{{trimV .Version}}_{{.Arch}}-{{.OS}}.tar.gz'
    format: tar.gz
    replacements:
      amd64: x86_64
      arm64: aarch64
      darwin: apple-darwin
      linux: unknown-linux-gnu

aqua g -i prompt-cult/total-recall works once that entry is in a registry aqua is configured to read.

Plain download

Prebuilt archives ship on every tagged release. Asset names are total-recall_<version>_<target-triple>.tar.gz — for example total-recall_0.7.0_x86_64-unknown-linux-gnu.tar.gz — and each archive holds one executable, total-recall, at its root.

VERSION=0.7.0
curl -fsSLO "https://github.com/prompt-cult/total-recall/releases/download/v${VERSION}/total-recall_${VERSION}_x86_64-unknown-linux-gnu.tar.gz"
curl -fsSLO "https://github.com/prompt-cult/total-recall/releases/download/v${VERSION}/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS
tar -xzf "total-recall_${VERSION}_x86_64-unknown-linux-gnu.tar.gz"
install -m 0755 total-recall /usr/local/bin/

Substitute the triple for your platform: aarch64-apple-darwin, x86_64-apple-darwin, x86_64-unknown-linux-gnu, or aarch64-unknown-linux-gnu.

Verifying a download

Every release carries a SHA256SUMS over all four archives, one line each, <64 hex digits> then two spaces then the bare asset filename. sha256sum --check SHA256SUMS verifies them all; --ignore-missing verifies only the archives you downloaded. mise reads the same file automatically. The archives are built reproducibly, so a given source tree always yields the same digest.

Build from source

Requires a stable Rust toolchain. The default build compiles both LLM vendors and needs an Inception API key for the LLM-backed tools:

cargo build --release
./target/release/total-recall --version

For a vendor-free build — no API key, no LLM calls, every log-mining tool working — see Configuration:

cargo build --release --no-default-features

API key

The key is only needed by the LLM-backed tools (prior_period_summary / prior-period-summary). Get one from https://platform.inceptionlabs.ai, then either:

• copy .env.template to .env and fill it in — the build-from-source CLI path, since the CLI reads .env relative to its working directory; or
• put the key in the MCP registration snippet’s environment block, which is what an MCP server actually reads: see MCP setup.

A .env file is only loaded by a default build, and only relative to the process’s working directory. A spawned MCP server’s working directory is chosen by the client, not by you, so the environment block is the reliable route for a server.

For --provider mistral, set MISTRAL_API_KEY instead.

Requirements

• Rust toolchain (stable), for building from source
• INCEPTION_API_KEY in the environment or .env, for the LLM-backed tools on a default build
• Optional: MISTRAL_API_KEY, for --provider mistral
• sqlite3 on PATH only for the store-vacuum runbook

Licensed MIT OR Apache-2.0.

MCP setup

The binary runs as an MCP stdio server:

total-recall mcp

One harness per server instance

Each server is bound to exactly ONE harness, set by the installing config via the HARNESS environment variable, or with --harness <name> on the mcp subcommand — the flag wins if both are given:

No tool takes a harness parameter; the harness tool reports the bound value. Register one server per store you want to mine. If the harness is unset or unknown the server refuses to start: it prints the reason to stdout and stderr and exits 2.

Keep the server key as total-recall

The MCP tool names derive from the server key — total-recall_prior_period_summary, total-recall_list_sessions, … — so a renamed server shows up in the agent’s tool surface under the wrong identity.

The environment block

The registration snippet’s environment block carries two entries:

• HARNESS — which store this server is bound to.
• INCEPTION_API_KEY — the Mercury key, for the LLM-backed tools. A spawned server only finds a .env file relative to its working directory, which the MCP client controls, not you. Alternatively export INCEPTION_API_KEY in the shell that launches the client; the server inherits it.

The log-mining tools (list_sessions, profile_session, the three extraction tools, she_said_he_said_action, index_sessions, do_android_dream_of_electric_sheep) need no key. Only prior_period_summary reads it.

Mistral Vibe

~/.vibe/config.toml:

[[mcp_servers]]
name = "total-recall"
transport = "stdio"
command = "/path/to/total-recall"
args = ["mcp"]

[mcp_servers.env]
HARNESS = "vibe"
INCEPTION_API_KEY = "sk_..."

OpenCode

~/.config/opencode/opencode.jsonc:

"mcp": {
  "total-recall": {
    "type": "local",
    "command": ["/path/to/total-recall", "mcp"],
    "environment": {"HARNESS": "opencode", "INCEPTION_API_KEY": "sk_..."}
  }
}

Which vendor the server calls

The MCP server constructs the default provider. --provider is a CLI flag and is not read by the mcp subcommand: a server that needs Mistral is not a supported configuration, and a vendor-free build answers the two LLM-backed tools with an error naming the feature to rebuild with — see Configuration.

Confirming the binding

The harness tool is the cheapest check that a registration is right: it returns the bound harness as JSON. If a tool answers “No sessions found”, the HARNESS value is the first thing to check, then the store root — a root override in the environment is honoured here exactly as it is on the CLI, see Configuration.

Tools

Every MCP tool has a CLI equivalent over the same adapter, except where a section states otherwise — harness (the CLI takes --harness explicitly) and extract_qa_pairs (MCP only) are the exceptions. Parameter surfaces are not duplicated here: the generated schemas are the authority — tools/list for the MCP tools (derived from the source descriptions) and total-recall --help for the CLI (derived from the clap definitions). Two generic contracts cover every tool, including ones added later:

• the scope contract — see The scope contract: the natural call is the cheap call; breadth is explicit, never a default, and every request outside the cheap forms is rejected with those forms named, never clamped, never silently run
• every response that can overflow the caller’s context is flood-capped — see Flood control

The scope contract

Work is scoped before it starts, and the bound lives where the data lives: a window or a directory is pushed into the store query, so a call costs what it asks for, not what the store holds.

Two regimes, one parameter:

• session tools — profile_session, extract_messages, extract_user_messages, extract_by_type, prior_period_summary, todo_history, summary_cache, summarise_from, cache_delete_latest: session_id (partial match; empty = the most recent session). One session is the smallest useful scope, so it is the default.
• window tools — list_sessions, she_said_he_said_action, extract_qa_pairs, index_sessions, do_android_dream_of_electric_sheep: session_id (partial match) selects sessions explicitly; when empty, the tool works its window — hours_back (a window in hours, 1 or more; each tool carries its own default) and optionally directory (substring). hours_back is never “no bound”: 0 means “omitted” and takes the tool’s default window, so a client that serialises an unset window as zero gets the default window and never the whole store.

Breadth beyond a window is explicit: the window tools take all: true for the whole store — every session of every project the store holds — as a deliberate opt-in, and all together with a real hours_back is rejected as the same contradiction. Every rejection names the cheap forms — pass session_id, or directory, or hours_back: 1 or more, or all: true on purpose — because the error is the documentation. Rejection, not clamping: a silent clamp would answer a smaller question than the one that was asked. The same house rule governs every other input a tool can be handed: limit above 1000, max_bytes above the 8 MiB ceiling, a 0-based line/start, an unknown parameter, and an empty search query.

Every params struct rejects unknown fields by name: a typo (hour_back, directorys) is a silently unscoped call otherwise, so the error names the unrecognised field and the fields the tool does accept. Every served schema states the bounds its handler enforces (minimum, maximum), so a schema-honouring client cannot send a value the handler will reject.

Listings are bounded the way reports are: the listing renders as many of the most recent rows as max_bytes holds — at most the 200 rows the store query bounds — and states how many rows the window holds but did not print. A default list_sessions therefore returns rows; when a single row is larger than the whole budget the response is handed to flood control whole, which is the same escape hatch every other report has.

The CLI keeps its own doctrine — stdout is a stream, 0 means unbounded, and the command you typed is the scope you asked for. The MCP tools are the ones an agent calls on the caller’s behalf, so they are the ones that reject the unscoped call.

Global CLI flags

Accepted before the subcommand:

Output format per subcommand, with no flag given: list and profile print a human-readable summary, extract prints one JSON object per line, user-messages prints a JSON array, extract-by-type always prints type,timestamp,json records, and todo-history always prints one JSONL todo-edit event per line. he-said-she-said, do-android-dream-of-electric-sheep and index print their own report format regardless of flags.

CLI extraction is unbounded by default — stdout is a stream, and 0 means unbounded. The MCP tools are always bounded. A CLI truncation notice goes to stderr so stdout stays machine-parseable.

Read tools

harness

No parameters. Returns the harness this server is bound to, as JSON. The cheapest confirmation that a registration is right. No CLI equivalent — the CLI takes --harness explicitly.

list_sessions

MCP: hours_back (a window in hours, 1 or more; 0 or omitted = the 240-hour default), directory (substring), all: true (the whole store), max_bytes (the response budget, default 16,384). window_count is how many rows the window holds, held_back how many of them the response did not print, and max_bytes the budget that decided it.

CLI: total-recall --harness <h> list [--json|--markdown] [--hours N] — --hours 0 (the default) is the unbounded full listing stream; --hours N with N >= 1 is the scoped listing (at most the 200 most recent rows of the window, held-back count on stderr).

Each entry: session_id, title, start_time, end_time, file_size, line_count, user_count, assistant_count, tool_count, has_compaction, directory, parent_session_id, child_sessions, child_session_count, has_tantivy_index, aliases, alias_count, notice and read_error when present.

A row whose session has cached summary blocks also carries a summary_cache marker, {through_id, through_ts} — the message id and timestamp the cache covers through, the same field the summary cache tools work with. The marker is absent when nothing is cached, so a listing alone tells the caller whether a summarise_from would start from id 1 or from the block edge.

child_sessions and aliases are bounded per row to the 10 most recent child ids and the 10 first alias names, with child_session_count / alias_count carrying the true count beside them and the row’s notice naming what it held back — a store that accumulates subagent children cannot spend the response budget on one row, so the row count a default call gets is a function of max_bytes; raise max_bytes for a wider slice.

aliases lists other session directory names that resolve to the same underlying rollout payload — the vibe store can hold two directories for one rollout, for example a resumed session that kept its content under a new directory name. One canonical entry is emitted per payload, and the rest are listed under aliases, so a caller paging the index never processes the same rollout twice. aliases is empty for a unique session.

read_error is set when the payload could not be read: the entry is still listed, with counts from whatever could be read, so damage is visible in the index rather than the payload silently appearing empty.

The listing is bounded the way the scope contract says: most recent rows first, at most the 200 the store query bounds and as many of those as max_bytes holds, with the held-back count stated. A default list_sessions returns rows, never the flood cap’s overflow marker; when a single row is larger than the whole budget the response is handed to flood control whole, which writes it out and returns the marker — the escape hatch every other report has. The bound lives where the data lives — the OpenCode implementation pushes the window, the directory substring and the row cap into SQL, so aggregates are computed only for the listed rows (through the store’s message(session_id, …) and part(session_id) indexes) and the window count is a bare COUNT(*); the file-based harnesses test mtime and directory before reading a payload. A scoped listing costs what it asks for, not what the store holds.

profile_session

CLI: total-recall --harness <h> --session <id> profile [--no-cache].

Returns session_id, file_size, line_count, first_ts, last_ts, role_counts, has_tantivy_index and interesting_events. An interesting event carries line_number, gap_lines, event_type and summary; event_type is one of Compaction, TodoCreate, TodoUpdate, TodoDelete, GitCommit, GitPush, GitTag, UserMessage.

The profile cache

On by default, staleness-checked; cache: false (MCP) or --no-cache (CLI) forces a recompute without touching the cache. The profile is served from a flat JSON file inside the adapter’s shadow root: <shadow root>/tr_<session-id>_meta.json, sibling of the per-session index folders. The folder is disposable, can be deleted at any time, and MUST NOT be committed.

Staleness is checked against one cheap time-updated source — for opencode a single SELECT time_updated row, never the full list_sessions aggregates; for the file-based harnesses the source file’s mtime. The cache is fresh when time_updated_ms <= cache_mtime_ms + 15_000 (15 s tolerance). Fresh → served from the cache; stale, missing, or corrupt → recomputed and the cache is rewritten. With the flag off, no cache file is created at all.

A cached payload is validated through an RFC 8927 JSON Type Definition schema, schemas/profile-cache.jtd, compiled by jtd-codegen into the standalone validator in src/profile_cache_types.rs. A payload that fails the gate — or whose event_type strings are not known EventType variant names — is treated as corrupt and recomputed.

Bounded output

extract_messages, extract_user_messages and extract_by_type return a JSON envelope, never a bare array, so a large session can never exceed what an MCP client can hold:

{ "session_id": "…", "harness": "vibe", "full": false,
  "bounds": { "total_records": 7525, "returned_records": 100, "offset": 7425,
              "limit": 100, "next_offset": null, "truncated": true,
              "truncation_reason": "record_limit", "max_bytes": 8388608,
              "bytes": 153220, "clamped_record_indices": [],
              "notice": "TRUNCATED: returned records 7425..7525 of 7525 …" },
  "messages": [ … ] }

Omit offset for the most recent limit; pass offset (from 0) and follow bounds.next_offset to page the whole session in bounded chunks. truncated and notice always state when output was cut and why (record_limit / byte_cap / record_clamp, comma-joined). An empty result window — an empty session, or paging past the end — never signals truncation. The payload is emitted as compact JSON and bounding is computed on that same compact serialization, so the cap holds exactly. When even one record cannot fit under the budget, the tool emits a descriptive error instead of silently breaking the cap: raise max_bytes, or set max_record_bytes to enable the clamp.

extract_messages

CLI: total-recall --harness <h> --session <id> extract [--full] [--limit N] [--offset N] [--max-bytes N] [--max-record-bytes N].

extract_user_messages

session_id, plus limit, offset, max_bytes, max_record_bytes. There is no full: user messages are always derived from the whole rollout. Injected (synthetic) user records are skipped; the envelope is identical to extract_messages with user_messages in place of messages.

CLI: total-recall --harness <h> --session <id> user-messages.

extract_by_type

A pure read of the raw store, for recovering data from large, partially corrupt, or poisoned sessions. Emits one record per line with no summarization or aggregation.

# {"session_id":"…","harness":"vibe","types":["user","tool"],"bounds":{…}}
user,2026-09-15T09:59:55Z,{"role":"user","content":"…","injected":false}
tool,2026-09-15T10:00:02Z,{"role":"tool","content":"…"}

Line 1 is a #-prefixed JSON header carrying the same bounds envelope, including next_offset and a truncation notice. Each record line is type,timestamp,json: timestamp is ISO8601, a unix epoch, or 0 when neither; json is the source record serialized compactly, so embedded newlines are escaped and the one-record-per-line invariant always holds. On truncation a final # TRUNCATED: … line is appended. The stable prefix lets callers filter and re-merge with standard Unix tools:

awk -F, '$1=="user"'        # filter by type
jq -R 'fromjson'            # re-parse the JSON column
sort -t, -k2                # order by timestamp

Injected records are skipped by default; the record itself carries the injected flag either way. For vibe the records are the byte-faithful source lines re-parsed from messages.jsonl; other harnesses derive entries from the normalized message stream.

CLI: total-recall --harness <h> --session <id> extract-by-type --type user --type thinking [--include-injected].

extract_qa_pairs

The question/answer pairs from sessions where the harness asked through a question tool: the question text, its header and options where the format carries them, paired with the answers the user picked or typed, chronologically:

=== ses_ee8e8f72 (opencode)
--- QA 1 (2026-10-12T08:15:04Z)
HEADER: Branch
OPTIONS:
  - main — the trunk, release when green
  - feature — cut a branch instead
ANSWERS:
  - main

For opencode the pairs are read store-side from the question tool parts (state.input.questions), the picks under state.metadata.answers and the tool result output as fallback. For vibe, ask_user_question arguments’ questions are paired with the following tool_result output’s answers by tool_call_id. A question that was never answered, or whose prompt was cancelled, yields no pair. Harnesses without question tools report no sections, never an error.

Session selection follows the window tools: session_id (partial match) selects one session; empty works the hours_back window (default 48, 0 = omitted) with an optional directory substring, and all: true is the explicit whole-store opt-in. In window mode sessions are ordered most-recent-first, one section each for the sessions that have pairs, the scanned count stated in the header. Each session’s section carries at most the most recent 100 pairs with the held-back count stated; the report is flood-capped at max_bytes (default 16384, 8 MiB ceiling) with the EOF marker and overflow file. MCP only — no CLI subcommand.

todo_history

Recovers a rollout’s todo-list edits: every todowrite flush replayed as the edits between consecutive list states, one compact JSON object per line, in ts order:

{"ts":"2026-10-08T04:14:18Z","action":"added","todo":"item00: ship the fix","status":"pending"}
{"ts":"2026-10-08T04:20:02Z","action":"in_progress","todo":"item00: ship the fix","status":"in_progress"}
{"ts":"2026-10-08T04:31:47Z","action":"removed","todo":"item01: later"}

action is added (first sight of an item), updated (reworded, same status), the item’s new status on a status change (pending, in_progress, completed, cancelled, …), or removed (the flush dropped it — status is omitted on a removal). Item identity is the itemNN: slug prefix this repo’s sessions carry, else the full content string. Folding the events reproduces the flushes. full defaults to true — the whole rollout’s todo history; false reads from the last compaction point. The MCP report carries a #-prefixed header line (filter it with grep -v '^#'); the CLI streams the bare JSONL, unbounded. The report is flood-capped at max_bytes (default 16384, 8 MiB ceiling) with the EOF marker and overflow file. Opencode only: every other harness refuses with an explicit error, never an empty list.

CLI: total-recall --harness opencode --session <id> todo-history [--no-full].

Matched dialogue and actions

she_said_he_said_action

Per session it extracts:

• HE SAID — user text parts matching any term
• SHE SAID — assistant text parts matching any term
• THEY DID — tool calls whose name or arguments match any term

Matching is pushed down into SQLite as a custom scalar function on a read-only connection: the database is never written, matching runs inside the query engine, and matching rows stream out in one pass. Synthetic parts (skill injections, compaction markers) are skipped. Output is a markdown report, sessions ordered most-recent first. An unmatched session_id is reported in the header, not fatal. Currently implemented for OpenCode; other harnesses return a clear unsupported error.

CLI: total-recall --harness opencode he-said-she-said --words git,branch,tag [--sessions <id>…] [--hours 48] [--directory <substr>].

Indexing and full-text search

index_sessions

Builds or refreshes a tantivy full-text index per session, from the normalized message stream of any harness. One document per message that carries text: content and thinking are both indexed; session_id, role, timestamp and the message sequence number are stored for display. For OpenCode, reasoning parts become the message’s thinking text (synthetic reasoning skipped); other harnesses currently have no thinking content.

The index lives in a shadow folder next to the session store, never inside it:

Each session gets <shadow root>/<session_id>/, plus a total-recall-meta.json marker carrying session_id, doc_count and built_at. Re-indexing a session replaces its index from scratch — the folder is disposable, can be deleted at any time, and MUST NOT be committed.

MCP: hours_back defaults to 24 — the natural call indexes the last day, not the whole store; all: true is the explicit whole-store opt-in.

CLI: total-recall --harness <h> index [--sessions <id>…] [--hours 24] [--directory <substr>]. list and profile carry the index presence flag alongside their usual output.

do_android_dream_of_electric_sheep

Searches the per-session indexes with a QueryParser over the content and thinking fields, merging the top 10 hits per session into one ranking ordered by score. The session_id resolves to the most recent match; an unmatched id is reported in the header, not fatal. Sessions without an index are listed as not indexed and skipped; a corrupt index is reported in the header, not fatal. Run index_sessions first.

Hit lines carry the matched text and are marked thinking when the term matched only a message’s thinking content:

ses_f5a4ea5e | 0.5395 | 2026-09-19T10:02:00Z | ASSISTANT (thinking) | contemplating the electric sheep dream

CLI: total-recall --harness opencode do-android-dream-of-electric-sheep --query 'shadow AND index' [--sessions <id>…] [--hours 48] [--directory <substr>].

The summary cache (blocks)

The summary cache makes what is cached an explicit, inspectable data structure: a per-session range ledger of blocks, each naming the exact message-id range it summarises. The ledger is <shadow root>/tr_<session-id>_summary.csv, one row per summarised part, 11 columns:

from_id,from_ts,to_id,to_ts,user,assistant,tool,thinking,chars,tokens_est,part_file

The ids are 1-based message sequence numbers in the session stream — stable for append-only stores, computed by enumerating the messages at read time, so they work on every harness. The type counts, chars and tokens_est describe the block’s message slice; the part text lives in tr_<session-id>_summary_parts/. There is no backward compatibility with the v0.15.0 3-column ledger (alpha): a malformed or short row invalidates the ledger — it is treated as empty and the file is dropped, and any orphaned part files may be clobbered later.

summary_cache

Read-only introspection: resolve the session, read the ledger, and return

{
  "session_id": "ses_…",
  "blocks": [
    {"from_id": 1, "from_ts": "…", "to_id": 4, "to_ts": "…", "user": 2,
     "assistant": 2, "tool": 0, "thinking": 0, "chars": 1200,
     "tokens_est": 300, "part_file": "…/part_001_004.md"}
  ],
  "tail": {"from_id": 5, "from_ts": "…", "count": 2, "user": 1,
           "assistant": 1, "tool": 0, "thinking": 0, "chars": 400,
           "tokens_est": 100}
}

tail is the first unsummarised id (the last block’s to_id + 1, or 1 with no blocks) plus the count, type counts, chars and token estimate over the messages from there to the session’s end — what a summarise would cost. tail is null when every message is already covered. The tool never calls the LLM and never writes.

CLI: total-recall --harness <h> --session <id> summary-cache.

summarise_from

Summarise from an explicit id: session_id (partial match) and from_id. The guard: from_id must be the first unsummarised id — one past the last cached block’s to_id, or 1 with no blocks. Anything else is an error in the house form naming the expected id, so a caller that has not seen summary_cache cannot guess. On success the messages from from_id to the session’s end are summarised, stored as a new block (part file plus ledger row), and the tool returns the composed parts — each named ### part N: from_ts → to_ts (ids from_id..to_id) — plus the summary cache: footer with the new block counted in.

Rollback is explicit too: cache_delete_latest deletes the newest block (ledger row and part file) and returns the new cache status; call it twice to roll back two blocks, then summarise_from redoes the period whole.

total-recall --harness <h> --session <id> summarise-from --from-id 5 [--provider mistral]

cache_delete_latest

Roll the cache back one block: the newest ledger row is removed and its part file deleted, then the same status JSON as summary_cache is returned. With no blocks it is a house-form error, not a no-op.

total-recall --harness <h> --session <id> cache-delete-latest

LLM-backed tools

prior_period_summary

An orientation summary of a prior period of work. The period is the work since the session’s last compaction boundary — the work after that boundary. A session with no compaction points has no earlier period, so the summary covers the whole session up to now.

The output carries, in deliberate ordering for autoregressive LLMs (metadata first, ground truth next, LLM summaries middle, action last):

1. Recent rollouts within hours_back, with the current session marked SUMMARISED
2. Plan/todo files from .tmp/delegation/ and ~/.vibe/plans/
3. The todo log of the period (opencode): items opened and closed with timings, the state-change history (opened → in_progress → completed/cancelled transitions), and a churn note when items open faster than they close — starting new things without finishing old ones may be causing churn — with the human’s recent steers read into it. On a harness without readable todowrite flushes the section says so instead of failing or silently reading as “no todos”.
4. The commits made in the period: the git commit tool calls with their timestamps.
5. The current-state summary and the user’s goals / tasks / steers (two parallel Mercury 2.5 calls).
6. A compaction-points FYI footer: the timestamps of the last few compaction points with the message and line counts since each — FYI only; mine the logs for anything deeper.
7. The instructions, then a fixed orientation warning appended to every output: summarising in-flight work in more detail is a fool’s errand and a moral hazard — the model would trust the summary and not double-check — so the warning makes re-orientation the model’s job.

CLI: total-recall --harness <h> --session <id> prior-period-summary [--provider mistral].

Flood control

Every response that can overflow the caller’s context is capped at max_bytes (default 16,384) — the mechanism is generic and applies to any tool that can produce a large response, not to a fixed list. An overflowing response is never lost and never floods the caller’s context:

• the full response is written to a private file under the user’s temp directory, $TMPDIR/total-recall/<tool>_<scope>_<unix-time>.md, mode 600
• the returned text is cut at a line boundary at or under max_bytes — JSON responses are never torn: an overflowing JSON response returns the marker alone
• the return ends with a line-oriented EOF marker naming the tool, the returned and total byte counts, the total line count, the full response’s path, and the prune policy
• the marker carries the file’s line histogram (the vendored line_histogram.awk), so the shape of what overflowed is visible without reading it
• every capped call first removes overflow files older than 24 hours from that directory, so the temp file needs no manual cleanup

Raise the window with max_bytes when a bigger response is wanted inline.

Bounded prompts

Flood control bounds what comes out. What goes in to an LLM is bounded the same way: every prompt a tool builds from a session is held to a byte budget, and the newest context is kept while the oldest is dropped. The drop is never silent — a marker line inside the prompt names how many items and how many bytes were dropped, and the tool’s timing comment carries the prompt byte sizes and dropped counts. A session too large for one call costs a bounded prompt rather than a request the vendor cannot ingest in time.

line_histogram

The companion for overflow files and any large dump on disk. Profiles a file by line-size distribution (histogram mode, ten buckets — a 2 MB file yields a 2 KB histogram) or extracts a line range (mode: extract with line, or start and end). Runs the vendored line_histogram.awk with a direct awk -f spawn — the shebang is never relied on. The line selectors are validated as a set: mode: extract needs line, or start with end; end alone, start past end, and a selector without mode: extract are rejected by name rather than answered with a histogram nobody asked for.

Ingestion guardrails

Mercury calls are guarded by measured, documented limits (probed against a Pay-As-You-Go key on 2026-09-15 with real rollout payloads, 5k/10k/20k tokens, concurrency ramped 1→64). The free tier carries the same caps — 1,000 requests/min, 1M input tokens/min, 100k output tokens/min — and exceeding any of them returns HTTP 429.

• The binding limit is ~1M input tokens/minute. Requests/min (1,000) and output tokens/min (100,000) never bind for compaction workloads.
• Per-call input cap: 234,000 tokens — 90% of Mercury 2.5’s 260K context window (~4 chars/token), leaving the rest for the completion. A 100MiB rollout is never slung in one call; an over-cap prompt is rejected with an error, never truncated. Practical calls stay far below the cap — tool results are snipped to 500 chars in prompt assembly, and a prompt built from a whole session is held to the byte budget above, so a session too large to ingest in one call costs a bounded prompt rather than a rejected request.
• Concurrency: 4 in-flight requests with ~10k-token prompts. Measured sweet spot: ~22k input tok/s with zero rejections, p50 latency 1.9s. Beyond concurrency 8 the 429 wall arrives with no throughput gain.
• 429 handling: at most 5 retries, then a descriptive error naming the status and the exhausted budget — never a silent drop. Retry-After (seconds form) is honoured in full and capped at 10s per attempt, so Retry-After: 0 comes back immediately; with no parseable header the wait doubles from 1s (1s, 2s, 4s, 8s, capped at 10s). Every 429 is logged via tracing.
• 5xx handling: 3 retries at ~1s spacing, then an error naming the status. The two budgets are independent: 429s never consume 5xx retries or vice versa.

Compaction prompt design

The compaction prompts and strategies are informed by badlogic’s context compaction research comparing Claude Code, Codex CLI, OpenCode, and Amp:

• Claude Code triggers at ~95% context capacity, uses a summary prompt
• Codex CLI preserves recent user messages (last ~20k tokens) alongside the summary
• OpenCode has a separate “prune” mechanism for tool outputs beyond 40k tokens
• Amp uses manual “handoff” instead of auto-compaction

This tool takes the best of each: it preserves recent messages, prunes large tool outputs, and uses Mercury 2.5 for fast summarization.

Configuration

Storage-root overrides

Each adapter resolves its storage root from $HOME by default — the live store. To point the CLI and the MCP server at fixtures or a scratch copy instead, without ever touching live sessions, set the harness’s root override.

Precedence: an explicit root (tests) > the env override (non-empty) > the $HOME-derived default. An empty value counts as unset. An override that resolves to nothing usable is an error, never a silent fall-back to $HOME.

The shadow index root follows the session root, so an override moves the tantivy indexes and the profile cache with it.

The sandbox guard

TOTAL_RECALL_SANDBOX=1 refuses to build any adapter whose root did not come from its override. With the sandbox armed and no override set, adapter construction fails closed with a descriptive error instead of reading the live store — a mechanical guarantee that sandboxed dev and test runs never read live sessions:

TOTAL_RECALL_SANDBOX is set but TOTAL_RECALL_VIBE_ROOT is not: refusing to read
the live store. Point TOTAL_RECALL_VIBE_ROOT at a fixture or scratch copy.

The guard is a constructor check, so it covers every tool: a sandboxed run cannot read the live store by any route.

API keys

A key reaches the binary one of two ways:

• Environment / MCP environment block. An MCP server finds a key only in its own environment, so the key belongs in the registration snippet’s environment map (or [mcp_servers.env]). Exporting it in the shell that launches the client works too — the server inherits it. This is the recommended path: a spawned server’s working directory is chosen by the MCP client, not by you, so a .env file may simply not be found.
• .env in the working directory, for the build-from-source CLI path: copy .env.template to .env and fill in the key. Only a default build reads it.

Build features

The LLM vendor is a compile-time cargo feature. A vendor is a third-party service that can be acquired, renamed or disappear entirely, so no build of this tool requires one: the default build has both vendors, and a vendor-free build is a first-class artifact.

[features]
default = ["mercury", "mistral"]
mercury = ["dep:dotenvy"]   # Inception Mercury 2.5 (the default provider)
mistral = ["dep:dotenvy"]   # Mistral, selected with --provider mistral

cargo build --release                        # both vendors (default)
cargo build --release --no-default-features   # vendor-free

A vendor-free build makes no LLM calls and needs no API key — it never reads INCEPTION_API_KEY or MISTRAL_API_KEY, and does not even load a .env file.

The LLM-backed MCP tool names stay in tools/list in every build — clients bind by name — and the call itself returns the error:

LLM provider `mercury` is not compiled into this build: it was built without the
`mercury` cargo feature. Rebuild with `cargo build --release --features mercury`,
or use the vendor-free build (`--no-default-features`) — it makes no LLM calls and
needs no API key, and every log-mining tool still works. Vendors compiled into
this build: none.

Provider selection

--provider selects between the vendors compiled into the build (mercury, mistral); a vendor that is not compiled in fails the same way, and an unrecognised name is refused rather than silently falling back to Mercury:

unknown LLM provider `gpt`: pass it to --provider. Known vendors: mercury,
mistral; compiled into this build: mercury, mistral.

Vendor identity (endpoint, model, key variable) is the gated part. The OpenAI-compatible chat transport itself compiles in every build but is unreachable without a vendor constructor.

--provider applies to the CLI’s summarise-from and prior-period-summary subcommands. The mcp subcommand constructs the default provider; see MCP setup.

OPENCODE_DB

Not read by the binary. scripts/opencode-db-vacuum.sh reads it to locate the store it operates on, defaulting to $HOME/.local/share/opencode/opencode.db. See the store-vacuum runbook.

What is written where

Nothing is written inside a session store. total-recall creates files only in the harness’s shadow root:

Both are disposable: delete the folder at any time, or override the session root to move it. A shadow root inside a git repository MUST NOT be committed.

Pricing

Mercury 2.5, launch (80% off) versus normal:

100M free tokens for new accounts.

Troubleshooting

The MCP server refuses to start, exit 2

The harness is unset or unknown. Each server is bound to exactly one harness by HARNESS in the registration snippet’s environment block, or --harness <name> on the mcp subcommand:

refusing to start MCP server: harness not defined. Set HARNESS=<vibe|codex|claude|opencode>
in the MCP config environment or pass --harness <name>.

refusing to start MCP server: unknown harness 'vibee'. Valid values:
vibe|codex|claude|opencode. Set HARNESS=<vibe|codex|claude|opencode> in the MCP
config environment or pass --harness <name>.

The server prints the reason to stdout and stderr and exits 2, because a server that starts with an unknown harness would bind every tool to a nonsense store.

401 “Incorrect API key” from prior_period_summary

The MCP server’s environment block is missing INCEPTION_API_KEY. List, profile, extract, index and search all work without a key — no LLM call — and the LLM-backed tool fails with the vendor’s 401.

Add the key to the snippet’s environment block, per MCP setup, or export it in the shell that launches the client so the server inherits it. A .env file is not a reliable route for a spawned server: it is only found relative to a working directory the MCP client chooses.

The same 401 on a CLI invocation means the key is neither exported nor in a .env in the current directory.

“LLM provider mercury is not compiled into this build”

The binary was built with --no-default-features, or without that vendor’s feature. Either rebuild:

cargo build --release --features mercury

or keep the vendor-free build and use the log-mining tools, which need no key. The MCP tool name stays in tools/list either way — clients bind by name, so a missing vendor is reported on the call, not by a disappearing tool. See Configuration.

unknown LLM provider '<name>'

The name is not a vendor. Valid values are mercury and mistral; an unrecognised name is refused rather than silently falling back to Mercury.

The sandbox guard refuses to build an adapter

TOTAL_RECALL_SANDBOX is set but TOTAL_RECALL_OPENCODE_ROOT is not: refusing to
read the live store. Point TOTAL_RECALL_OPENCODE_ROOT at a fixture or scratch
copy.

TOTAL_RECALL_SANDBOX=1 is armed and this harness has no root override. Set the harness’s override — see Configuration — or unset the sandbox. The guard never falls back to the live store.

An override resolves to nothing usable

An override that does not exist, or is empty, is an error naming the path — not a silent fall-back to $HOME. Check for a typo in the variable name and in the path, and remember an empty value counts as unset.

“No sessions found”

The store resolved to an empty or non-existent directory. Check, in order: the HARNESS value, then the store root the tool resolved for it (see Configuration), then that the harness has actually written sessions there.

The CLI exits 2 with a path in the message

A rollout payload could not be resolved or read. Read paths return Result<T, String>: unreadable data is an error, never an empty session. The usual causes are permissions on the store, or the session id not resolving. A JSONL line that fails to parse is not fatal — it is skipped with a WARN carrying its line number, visible under --verbose.

list_sessions is the deliberate exception: a session whose payload cannot be read is still listed, with counts from whatever could be read, and carries a read_error field.

A tool reports “No sessions found” but the store has sessions

A partial session_id that matches nothing resolves to no session, and the tools answer with that error rather than guessing. Confirm the id against list_sessions; partial ids are matched by substring, and the most recent match wins.

do_android_dream_of_electric_sheep reports “not indexed”

The session has no shadow index yet. Run index_sessions (or total-recall --harness <h> index) first — the search reports unindexed sessions in its header and skips them, rather than returning nothing as if the terms were absent. A corrupt index is reported in the header, also not fatal.

she_said_he_said_action fails on a non-OpenCode harness

The tool is implemented for OpenCode only; other harnesses return a clear unsupported error. Matching runs inside SQLite as a custom scalar function on a read-only connection, which is an OpenCode-store property.

A tool name looks wrong in the agent’s surface

The MCP server key was renamed. The tool names derive from it — total-recall_list_sessions, … — so keep the key as total-recall in both registration snippets.

Extraction came back truncated

Expected: the MCP tools are bounded by design. Read bounds.truncation_reason and bounds.notice — record_limit, byte_cap or record_clamp — and page with offset from 0, following bounds.next_offset. See Tools.

CLI extraction is unbounded unless you pass --limit / --max-bytes; when you do, the truncation notice goes to stderr so stdout stays machine-parseable.

“even one record … exceeds max_bytes”

One record is larger than the whole budget. Raise max_bytes, or set max_record_bytes to enable the per-record clamp. The tool errors rather than silently breaking the cap.

The store is huge and list is slow

A scoped listing is bounded where the data lives: the window, the directory substring and the row cap are pushed into the store query, so a scoped list costs what it asks for, not what the store holds. An unscoped list is the full aggregate scan over the whole store — a disk-read question, not a CPU one. A store that grew without bound is the subject of the store-vacuum runbook.

held_back is most of the window

Expected: list_sessions renders as many of the window’s most recent rows as max_bytes holds, and states the rest in held_back. Each row carries at most the 10 most recent child_sessions ids and the 10 first aliases, with child_session_count / alias_count stating the true totals and the row’s notice naming what it held back — a bounded row means the count is set by the budget, not by any session’s size. Raise max_bytes for a wider slice, or narrow the window with hours_back or directory.

“unrecognised field hour_back”

A parameter name the tool does not accept. Every tool rejects unknown fields by name and lists the fields it does accept: a typo is a silently unscoped call otherwise, so the call is answered with the accepted names rather than run on the default scope.

“limit is out of range” / “max_bytes is out of range”

The value is outside the bound the tool enforces — limit runs 1 to 1000, max_bytes 1 to the 8 MiB ceiling. Either value is stated in the tool’s schema as minimum/maximum, and 0 means “omitted” for both: limit: 0 takes the default 100, max_bytes: 0 takes the tool’s default. Page with offset/limit instead of raising limit.

opencode store: vacuum rehearsal and procedure

This document is the write-up for scripts/opencode-db-vacuum.sh, which implements the procedure below as subcommands (check, counts, backup, vacuum-into, prune-events, swap, verify). Read this for the measurements and the reasoning; run the script for the mechanics.

The opencode session store (~/.local/share/opencode/opencode.db) was 31 GiB and looked bloated: message+part payloads sum to only ~4.4 GB. This document records a dress rehearsal of a vacuum/swap, what it disproved, what the real space consumer is, and the verified procedure for shrinking the store safely.

Measured facts (2026-09-20/21 rehearsal)

Where the space actually goes (dbstat, copy)

Earlier length(data) sums understated payloads because length() on TEXT counts characters, not bytes — always use length(CAST(data AS BLOB)) for byte truth.

The event log is the whale

1,835,459 event rows. opencode writes a full JSON snapshot per update:

Every message/part edit appends a complete JSON snapshot — the log grows unboundedly as sessions run. The event table has no time column (id, aggregate_id, seq, type, data), so age-based pruning needs care.

Consequence: plain VACUUM reclaims nothing on this store (freelist is 0). Space is only reclaimed by pruning event first, then vacuuming. This is an opencode upstream growth issue, not user data: sessions/messages/parts (what total-recall reads) are 4.4 GB of the 36.

Procedure

Run from a plain terminal. The store is cold only when every opencode TUI is quit — including the one an agent would be running inside, so an agent cannot execute the swap itself; it can only verify before/after.

0. Guard — cold check

pgrep -fl opencode   # must output nothing

1. Counts before (write down)

sqlite3 ~/.local/share/opencode/opencode.db <<'SQL'
SELECT 'sessions', COUNT(*) FROM session;
SELECT 'messages', COUNT(*) FROM message;
SELECT 'parts', COUNT(*) FROM part;
SELECT 'events', COUNT(*) FROM event;
SELECT 'message_bytes', SUM(length(CAST(data AS BLOB))) FROM message;
SELECT 'part_bytes', SUM(length(CAST(data AS BLOB))) FROM part;
SELECT 'orphan_parts(no message)', COUNT(*) FROM part p WHERE NOT EXISTS (SELECT 1 FROM message m WHERE m.id = p.message_id);
SELECT 'orphan_parts(no session)', COUNT(*) FROM part p WHERE NOT EXISTS (SELECT 1 FROM session s WHERE s.id = p.session_id);
SELECT 'orphan_messages(no session)', COUNT(*) FROM message m WHERE NOT EXISTS (SELECT 1 FROM session s WHERE s.id = m.session_id);
SQL

Expect: 0 orphans in all three. Record all numbers.

2. Backup (rename the old store aside)

Quitting opencode closes the WAL; a checkpoint happens on clean exit. Verify opencode.db-wal is gone (or tiny) before proceeding.

cd ~/.local/share/opencode
mv opencode.db opencode.db.pre-vacuum-$(date +%Y%m%d-%H%M%S)

3. Vacuum to a copy, swap the compact one in

sqlite3 opencode.db.pre-vacuum-* "VACUUM INTO '$HOME/.local/share/opencode/opencode.db.new';"
mv opencode.db.new opencode.db

Measured: ~4 min for 36 GiB on this machine; expect the output to be ~36 GiB unless events were pruned first (step 4).

4. (Optional, the actual win) Prune the event log before vacuuming

The event log is a bus log — the TUI reads current state from the session/message/part tables, not from historical events. Prune keeps the newest N events per aggregate and drops the rest (default below keeps 50):

sqlite3 opencode.db.pre-vacuum-* "
DELETE FROM event WHERE seq NOT IN (
  SELECT seq FROM (
    SELECT aggregate_id, seq,
           ROW_NUMBER() OVER (PARTITION BY aggregate_id ORDER BY seq DESC) rn
    FROM event
  ) WHERE rn <= 50
);"

Run VACUUM (or step 3’s VACUUM INTO) after this; then the output should be ~5 GiB instead of ~36.

5. Verify after — counts must reconcile

Re-run the step-1 counts against the new store. Acceptance criteria:

• integrity_check → ok
• orphans still 0
• sessions: identical (no sessions lost)
• messages/parts: identical on a cold run (a warm snapshot copy will be behind by whatever the live sessions wrote during the vacuum — rehearsal measured −24 messages / −87 parts; that drift is the snapshot window, not data loss)
• events: only reduced if you pruned, and only by the prune predicate

Rehearsal verification on the copy: integrity ok, 0 orphans everywhere, sessions identical (3205), messages/parts behind by the snapshot drift.

6. Rollback

cd ~/.local/share/opencode
mv opencode.db opencode.db.vacuumed
mv opencode.db.pre-vacuum-<stamp> opencode.db

opencode recreates opencode.db-wal and opencode.db-shm on next launch; never copy stale -wal/-shm files onto a swapped-in database.

Source and script

This page reproduces docs/opencode-store-vacuum.md, which is the canonical copy in the repository. The procedure is implemented by scripts/opencode-db-vacuum.sh, available here for download, with subcommands check, counts, backup, vacuum-into, prune-events, swap, verify. Its store path comes from the OPENCODE_DB environment variable, defaulting to $HOME/.local/share/opencode/opencode.db.