The operator’smanual.

Install

Python ≥ 3.11. macOS, Linux and Windows — the full suite runs on all three in CI, including the Windows-specific behaviours (OneDrive placeholder detection, junction refusal at the root boundary).

Email ingestion is optional; the extra depends on the format you have. .msg resolves to wheels everywhere; .pst needs libpff-python, which compiles from C source and requires a build toolchain.

uv tool install "shelfmark[msg]"      # .msg — no compiler needed
uv tool install "shelfmark[pst]"      # .pst — compiles from C source
uv tool install "shelfmark[email]"    # both

Quickstart

Three commands, and init finds your documents for you — when the default root misses, it sweeps for the folders that do hold documents, and one keypress fixes the config.

$ shelfmark init
Wrote ~/.config/shelfmark/config.toml

$ shelfmark refresh
cataloguing ~/Paperwork -> ~/.local/share/shelfmark/catalog.db
seen 6  new 6  updated 0  unchanged 0  rematerialised 0
evicted 0  corrupt 0  restricted 0

$ claude mcp add shelfmark -s user -- shelfmark-mcp

That's install to connected. The agent's first call then looks like this — a real corpus_stats() answer:

# shelfmark corpus
6 files · 0.0 GB
✓ index fresh — matches disk, last refresh 0 min ago

## Roots
root                     files  own+shareable
Clients                      4              0
Decks                        2              0

## Rights × confidential
  REFERENCE    may leave                5
  UNKNOWN      unreviewed → held        1

Optional but worth five minutes: shelfmark review settles rights (see governance), shelfmark stats prints the census any time, and shelfmark config shows every rights rule in the order it is checked, with the number of files each one currently claims.

Configuration

Everything corpus-specific lives in config.toml — the code ships with neutral defaults only. Resolution order: --config flag → $SHELFMARK_CONFIG → ~/.config/shelfmark/config.toml.

Rights & governance

Shelfmark separates two questions that usually get confused:

A method may belong to you while the client deliverable containing it stays confidential. shareable_only=True means positively classified — confidential=0 AND rights IN (OWN, REFERENCE). Never-reviewed files are held back: unreviewed is not the same as cleared.

Getting rights set: shelfmark review

Most files carry no OOXML author, so path rules are the only mechanism that can classify them — and that knowledge is yours, not the corpus's. review asks for it, biggest win first, with what the catalogue knows on screen:

[1/8] Projects
      493 unclassified of 662 files · folder reads as: pitch
      is: note 202, document 89, code 88, pdf 51
      authored by: R. Okonjo (31), A. Lindqvist (12), openpyxl (9)
      e.g. programme-overview.pptx · phase-two-proposal.pptx
      answer [skip]:

Answer own, own-private, reference, client, personal, scratch or skip. On a ~1,900-file corpus, five answers settled 80% of the unclassified files.

shelfmark review                 # dry run — shows what it would write
shelfmark review --apply         # writes the prefixes, re-derives rights

It writes config, never rights values in the data — any answer is undone by editing the config and re-running shelfmark rights.

Keeping it current

shelfmark refresh                # build + rights + prune + assertions
shelfmark refresh --if-needed    # only if a write landed or the index is old

refresh is incremental by (size, mtime, cloud-residency) — a couple of seconds over a ~30k-file tree — and asserts its own correctness on every run: walk coverage, guarded pruning, and governance invariants. Results land in REFRESH_STATUS.json next to the DB.

You do not schedule this

The MCP server keeps its own index current: it refreshes once on startup and then whenever refresh.max_age_seconds has passed. Nothing to install, no timer to configure. If you maintain the catalogue some other way, shelfmark-mcp --no-auto-refresh leaves it alone. When the index cannot be trusted, every tool says so above its answer — an old snapshot is never presented as complete knowledge.

When a guard stops you

The size guards cannot tell "the root was unreadable" from "those files really were deleted" — both look like a short walk. So they refuse, name both possibilities, and leave the index untouched:

prune REFUSED — 60 of 200 rows are no longer on disk, over the 2% ceiling.
  Nothing was deleted; the index still lists them. A real deletion, an
  unreadable subtree, or an upgrade widening the default skip list all look
  like this. Check what went missing, then re-run with --force to accept it.

Check which it was. If the files are genuinely gone, shelfmark refresh --force accepts it — after backing the catalogue up to catalog.db.bak-preprune. If a root was merely unmounted or unreadable, fix that and refresh normally: the rows are still there.

MCP tools

Result lists always say when they are cut (showing 100 of 195 …), unknown filter values are reported as bad filters with suggestions — never as an empty corpus — and excerpts mark their truncation point.

Classification rules

Two things people trip over, learned the hard way:

1. Anchor short regex alternatives on both sides. An unanchored rfi matches inside "Dockerfile" and the Spanish word "perfil". Before adding or deleting an alternative, list the filenames it actually matches.
2. A rule edit does not relabel existing files. The builder is incremental, so after any rule change:

   shelfmark reclassify all          # dry run — shows what would change
   shelfmark reclassify all --apply

   Do not use build --rebuild for this: it re-walks everything (and on cloud-synced trees drags every evicted file back down) when the filenames are already in the DB.

Diagnostics

shelfmark doctor

Checks the setup for the failures that stay quiet: a catalogue inside a cloud-synced folder (a torn sync-restore opens and answers wrongly), a root the process cannot read — on macOS usually Full Disk Access — a database folder that is not writable, a corpus that is mostly cloud-evicted, and any email format in the catalogue with no reader installed. It names the fix rather than the fault, prints the evidence behind each guess, and exits non-zero on anything fatal so it can gate a setup script.

shelfmark doctor --report emits the same verdicts as JSON with the corpus taken out — counts, states and verdict codes, never a path, filename, root label or config string. At roughly 1.5 KB it fits in an issue, and it carries the failure streak: the difference between "it is broken" and "it has been broken since Tuesday, 113 runs".

shelfmark misses

Searches that found nothing are recorded locally, and the report answers one question: could metadata search ever have found it?

73 searches returned nothing   (2026-06-02 → 2026-08-07)

Most-missed terms:
    9  abatement
    7  timeline      (nowhere in your metadata)
    6  commitments   (nowhere in your metadata)

41 of 62 distinct terms (66%) appear nowhere in your filenames,
paths, authors, titles or slide titles.

A term appearing in no filename, path, author, title or slide title was unreachable however it was phrased — that is the pattern that should reopen content extraction. It stays local, is capped, and never leaves your machine; turn it off with [misses] enabled = false.

Content hashing & duplicates

The refresh never reads file contents — it must stay fast. To populate hashes for duplicate detection:

shelfmark hash                   # reads every unhashed, non-sensitive file
shelfmark hash --limit 2000      # chip away at it

Sensitive rows are never opened, and neither are symlinks. On cloud-synced trees, dataless placeholder files are skipped — reading one silently yields the hash of the empty string, which would make distinct files look identical.

Claude Code hooks

The MCP server already keeps the index current; hooks are an optimisation that cuts the delay between an agent writing a file and that file being searchable, from one refresh interval to the end of the turn. In ~/.claude/settings.json:

{
  "hooks": {
    "SessionStart": [{"hooks": [{"type": "command",
      "command": "shelfmark hook session-start", "timeout": 60}]}],
    "PostToolUse": [{"matcher": "Write|Edit|MultiEdit|NotebookEdit",
      "hooks": [{"type": "command", "command": "shelfmark mark-dirty"}]}],
    "Stop": [{"hooks": [{"type": "command",
      "command": "shelfmark hook stop"}]}]
  }
}

FAQ

Full reference: README on GitHub ↗ · PyPI ↗