perf: index filename candidates and avoid full-corpus retrieval scans

crates/searchi-core/README.md

@@ -4,7 +4,7 @@ The core has no desktop or inference dependencies. Its current retrieval engine
 combines literal filename matching with SQLite FTS5 BM25 keyword search. It does
 not implement semantic search.
 
-`SearchIndex::open` creates or migrates schema version 4. Current-schema opens do
+`SearchIndex::open` creates or migrates schema version 7. Current-schema opens do
 not acquire a write lock. WAL allows independent readers to use the committed
 index while reconciliation runs. All indexing changes are committed atomically;
 a traversal or plain-text file read error rolls back the entire reconciliation.
@@ -73,10 +73,39 @@ apply before limiting results. `search` retains the unfiltered API. Query text i
 escaped as literal FTS terms, not interpreted as operators. Results are capped at
 1,000 and queries at 4,096 bytes.
 
+Retrieval runs complete SQL-filtered lanes: indexed exact filename equality,
+literal filename substring, literal path substring, and content-only BM25.
+Multiword queries also have a tokenized name/path fallback, so `annual report`
+can find `annual_report.md`. Single-token filename identifiers are matched
+literally; the text tokenizer still handles punctuation inside document content.
+Exact filenames always precede other lanes. Ties use persistent file row IDs,
+and keyword ties use relevance then row ID. Selected rows are deduplicated.
+
+Filename/path substrings of at least three Unicode characters use an
+external-content trigram FTS index. The rarest known query trigram supplies a
+candidate superset; the whole literal substring is verified on the candidate.
+At most 256 unique query trigrams are inspected. This lookup budget does not cap
+candidate rows or hide matching results. All size/type/folder/date filters apply
+before each lane's SQL LIMIT. Shorter substrings retain complete metadata scans
+and can be slow. Broad queries under restrictive filters can also examine many
+matching candidates; no universal filtered-query latency claim is made.
+
+Trigram frequencies use the FTS vocabulary and a bounded 4,096-term cache on each
+`SearchIndex`. Initial frequency lookups can be expensive; the cache is cleared
+after reconciliation. Keep an independent reader engine alive across searches to
+benefit from warm behavior. These frequency estimates affect candidate selection
+cost, not correctness. A source query batches all finalists' indexed passage IDs
+and selects actual matching passages before choosing one source per file.
+
+Migration to the new indexes preserves file content and passage IDs and rebuilds
+both filename/path and passage indexes atomically. Large migrations can take time
+and increase database size; they must not be confused with normal current-schema
+opens. The measured Windows synthetic fixture grew from about 295 MiB to 605 MiB.
+
 ## Known limits
 
-- Filename substring lookup uses `instr` and scans metadata. A dedicated substring
-  index and benchmarks are needed before claiming million-file latency targets.
+- Short substring queries and broad restrictive filters can still scan many
+  metadata candidates. Cold frequency-cache fills exceed the warm filename target.
 - Files changed without changing size or modification time require a future
   forced refresh/content-fingerprint option. Timestamp precision depends on the
   filesystem. Indexing does not yet run a filesystem watcher.
@@ -91,5 +120,5 @@ escaped as literal FTS terms, not interpreted as operators. Results are capped a
   Bounded worker isolation, persistent jobs, and crash-resumable scans remain
   production milestones.
 - OCR, media extraction, and real model inference belong to subsequent modules.
-- Passage selection currently adds a bounded query per result; query batching and
-  passage ranking benchmarks remain necessary for large result lists.
+- Linux/macOS measurements, realistic mixed-document corpora, cold startup,
+  memory budgets, and indexing performance remain unverified.