searchi/crates/searchi-core/README.md

# Searchi core

The core has no desktop or inference dependencies. Its current retrieval engine
combines literal filename matching with SQLite FTS5 BM25 keyword search. It does
not implement semantic search.

`SearchIndex::open` creates or migrates schema version 9. Current-schema opens do
not acquire a write lock. WAL allows independent readers to use the committed
index while reconciliation runs. All indexing changes are committed atomically;
a traversal or plain-text file read error rolls back the entire reconciliation.
PDF/DOCX parser or limit failures retain a filename record with a persisted
`SearchResult.extraction_error`, clear stale content, and increment
`IndexReport.failures`. This counts attempts in this reconciliation, so an
unchanged failed record retains its reason without incrementing the count again.

`index_folder` applies `IndexOptions::default()`: directories named `.git`,
`node_modules`, and `target` are pruned. `index_folder_with_options` lets callers
replace those names. Matching directory names are ASCII case insensitive. An
explicit root is always traversed, even if its name is excluded. Symlinks are not
followed. The index's own database, WAL, and shared-memory files are excluded.

Metadata includes byte size and modification time at nanosecond resolution.
Unchanged files bypass extraction. `IndexReport.indexed` counts newly extracted
or updated records; `unchanged` counts reused records; `skipped` counts new or
changed records whose content could not be extracted because of format, encoding,
or size. Unsupported files remain searchable by filename. Content reads are
limited to 2 MiB. Reconciliation tracks seen files in SQLite rather than retaining
every path in process memory.

PDF text extraction uses `lopdf` with strict parsing and bounded load/page/font
stream decompression. DOCX uses `zip` and namespace-aware `quick-xml` to read the
main document body, including table paragraphs. The extractor does not execute
macros or follow external document relationships. It supports UTF-8 DOCX XML;
UTF-16 document parts, nested paragraphs, and XML document types are explicit
failures. Headers, footers, comments, drawings, and tracked-deletion semantics
are not yet covered. Encrypted PDFs are explicitly unsupported. Scanned PDFs
without a text layer remain filename searchable; OCR is not implemented.

| Limit | Value |
| --- | --- |
| PDF/DOCX file input, before parsing | 32 MiB |
| PDF load stream / page content / font decoding | 4 MiB per stream or page budget |
| PDF page count, after parsing | 1,000 |
| DOCX main XML, including decompressed ZIP output | 8 MiB |
| DOCX ZIP entries / paragraphs | 10,000 each |
| DOCX XML depth | 128 |
| Extracted text per document | 2 MiB |

Input reads and DOCX XML decompression are capped while reading. PDF's loader and
text extractor use the library's bounded decoding APIs. Page/entry counts and
total PDF extracted text are checked after the parser produces them. These are
not global process memory or CPU-time limits: multiple PDF objects/fonts can each
consume their budget, and parsing may still be expensive. Process isolation and
timeouts remain required before handling arbitrary untrusted documents in a
production release.

Actual page/paragraph text is stored separately in an FTS passage index. Search
results use the best passage that matches all literal query terms and expose
`source: { kind, number }`, with one-based PDF page, DOCX paragraph, or plain-text
line-range start. Paragraph numbers include empty paragraphs. A filename-only
match or a match spanning several passages can have no source; the engine does
not invent a location. Snippets with a source are drawn from that passage.

`passage_batch(after_id, limit)` exposes at most 256 `IndexedPassage` records for
real embedding runtimes. IDs represent current rows and can be reused after
reconciliation; downstream embedding caches must validate content and model
fingerprints. Consumers must tokenize long pages/paragraphs into overlapping
windows rather than silently truncate them to a model token limit.

`search_with_options` supports canonical folder boundaries, extension, inclusive
size bounds, and inclusive Unix millisecond modification-time bounds. Filters
apply before limiting results. `search` retains the unfiltered API. Query text is
escaped as literal FTS terms, not interpreted as operators. Results are capped at
1,000 and queries at 4,096 bytes.

Retrieval runs complete SQL-filtered lanes: indexed exact filename equality,
literal filename substring, literal path substring, and content-only BM25.
Multiword queries also have a tokenized name/path fallback, so `annual report`
can find `annual_report.md`. Single-token filename identifiers are matched
literally; the text tokenizer still handles punctuation inside document content.
Exact filenames always precede other lanes. Ties use persistent file row IDs,
and keyword ties use relevance then row ID. Selected rows are deduplicated.

Filename/path substrings of at least three Unicode characters use an
external-content trigram FTS index. A positional anchor grows around the rarest
known query trigram, admitting only neighboring grams within four times its
document frequency (with a small floor) and at most 32 Unicode characters.
This rejects repeated-identifier false candidates inside FTS without extending
through dense common suffix postings. The anchor supplies a candidate superset;
the whole literal substring is still verified on each candidate.
At most 256 unique query trigrams are inspected. This lookup budget does not cap
candidate rows or hide matching results. All size/type/folder/date filters apply
before each lane's SQL LIMIT. Shorter substrings retain complete metadata scans
and can be slow. Broad queries under restrictive filters can also examine many
matching candidates; no universal filtered-query latency claim is made.

`prewarm()` refreshes persisted trigram frequencies from the FTS vocabulary only
when names/path membership is dirty, then loads a bounded 4,096-term reader cache.
Foreground frequency misses use an indexed persisted table, not vocabulary
postings scans. Name/path insert/delete/change marks statistics dirty; content
changes alone do not. Statistics affect candidate cost, not match correctness.
Background preparation exercises actual first/last filenames, alphabetic name
tokens, and three words of the first document. UI callers should complete this
preparation before enabling interaction. Representative inputs are disclosed in
the benchmark report; preparation is not counted as foreground query time.

Unfiltered content retrieval materializes ranked row IDs before reading finalist
metadata and snippets. Filtered requests retain complete filter-before-LIMIT
ranking. Tests compare ordering, scores, snippets, and sources between both paths.
A source query batches all finalists' indexed passage IDs and chooses one actual
matching source per file. Keep an independent reader alive across searches.

Migration to the new indexes preserves file content and passage IDs and rebuilds
both filename/path and passage indexes atomically. Large migrations can take time
and increase database size; they must not be confused with normal current-schema
opens. The measured Windows synthetic fixture grew from about 295 MiB to 605 MiB.

## Known limits

- Short substring queries and broad restrictive filters can still scan many
  metadata candidates. Statistics preparation may take seconds after large scans;
  arbitrary unseen or OS-cold queries have no universal latency guarantee.
- Files changed without changing size or modification time require a future
  forced refresh/content-fingerprint option. Timestamp precision depends on the
  filesystem. Indexing does not yet run a filesystem watcher.
- Metadata modification times before the Unix epoch or beyond the nanosecond
  integer range are explicit errors. Unicode filename case folding is currently
  limited by SQLite's built-in `lower` function.
- Lossy path conversion does not preserve non-Unicode native filenames. This must
  be addressed before Linux production support for arbitrary byte filenames.
- The `files.root` ownership model does not yet support overlapping selected
  roots independently. Use disjoint roots until root memberships are implemented.
- Extraction runs in the caller and a large reconciliation holds one transaction.
  Bounded worker isolation, persistent jobs, and crash-resumable scans remain
  production milestones.
- OCR, media extraction, and real model inference belong to subsequent modules.
- Linux/macOS measurements, realistic mixed-document corpora, cold startup,
  memory budgets, and indexing performance remain unverified.