The core has no desktop or inference dependencies. Its current retrieval engine combines literal filename matching with SQLite FTS5 BM25 keyword search. It does not implement semantic search.

SearchIndex::open creates or migrates schema version 7. Current-schema opens do not acquire a write lock. WAL allows independent readers to use the committed index while reconciliation runs. All indexing changes are committed atomically; a traversal or plain-text file read error rolls back the entire reconciliation. PDF/DOCX parser or limit failures retain a filename record with a persisted SearchResult.extraction_error, clear stale content, and increment IndexReport.failures. This counts attempts in this reconciliation, so an unchanged failed record retains its reason without incrementing the count again.

index_folder applies IndexOptions::default(): directories named .git, node_modules, and target are pruned. index_folder_with_options lets callers replace those names. Matching directory names are ASCII case insensitive. An explicit root is always traversed, even if its name is excluded. Symlinks are not followed. The index's own database, WAL, and shared-memory files are excluded.

Metadata includes byte size and modification time at nanosecond resolution. Unchanged files bypass extraction. IndexReport.indexed counts newly extracted or updated records; unchanged counts reused records; skipped counts new or changed records whose content could not be extracted because of format, encoding, or size. Unsupported files remain searchable by filename. Content reads are limited to 2 MiB. Reconciliation tracks seen files in SQLite rather than retaining every path in process memory.

PDF text extraction uses lopdf with strict parsing and bounded load/page/font stream decompression. DOCX uses zip and namespace-aware quick-xml to read the main document body, including table paragraphs. The extractor does not execute macros or follow external document relationships. It supports UTF-8 DOCX XML; UTF-16 document parts, nested paragraphs, and XML document types are explicit failures. Headers, footers, comments, drawings, and tracked-deletion semantics are not yet covered. Encrypted PDFs are explicitly unsupported. Scanned PDFs without a text layer remain filename searchable; OCR is not implemented.

Limit	Value
PDF/DOCX file input, before parsing	32 MiB
PDF load stream / page content / font decoding	4 MiB per stream or page budget
PDF page count, after parsing	1,000
DOCX main XML, including decompressed ZIP output	8 MiB
DOCX ZIP entries / paragraphs	10,000 each
DOCX XML depth	128
Extracted text per document	2 MiB

Input reads and DOCX XML decompression are capped while reading. PDF's loader and text extractor use the library's bounded decoding APIs. Page/entry counts and total PDF extracted text are checked after the parser produces them. These are not global process memory or CPU-time limits: multiple PDF objects/fonts can each consume their budget, and parsing may still be expensive. Process isolation and timeouts remain required before handling arbitrary untrusted documents in a production release.

Actual page/paragraph text is stored separately in an FTS passage index. Search results use the best passage that matches all literal query terms and expose source: { kind, number }, with one-based PDF page, DOCX paragraph, or plain-text line-range start. Paragraph numbers include empty paragraphs. A filename-only match or a match spanning several passages can have no source; the engine does not invent a location. Snippets with a source are drawn from that passage.

passage_batch(after_id, limit) exposes at most 256 IndexedPassage records for real embedding runtimes. IDs represent current rows and can be reused after reconciliation; downstream embedding caches must validate content and model fingerprints. Consumers must tokenize long pages/paragraphs into overlapping windows rather than silently truncate them to a model token limit.

search_with_options supports canonical folder boundaries, extension, inclusive size bounds, and inclusive Unix millisecond modification-time bounds. Filters apply before limiting results. search retains the unfiltered API. Query text is escaped as literal FTS terms, not interpreted as operators. Results are capped at 1,000 and queries at 4,096 bytes.

Retrieval runs complete SQL-filtered lanes: indexed exact filename equality, literal filename substring, literal path substring, and content-only BM25. Multiword queries also have a tokenized name/path fallback, so annual report can find annual_report.md. Single-token filename identifiers are matched literally; the text tokenizer still handles punctuation inside document content. Exact filenames always precede other lanes. Ties use persistent file row IDs, and keyword ties use relevance then row ID. Selected rows are deduplicated.

Filename/path substrings of at least three Unicode characters use an external-content trigram FTS index. The rarest known query trigram supplies a candidate superset; the whole literal substring is verified on the candidate. At most 256 unique query trigrams are inspected. This lookup budget does not cap candidate rows or hide matching results. All size/type/folder/date filters apply before each lane's SQL LIMIT. Shorter substrings retain complete metadata scans and can be slow. Broad queries under restrictive filters can also examine many matching candidates; no universal filtered-query latency claim is made.

Trigram frequencies use the FTS vocabulary and a bounded 4,096-term cache on each SearchIndex. Initial frequency lookups can be expensive; the cache is cleared after reconciliation. Keep an independent reader engine alive across searches to benefit from warm behavior. These frequency estimates affect candidate selection cost, not correctness. A source query batches all finalists' indexed passage IDs and selects actual matching passages before choosing one source per file.

Migration to the new indexes preserves file content and passage IDs and rebuilds both filename/path and passage indexes atomically. Large migrations can take time and increase database size; they must not be confused with normal current-schema opens. The measured Windows synthetic fixture grew from about 295 MiB to 605 MiB.

Known limits

Short substring queries and broad restrictive filters can still scan many metadata candidates. Cold frequency-cache fills exceed the warm filename target.
Files changed without changing size or modification time require a future forced refresh/content-fingerprint option. Timestamp precision depends on the filesystem. Indexing does not yet run a filesystem watcher.
Metadata modification times before the Unix epoch or beyond the nanosecond integer range are explicit errors. Unicode filename case folding is currently limited by SQLite's built-in lower function.
Lossy path conversion does not preserve non-Unicode native filenames. This must be addressed before Linux production support for arbitrary byte filenames.
The files.root ownership model does not yet support overlapping selected roots independently. Use disjoint roots until root memberships are implemented.
Extraction runs in the caller and a large reconciliation holds one transaction. Bounded worker isolation, persistent jobs, and crash-resumable scans remain production milestones.
OCR, media extraction, and real model inference belong to subsequent modules.
Linux/macOS measurements, realistic mixed-document corpora, cold startup, memory budgets, and indexing performance remain unverified.