feat: add desktop document search and verified native CPU embeddings

crates/searchi-core/README.md

@@ -4,10 +4,14 @@ The core has no desktop or inference dependencies. Its current retrieval engine
 combines literal filename matching with SQLite FTS5 BM25 keyword search. It does
 not implement semantic search.
 
-`SearchIndex::open` creates or migrates schema version 2. Current-schema opens do
+`SearchIndex::open` creates or migrates schema version 4. Current-schema opens do
 not acquire a write lock. WAL allows independent readers to use the committed
 index while reconciliation runs. All indexing changes are committed atomically;
-a traversal or supported-file read error rolls back the entire reconciliation.
+a traversal or plain-text file read error rolls back the entire reconciliation.
+PDF/DOCX parser or limit failures retain a filename record with a persisted
+`SearchResult.extraction_error`, clear stale content, and increment
+`IndexReport.failures`. This counts attempts in this reconciliation, so an
+unchanged failed record retains its reason without incrementing the count again.
 
 `index_folder` applies `IndexOptions::default()`: directories named `.git`,
 `node_modules`, and `target` are pruned. `index_folder_with_options` lets callers
@@ -23,6 +27,46 @@ or size. Unsupported files remain searchable by filename. Content reads are
 limited to 2 MiB. Reconciliation tracks seen files in SQLite rather than retaining
 every path in process memory.
 
+PDF text extraction uses `lopdf` with strict parsing and bounded load/page/font
+stream decompression. DOCX uses `zip` and namespace-aware `quick-xml` to read the
+main document body, including table paragraphs. The extractor does not execute
+macros or follow external document relationships. It supports UTF-8 DOCX XML;
+UTF-16 document parts, nested paragraphs, and XML document types are explicit
+failures. Headers, footers, comments, drawings, and tracked-deletion semantics
+are not yet covered. Encrypted PDFs are explicitly unsupported. Scanned PDFs
+without a text layer remain filename searchable; OCR is not implemented.
+
+| Limit | Value |
+| --- | --- |
+| PDF/DOCX file input, before parsing | 32 MiB |
+| PDF load stream / page content / font decoding | 4 MiB per stream or page budget |
+| PDF page count, after parsing | 1,000 |
+| DOCX main XML, including decompressed ZIP output | 8 MiB |
+| DOCX ZIP entries / paragraphs | 10,000 each |
+| DOCX XML depth | 128 |
+| Extracted text per document | 2 MiB |
+
+Input reads and DOCX XML decompression are capped while reading. PDF's loader and
+text extractor use the library's bounded decoding APIs. Page/entry counts and
+total PDF extracted text are checked after the parser produces them. These are
+not global process memory or CPU-time limits: multiple PDF objects/fonts can each
+consume their budget, and parsing may still be expensive. Process isolation and
+timeouts remain required before handling arbitrary untrusted documents in a
+production release.
+
+Actual page/paragraph text is stored separately in an FTS passage index. Search
+results use the best passage that matches all literal query terms and expose
+`source: { kind, number }`, with one-based PDF page, DOCX paragraph, or plain-text
+line-range start. Paragraph numbers include empty paragraphs. A filename-only
+match or a match spanning several passages can have no source; the engine does
+not invent a location. Snippets with a source are drawn from that passage.
+
+`passage_batch(after_id, limit)` exposes at most 256 `IndexedPassage` records for
+real embedding runtimes. IDs represent current rows and can be reused after
+reconciliation; downstream embedding caches must validate content and model
+fingerprints. Consumers must tokenize long pages/paragraphs into overlapping
+windows rather than silently truncate them to a model token limit.
+
 `search_with_options` supports canonical folder boundaries, extension, inclusive
 size bounds, and inclusive Unix millisecond modification-time bounds. Filters
 apply before limiting results. `search` retains the unfiltered API. Query text is
@@ -46,4 +90,6 @@ escaped as literal FTS terms, not interpreted as operators. Results are capped a
 - Extraction runs in the caller and a large reconciliation holds one transaction.
   Bounded worker isolation, persistent jobs, and crash-resumable scans remain
   production milestones.
-- PDF, DOCX, OCR, media extraction, and real model inference remain unimplemented.
+- OCR, media extraction, and real model inference belong to subsequent modules.
+- Passage selection currently adds a bounded query per result; query batching and
+  passage ranking benchmarks remain necessary for large result lists.