searchi/docs/INCREMENTAL-SEMANTIC.md

Incremental semantic indexing and recovery

Status: completed-source reuse and compatible staging recovery implemented in schema 8 (completed-source tasks were introduced in schema 7). Unchanged vectors are copied with current citations; the ANN graph is rebuilt. Incomplete source groups are discarded before retry. The previous ready generation and its reader policy remain available on failures.

CPU tests use the normal hardware-detecting runtime forced to CPU. Recorded integration evidence and remaining gates are below. This is not a production readiness or media relevance claim.

Reuse requires complete sources

Document vectors depend on the extracted text, the bounded filename title, model, prompt and token-window policy. The current passage content hash alone is insufficient: a basename rename changes Google's document prompt. Passage IDs are recreated by core reconciliation, so reuse must remap current IDs, paths and page/line locations rather than retain old citations.

Media reuse requires a verified hash of the actual file bytes and matching encoder/preprocessing/sampling policy. Size and modification time alone are insufficient. Existing media chunks are written progressively; their presence does not prove that a file completed. A crash can leave an incomplete group.

The versioned completion manifest records per source group: generation, task key, source type, encoder/policy key, source hash, title hash where applicable, expected chunk count and completion state. Associate chunks with their group. Only completed groups can supply reusable vectors. Historical generations without completion evidence remain searchable but require one rebuild before their vectors qualify for reuse.

Start with matching complete-generation policies. Component encoder keys can later let unchanged document vectors survive an audio-sampling policy change. Do not silently omit image, audio or video work to make a rebuild cheaper.

Durable work and atomic publication

Persist bounded pending/running/done/failed tasks in SQLite. Tasks retain their source identity, attempts and a bounded error. Obtain the existing OS build lock before starting or resuming a compatible staging generation; only then can interrupted running tasks be reset. Keep inference outside write transactions. Commit source completion, verified chunk membership and task state atomically. Incomplete groups must be discarded or resumed without qualifying as completed reuse sources.

Discover work in bounded keyset batches. Before publication, reconcile staging against current indexed sources: remove deleted sources, invalidate changed identities and enqueue additions. Verify exact document window slices and remap citations. For media, confirm the file is unchanged after extraction and inference. Retain the 100,000-chunk budget and explicit per-file media failures.

Build the ANN graph only from eligible completed groups. Publish under a transaction that checks the reconciled corpus revision; a concurrent source change must trigger reconciliation/retry. Storage or activation failure must preserve both the previous active generation and the same reader's policy.

Acceptance gates

• Schema-6 storage migration preserves active-generation search; legacy chunks without completion evidence cannot supply reusable vectors.
• Rebuild mixed text/code/image/audio/video fixtures unchanged with zero document/media inference calls and identical vectors and source locations.
• Change one document and one image; re-encode only affected groups. Image regression preserves file size and exact mtime to exercise SHA identity.
• Re-encode text after a basename rename; reuse a folder-only move with current citations.
• Verify media bytes and current citations after a media-file rename.
• Regenerate affected media when sampling policy changes; retain eligible text groups when component encoder-key reuse is implemented.
• Terminate after a committed batch and resume without repeating completed inference. Never reuse or publish an unfinished source group.
• Same-size/mtime text mutation during genuine inference rejects publication; a mutation before rebuilding fails before text encoding and ordinary rescans reextract it. Unaffected vectors survive resumed retry.
• Concurrent deletion/media mutation and comprehensive queued/running source search freshness tests.
• Inject activation failure, preserve existing results and resume completed staging with zero document inference; cleanup-failure unit regression retains the primary error and old reader policy.
• Use genuine Google CPU inference for reuse integration checks. Storage fixtures alone do not establish retrieval quality or reuse equivalence.
• Measure preparation, the first foreground query and warm p95 while resumed indexing runs; record actual inference/reuse counts and resource use.

Foreground query priority and bounded persistent workers remain part of the first-use requirement. Faster unchanged rebuilds alone do not prove responsive search while indexing.

Current limits

Reuse requires the complete generation policy to match. Component-level reuse across media sampling changes remains open. Durable staging resumes at completed source boundaries; a real process-kill test during a multi-batch source remains open. Storage tests cover discarded partial groups and corpus revision races.

Full source hashes are checked after ANN construction and serialization, before opening the publication write transaction. Indexed changes are also rejected by the transaction's revision guard. This narrows the filesystem race; it does not make file reads and SQLite publication one atomic filesystem operation.

Core schema 12 records the SHA-256 of the exact bounded bytes consumed by each parser. Semantic builds verify that provenance before reusing or encoding text. Same-size/mtime changes are detected by normal folder rescanning. Legacy records retain lexical search but need a rescan before semantic rebuilding or serving agent excerpts. Excerpts reject missing or mismatched source provenance.

Bounded discovery now prunes obsolete resumed groups before inference. Compatible renamed staging groups survive with current memberships through a transactional, disk-backed SQLite snapshot; active vectors are untouched. Genuine duplicate current groups still count toward the 100,000-chunk ceiling. Hashing, discovery, media metadata probes, snapshot copying and graph rebuilding have costs: zero encoder calls does not mean zero work or prove fast first input.

Synthetic colored-image and sine-wave fixtures prove actual vector reuse and citation equality; they do not establish useful media relevance.

Current Windows CPU integration rerun, 2026-10-13

At source 66bf7d7b4757c1454e4cd61e514ff9376bddd728 with a clean worktree, the normal current worker SHA-256 c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73 ran all three incremental_google_cpu tests successfully in 75.47 seconds. Actual Google inference was explicitly CPU-only with one thread. Verified Google assets and the pinned Windows FFmpeg/FFprobe helpers were used.

The checks cover completed-vector reuse after failed activation and reopen, document/code moves with preserved citations, title-change reencoding, mixed document/image/audio/video reuse, and byte-changing media overwrites with identical size/mtime. The prepared-reader test also passed in 14.04 seconds: publication preserved the real query-vector LRU and shared prepared ANN graph, returned updated source content, and a distinct query executed genuine inference. Observed service times were 2.962 ms for the cached query and 84.808 ms for the distinct query, on a one-document fixture. These are debug-profile integration observations, not desktop first-input, scale or release latency acceptance.

Command: cargo test -p searchi-service --test prepared_google_cpu --test incremental_google_cpu --locked -- --ignored --nocapture --test-threads=1, with SEARCHI_FORCE_CPU=1 and the verified model/worker/decoder environment variables described in SKILLS.md. All four tests executed, none skipped. The running installed user application was left untouched. Other platform, storage-contention and representative-corpus gates remain open.