4.2 KiB
Incremental semantic indexing and recovery
Status: implementation design and acceptance gates; not implemented. Current semantic rebuilds regenerate vectors and the ANN graph. Interrupted staging is discarded by the next rebuild. The previous ready generation remains searchable.
Reuse requires complete sources
Document vectors depend on the extracted text, the bounded filename title, model, prompt and token-window policy. The current passage content hash alone is insufficient: a basename rename changes Google's document prompt. Passage IDs are recreated by core reconciliation, so reuse must remap current IDs, paths and page/line locations rather than retain old citations.
Media reuse requires a verified hash of the actual file bytes and matching encoder/preprocessing/sampling policy. Size and modification time alone are insufficient. Existing media chunks are written progressively; their presence does not prove that a file completed. A crash can leave an incomplete group.
Add a versioned completion manifest per source group: generation, task key, source type, encoder/policy key, source hash, title hash where applicable, expected chunk count and completion state. Associate chunks with their group. Only completed groups can supply reusable vectors. Historical generations without completion evidence remain searchable but require one rebuild before their vectors qualify for reuse.
Start with matching complete-generation policies. Component encoder keys can later let unchanged document vectors survive an audio-sampling policy change. Do not silently omit image, audio or video work to make a rebuild cheaper.
Durable work and atomic publication
Persist bounded pending/running/done/failed tasks in SQLite. Tasks retain their source identity, attempts and a bounded error. Obtain the existing OS build lock before starting or resuming a compatible staging generation; only then can interrupted running tasks be reset. Keep inference outside write transactions. Commit source completion, verified chunk membership and task state atomically. Incomplete groups must be discarded or resumed without qualifying as completed reuse sources.
Discover work in bounded keyset batches. Before publication, reconcile staging against current indexed sources: remove deleted sources, invalidate changed identities and enqueue additions. Verify exact document window slices and remap citations. For media, confirm the file is unchanged after extraction and inference. Retain the 100,000-chunk budget and explicit per-file media failures.
Build the ANN graph only from eligible completed groups. Publish under a transaction that checks the reconciled corpus revision; a concurrent source change must trigger reconciliation/retry. Storage or activation failure must preserve both the previous active generation and the same reader's policy.
Acceptance gates
- Migrate existing databases while preserving active-generation search.
- Rebuild mixed text/code/image/audio/video fixtures unchanged with zero document/media inference calls and identical vectors and source locations.
- Change one document and one media file; re-encode only affected groups.
- Re-encode text after a basename rename; reuse a folder-only move with current citations. Verify media bytes before reusing a renamed media file.
- Regenerate affected media when sampling policy changes; retain eligible text groups when component encoder-key reuse is implemented.
- Terminate after a committed batch and resume without repeating completed inference. Never reuse or publish an unfinished source group.
- Mutate or delete a queued/running source and prove stale vectors cannot activate or become readable.
- Inject storage/activation failure, preserve existing results and resume.
- Use genuine Google CPU inference for these integration checks. Storage fixtures alone do not establish retrieval quality or reuse equivalence.
- Measure preparation, the first foreground query and warm p95 while resumed indexing runs; record actual inference/reuse counts and resource use.
Foreground query priority and bounded persistent workers remain part of the first-use requirement. Faster unchanged rebuilds alone do not prove responsive search while indexing.