Package portable Linux runtime and verify Google CPU search

docs/INCREMENTAL-SEMANTIC.md

@@ -0,0 +1,74 @@
+# Incremental semantic indexing and recovery
+
+Status: implementation design and acceptance gates; not implemented. Current
+semantic rebuilds regenerate vectors and the ANN graph. Interrupted staging is
+discarded by the next rebuild. The previous ready generation remains searchable.
+
+## Reuse requires complete sources
+
+Document vectors depend on the extracted text, the bounded filename title,
+model, prompt and token-window policy. The current passage content hash alone
+is insufficient: a basename rename changes Google's document prompt. Passage
+IDs are recreated by core reconciliation, so reuse must remap current IDs,
+paths and page/line locations rather than retain old citations.
+
+Media reuse requires a verified hash of the actual file bytes and matching
+encoder/preprocessing/sampling policy. Size and modification time alone are
+insufficient. Existing media chunks are written progressively; their presence
+does not prove that a file completed. A crash can leave an incomplete group.
+
+Add a versioned completion manifest per source group: generation, task key,
+source type, encoder/policy key, source hash, title hash where applicable,
+expected chunk count and completion state. Associate chunks with their group.
+Only completed groups can supply reusable vectors. Historical generations
+without completion evidence remain searchable but require one rebuild before
+their vectors qualify for reuse.
+
+Start with matching complete-generation policies. Component encoder keys can
+later let unchanged document vectors survive an audio-sampling policy change.
+Do not silently omit image, audio or video work to make a rebuild cheaper.
+
+## Durable work and atomic publication
+
+Persist bounded pending/running/done/failed tasks in SQLite. Tasks retain their
+source identity, attempts and a bounded error. Obtain the existing OS build
+lock before starting or resuming a compatible staging generation; only then
+can interrupted running tasks be reset. Keep inference outside write
+transactions. Commit source completion, verified chunk membership and task
+state atomically. Incomplete groups must be discarded or resumed without
+qualifying as completed reuse sources.
+
+Discover work in bounded keyset batches. Before publication, reconcile staging
+against current indexed sources: remove deleted sources, invalidate changed
+identities and enqueue additions. Verify exact document window slices and
+remap citations. For media, confirm the file is unchanged after extraction and
+inference. Retain the 100,000-chunk budget and explicit per-file media failures.
+
+Build the ANN graph only from eligible completed groups. Publish under a
+transaction that checks the reconciled corpus revision; a concurrent source
+change must trigger reconciliation/retry. Storage or activation failure must
+preserve both the previous active generation and the same reader's policy.
+
+## Acceptance gates
+
+- [ ] Migrate existing databases while preserving active-generation search.
+- [ ] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
+  document/media inference calls and identical vectors and source locations.
+- [ ] Change one document and one media file; re-encode only affected groups.
+- [ ] Re-encode text after a basename rename; reuse a folder-only move with
+  current citations. Verify media bytes before reusing a renamed media file.
+- [ ] Regenerate affected media when sampling policy changes; retain eligible
+  text groups when component encoder-key reuse is implemented.
+- [ ] Terminate after a committed batch and resume without repeating completed
+  inference. Never reuse or publish an unfinished source group.
+- [ ] Mutate or delete a queued/running source and prove stale vectors cannot
+  activate or become readable.
+- [ ] Inject storage/activation failure, preserve existing results and resume.
+- [ ] Use genuine Google CPU inference for these integration checks. Storage
+  fixtures alone do not establish retrieval quality or reuse equivalence.
+- [ ] Measure preparation, the first foreground query and warm p95 while
+  resumed indexing runs; record actual inference/reuse counts and resource use.
+
+Foreground query priority and bounded persistent workers remain part of the
+first-use requirement. Faster unchanged rebuilds alone do not prove responsive
+search while indexing.