Freeze mixed-file retrieval diagnostic and shared-engine evaluator

fixtures/mixed/sources/docs__INCREMENTAL-SEMANTIC.md

@@ -0,0 +1,115 @@
+# Incremental semantic indexing and recovery
+
+Status: completed-source reuse and compatible staging recovery implemented in
+schema 8 (completed-source tasks were introduced in schema 7). Unchanged vectors are copied with current citations; the ANN graph
+is rebuilt. Incomplete source groups are discarded before retry. The previous
+ready generation and its reader policy remain available on failures.
+
+CPU tests use the normal hardware-detecting runtime forced to CPU.
+Recorded integration evidence and remaining gates are below. This is not a
+production readiness or media relevance claim.
+
+## Reuse requires complete sources
+
+Document vectors depend on the extracted text, the bounded filename title,
+model, prompt and token-window policy. The current passage content hash alone
+is insufficient: a basename rename changes Google's document prompt. Passage
+IDs are recreated by core reconciliation, so reuse must remap current IDs,
+paths and page/line locations rather than retain old citations.
+
+Media reuse requires a verified hash of the actual file bytes and matching
+encoder/preprocessing/sampling policy. Size and modification time alone are
+insufficient. Existing media chunks are written progressively; their presence
+does not prove that a file completed. A crash can leave an incomplete group.
+
+The versioned completion manifest records per source group: generation, task key,
+source type, encoder/policy key, source hash, title hash where applicable,
+expected chunk count and completion state. Associate chunks with their group.
+Only completed groups can supply reusable vectors. Historical generations
+without completion evidence remain searchable but require one rebuild before
+their vectors qualify for reuse.
+
+Start with matching complete-generation policies. Component encoder keys can
+later let unchanged document vectors survive an audio-sampling policy change.
+Do not silently omit image, audio or video work to make a rebuild cheaper.
+
+## Durable work and atomic publication
+
+Persist bounded pending/running/done/failed tasks in SQLite. Tasks retain their
+source identity, attempts and a bounded error. Obtain the existing OS build
+lock before starting or resuming a compatible staging generation; only then
+can interrupted running tasks be reset. Keep inference outside write
+transactions. Commit source completion, verified chunk membership and task
+state atomically. Incomplete groups must be discarded or resumed without
+qualifying as completed reuse sources.
+
+Discover work in bounded keyset batches. Before publication, reconcile staging
+against current indexed sources: remove deleted sources, invalidate changed
+identities and enqueue additions. Verify exact document window slices and
+remap citations. For media, confirm the file is unchanged after extraction and
+inference. Retain the 100,000-chunk budget and explicit per-file media failures.
+
+Build the ANN graph only from eligible completed groups. Publish under a
+transaction that checks the reconciled corpus revision; a concurrent source
+change must trigger reconciliation/retry. Storage or activation failure must
+preserve both the previous active generation and the same reader's policy.
+
+## Acceptance gates
+
+- [x] Schema-6 storage migration preserves active-generation search; legacy
+  chunks without completion evidence cannot supply reusable vectors.
+- [x] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
+  document/media inference calls and identical vectors and source locations.
+- [x] Change one document and one image; re-encode only affected groups.
+  Image regression preserves file size and exact mtime to exercise SHA identity.
+- [x] Re-encode text after a basename rename; reuse a folder-only move with
+  current citations.
+- [ ] Verify media bytes and current citations after a media-file rename.
+- [ ] Regenerate affected media when sampling policy changes; retain eligible
+  text groups when component encoder-key reuse is implemented.
+- [ ] Terminate after a committed batch and resume without repeating completed
+  inference. Never reuse or publish an unfinished source group.
+- [x] Same-size/mtime text mutation during genuine inference rejects publication;
+  a mutation before rebuilding fails before text encoding and ordinary rescans
+  reextract it. Unaffected vectors survive resumed retry.
+- [ ] Concurrent deletion/media mutation and comprehensive queued/running source
+  search freshness tests.
+- [x] Inject activation failure, preserve existing results and resume completed
+  staging with zero document inference; cleanup-failure unit regression retains
+  the primary error and old reader policy.
+- [x] Use genuine Google CPU inference for reuse integration checks. Storage
+  fixtures alone do not establish retrieval quality or reuse equivalence.
+- [ ] Measure preparation, the first foreground query and warm p95 while
+  resumed indexing runs; record actual inference/reuse counts and resource use.
+
+Foreground query priority and bounded persistent workers remain part of the
+first-use requirement. Faster unchanged rebuilds alone do not prove responsive
+search while indexing.
+
+## Current limits
+
+Reuse requires the complete generation policy to match. Component-level reuse
+across media sampling changes remains open. Durable staging resumes at completed
+source boundaries; a real process-kill test during a multi-batch source remains
+open. Storage tests cover discarded partial groups and corpus revision races.
+
+Full source hashes are checked after ANN construction and serialization, before
+opening the publication write transaction. Indexed changes are also rejected by
+the transaction's revision guard. This narrows the filesystem race; it does not
+make file reads and SQLite publication one atomic filesystem operation.
+
+Core schema 12 records the SHA-256 of the exact bounded bytes consumed by each
+parser. Semantic builds verify that provenance before reusing or encoding text.
+Same-size/mtime changes are detected by normal folder rescanning. Legacy records
+retain lexical search but need a rescan before semantic rebuilding or serving
+agent excerpts. Excerpts reject missing or mismatched source provenance.
+
+Bounded discovery now prunes obsolete resumed groups before inference. Compatible
+renamed staging groups survive with current memberships through a transactional,
+disk-backed SQLite snapshot; active vectors are untouched. Genuine duplicate
+current groups still count toward the 100,000-chunk ceiling. Hashing, discovery,
+media metadata probes, snapshot copying and graph rebuilding have costs: zero
+encoder calls does not mean zero work or prove fast first input.
+
+Synthetic colored-image and sine-wave fixtures prove actual vector reuse and
+citation equality; they do not establish useful media relevance.