Reuse completed Google embeddings and resume semantic staging

docs/INCREMENTAL-SEMANTIC.md

@@ -1,8 +1,13 @@
 # Incremental semantic indexing and recovery
 
-Status: implementation design and acceptance gates; not implemented. Current
-semantic rebuilds regenerate vectors and the ANN graph. Interrupted staging is
-discarded by the next rebuild. The previous ready generation remains searchable.
+Status: completed-source reuse and compatible staging recovery implemented in
+schema 7. Unchanged vectors are copied with current citations; the ANN graph
+is rebuilt. Incomplete source groups are discarded before retry. The previous
+ready generation and its reader policy remain available on failures.
+
+CPU tests use the normal hardware-detecting runtime forced to CPU.
+Recorded integration evidence and remaining gates are below. This is not a
+production readiness or media relevance claim.
 
 ## Reuse requires complete sources
 
@@ -17,7 +22,7 @@ encoder/preprocessing/sampling policy. Size and modification time alone are
 insufficient. Existing media chunks are written progressively; their presence
 does not prove that a file completed. A crash can leave an incomplete group.
 
-Add a versioned completion manifest per source group: generation, task key,
+The versioned completion manifest records per source group: generation, task key,
 source type, encoder/policy key, source hash, title hash where applicable,
 expected chunk count and completion state. Associate chunks with their group.
 Only completed groups can supply reusable vectors. Historical generations
@@ -51,20 +56,25 @@ preserve both the previous active generation and the same reader's policy.
 
 ## Acceptance gates
 
-- [ ] Migrate existing databases while preserving active-generation search.
-- [ ] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
+- [x] Schema-6 storage migration preserves active-generation search; legacy
+  chunks without completion evidence cannot supply reusable vectors.
+- [x] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
   document/media inference calls and identical vectors and source locations.
-- [ ] Change one document and one media file; re-encode only affected groups.
-- [ ] Re-encode text after a basename rename; reuse a folder-only move with
-  current citations. Verify media bytes before reusing a renamed media file.
+- [x] Change one document and one image; re-encode only affected groups.
+  Image regression preserves file size and exact mtime to exercise SHA identity.
+- [x] Re-encode text after a basename rename; reuse a folder-only move with
+  current citations.
+- [ ] Verify media bytes and current citations after a media-file rename.
 - [ ] Regenerate affected media when sampling policy changes; retain eligible
   text groups when component encoder-key reuse is implemented.
 - [ ] Terminate after a committed batch and resume without repeating completed
   inference. Never reuse or publish an unfinished source group.
 - [ ] Mutate or delete a queued/running source and prove stale vectors cannot
   activate or become readable.
-- [ ] Inject storage/activation failure, preserve existing results and resume.
-- [ ] Use genuine Google CPU inference for these integration checks. Storage
+- [x] Inject activation failure, preserve existing results and resume completed
+  staging with zero document inference; cleanup-failure unit regression retains
+  the primary error and old reader policy.
+- [x] Use genuine Google CPU inference for reuse integration checks. Storage
   fixtures alone do not establish retrieval quality or reuse equivalence.
 - [ ] Measure preparation, the first foreground query and warm p95 while
   resumed indexing runs; record actual inference/reuse counts and resource use.
@@ -72,3 +82,24 @@ preserve both the previous active generation and the same reader's policy.
 Foreground query priority and bounded persistent workers remain part of the
 first-use requirement. Faster unchanged rebuilds alone do not prove responsive
 search while indexing.
+
+## Current limits
+
+Reuse requires the complete generation policy to match. Component-level reuse
+across media sampling changes remains open. Durable staging resumes at completed
+source boundaries; a real process-kill test during a multi-batch source remains
+open. Storage tests cover discarded partial groups and corpus revision races.
+
+Full source hashes are checked before ANN construction; indexed changes during
+ANN construction are rejected by the final transaction's revision guard. A direct
+filesystem mutation during ANN construction before watcher reconciliation still
+needs a later validation callback and concurrent real-model regression.
+
+Obsolete completed groups in resumed staging are removed at final reconciliation.
+Near the 100,000-chunk ceiling, replacement work can temporarily hit the budget
+before this cleanup and fail safely. Bounded discovery/pruning before inference
+is the next improvement. Hashing, media metadata probes and graph rebuilding still
+have costs: zero encoder calls does not mean zero work or prove fast first input.
+
+Synthetic colored-image and sine-wave fixtures prove actual vector reuse and
+citation equality; they do not establish useful media relevance.