Reuse completed Google embeddings and resume semantic staging

docs/INCREMENTAL-SEMANTIC-VALIDATION.md

@@ -0,0 +1,63 @@
+# Completed-source reuse validation
+
+2026-10-13. Windows VM, CPU-only inference (`SEARCHI_FORCE_CPU=1`), normal
+hardware-detecting worker, pinned Google EmbeddingGemma 2 backbone/projector,
+one inference thread. FFmpeg/ffprobe are the verified app-local helpers.
+
+## Genuine Google integration
+
+`cargo test -p searchi-service --test incremental_google_cpu --test google_cpu
+--locked -- --ignored --nocapture --test-threads=1` executes actual models.
+A counting adapter forwards every operation to `GemmaEmbedder`; no synthetic
+vectors satisfy the integration assertions.
+
+- Document and code initial encoding: two calls. Unchanged rebuild and folder
+  move: zero extra calls, bit-identical vectors, remapped current paths/locations.
+  A basename rename adds one call because the document title is in Google's
+  prompt. A single content change adds one call; unrelated code stays identical.
+- Mixed document/code/image/audio/video fixture: five source groups. Unchanged
+  rebuild makes zero additional document/image/audio encoder calls and retains
+  exact vector bytes and citations. Replacing the BMP while preserving its
+  size and exact mtime adds one image call; the other four groups stay identical.
+- Rejected activation retains the prior active generation. Reopening resumes
+  completed changed-document vectors with zero document calls; updated text
+  citations are returned after publication. Existing policy-change regression
+  covers lock admission, generation creation, repeated activation failures,
+  same-instance reader policy and resumed retry in both directions.
+
+Synthetic colors/sine waves in media fixtures prove reuse, not media relevance.
+Tests are correctness checks, not latency or throughput benchmarks.
+
+## Storage and failure boundaries
+
+Four unit regressions cover schema-6 legacy search without unsafe reuse,
+completed versus partial restart state, revision-checked publication,
+transactional completion failure and malformed stored text windows. Another
+fault-only pipeline test covers primary inference error preservation when
+partial-source cleanup also fails. Its provider returns no simulated vectors.
+
+## Executed checks
+
+- Windows workspace: 98 passed, 15 explicitly ignored; no failures.
+- Native Ubuntu 24.04 headless workspace: 98 passed, 15 explicitly ignored.
+- Windows native desktop unit suite: 17 passed.
+- Strict Windows/Linux workspace and Windows native-desktop Clippy passed;
+  Windows workspace/desktop formatting passed.
+- Final-source Windows real Google tests above: five passed. The two existing
+  lifecycle/policy tests took 25.31 s and the three incremental tests took
+  138.90 s. Concurrent compilation/resource contention was present; these
+  elapsed suite times are not application search benchmarks.
+- Final-source native Linux actual Google integration: the same five tests
+  passed; existing lifecycle/policy suite 49.77 s, incremental suite 173.59 s.
+  Ubuntu 24.04 WSL x86-64 with two allocated CPUs, one inference thread,
+  isolated verified models/normal auto worker and pinned native FFmpeg helpers.
+  Cold Clippy compilation ran concurrently; these are not latency comparisons.
+
+## Remaining gates
+
+Actual process termination during a multi-batch source, simultaneous real-model
+source mutation, wider policy changes, near-limit pruning, minimum-ABI/older-CPU
+checks and first-query/foreground latency during indexing remain open. Full file
+hash validation currently precedes ANN construction; direct file mutation during
+that construction before watcher reconciliation needs a later validation callback.
+See [INCREMENTAL-SEMANTIC.md](INCREMENTAL-SEMANTIC.md).

docs/INCREMENTAL-SEMANTIC.md

@@ -1,8 +1,13 @@
 # Incremental semantic indexing and recovery
 
-Status: implementation design and acceptance gates; not implemented. Current
-semantic rebuilds regenerate vectors and the ANN graph. Interrupted staging is
-discarded by the next rebuild. The previous ready generation remains searchable.
+Status: completed-source reuse and compatible staging recovery implemented in
+schema 7. Unchanged vectors are copied with current citations; the ANN graph
+is rebuilt. Incomplete source groups are discarded before retry. The previous
+ready generation and its reader policy remain available on failures.
+
+CPU tests use the normal hardware-detecting runtime forced to CPU.
+Recorded integration evidence and remaining gates are below. This is not a
+production readiness or media relevance claim.
 
 ## Reuse requires complete sources
 
@@ -17,7 +22,7 @@ encoder/preprocessing/sampling policy. Size and modification time alone are
 insufficient. Existing media chunks are written progressively; their presence
 does not prove that a file completed. A crash can leave an incomplete group.
 
-Add a versioned completion manifest per source group: generation, task key,
+The versioned completion manifest records per source group: generation, task key,
 source type, encoder/policy key, source hash, title hash where applicable,
 expected chunk count and completion state. Associate chunks with their group.
 Only completed groups can supply reusable vectors. Historical generations
@@ -51,20 +56,25 @@ preserve both the previous active generation and the same reader's policy.
 
 ## Acceptance gates
 
-- [ ] Migrate existing databases while preserving active-generation search.
-- [ ] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
+- [x] Schema-6 storage migration preserves active-generation search; legacy
+  chunks without completion evidence cannot supply reusable vectors.
+- [x] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
   document/media inference calls and identical vectors and source locations.
-- [ ] Change one document and one media file; re-encode only affected groups.
-- [ ] Re-encode text after a basename rename; reuse a folder-only move with
-  current citations. Verify media bytes before reusing a renamed media file.
+- [x] Change one document and one image; re-encode only affected groups.
+  Image regression preserves file size and exact mtime to exercise SHA identity.
+- [x] Re-encode text after a basename rename; reuse a folder-only move with
+  current citations.
+- [ ] Verify media bytes and current citations after a media-file rename.
 - [ ] Regenerate affected media when sampling policy changes; retain eligible
   text groups when component encoder-key reuse is implemented.
 - [ ] Terminate after a committed batch and resume without repeating completed
   inference. Never reuse or publish an unfinished source group.
 - [ ] Mutate or delete a queued/running source and prove stale vectors cannot
   activate or become readable.
-- [ ] Inject storage/activation failure, preserve existing results and resume.
-- [ ] Use genuine Google CPU inference for these integration checks. Storage
+- [x] Inject activation failure, preserve existing results and resume completed
+  staging with zero document inference; cleanup-failure unit regression retains
+  the primary error and old reader policy.
+- [x] Use genuine Google CPU inference for reuse integration checks. Storage
   fixtures alone do not establish retrieval quality or reuse equivalence.
 - [ ] Measure preparation, the first foreground query and warm p95 while
   resumed indexing runs; record actual inference/reuse counts and resource use.
@@ -72,3 +82,24 @@ preserve both the previous active generation and the same reader's policy.
 Foreground query priority and bounded persistent workers remain part of the
 first-use requirement. Faster unchanged rebuilds alone do not prove responsive
 search while indexing.
+
+## Current limits
+
+Reuse requires the complete generation policy to match. Component-level reuse
+across media sampling changes remains open. Durable staging resumes at completed
+source boundaries; a real process-kill test during a multi-batch source remains
+open. Storage tests cover discarded partial groups and corpus revision races.
+
+Full source hashes are checked before ANN construction; indexed changes during
+ANN construction are rejected by the final transaction's revision guard. A direct
+filesystem mutation during ANN construction before watcher reconciliation still
+needs a later validation callback and concurrent real-model regression.
+
+Obsolete completed groups in resumed staging are removed at final reconciliation.
+Near the 100,000-chunk ceiling, replacement work can temporarily hit the budget
+before this cleanup and fail safely. Bounded discovery/pruning before inference
+is the next improvement. Hashing, media metadata probes and graph rebuilding still
+have costs: zero encoder calls does not mean zero work or prove fast first input.
+
+Synthetic colored-image and sine-wave fixtures prove actual vector reuse and
+citation equality; they do not establish useful media relevance.

docs/ROADMAP.md

@@ -100,7 +100,7 @@ Status: implementation started; no production release yet. Windows VM tests use
   Warm p95 alone does not satisfy the first-use requirement.
 - [ ] Test migrations, interrupted work, invalid documents, inaccessible/offline roots and cloud placeholders.
 - [ ] Automated native build matrix and documented supported cross-compilation paths.
-- [x] Native Linux headless workspace tests: 93 passed, 11 explicitly ignored.
+- [x] Native Linux headless workspace tests: 98 passed, 15 explicitly ignored.
 - [x] Native Linux Google text/code/media reference parity and service
   lifecycle/rebuild recovery: seven genuine CPU integration tests passed.
 - [x] Unsigned Linux x86-64 headless archive: verify extracted checksums/modes,
@@ -143,7 +143,13 @@ Audit and measured priorities: [EMBEDDINGGEMMA2-AUDIT.md](EMBEDDINGGEMMA2-AUDIT.
 - [x] Windows x64 baseline/optimized CPU runtime selection and OS AVX-state
   checks, verified with real packaged baseline and optimized inference.
 - [ ] Validate CPU selection on older hardware and remaining architectures.
-- [ ] Incrementally reuse unchanged document/media vectors in atomic generations.
+- [x] Completed-source vector reuse in atomic schema-7 generations, verified
+  with genuine Google CPU document/code/image/audio/video fixtures.
+- [x] Resume completed staging sources after failed publication; incomplete
+  groups are purged under the OS build lock before retry.
+- [ ] Process-kill/concurrent-source integration, pre-inference pruning near
+  chunk limits, component-policy reuse and indexing/foreground measurements.
+  See [INCREMENTAL-SEMANTIC.md](INCREMENTAL-SEMANTIC.md).
 - [ ] Benchmark genuine token-budgeted multi-sequence batching and isolation.
 - [ ] Implement and measure shared foreground/indexer resource budgets/priority.
 - [ ] Attribute startup cost; measure smaller query context and selective encoders.