Verify extraction provenance and prune semantic staging before inference

docs/INCREMENTAL-SEMANTIC-VALIDATION.md

@@ -36,7 +36,7 @@ transactional completion failure and malformed stored text windows. Another
 fault-only pipeline test covers primary inference error preservation when
 partial-source cleanup also fails. Its provider returns no simulated vectors.
 
-## Executed checks
+## Original reuse commit checks (5c18e7b)
 
 - Windows workspace: 98 passed, 15 explicitly ignored; no failures.
 - Native Ubuntu 24.04 headless workspace: 98 passed, 15 explicitly ignored.
@@ -53,11 +53,50 @@ partial-source cleanup also fails. Its provider returns no simulated vectors.
   isolated verified models/normal auto worker and pinned native FFmpeg helpers.
   Cold Clippy compilation ran concurrently; these are not latency comparisons.
 
+## Publication and provenance follow-up
+
+The follow-up moves full source validation after ANN construction/serialization,
+before opening the publication write transaction. A unit regression confirms
+an independent writer can acquire the database while validation runs; rejection
+keeps the previous fingerprint/cache and completed resumable staging.
+
+Pre-inference pruning uses a bounded, disk-backed transactional SQLite snapshot.
+The 100,000-chunk regression checks renamed completed staging without duplicate
+allocation, genuine duplicate current groups exceeding the budget, rollback,
+obsolete replacement cleanup and preservation of active vectors. This is a
+storage fixture, not an ANN latency or neural relevance benchmark.
+
+Core schema 12 hashes the exact bounded parser input, without reopening the file
+for parsing. Four tests cover immutable input snapshots, same-size/mtime mutation,
+real schema-11 migration preserving lexical content, stale agent excerpts and
+filename-only oversized/unsupported inputs. Supported rescans read and hash
+bounded bytes even when metadata is unchanged; matching hashes skip parsing.
+Legacy rows need reindexing to establish provenance before semantic rebuilding
+or serving agent excerpts.
+
+## Follow-up executed checks
+
+- Windows and native Linux workspaces: 104 passed each, 16 explicitly ignored; no failures.
+- Windows desktop: 17 passed.
+- Strict Windows/Linux workspace and Windows desktop Clippy passed; workspace formatting passed.
+- Six genuine Windows Google CPU tests passed: existing lifecycle/policy tests
+  32.52 s; mixed incremental tests 182.03 s; new publication mutation test
+  11.02 s. Compiler/resource contention was present. These are correctness-suite
+  times and are not search-latency measurements.
+- The same six genuine native Linux Google CPU tests passed: lifecycle/policy
+  40.16 s, incremental/media 135.42 s, publication mutation 11.98 s. These
+  elapsed suites are correctness evidence, not comparative performance claims.
+- The mutation test overwrites the file after actual document inference,
+  restores the exact mtime, rejects publication and preserves the active ID.
+  An ordinary folder rescan then resumes with only the changed source encoded.
+  Another overwrite before rebuilding rejects stale extraction before any
+  additional document encoder calls. Unaffected stored vector bytes and
+  citations remain identical.
+
 ## Remaining gates
 
-Actual process termination during a multi-batch source, simultaneous real-model
-source mutation, wider policy changes, near-limit pruning, minimum-ABI/older-CPU
-checks and first-query/foreground latency during indexing remain open. Full file
-hash validation currently precedes ANN construction; direct file mutation during
-that construction before watcher reconciliation needs a later validation callback.
+Actual process termination during a multi-batch source, wider policy changes,
+minimum-ABI/older-CPU checks and first-query/foreground latency during indexing
+remain open. Live filesystem reads and SQLite publication are not one atomic
+operation; the later check narrows that race rather than eliminating it.
 See [INCREMENTAL-SEMANTIC.md](INCREMENTAL-SEMANTIC.md).

docs/INCREMENTAL-SEMANTIC.md

@@ -69,8 +69,11 @@ preserve both the previous active generation and the same reader's policy.
   text groups when component encoder-key reuse is implemented.
 - [ ] Terminate after a committed batch and resume without repeating completed
   inference. Never reuse or publish an unfinished source group.
-- [ ] Mutate or delete a queued/running source and prove stale vectors cannot
-  activate or become readable.
+- [x] Same-size/mtime text mutation during genuine inference rejects publication;
+  a mutation before rebuilding fails before text encoding and ordinary rescans
+  reextract it. Unaffected vectors survive resumed retry.
+- [ ] Concurrent deletion/media mutation and comprehensive queued/running source
+  search freshness tests.
 - [x] Inject activation failure, preserve existing results and resume completed
   staging with zero document inference; cleanup-failure unit regression retains
   the primary error and old reader policy.
@@ -90,16 +93,23 @@ across media sampling changes remains open. Durable staging resumes at completed
 source boundaries; a real process-kill test during a multi-batch source remains
 open. Storage tests cover discarded partial groups and corpus revision races.
 
-Full source hashes are checked before ANN construction; indexed changes during
-ANN construction are rejected by the final transaction's revision guard. A direct
-filesystem mutation during ANN construction before watcher reconciliation still
-needs a later validation callback and concurrent real-model regression.
+Full source hashes are checked after ANN construction and serialization, before
+opening the publication write transaction. Indexed changes are also rejected by
+the transaction's revision guard. This narrows the filesystem race; it does not
+make file reads and SQLite publication one atomic filesystem operation.
 
-Obsolete completed groups in resumed staging are removed at final reconciliation.
-Near the 100,000-chunk ceiling, replacement work can temporarily hit the budget
-before this cleanup and fail safely. Bounded discovery/pruning before inference
-is the next improvement. Hashing, media metadata probes and graph rebuilding still
-have costs: zero encoder calls does not mean zero work or prove fast first input.
+Core schema 12 records the SHA-256 of the exact bounded bytes consumed by each
+parser. Semantic builds verify that provenance before reusing or encoding text.
+Same-size/mtime changes are detected by normal folder rescanning. Legacy records
+retain lexical search but need a rescan before semantic rebuilding or serving
+agent excerpts. Excerpts reject missing or mismatched source provenance.
+
+Bounded discovery now prunes obsolete resumed groups before inference. Compatible
+renamed staging groups survive with current memberships through a transactional,
+disk-backed SQLite snapshot; active vectors are untouched. Genuine duplicate
+current groups still count toward the 100,000-chunk ceiling. Hashing, discovery,
+media metadata probes, snapshot copying and graph rebuilding have costs: zero
+encoder calls does not mean zero work or prove fast first input.
 
 Synthetic colored-image and sine-wave fixtures prove actual vector reuse and
 citation equality; they do not establish useful media relevance.

docs/ROADMAP.md

@@ -100,7 +100,7 @@ Status: implementation started; no production release yet. Windows VM tests use
   Warm p95 alone does not satisfy the first-use requirement.
 - [ ] Test migrations, interrupted work, invalid documents, inaccessible/offline roots and cloud placeholders.
 - [ ] Automated native build matrix and documented supported cross-compilation paths.
-- [x] Native Linux headless workspace tests: 98 passed, 15 explicitly ignored.
+- [x] Native Linux headless workspace tests: 104 passed, 16 explicitly ignored.
 - [x] Native Linux Google text/code/media reference parity and service
   lifecycle/rebuild recovery: seven genuine CPU integration tests passed.
 - [x] Unsigned Linux x86-64 headless archive: verify extracted checksums/modes,
@@ -147,8 +147,14 @@ Audit and measured priorities: [EMBEDDINGGEMMA2-AUDIT.md](EMBEDDINGGEMMA2-AUDIT.
   with genuine Google CPU document/code/image/audio/video fixtures.
 - [x] Resume completed staging sources after failed publication; incomplete
   groups are purged under the OS build lock before retry.
-- [ ] Process-kill/concurrent-source integration, pre-inference pruning near
-  chunk limits, component-policy reuse and indexing/foreground measurements.
+- [x] Pre-inference staging pruning with compatible rename remapping and
+  100,000-chunk storage boundary; late validation after ANN preparation.
+- [x] Exact parser-input SHA provenance and same-metadata overwrite detection
+  on normal rescans; stale/missing agent excerpts fail closed.
+- [x] Actual Google CPU same-metadata text mutation before/during inference:
+  reject publication, rescan without metadata workarounds and reuse unaffected vectors.
+- [ ] Process-kill/concurrent deletion/media mutation, component-policy reuse and
+  indexing/foreground measurements.
   See [INCREMENTAL-SEMANTIC.md](INCREMENTAL-SEMANTIC.md).
 - [ ] Benchmark genuine token-budgeted multi-sequence batching and isolation.
 - [ ] Implement and measure shared foreground/indexer resource budgets/priority.