Verify extraction provenance and prune semantic staging before inference

This commit is contained in:
karl committed 2026-10-13 01:14:40 +02:00
1 parent 5c18e7b49c
commit 423a1ea57d
17 files changed
+900 -78

No files matched your search

+45 -6
View File
@@ -36,7 +36,7 @@ transactional completion failure and malformed stored text windows. Another
fault-only pipeline test covers primary inference error preservation when
partial-source cleanup also fails. Its provider returns no simulated vectors.
## Executed checks
## Original reuse commit checks (5c18e7b)
- Windows workspace: 98 passed, 15 explicitly ignored; no failures.
- Native Ubuntu 24.04 headless workspace: 98 passed, 15 explicitly ignored.
@@ -53,11 +53,50 @@ partial-source cleanup also fails. Its provider returns no simulated vectors.
isolated verified models/normal auto worker and pinned native FFmpeg helpers.
Cold Clippy compilation ran concurrently; these are not latency comparisons.
## Publication and provenance follow-up
The follow-up moves full source validation after ANN construction/serialization,
before opening the publication write transaction. A unit regression confirms
an independent writer can acquire the database while validation runs; rejection
keeps the previous fingerprint/cache and completed resumable staging.
Pre-inference pruning uses a bounded, disk-backed transactional SQLite snapshot.
The 100,000-chunk regression checks renamed completed staging without duplicate
allocation, genuine duplicate current groups exceeding the budget, rollback,
obsolete replacement cleanup and preservation of active vectors. This is a
storage fixture, not an ANN latency or neural relevance benchmark.
Core schema 12 hashes the exact bounded parser input, without reopening the file
for parsing. Four tests cover immutable input snapshots, same-size/mtime mutation,
real schema-11 migration preserving lexical content, stale agent excerpts and
filename-only oversized/unsupported inputs. Supported rescans read and hash
bounded bytes even when metadata is unchanged; matching hashes skip parsing.
Legacy rows need reindexing to establish provenance before semantic rebuilding
or serving agent excerpts.
## Follow-up executed checks
- Windows and native Linux workspaces: 104 passed each, 16 explicitly ignored; no failures.
- Windows desktop: 17 passed.
- Strict Windows/Linux workspace and Windows desktop Clippy passed; workspace formatting passed.
- Six genuine Windows Google CPU tests passed: existing lifecycle/policy tests
32.52 s; mixed incremental tests 182.03 s; new publication mutation test
11.02 s. Compiler/resource contention was present. These are correctness-suite
times and are not search-latency measurements.
- The same six genuine native Linux Google CPU tests passed: lifecycle/policy
40.16 s, incremental/media 135.42 s, publication mutation 11.98 s. These
elapsed suites are correctness evidence, not comparative performance claims.
- The mutation test overwrites the file after actual document inference,
restores the exact mtime, rejects publication and preserves the active ID.
An ordinary folder rescan then resumes with only the changed source encoded.
Another overwrite before rebuilding rejects stale extraction before any
additional document encoder calls. Unaffected stored vector bytes and
citations remain identical.
## Remaining gates
Actual process termination during a multi-batch source, simultaneous real-model
source mutation, wider policy changes, near-limit pruning, minimum-ABI/older-CPU
checks and first-query/foreground latency during indexing remain open. Full file
hash validation currently precedes ANN construction; direct file mutation during
that construction before watcher reconciliation needs a later validation callback.
Actual process termination during a multi-batch source, wider policy changes,
minimum-ABI/older-CPU checks and first-query/foreground latency during indexing
remain open. Live filesystem reads and SQLite publication are not one atomic
operation; the later check narrows that race rather than eliminating it.
See [INCREMENTAL-SEMANTIC.md](INCREMENTAL-SEMANTIC.md).