Verify extraction provenance and prune semantic staging before inference
This commit is contained in:
1 parent
5c18e7b49c
commit
423a1ea57d
17 files changed
+900
-78
No files matched your search
@@ -36,7 +36,7 @@ transactional completion failure and malformed stored text windows. Another
|
||||
fault-only pipeline test covers primary inference error preservation when
|
||||
partial-source cleanup also fails. Its provider returns no simulated vectors.
|
||||
|
||||
## Executed checks
|
||||
## Original reuse commit checks (5c18e7b)
|
||||
|
||||
- Windows workspace: 98 passed, 15 explicitly ignored; no failures.
|
||||
- Native Ubuntu 24.04 headless workspace: 98 passed, 15 explicitly ignored.
|
||||
@@ -53,11 +53,50 @@ partial-source cleanup also fails. Its provider returns no simulated vectors.
|
||||
isolated verified models/normal auto worker and pinned native FFmpeg helpers.
|
||||
Cold Clippy compilation ran concurrently; these are not latency comparisons.
|
||||
|
||||
## Publication and provenance follow-up
|
||||
|
||||
The follow-up moves full source validation after ANN construction/serialization,
|
||||
before opening the publication write transaction. A unit regression confirms
|
||||
an independent writer can acquire the database while validation runs; rejection
|
||||
keeps the previous fingerprint/cache and completed resumable staging.
|
||||
|
||||
Pre-inference pruning uses a bounded, disk-backed transactional SQLite snapshot.
|
||||
The 100,000-chunk regression checks renamed completed staging without duplicate
|
||||
allocation, genuine duplicate current groups exceeding the budget, rollback,
|
||||
obsolete replacement cleanup and preservation of active vectors. This is a
|
||||
storage fixture, not an ANN latency or neural relevance benchmark.
|
||||
|
||||
Core schema 12 hashes the exact bounded parser input, without reopening the file
|
||||
for parsing. Four tests cover immutable input snapshots, same-size/mtime mutation,
|
||||
real schema-11 migration preserving lexical content, stale agent excerpts and
|
||||
filename-only oversized/unsupported inputs. Supported rescans read and hash
|
||||
bounded bytes even when metadata is unchanged; matching hashes skip parsing.
|
||||
Legacy rows need reindexing to establish provenance before semantic rebuilding
|
||||
or serving agent excerpts.
|
||||
|
||||
## Follow-up executed checks
|
||||
|
||||
- Windows and native Linux workspaces: 104 passed each, 16 explicitly ignored; no failures.
|
||||
- Windows desktop: 17 passed.
|
||||
- Strict Windows/Linux workspace and Windows desktop Clippy passed; workspace formatting passed.
|
||||
- Six genuine Windows Google CPU tests passed: existing lifecycle/policy tests
|
||||
32.52 s; mixed incremental tests 182.03 s; new publication mutation test
|
||||
11.02 s. Compiler/resource contention was present. These are correctness-suite
|
||||
times and are not search-latency measurements.
|
||||
- The same six genuine native Linux Google CPU tests passed: lifecycle/policy
|
||||
40.16 s, incremental/media 135.42 s, publication mutation 11.98 s. These
|
||||
elapsed suites are correctness evidence, not comparative performance claims.
|
||||
- The mutation test overwrites the file after actual document inference,
|
||||
restores the exact mtime, rejects publication and preserves the active ID.
|
||||
An ordinary folder rescan then resumes with only the changed source encoded.
|
||||
Another overwrite before rebuilding rejects stale extraction before any
|
||||
additional document encoder calls. Unaffected stored vector bytes and
|
||||
citations remain identical.
|
||||
|
||||
## Remaining gates
|
||||
|
||||
Actual process termination during a multi-batch source, simultaneous real-model
|
||||
source mutation, wider policy changes, near-limit pruning, minimum-ABI/older-CPU
|
||||
checks and first-query/foreground latency during indexing remain open. Full file
|
||||
hash validation currently precedes ANN construction; direct file mutation during
|
||||
that construction before watcher reconciliation needs a later validation callback.
|
||||
Actual process termination during a multi-batch source, wider policy changes,
|
||||
minimum-ABI/older-CPU checks and first-query/foreground latency during indexing
|
||||
remain open. Live filesystem reads and SQLite publication are not one atomic
|
||||
operation; the later check narrows that race rather than eliminating it.
|
||||
See [INCREMENTAL-SEMANTIC.md](INCREMENTAL-SEMANTIC.md).
|
||||
Reference in new issue
Block a user