Verify extraction provenance and prune semantic staging before inference

This commit is contained in:
karl committed 2026-10-13 01:14:40 +02:00
1 parent 5c18e7b49c
commit 423a1ea57d
17 files changed
+900 -78

No files matched your search

+21 -11
View File
@@ -69,8 +69,11 @@ preserve both the previous active generation and the same reader's policy.
text groups when component encoder-key reuse is implemented.
- [ ] Terminate after a committed batch and resume without repeating completed
inference. Never reuse or publish an unfinished source group.
- [ ] Mutate or delete a queued/running source and prove stale vectors cannot
activate or become readable.
- [x] Same-size/mtime text mutation during genuine inference rejects publication;
a mutation before rebuilding fails before text encoding and ordinary rescans
reextract it. Unaffected vectors survive resumed retry.
- [ ] Concurrent deletion/media mutation and comprehensive queued/running source
search freshness tests.
- [x] Inject activation failure, preserve existing results and resume completed
staging with zero document inference; cleanup-failure unit regression retains
the primary error and old reader policy.
@@ -90,16 +93,23 @@ across media sampling changes remains open. Durable staging resumes at completed
source boundaries; a real process-kill test during a multi-batch source remains
open. Storage tests cover discarded partial groups and corpus revision races.
Full source hashes are checked before ANN construction; indexed changes during
ANN construction are rejected by the final transaction's revision guard. A direct
filesystem mutation during ANN construction before watcher reconciliation still
needs a later validation callback and concurrent real-model regression.
Full source hashes are checked after ANN construction and serialization, before
opening the publication write transaction. Indexed changes are also rejected by
the transaction's revision guard. This narrows the filesystem race; it does not
make file reads and SQLite publication one atomic filesystem operation.
Obsolete completed groups in resumed staging are removed at final reconciliation.
Near the 100,000-chunk ceiling, replacement work can temporarily hit the budget
before this cleanup and fail safely. Bounded discovery/pruning before inference
is the next improvement. Hashing, media metadata probes and graph rebuilding still
have costs: zero encoder calls does not mean zero work or prove fast first input.
Core schema 12 records the SHA-256 of the exact bounded bytes consumed by each
parser. Semantic builds verify that provenance before reusing or encoding text.
Same-size/mtime changes are detected by normal folder rescanning. Legacy records
retain lexical search but need a rescan before semantic rebuilding or serving
agent excerpts. Excerpts reject missing or mismatched source provenance.
Bounded discovery now prunes obsolete resumed groups before inference. Compatible
renamed staging groups survive with current memberships through a transactional,
disk-backed SQLite snapshot; active vectors are untouched. Genuine duplicate
current groups still count toward the 100,000-chunk ceiling. Hashing, discovery,
media metadata probes, snapshot copying and graph rebuilding have costs: zero
encoder calls does not mean zero work or prove fast first input.
Synthetic colored-image and sine-wave fixtures prove actual vector reuse and
citation equality; they do not establish useful media relevance.