Verify extraction provenance and prune semantic staging before inference
This commit is contained in:
1 parent
5c18e7b49c
commit
423a1ea57d
17 files changed
+900
-78
No files matched your search
@@ -69,8 +69,11 @@ preserve both the previous active generation and the same reader's policy.
|
||||
text groups when component encoder-key reuse is implemented.
|
||||
- [ ] Terminate after a committed batch and resume without repeating completed
|
||||
inference. Never reuse or publish an unfinished source group.
|
||||
- [ ] Mutate or delete a queued/running source and prove stale vectors cannot
|
||||
activate or become readable.
|
||||
- [x] Same-size/mtime text mutation during genuine inference rejects publication;
|
||||
a mutation before rebuilding fails before text encoding and ordinary rescans
|
||||
reextract it. Unaffected vectors survive resumed retry.
|
||||
- [ ] Concurrent deletion/media mutation and comprehensive queued/running source
|
||||
search freshness tests.
|
||||
- [x] Inject activation failure, preserve existing results and resume completed
|
||||
staging with zero document inference; cleanup-failure unit regression retains
|
||||
the primary error and old reader policy.
|
||||
@@ -90,16 +93,23 @@ across media sampling changes remains open. Durable staging resumes at completed
|
||||
source boundaries; a real process-kill test during a multi-batch source remains
|
||||
open. Storage tests cover discarded partial groups and corpus revision races.
|
||||
|
||||
Full source hashes are checked before ANN construction; indexed changes during
|
||||
ANN construction are rejected by the final transaction's revision guard. A direct
|
||||
filesystem mutation during ANN construction before watcher reconciliation still
|
||||
needs a later validation callback and concurrent real-model regression.
|
||||
Full source hashes are checked after ANN construction and serialization, before
|
||||
opening the publication write transaction. Indexed changes are also rejected by
|
||||
the transaction's revision guard. This narrows the filesystem race; it does not
|
||||
make file reads and SQLite publication one atomic filesystem operation.
|
||||
|
||||
Obsolete completed groups in resumed staging are removed at final reconciliation.
|
||||
Near the 100,000-chunk ceiling, replacement work can temporarily hit the budget
|
||||
before this cleanup and fail safely. Bounded discovery/pruning before inference
|
||||
is the next improvement. Hashing, media metadata probes and graph rebuilding still
|
||||
have costs: zero encoder calls does not mean zero work or prove fast first input.
|
||||
Core schema 12 records the SHA-256 of the exact bounded bytes consumed by each
|
||||
parser. Semantic builds verify that provenance before reusing or encoding text.
|
||||
Same-size/mtime changes are detected by normal folder rescanning. Legacy records
|
||||
retain lexical search but need a rescan before semantic rebuilding or serving
|
||||
agent excerpts. Excerpts reject missing or mismatched source provenance.
|
||||
|
||||
Bounded discovery now prunes obsolete resumed groups before inference. Compatible
|
||||
renamed staging groups survive with current memberships through a transactional,
|
||||
disk-backed SQLite snapshot; active vectors are untouched. Genuine duplicate
|
||||
current groups still count toward the 100,000-chunk ceiling. Hashing, discovery,
|
||||
media metadata probes, snapshot copying and graph rebuilding have costs: zero
|
||||
encoder calls does not mean zero work or prove fast first input.
|
||||
|
||||
Synthetic colored-image and sine-wave fixtures prove actual vector reuse and
|
||||
citation equality; they do not establish useful media relevance.
|
||||
Reference in new issue
Block a user