142 lines
8.2 KiB
Markdown
142 lines
8.2 KiB
Markdown
# Incremental semantic indexing and recovery
|
|
|
|
Status: completed-source reuse and compatible staging recovery implemented in
|
|
schema 8 (completed-source tasks were introduced in schema 7). Unchanged vectors are copied with current citations; the ANN graph
|
|
is rebuilt. Incomplete source groups are discarded before retry. The previous
|
|
ready generation and its reader policy remain available on failures.
|
|
|
|
CPU tests use the normal hardware-detecting runtime forced to CPU.
|
|
Recorded integration evidence and remaining gates are below. This is not a
|
|
production readiness or media relevance claim.
|
|
|
|
## Reuse requires complete sources
|
|
|
|
Document vectors depend on the extracted text, the bounded filename title,
|
|
model, prompt and token-window policy. The current passage content hash alone
|
|
is insufficient: a basename rename changes Google's document prompt. Passage
|
|
IDs are recreated by core reconciliation, so reuse must remap current IDs,
|
|
paths and page/line locations rather than retain old citations.
|
|
|
|
Media reuse requires a verified hash of the actual file bytes and matching
|
|
encoder/preprocessing/sampling policy. Size and modification time alone are
|
|
insufficient. Existing media chunks are written progressively; their presence
|
|
does not prove that a file completed. A crash can leave an incomplete group.
|
|
|
|
The versioned completion manifest records per source group: generation, task key,
|
|
source type, encoder/policy key, source hash, title hash where applicable,
|
|
expected chunk count and completion state. Associate chunks with their group.
|
|
Only completed groups can supply reusable vectors. Historical generations
|
|
without completion evidence remain searchable but require one rebuild before
|
|
their vectors qualify for reuse.
|
|
|
|
Start with matching complete-generation policies. Component encoder keys can
|
|
later let unchanged document vectors survive an audio-sampling policy change.
|
|
Do not silently omit image, audio or video work to make a rebuild cheaper.
|
|
|
|
## Durable work and atomic publication
|
|
|
|
Persist bounded pending/running/done/failed tasks in SQLite. Tasks retain their
|
|
source identity, attempts and a bounded error. Obtain the existing OS build
|
|
lock before starting or resuming a compatible staging generation; only then
|
|
can interrupted running tasks be reset. Keep inference outside write
|
|
transactions. Commit source completion, verified chunk membership and task
|
|
state atomically. Incomplete groups must be discarded or resumed without
|
|
qualifying as completed reuse sources.
|
|
|
|
Discover work in bounded keyset batches. Before publication, reconcile staging
|
|
against current indexed sources: remove deleted sources, invalidate changed
|
|
identities and enqueue additions. Verify exact document window slices and
|
|
remap citations. For media, confirm the file is unchanged after extraction and
|
|
inference. Retain the 100,000-chunk budget and explicit per-file media failures.
|
|
|
|
Build the ANN graph only from eligible completed groups. Publish under a
|
|
transaction that checks the reconciled corpus revision; a concurrent source
|
|
change must trigger reconciliation/retry. Storage or activation failure must
|
|
preserve both the previous active generation and the same reader's policy.
|
|
|
|
## Acceptance gates
|
|
|
|
- [x] Schema-6 storage migration preserves active-generation search; legacy
|
|
chunks without completion evidence cannot supply reusable vectors.
|
|
- [x] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
|
|
document/media inference calls and identical vectors and source locations.
|
|
- [x] Change one document and one image; re-encode only affected groups.
|
|
Image regression preserves file size and exact mtime to exercise SHA identity.
|
|
- [x] Re-encode text after a basename rename; reuse a folder-only move with
|
|
current citations.
|
|
- [ ] Verify media bytes and current citations after a media-file rename.
|
|
- [ ] Regenerate affected media when sampling policy changes; retain eligible
|
|
text groups when component encoder-key reuse is implemented.
|
|
- [ ] Terminate after a committed batch and resume without repeating completed
|
|
inference. Never reuse or publish an unfinished source group.
|
|
- [x] Same-size/mtime text mutation during genuine inference rejects publication;
|
|
a mutation before rebuilding fails before text encoding and ordinary rescans
|
|
reextract it. Unaffected vectors survive resumed retry.
|
|
- [ ] Concurrent deletion/media mutation and comprehensive queued/running source
|
|
search freshness tests.
|
|
- [x] Inject activation failure, preserve existing results and resume completed
|
|
staging with zero document inference; cleanup-failure unit regression retains
|
|
the primary error and old reader policy.
|
|
- [x] Use genuine Google CPU inference for reuse integration checks. Storage
|
|
fixtures alone do not establish retrieval quality or reuse equivalence.
|
|
- [ ] Measure preparation, the first foreground query and warm p95 while
|
|
resumed indexing runs; record actual inference/reuse counts and resource use.
|
|
|
|
Foreground query priority and bounded persistent workers remain part of the
|
|
first-use requirement. Faster unchanged rebuilds alone do not prove responsive
|
|
search while indexing.
|
|
|
|
## Current limits
|
|
|
|
Reuse requires the complete generation policy to match. Component-level reuse
|
|
across media sampling changes remains open. Durable staging resumes at completed
|
|
source boundaries; a real process-kill test during a multi-batch source remains
|
|
open. Storage tests cover discarded partial groups and corpus revision races.
|
|
|
|
Full source hashes are checked after ANN construction and serialization, before
|
|
opening the publication write transaction. Indexed changes are also rejected by
|
|
the transaction's revision guard. This narrows the filesystem race; it does not
|
|
make file reads and SQLite publication one atomic filesystem operation.
|
|
|
|
Core schema 12 records the SHA-256 of the exact bounded bytes consumed by each
|
|
parser. Semantic builds verify that provenance before reusing or encoding text.
|
|
Same-size/mtime changes are detected by normal folder rescanning. Legacy records
|
|
retain lexical search but need a rescan before semantic rebuilding or serving
|
|
agent excerpts. Excerpts reject missing or mismatched source provenance.
|
|
|
|
Bounded discovery now prunes obsolete resumed groups before inference. Compatible
|
|
renamed staging groups survive with current memberships through a transactional,
|
|
disk-backed SQLite snapshot; active vectors are untouched. Genuine duplicate
|
|
current groups still count toward the 100,000-chunk ceiling. Hashing, discovery,
|
|
media metadata probes, snapshot copying and graph rebuilding have costs: zero
|
|
encoder calls does not mean zero work or prove fast first input.
|
|
|
|
Synthetic colored-image and sine-wave fixtures prove actual vector reuse and
|
|
citation equality; they do not establish useful media relevance.
|
|
|
|
## Current Windows CPU integration rerun, 2026-10-13
|
|
|
|
At source `66bf7d7b4757c1454e4cd61e514ff9376bddd728` with a clean worktree,
|
|
the normal current worker SHA-256
|
|
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`
|
|
ran all three `incremental_google_cpu` tests successfully in 75.47 seconds.
|
|
Actual Google inference was explicitly CPU-only with one thread. Verified
|
|
Google assets and the pinned Windows FFmpeg/FFprobe helpers were used.
|
|
|
|
The checks cover completed-vector reuse after failed activation and reopen,
|
|
document/code moves with preserved citations, title-change reencoding, mixed
|
|
document/image/audio/video reuse, and byte-changing media overwrites with
|
|
identical size/mtime. The prepared-reader test also passed in 14.04 seconds:
|
|
publication preserved the real query-vector LRU and shared prepared ANN graph,
|
|
returned updated source content, and a distinct query executed genuine inference.
|
|
Observed service times were 2.962 ms for the cached query and 84.808 ms for the
|
|
distinct query, on a one-document fixture. These are debug-profile integration
|
|
observations, not desktop first-input, scale or release latency acceptance.
|
|
|
|
Command: `cargo test -p searchi-service --test prepared_google_cpu --test
|
|
incremental_google_cpu --locked -- --ignored --nocapture --test-threads=1`,
|
|
with `SEARCHI_FORCE_CPU=1` and the verified model/worker/decoder environment
|
|
variables described in SKILLS.md. All four tests executed, none skipped.
|
|
The running installed user application was left untouched. Other platform,
|
|
storage-contention and representative-corpus gates remain open.
|