74 lines
4.2 KiB
Markdown
74 lines
4.2 KiB
Markdown
|
# Incremental semantic indexing and recovery
|
||
|
|
|
||
|
|
Status: implementation design and acceptance gates; not implemented. Current
|
||
|
|
semantic rebuilds regenerate vectors and the ANN graph. Interrupted staging is
|
||
|
|
discarded by the next rebuild. The previous ready generation remains searchable.
|
||
|
|
|
||
|
|
## Reuse requires complete sources
|
||
|
|
|
||
|
|
Document vectors depend on the extracted text, the bounded filename title,
|
||
|
|
model, prompt and token-window policy. The current passage content hash alone
|
||
|
|
is insufficient: a basename rename changes Google's document prompt. Passage
|
||
|
|
IDs are recreated by core reconciliation, so reuse must remap current IDs,
|
||
|
|
paths and page/line locations rather than retain old citations.
|
||
|
|
|
||
|
|
Media reuse requires a verified hash of the actual file bytes and matching
|
||
|
|
encoder/preprocessing/sampling policy. Size and modification time alone are
|
||
|
|
insufficient. Existing media chunks are written progressively; their presence
|
||
|
|
does not prove that a file completed. A crash can leave an incomplete group.
|
||
|
|
|
||
|
|
Add a versioned completion manifest per source group: generation, task key,
|
||
|
|
source type, encoder/policy key, source hash, title hash where applicable,
|
||
|
|
expected chunk count and completion state. Associate chunks with their group.
|
||
|
|
Only completed groups can supply reusable vectors. Historical generations
|
||
|
|
without completion evidence remain searchable but require one rebuild before
|
||
|
|
their vectors qualify for reuse.
|
||
|
|
|
||
|
|
Start with matching complete-generation policies. Component encoder keys can
|
||
|
|
later let unchanged document vectors survive an audio-sampling policy change.
|
||
|
|
Do not silently omit image, audio or video work to make a rebuild cheaper.
|
||
|
|
|
||
|
|
## Durable work and atomic publication
|
||
|
|
|
||
|
|
Persist bounded pending/running/done/failed tasks in SQLite. Tasks retain their
|
||
|
|
source identity, attempts and a bounded error. Obtain the existing OS build
|
||
|
|
lock before starting or resuming a compatible staging generation; only then
|
||
|
|
can interrupted running tasks be reset. Keep inference outside write
|
||
|
|
transactions. Commit source completion, verified chunk membership and task
|
||
|
|
state atomically. Incomplete groups must be discarded or resumed without
|
||
|
|
qualifying as completed reuse sources.
|
||
|
|
|
||
|
|
Discover work in bounded keyset batches. Before publication, reconcile staging
|
||
|
|
against current indexed sources: remove deleted sources, invalidate changed
|
||
|
|
identities and enqueue additions. Verify exact document window slices and
|
||
|
|
remap citations. For media, confirm the file is unchanged after extraction and
|
||
|
|
inference. Retain the 100,000-chunk budget and explicit per-file media failures.
|
||
|
|
|
||
|
|
Build the ANN graph only from eligible completed groups. Publish under a
|
||
|
|
transaction that checks the reconciled corpus revision; a concurrent source
|
||
|
|
change must trigger reconciliation/retry. Storage or activation failure must
|
||
|
|
preserve both the previous active generation and the same reader's policy.
|
||
|
|
|
||
|
|
## Acceptance gates
|
||
|
|
|
||
|
|
- [ ] Migrate existing databases while preserving active-generation search.
|
||
|
|
- [ ] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
|
||
|
|
document/media inference calls and identical vectors and source locations.
|
||
|
|
- [ ] Change one document and one media file; re-encode only affected groups.
|
||
|
|
- [ ] Re-encode text after a basename rename; reuse a folder-only move with
|
||
|
|
current citations. Verify media bytes before reusing a renamed media file.
|
||
|
|
- [ ] Regenerate affected media when sampling policy changes; retain eligible
|
||
|
|
text groups when component encoder-key reuse is implemented.
|
||
|
|
- [ ] Terminate after a committed batch and resume without repeating completed
|
||
|
|
inference. Never reuse or publish an unfinished source group.
|
||
|
|
- [ ] Mutate or delete a queued/running source and prove stale vectors cannot
|
||
|
|
activate or become readable.
|
||
|
|
- [ ] Inject storage/activation failure, preserve existing results and resume.
|
||
|
|
- [ ] Use genuine Google CPU inference for these integration checks. Storage
|
||
|
|
fixtures alone do not establish retrieval quality or reuse equivalence.
|
||
|
|
- [ ] Measure preparation, the first foreground query and warm p95 while
|
||
|
|
resumed indexing runs; record actual inference/reuse counts and resource use.
|
||
|
|
|
||
|
|
Foreground query priority and bounded persistent workers remain part of the
|
||
|
|
first-use requirement. Faster unchanged rebuilds alone do not prove responsive
|
||
|
|
search while indexing.
|