searchi/docs/INCREMENTAL-SEMANTIC.md

# Incremental semantic indexing and recovery

Status: completed-source reuse and compatible staging recovery implemented in
schema 7. Unchanged vectors are copied with current citations; the ANN graph
is rebuilt. Incomplete source groups are discarded before retry. The previous
ready generation and its reader policy remain available on failures.

CPU tests use the normal hardware-detecting runtime forced to CPU.
Recorded integration evidence and remaining gates are below. This is not a
production readiness or media relevance claim.

## Reuse requires complete sources

Document vectors depend on the extracted text, the bounded filename title,
model, prompt and token-window policy. The current passage content hash alone
is insufficient: a basename rename changes Google's document prompt. Passage
IDs are recreated by core reconciliation, so reuse must remap current IDs,
paths and page/line locations rather than retain old citations.

Media reuse requires a verified hash of the actual file bytes and matching
encoder/preprocessing/sampling policy. Size and modification time alone are
insufficient. Existing media chunks are written progressively; their presence
does not prove that a file completed. A crash can leave an incomplete group.

The versioned completion manifest records per source group: generation, task key,
source type, encoder/policy key, source hash, title hash where applicable,
expected chunk count and completion state. Associate chunks with their group.
Only completed groups can supply reusable vectors. Historical generations
without completion evidence remain searchable but require one rebuild before
their vectors qualify for reuse.

Start with matching complete-generation policies. Component encoder keys can
later let unchanged document vectors survive an audio-sampling policy change.
Do not silently omit image, audio or video work to make a rebuild cheaper.

## Durable work and atomic publication

Persist bounded pending/running/done/failed tasks in SQLite. Tasks retain their
source identity, attempts and a bounded error. Obtain the existing OS build
lock before starting or resuming a compatible staging generation; only then
can interrupted running tasks be reset. Keep inference outside write
transactions. Commit source completion, verified chunk membership and task
state atomically. Incomplete groups must be discarded or resumed without
qualifying as completed reuse sources.

Discover work in bounded keyset batches. Before publication, reconcile staging
against current indexed sources: remove deleted sources, invalidate changed
identities and enqueue additions. Verify exact document window slices and
remap citations. For media, confirm the file is unchanged after extraction and
inference. Retain the 100,000-chunk budget and explicit per-file media failures.

Build the ANN graph only from eligible completed groups. Publish under a
transaction that checks the reconciled corpus revision; a concurrent source
change must trigger reconciliation/retry. Storage or activation failure must
preserve both the previous active generation and the same reader's policy.

## Acceptance gates

- [x] Schema-6 storage migration preserves active-generation search; legacy
  chunks without completion evidence cannot supply reusable vectors.
- [x] Rebuild mixed text/code/image/audio/video fixtures unchanged with zero
  document/media inference calls and identical vectors and source locations.
- [x] Change one document and one image; re-encode only affected groups.
  Image regression preserves file size and exact mtime to exercise SHA identity.
- [x] Re-encode text after a basename rename; reuse a folder-only move with
  current citations.
- [ ] Verify media bytes and current citations after a media-file rename.
- [ ] Regenerate affected media when sampling policy changes; retain eligible
  text groups when component encoder-key reuse is implemented.
- [ ] Terminate after a committed batch and resume without repeating completed
  inference. Never reuse or publish an unfinished source group.
- [x] Same-size/mtime text mutation during genuine inference rejects publication;
  a mutation before rebuilding fails before text encoding and ordinary rescans
  reextract it. Unaffected vectors survive resumed retry.
- [ ] Concurrent deletion/media mutation and comprehensive queued/running source
  search freshness tests.
- [x] Inject activation failure, preserve existing results and resume completed
  staging with zero document inference; cleanup-failure unit regression retains
  the primary error and old reader policy.
- [x] Use genuine Google CPU inference for reuse integration checks. Storage
  fixtures alone do not establish retrieval quality or reuse equivalence.
- [ ] Measure preparation, the first foreground query and warm p95 while
  resumed indexing runs; record actual inference/reuse counts and resource use.

Foreground query priority and bounded persistent workers remain part of the
first-use requirement. Faster unchanged rebuilds alone do not prove responsive
search while indexing.

## Current limits

Reuse requires the complete generation policy to match. Component-level reuse
across media sampling changes remains open. Durable staging resumes at completed
source boundaries; a real process-kill test during a multi-batch source remains
open. Storage tests cover discarded partial groups and corpus revision races.

Full source hashes are checked after ANN construction and serialization, before
opening the publication write transaction. Indexed changes are also rejected by
the transaction's revision guard. This narrows the filesystem race; it does not
make file reads and SQLite publication one atomic filesystem operation.

Core schema 12 records the SHA-256 of the exact bounded bytes consumed by each
parser. Semantic builds verify that provenance before reusing or encoding text.
Same-size/mtime changes are detected by normal folder rescanning. Legacy records
retain lexical search but need a rescan before semantic rebuilding or serving
agent excerpts. Excerpts reject missing or mismatched source provenance.

Bounded discovery now prunes obsolete resumed groups before inference. Compatible
renamed staging groups survive with current memberships through a transactional,
disk-backed SQLite snapshot; active vectors are untouched. Genuine duplicate
current groups still count toward the 100,000-chunk ceiling. Hashing, discovery,
media metadata probes, snapshot copying and graph rebuilding have costs: zero
encoder calls does not mean zero work or prove fast first input.

Synthetic colored-image and sine-wave fixtures prove actual vector reuse and
citation equality; they do not establish useful media relevance.