searchi/docs/INCREMENTAL-SEMANTIC-VALIDATION.md

Completed-source reuse validation

2026-10-13. Windows VM, CPU-only inference (SEARCHI_FORCE_CPU=1), normal hardware-detecting worker, pinned Google EmbeddingGemma 2 backbone/projector, one inference thread. FFmpeg/ffprobe are the verified app-local helpers.

Genuine Google integration

cargo test -p searchi-service --test incremental_google_cpu --test google_cpu --locked -- --ignored --nocapture --test-threads=1 executes actual models. A counting adapter forwards every operation to GemmaEmbedder; no synthetic vectors satisfy the integration assertions.

• Document and code initial encoding: two calls. Unchanged rebuild and folder move: zero extra calls, bit-identical vectors, remapped current paths/locations. A basename rename adds one call because the document title is in Google's prompt. A single content change adds one call; unrelated code stays identical.
• Mixed document/code/image/audio/video fixture: five source groups. Unchanged rebuild makes zero additional document/image/audio encoder calls and retains exact vector bytes and citations. Replacing the BMP while preserving its size and exact mtime adds one image call; the other four groups stay identical.
• Rejected activation retains the prior active generation. Reopening resumes completed changed-document vectors with zero document calls; updated text citations are returned after publication. Existing policy-change regression covers lock admission, generation creation, repeated activation failures, same-instance reader policy and resumed retry in both directions.

Synthetic colors/sine waves in media fixtures prove reuse, not media relevance. Tests are correctness checks, not latency or throughput benchmarks.

Storage and failure boundaries

Four unit regressions cover schema-6 legacy search without unsafe reuse, completed versus partial restart state, revision-checked publication, transactional completion failure and malformed stored text windows. Another fault-only pipeline test covers primary inference error preservation when partial-source cleanup also fails. Its provider returns no simulated vectors.

Original reuse commit checks (5c18e7b)

• Windows workspace: 98 passed, 15 explicitly ignored; no failures.
• Native Ubuntu 24.04 headless workspace: 98 passed, 15 explicitly ignored.
• Windows native desktop unit suite: 17 passed.
• Strict Windows/Linux workspace and Windows native-desktop Clippy passed; Windows workspace/desktop formatting passed.
• Final-source Windows real Google tests above: five passed. The two existing lifecycle/policy tests took 25.31 s and the three incremental tests took 138.90 s. Concurrent compilation/resource contention was present; these elapsed suite times are not application search benchmarks.
• Final-source native Linux actual Google integration: the same five tests passed; existing lifecycle/policy suite 49.77 s, incremental suite 173.59 s. Ubuntu 24.04 WSL x86-64 with two allocated CPUs, one inference thread, isolated verified models/normal auto worker and pinned native FFmpeg helpers. Cold Clippy compilation ran concurrently; these are not latency comparisons.

Publication and provenance follow-up

The follow-up moves full source validation after ANN construction/serialization, before opening the publication write transaction. A unit regression confirms an independent writer can acquire the database while validation runs; rejection keeps the previous fingerprint/cache and completed resumable staging.

Pre-inference pruning uses a bounded, disk-backed transactional SQLite snapshot. The 100,000-chunk regression checks renamed completed staging without duplicate allocation, genuine duplicate current groups exceeding the budget, rollback, obsolete replacement cleanup and preservation of active vectors. This is a storage fixture, not an ANN latency or neural relevance benchmark.

Core schema 12 hashes the exact bounded parser input, without reopening the file for parsing. Four tests cover immutable input snapshots, same-size/mtime mutation, real schema-11 migration preserving lexical content, stale agent excerpts and filename-only oversized/unsupported inputs. Supported rescans read and hash bounded bytes even when metadata is unchanged; matching hashes skip parsing. Legacy rows need reindexing to establish provenance before semantic rebuilding or serving agent excerpts.

Follow-up executed checks

• Windows and native Linux workspaces: 104 passed each, 16 explicitly ignored; no failures.
• Windows desktop: 17 passed.
• Strict Windows/Linux workspace and Windows desktop Clippy passed; workspace formatting passed.
• Six genuine Windows Google CPU tests passed: existing lifecycle/policy tests 32.52 s; mixed incremental tests 182.03 s; new publication mutation test 11.02 s. Compiler/resource contention was present. These are correctness-suite times and are not search-latency measurements.
• The same six genuine native Linux Google CPU tests passed: lifecycle/policy 40.16 s, incremental/media 135.42 s, publication mutation 11.98 s. These elapsed suites are correctness evidence, not comparative performance claims.
• The mutation test overwrites the file after actual document inference, restores the exact mtime, rejects publication and preserves the active ID. An ordinary folder rescan then resumes with only the changed source encoded. Another overwrite before rebuilding rejects stale extraction before any additional document encoder calls. Unaffected stored vector bytes and citations remain identical.

Remaining gates

Actual process termination during a multi-batch source, wider policy changes, minimum-ABI/older-CPU checks and first-query/foreground latency during indexing remain open. Live filesystem reads and SQLite publication are not one atomic operation; the later check narrows that race rather than eliminating it. See INCREMENTAL-SEMANTIC.md.