103 lines
5.8 KiB
Markdown
103 lines
5.8 KiB
Markdown
# Completed-source reuse validation
|
|
|
|
2026-10-13. Windows VM, CPU-only inference (`SEARCHI_FORCE_CPU=1`), normal
|
|
hardware-detecting worker, pinned Google EmbeddingGemma 2 backbone/projector,
|
|
one inference thread. FFmpeg/ffprobe are the verified app-local helpers.
|
|
|
|
## Genuine Google integration
|
|
|
|
`cargo test -p searchi-service --test incremental_google_cpu --test google_cpu
|
|
--locked -- --ignored --nocapture --test-threads=1` executes actual models.
|
|
A counting adapter forwards every operation to `GemmaEmbedder`; no synthetic
|
|
vectors satisfy the integration assertions.
|
|
|
|
- Document and code initial encoding: two calls. Unchanged rebuild and folder
|
|
move: zero extra calls, bit-identical vectors, remapped current paths/locations.
|
|
A basename rename adds one call because the document title is in Google's
|
|
prompt. A single content change adds one call; unrelated code stays identical.
|
|
- Mixed document/code/image/audio/video fixture: five source groups. Unchanged
|
|
rebuild makes zero additional document/image/audio encoder calls and retains
|
|
exact vector bytes and citations. Replacing the BMP while preserving its
|
|
size and exact mtime adds one image call; the other four groups stay identical.
|
|
- Rejected activation retains the prior active generation. Reopening resumes
|
|
completed changed-document vectors with zero document calls; updated text
|
|
citations are returned after publication. Existing policy-change regression
|
|
covers lock admission, generation creation, repeated activation failures,
|
|
same-instance reader policy and resumed retry in both directions.
|
|
|
|
Synthetic colors/sine waves in media fixtures prove reuse, not media relevance.
|
|
Tests are correctness checks, not latency or throughput benchmarks.
|
|
|
|
## Storage and failure boundaries
|
|
|
|
Four unit regressions cover schema-6 legacy search without unsafe reuse,
|
|
completed versus partial restart state, revision-checked publication,
|
|
transactional completion failure and malformed stored text windows. Another
|
|
fault-only pipeline test covers primary inference error preservation when
|
|
partial-source cleanup also fails. Its provider returns no simulated vectors.
|
|
|
|
## Original reuse commit checks (5c18e7b)
|
|
|
|
- Windows workspace: 98 passed, 15 explicitly ignored; no failures.
|
|
- Native Ubuntu 24.04 headless workspace: 98 passed, 15 explicitly ignored.
|
|
- Windows native desktop unit suite: 17 passed.
|
|
- Strict Windows/Linux workspace and Windows native-desktop Clippy passed;
|
|
Windows workspace/desktop formatting passed.
|
|
- Final-source Windows real Google tests above: five passed. The two existing
|
|
lifecycle/policy tests took 25.31 s and the three incremental tests took
|
|
138.90 s. Concurrent compilation/resource contention was present; these
|
|
elapsed suite times are not application search benchmarks.
|
|
- Final-source native Linux actual Google integration: the same five tests
|
|
passed; existing lifecycle/policy suite 49.77 s, incremental suite 173.59 s.
|
|
Ubuntu 24.04 WSL x86-64 with two allocated CPUs, one inference thread,
|
|
isolated verified models/normal auto worker and pinned native FFmpeg helpers.
|
|
Cold Clippy compilation ran concurrently; these are not latency comparisons.
|
|
|
|
## Publication and provenance follow-up
|
|
|
|
The follow-up moves full source validation after ANN construction/serialization,
|
|
before opening the publication write transaction. A unit regression confirms
|
|
an independent writer can acquire the database while validation runs; rejection
|
|
keeps the previous fingerprint/cache and completed resumable staging.
|
|
|
|
Pre-inference pruning uses a bounded, disk-backed transactional SQLite snapshot.
|
|
The 100,000-chunk regression checks renamed completed staging without duplicate
|
|
allocation, genuine duplicate current groups exceeding the budget, rollback,
|
|
obsolete replacement cleanup and preservation of active vectors. This is a
|
|
storage fixture, not an ANN latency or neural relevance benchmark.
|
|
|
|
Core schema 12 hashes the exact bounded parser input, without reopening the file
|
|
for parsing. Four tests cover immutable input snapshots, same-size/mtime mutation,
|
|
real schema-11 migration preserving lexical content, stale agent excerpts and
|
|
filename-only oversized/unsupported inputs. Supported rescans read and hash
|
|
bounded bytes even when metadata is unchanged; matching hashes skip parsing.
|
|
Legacy rows need reindexing to establish provenance before semantic rebuilding
|
|
or serving agent excerpts.
|
|
|
|
## Follow-up executed checks
|
|
|
|
- Windows and native Linux workspaces: 104 passed each, 16 explicitly ignored; no failures.
|
|
- Windows desktop: 17 passed.
|
|
- Strict Windows/Linux workspace and Windows desktop Clippy passed; workspace formatting passed.
|
|
- Six genuine Windows Google CPU tests passed: existing lifecycle/policy tests
|
|
32.52 s; mixed incremental tests 182.03 s; new publication mutation test
|
|
11.02 s. Compiler/resource contention was present. These are correctness-suite
|
|
times and are not search-latency measurements.
|
|
- The same six genuine native Linux Google CPU tests passed: lifecycle/policy
|
|
40.16 s, incremental/media 135.42 s, publication mutation 11.98 s. These
|
|
elapsed suites are correctness evidence, not comparative performance claims.
|
|
- The mutation test overwrites the file after actual document inference,
|
|
restores the exact mtime, rejects publication and preserves the active ID.
|
|
An ordinary folder rescan then resumes with only the changed source encoded.
|
|
Another overwrite before rebuilding rejects stale extraction before any
|
|
additional document encoder calls. Unaffected stored vector bytes and
|
|
citations remain identical.
|
|
|
|
## Remaining gates
|
|
|
|
Actual process termination during a multi-batch source, wider policy changes,
|
|
minimum-ABI/older-CPU checks and first-query/foreground latency during indexing
|
|
remain open. Live filesystem reads and SQLite publication are not one atomic
|
|
operation; the later check narrows that race rather than eliminating it.
|
|
See [INCREMENTAL-SEMANTIC.md](INCREMENTAL-SEMANTIC.md).
|