searchi/docs/RECOVERY-AND-HANDOFF.md

Process recovery and prepared-reader handoff

2026-10-13 development validation. Production release gates remain open.

Abrupt process termination

crates/searchi-cli/tests/crash_recovery.rs invokes the real installed-layout CLI adapter with verified Google EmbeddingGemma 2 assets, one CPU thread and actual FFmpeg helpers. It creates two document/code source groups and a 65-second PCM WAV with three normal audio windows. It polls SQLite for both completed document groups plus a committed, unowned chunk belonging to the running media task, then terminates only its owned process tree.

Windows uses PID-scoped taskkill /T /F; Unix assigns a dedicated process group and kills that group. Cleanup does not kill processes by name and does not panic during test unwinding. No user app/settings/index is used.

Assertions verify that the previous active generation and completed vector bytes remain intact. Restart resumes the same generation, makes zero document encoder calls, re-encodes all three windows of the unfinished audio source, and publishes five unique chunks without leaked partial vectors. A real Google CodeRetrieval query checks the current source line citation.

Windows and native Linux actual runs passed in 157.86 s and 233.65 s. These are suite elapsed times with compiler contention, not a search benchmark. Power loss, arbitrary batch boundaries and all crash scenarios are not established by one process-termination case.

Prepared graphs and warmed query model

The desktop already used separate background and query model workers, but publication could send the next foreground request through blob reading, SHA verification and ANN deserialization. It also withdrew readiness and repeated query warmup while holding the retained session mutex.

The publisher now shares its immutable prepared Arc<usearch::Index> before SQL activation through a process-local PreparedGraphs handle. That handle binds to the canonical database and vector dimension; entries match generation, full policy fingerprint and persisted ANN checksum. At most two registry entries are retained. Failed SQL activation removes only its unpublished entry; readers retain their own Arc when an old registry entry is evicted.

The desktop query worker keeps its model and query-vector LRU ready throughout publication. Detailed traces include semantic.prepared_graph_lookup; the new active graph is a cache hit instead of foreground blob loading. Existing source eligibility, strict filtering and generation snapshot checks still apply.

Storage regressions cover identity/dimension/checksum mismatches, bounded eviction, retained-reader lifetime and SQL activation rejection/retry. The real Google CPU test drops the background worker, searches the updated document and asserts no blob-read/checksum/deserialization spans. It verifies both a retained query-vector hit and an unseen query with actual neural inference.

Windows one-document fixture: retained query 4.389 ms, unseen query 111.086 ms, 29.05 s suite elapsed. These measure the shared engine request in debug mode, not native UI first input, application startup, a representative corpus or p95. The same native Linux fixture passed: retained query 2.107 ms, unseen query 79.614 ms, suite 21.98 s. Different cache/contention conditions prevent treating these numbers as Windows-versus-Linux comparative performance.

Windows package media provenance

The packager now verifies the pinned original decoder ZIP size and checksum, then compares the staged manifest, helpers and notices with bounded archive members. Valid PE headers and a freshly recorded package checksum are not accepted as decoder provenance. Fifteen packaging tests cover changed helpers, notices, manifests, archive members and changes during ZIP writing; failure preserves the previous output. Media staging was refreshed from the verified cached archive before packaging.

Current embedded-frontend Windows release built successfully. Its 145,265,448-byte unsigned ZIP contains 39 verified files; optimized and baseline-only packaged CPU inference and decoder startup passed. Archive SHA-256: cf7953827be90255f182932fe53c16faf76701098d2636c8860b6ed4e0eb6ec4. Both worker smoke tests request auto with the GPU provider disabled only in the child process. Actual GPU, native UI interaction and clean-machine installation are not certified by those smoke tests.

Remaining performance/release work

Background and query workers still receive independent thread budgets; shared CPU admission and foreground priority are open. Publication no longer charges foreground graph deserialization, but eligibility refresh, indexing I/O and CPU contention need representative measurements. Native compositor/input timing, large-corpus p95, older CPUs, actual GPU/device-loss, power-loss durability, installers/upgrades/signatures and macOS packaging remain open.

Executed checks

• Windows and native Linux workspace tests: 106 passed each, 18 ignored.
• Windows native desktop unit suite: 17 passed.
• Strict Windows/Linux workspace and Windows desktop Clippy plus both formatting checks passed.
• Real Google CPU process-kill and prepared-reader regressions passed on both OSes.
• Windows packager tests: 15 passed. Current embedded-frontend release build, archive integrity, packaged optimized/baseline CPU inference and decoder startup passed; no accelerator was used.

The prepared graph handle is process-local: foreign CLI/MCP index writers cannot share its native Arc. Cold CLI initialization and cross-process reader refresh remain separate costs. Representative concurrent-indexing/foreground measurements and shared CPU thread admission remain the next performance milestone.