Harden worker IPC and recoverable runtime staging

docs/BUILDING.md

@@ -35,8 +35,13 @@ cargo test -p searchi-service --test google_cpu -- --ignored --nocapture
 cargo test -p searchi-service --lib google_cpu_preserves -- --ignored --nocapture
 ```
 
-The worker is statically linked with the MSVC CRT on Windows. Staging copies
-it and the llama.cpp license into the desktop resources directory. Add
+The explicit internal CPU worker uses the static MSVC CRT on Windows; normal
+auto builds use the shared runtime described below. Staging verifies a complete
+candidate resource directory, then swaps it into place under a process lock.
+Its recovery journal restores an interrupted swap on the next invocation;
+failed restoration retains the previous backup. Existing unrelated fonts and
+media resources are preserved. Fault-injected process interruption is tested;
+power-loss durability is not certified. Add
 `--bundle-model` to staging for an offline model bundle after downloading;
 otherwise desktop setup provisions verified models on first launch. Search
 itself performs no network inference or downloads. The worker now uses the verified media projector for genuine image/audio embeddings. The FFmpeg script stages verified standalone decoders and their notices.
@@ -59,25 +64,55 @@ supports `--backend auto|metal|vulkan|cuda` builds with CPU fallback; do not
 exercise those acceleration backends in this VM. Their execution and failure
 recovery require validation on the corresponding release hardware.
 
+### Isolated native desktop validation
+
+Set `SEARCHI_DATA_DIR` to an absolute directory to isolate the desktop's
+settings and index for release QA or portable use. Normal launches use the OS
+app-data directory. Invalid overrides fail startup instead of falling back to
+the user's regular data. Model downloads still use the configured model
+directory or managed cache, so isolated tests should also configure a verified
+fixture model directory in their own settings.
+
+```powershell
+$env:SEARCHI_DATA_DIR = 'C:\searchi-qa-data'
+$env:SEARCHI_FORCE_CPU = '1'
+$env:WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS = '--disable-gpu --disable-accelerated-video-decode'
+```
+
+Extract and launch the entire Release archive outside the source checkout.
+Direct Cargo release builds must embed the frontend:
+
+```powershell
+cargo build --manifest-path apps/desktop/src-tauri/Cargo.toml --release --features custom-protocol --locked
+```
+
+The crate rejects Release compilation without this feature; the Tauri build
+CLI enables it automatically. A development-server URL is not a release gate.
+Record native window interaction separately from process startup, browser
+screenshots and worker inference; those are different release gates.
+
 ## Headless Linux Google search
 
 These are the current script and CLI interfaces, not evidence of an executed
-Linux release. On a native Linux build host, run from the repository root:
+Linux release. On a native Linux build host with Vulkan development libraries
+and `glslc`, run from the repository root:
 
 ```sh
-python3 tools/embeddinggemma2/build_runtime.py --backend cpu --jobs 2
+python3 tools/embeddinggemma2/build_runtime.py --backend auto --jobs 2
 python3 tools/embeddinggemma2/download.py
 python3 tools/media-runtime/provision.py
-export SEARCHI_FORCE_CPU=1
 cargo build --release -p searchi-cli --locked
 ./target/release/searchi index --db searchi.sqlite --root /path/to/documents
-./target/release/searchi embed --db searchi.sqlite --model-dir .searchi/models/embeddinggemma2 --embedding-worker .searchi/runtime/cpu/searchi-gemma-worker --ffmpeg .searchi/media-runtime/bin/ffmpeg --ffprobe .searchi/media-runtime/bin/ffprobe --cpu-threads 1
-./target/release/searchi search --db searchi.sqlite 'supplier cancellation terms' --model-dir .searchi/models/embeddinggemma2 --embedding-worker .searchi/runtime/cpu/searchi-gemma-worker --cpu-threads 1
+./target/release/searchi embed --db searchi.sqlite --model-dir .searchi/models/embeddinggemma2 --embedding-worker .searchi/runtime/auto/searchi-gemma-worker --ffmpeg .searchi/media-runtime/bin/ffmpeg --ffprobe .searchi/media-runtime/bin/ffprobe --cpu-threads 1
+./target/release/searchi search --db searchi.sqlite 'supplier cancellation terms' --model-dir .searchi/models/embeddinggemma2 --embedding-worker .searchi/runtime/auto/searchi-gemma-worker --cpu-threads 1
 ```
 
 `embed` builds document, image, video and audio/music vectors by default. Use
 `--text-only` explicitly to omit media. Default media indexing needs paired
 FFmpeg/FFprobe paths or packaged helpers beside the installed CLI.
+This normal worker detects usable acceleration and falls back to CPU. Internal
+CPU-only model tests may use `--backend cpu` and `SEARCHI_FORCE_CPU=1`; those
+test artifacts are not a separate distribution edition.
 
 `searchi setup --model-dir PATH` is an alternative verified model-provisioning
 command. It provisions model assets, not the native worker. Package the worker

docs/ROADMAP.md

@@ -67,11 +67,16 @@ Status: implementation started; no production release yet. Windows VM tests use
 - [x] Windows desktop archive packager includes explicit runtime/helper files
   and notices, checks architecture and records per-file checksums; excludes
   model caches and user data. Controlled packaging regression tests pass.
-- [ ] Verify embedded-frontend Release archive launch outside the source tree,
-  clean-machine prerequisites, GPU dependencies, upgrades and signing.
+- [x] Embedded-frontend Windows Release archive launches outside the source tree
+  with isolated data and a ready real Google CPU semantic generation.
+- [ ] Verify native interactions, clean-machine prerequisites, GPU dependencies,
+  upgrades and signing.
 - [x] Preflight native license and the complete offline Google model pair before
   modifying staged resources; corrupted/missing assets preserve existing worker.
-- [ ] Atomic resource staging and interrupted-copy recovery for release bundles.
+- [x] Transactional resource staging, verified copied bytes, process locking,
+  interrupted-copy/swap recovery and backup preservation on failed rollback.
+- [ ] Verify power-loss durability and release-tool interaction with resource
+  directory swaps on each OS; fault-injected process interruption is narrower.
 - [x] Shared monotonic Basic/Debug search metrics, granular bounded engine traces,
   actual worker query diagnostics, and headless JSON performance output.
 - [ ] Verify detailed performance UI interaction on native desktop platforms;
@@ -84,7 +89,9 @@ Status: implementation started; no production release yet. Windows VM tests use
   Warm p95 alone does not satisfy the first-use requirement.
 - [ ] Test migrations, interrupted work, invalid documents, inaccessible/offline roots and cloud placeholders.
 - [ ] Automated native build matrix and documented supported cross-compilation paths.
-- [ ] Execute Linux/macOS platform tests; create Windows/macOS/Linux installers.
+- [x] Native Linux headless workspace tests: 93 passed, 11 explicitly ignored.
+- [ ] Execute Linux native Google worker and desktop/macOS platform tests;
+  create Windows/macOS/Linux installers.
 - [ ] Signing/notarization configuration, upgrade/rollback validation, release instructions.
 
 ## 6. Local agents and optional answers
@@ -115,8 +122,11 @@ Audit and measured priorities: [EMBEDDINGGEMMA2-AUDIT.md](EMBEDDINGGEMMA2-AUDIT.
   preserve filtering, source freshness, failures and actual warmup probes.
 - [x] Reduce semantic settle delay to 120 ms and retain old ready generation
   during rebuilding; test initial-generation and activation readiness boundaries.
-- [ ] Bound whole IPC write/read deadlines and recover from a nonreading helper.
-- [ ] Declare/test x86 CPU minimum or baseline/AVX2 runtime selection.
+- [x] Bound whole IPC write/read deadlines and recover from a nonreading helper;
+  CPU-only real helper-process regressions cover blocked writes and responses.
+- [x] Windows x64 baseline/optimized CPU runtime selection and OS AVX-state
+  checks, verified with real packaged baseline and optimized inference.
+- [ ] Validate CPU selection on older hardware and remaining architectures.
 - [ ] Incrementally reuse unchanged document/media vectors in atomic generations.
 - [ ] Benchmark genuine token-budgeted multi-sequence batching and isolation.
 - [ ] Implement and measure shared foreground/indexer resource budgets/priority.

docs/SEMANTIC.md

@@ -20,6 +20,12 @@ under the opt-in minilm-proof feature and tools/semantic-proof/HISTORICAL.md.
 The persistent native helper uses bounded NDJSON over stdin/stdout. It has no
 HTTP server or download code. Rust verifies the text model's length and SHA256
 before starting it. Setup provisions assets; search never downloads them.
+Each request has one 180-second IPC deadline across pipe writing and response
+reading. Single-slot queues bound pending work. A stalled write or response
+kills/reaps the helper and joins both IPC threads; only transport failures
+qualify for a bounded CPU restart. Malformed/oversized responses remain
+protocol errors. Cleanup assumes the shipped helper spawns no descendants
+holding its pipe handles, matching the native implementation.
 
 Production device mode is auto, with available compiled acceleration and CPU
 fallback. Model/context initialization and returned inference/nonfinite-output

docs/VALIDATION.md

@@ -7,6 +7,39 @@ and `SEARCHI_FORCE_CPU=1`.
 
 ## Current integration checkpoint (2026-10-12)
 
+The reliability checkpoint passes 93 workspace tests on Windows and 93 on
+native Linux (Ubuntu 24.04 under WSL); each explicitly ignores 11 optional
+integration tests. Linux uses its native Rust toolchain and an isolated ext4
+build cache. This verifies headless core portability, not a Linux desktop,
+Linux native Google worker, GPU or installer. Windows desktop passes 17 tests;
+strict workspace and desktop Clippy checks pass. Five genuine Google model
+integration tests pass in forced-CPU mode.
+
+Worker IPC now applies one deadline across sending and receiving, with bounded
+queues and owned writer/reader threads. Transport failures terminate and reap
+the worker; eligible GPU transport failures retry on CPU. Invalid protocol
+responses remain errors. Fifteen semantic unit tests pass, including a blocked
+pipe and disconnected writer; one optional integration test is ignored.
+
+Resource staging passes 24 tests on Windows (one symlink-permission skip) and
+23 on Linux (two Windows-only skips). Locking, recovery, rollback preservation,
+link rejection and concurrent edits before the final swap are covered. Actual
+automatic-backend staging also succeeds. These are process-failure checks;
+power-loss durability and concurrent arbitrary writes after the final check
+remain release gates.
+
+The latest unsigned Windows archive launches outside the checkout with an
+embedded frontend at `http://tauri.localhost/`, isolated data, one real indexed
+document and a ready Google CPU semantic generation. The archive is
+145,217,175 bytes, SHA-256
+`fb454a9dbf258f51044796ab24a5942a8f90804d0824ca0e2dc5f59b7847ec3b`.
+Native accessibility confirms startup and status only. Input and screenshot
+capture remain denied by the environment; native interaction, appearance,
+clean-machine installation and GPU behavior are unverified. Release builds
+without the embedded-frontend feature now fail explicitly.
+
+The following package-dispatch evidence predates this reliability checkpoint:
+
 The normal Windows x64 package now includes baseline and optimized CPU DLLs
 with CPU/OS feature checks, plus the Vulkan backend. Shared DLLs use a common
 MSVC CRT: a first `/MT` prototype crashed because model file handles crossed