Production readiness roadmap

Status: implementation started; no production release yet. Windows VM tests use CPU only.

Required product scope (user clarification)

Desktop UI is primary; semantic setup/indexing/search must require no terminal.
Google EmbeddingGemma 2 is the required production model, matching DigUp.
Local document, image, audio and video search are required product milestones.
A separate CLI-only build supports headless Linux without desktop dependencies.
Production automatically uses available hardware acceleration with CPU fallback; this Windows VM is tested in forced-CPU mode only.
Installer ships the native worker. First launch automatically downloads/verifies missing model files and manages indexing in the UI; no terminal setup is required.
Earlier MiniLM proof is an experiment and does not satisfy the model requirement.
First interaction must be responsive: prepare indexes and a real Google query in the background, retain ready sessions, and show immediate keyword results. Model download and initialization must have honest progress and retry states.

1. Search foundation

Rust workspace, reusable engine, CLI, versioned SQLite storage.
Filename/path and ranked full-text retrieval with strict filters.
Safe folder discovery, exclusions, bounded extraction, passage locations.
Incremental updates, rename/deletion reconciliation, crash recovery.
Persistence and filesystem lifecycle integration tests (Windows CPU).

2. Document and semantic retrieval

Text, Markdown, code, PDF and DOCX extraction with real fixtures (Windows CPU).
Validate actual embedding model artifacts, license, CPU runtime and quantization.
Versioned embeddings, ANN index, exact-match priority and hybrid fusion.
English/German relevance corpus and measured top-10 recall.
Atomic model/index replacement and resumable persistent work queues.
Completed-source manifests with title/model/window policy keys and citation remapping; reuse only verified complete document/media groups.
Durable generation tasks, crash resume and final corpus reconciliation; see incremental semantic design.

3. Desktop

Tauri shell, accessible keyboard search, progressive results and previews (shell built; UI interaction verification pending).
Folder selection, exclusion controls, resource budgets and indexing status.
Global shortcut, opening/revealing source files, PDF page navigation.
Watchers and reconciliation across Windows, macOS and Linux.
- Native notification coordinator, bounded targeted reconciliation, periodic verification and CPU-tested Windows/Linux create/update/rename/delete.
- Durable same-stat media hint invalidation, guarded metadata writes and real Google reindex/reuse regression; owned decoder scratch avoids feedback.
- Native Linux watcher-driven UI refresh, independent refresh timing, temporary selection restoration and explicit selection override.
- Native macOS events, Windows/macOS watcher-driven UI refresh and representative storage failure/contention checks; see automatic indexing.

4. Media

Genuine Google image, audio and video embeddings integrated with the shared search service and CPU-tested FFmpeg decoding; music tags and duration filters.
Frozen independent media corpus with retrieval, timestamp, negative-query and indexing-cost measurements (see MEDIA-QUALITY.md).
Improve precise spoken-passage localization, music hard negatives and calibrated handling of queries without a relevant match.
Package and verify decoders on every release OS; test production acceleration.
Native Linux x86-64 pinned decoder provenance, startup, WAV probe and CPU decode; broader codec/ABI coverage and desktop packaging remain open.
CPU OCR, image retrieval, audio transcription, video segment locations.
Isolated extraction workers with timeouts and resource ceilings.

5. Release gates

Normal worker build/staging defaults to hardware-detecting auto with CPU fallback; missing Vulkan/glslc fails configuration rather than hiding GPU loss.
Provision release-build GPU SDKs and verify one hardware-detecting package per OS/architecture, including missing-driver fallback and dependency closure.
Windows-x64 official pinned SDK copied into workspace with verified cache provenance; universal CPU/Vulkan worker and embedded-frontend desktop built.
Real Windows archive integrity, packaged decoder startup and auto-mode CPU inference with a controlled no-GPU-provider registry; Google CPU parity.
Normal Windows x64 archive includes baseline/optimized CPU dispatch, OS AVX-state guards, shared app-local MSVC CRT and hashed DLL inventory; real optimized and baseline-only package inference passes in the CPU VM.
Normal native Linux x86-64 CPU/Vulkan worker build, hashed 19-library staging and genuine optimized/baseline-only CPU code-query inference outside the build tree; desktop, ABI/older-hardware and GPU validation remain open.
Actual Linux automatic CPU fallback with Vulkan loader, ICDs/drivers and GPU devices absent in a verified private namespace; no GPU provider loaded.
Verify CPU dispatch on older physical CPUs and a clean Windows machine; dependency inspection and baseline execution on a modern VM are insufficient.
Actual missing/incompatible GPU drivers, precision, device loss and memory pressure on release hardware; native GUI and clean-machine archive launch.
Windows desktop archive packager includes explicit runtime/helper files and notices, checks architecture and records per-file checksums; excludes model caches and user data. Controlled packaging regression tests pass.
Embedded-frontend Windows Release archive launches outside the source tree with isolated data and a ready real Google CPU semantic generation.
Verify native interactions, clean-machine prerequisites, GPU dependencies, upgrades and signing.
Preflight native license and the complete offline Google model pair before modifying staged resources; corrupted/missing assets preserve existing worker.
Transactional resource staging, verified copied bytes, process locking, interrupted-copy/swap recovery and backup preservation on failed rollback.
Verify power-loss durability and release-tool interaction with resource directory swaps on each OS; fault-injected process interruption is narrower.
Shared monotonic Basic/Debug search metrics, granular bounded engine traces, actual worker query diagnostics, and headless JSON performance output.
Verify detailed performance UI interaction on native desktop platforms; repeat overhead measurements on representative libraries and hardware.
Formatting, lint, tests and security/dependency review.
CPU benchmarks: filename p95 <50 ms/1M records; text <150 ms/100K documents; semantic <500 ms on declared hardware.
Measure cold start, indexing throughput, RAM, disk and idle CPU.
Record the very first foreground query and distinct unseen queries after production background preparation; report initialization time separately. Warm p95 alone does not satisfy the first-use requirement.
Test migrations, interrupted work, invalid documents, inaccessible/offline roots and cloud placeholders.
Automated native build matrix and documented supported cross-compilation paths.
Native Linux headless workspace tests: 106 passed, 18 explicitly ignored.
Native Linux Google text/code/media reference parity and service lifecycle/rebuild recovery: seven genuine CPU integration tests passed.
Unsigned Linux x86-64 headless archive: verify extracted checksums/modes, mixed document/code/audio indexing, decoder discovery, semantic retrieval, line citations and strict filters outside the source/build directories.
Execute native Linux video/codec coverage and Linux desktop/macOS platform tests; create Windows/macOS/Linux installers.
Signing/notarization configuration, upgrade/rollback validation, release instructions.

6. Local agents and optional answers

Read-only MCP stdio server using the shared search service, persistent Google worker, structured filters, source citations and measured search stats.
End-to-end MCP initialization, discovery, search, invalid-input and source-access tests; no protocol text mixed with diagnostic stdout.
Installable agent skill and copyable MCP client configuration, with clear database scope, readiness and untrusted-source handling.
Optional OpenAI-compatible summaries/questions, disabled by default; explicit selected-source action, endpoint disclosure, bounded payloads, credentials kept out of persistent settings, diagnostics and error messages.
Verify real HTTP wire format against a local test endpoint; reject invalid settings, stale sources and unauthorized citations; preserve search on failure.
Native UI interaction and chosen-provider quality/latency checks before claiming production-ready answers or compatibility with every provider.

7. EmbeddingGemma 2 efficiency

Audit and measured priorities: EMBEDDINGGEMMA2-AUDIT.md.

Audit actual required model, task prefixes, pooling/normalization, native sequence processing and CPU/runtime policies against primary sources.
Record real CPU 1/2/4-thread query/first-use/document measurements and paired repeated-query/filter-change results with unchanged retrieval evidence.
Bounded provider-scoped actual-query vector LRU with accurate cache traces; preserve filtering, source freshness, failures and actual warmup probes.
Bounded shared prepared-ANN handoff before SQL activation; retain query model/LRU and readiness across publication. Actual CPU first new query has no foreground graph load; recovery/handoff evidence.
Reduce semantic settle delay to 120 ms and retain old ready generation during rebuilding; test initial-generation and activation readiness boundaries.
Bound whole IPC write/read deadlines and recover from a nonreading helper; CPU-only real helper-process regressions cover blocked writes and responses.
Windows x64 baseline/optimized CPU runtime selection and OS AVX-state checks, verified with real packaged baseline and optimized inference.
Validate CPU selection on older hardware and remaining architectures.
Completed-source vector reuse in atomic schema-7 generations, verified with genuine Google CPU document/code/image/audio/video fixtures.
Resume completed staging sources after failed publication; incomplete groups are purged under the OS build lock before retry.
Pre-inference staging pruning with compatible rename remapping and 100,000-chunk storage boundary; late validation after ANN preparation.
Exact parser-input SHA provenance and same-metadata overwrite detection on normal rescans; stale/missing agent excerpts fail closed.
Actual Google CPU same-metadata text mutation before/during inference: reject publication, rescan without metadata workarounds and reuse unaffected vectors.
Abrupt owned CLI termination after a committed partial audio chunk; resume completed documents and reencode unfinished media without duplicates. Process termination does not prove power-loss durability.
Concurrent deletion/media mutation, component-policy reuse and indexing/foreground measurements. See INCREMENTAL-SEMANTIC.md.
Benchmark genuine token-budgeted multi-sequence batching and isolation.
Implement and measure shared foreground/indexer resource budgets/priority.
- Shared inference admission, foreground priority between background encodes, total-thread settings migration and retained warm sessions on save failure; real Google CPU overlap regression passed on Windows.
- Measure representative mixed-media contention, single-thread latency, extraction/decoder/ANN work and native UI p95 before closing this gate.
Attribute startup cost; measure smaller query context and selective encoders.
- Query-only 512-token context in the normal worker, desktop/CLI/MCP integration and actual Windows CPU parity/boundary/reference tests. Paired samples reduced private commitment by 183 MB; resident memory and cached startup were essentially unchanged. See QUERY-CONTEXT.md.
- Verify native Linux/macOS, GPU parity and updated installed packages.
- Evaluate text readiness before the larger media projector finishes downloading; preserve verified assets and honest media setup status.
Buffered bounded media IPC and linear UTF-8 source-window offset processing.
Evaluate 256d MRL and content-specific vision budgets on independent data.
Validate GPU precision, synchronization-aware timings, parity and fallback on actual accelerator hardware; CPU VM checks do not satisfy this gate.

Completion requires recorded evidence; targets are not measured results.

8. Source-code retrieval

Shared code classification, strict lexical/ANN filtering before top-k, expanded source/build formats, bounded overlapping passages and actual lines.
Desktop Code scope, CLI --code-only, read-only MCP code_only, immutable scope snapshots and code-format-preserving previews.
Google's CodeRetrieval query instruction, explicit worker capabilities, task-scoped vector cache and genuine CPU prompt/debug parity.
Exact original source windows when Google's tokenizer normalizes text; verified UTF-8 offsets, full tail coverage and actual token-budget checks.
Schema-11 migration schedules unchanged source for re-extraction; tests preserve ordinary documents and filename availability through the upgrade.
Frozen 9-query real Google CPU regression with lexical comparison, ordinary-document exclusion and exact MCP source excerpt verification.
Evaluate independent real repositories, more languages, overloaded symbols, near-duplicate routines and queries with no relevant implementation.
Validate code search UI interaction on native Windows/macOS/Linux and actual GPU hardware. Tiny CPU fixtures do not close those release gates.

Completed foundation details

SQLite FTS5 with exact filename priority, text snippets and strict filters.
Default directory exclusions and protection against indexing the index itself.
Metadata reuse, atomic root reconciliation and schema-v1 migration coverage.
Actual CLI process integration tests and independent-reader concurrency test.
TypeScript desktop frontend build and dependency audit passed; native validation underway.
Initial README commit pushed to Gitea main via protected Infisical injection.
Native desktop build and guarded opening tests passed on Windows.
Genuine multilingual CPU model proof: 5/6 top-1, 6/6 recall@3; quality gate remains open.
Native Rust CPU embedding adapter verifies pinned model/tokenizer assets; integration with the search index and platform runtime packaging remain open.
PDF page and DOCX paragraph matches persist; corrupt/encrypted/over-limit documents remain filename-searchable and report extraction errors.
CLI exposes folder/type/size/date filters and additional directory exclusions.

The partial features above do not close broader production milestones such as crash-resumable jobs, fuzzy retrieval, watcher reliability or performance targets.

15 KiB Raw Blame History