searchi/docs/ROADMAP.md

Production readiness roadmap

Status: implementation started; no production release yet. Windows VM tests use CPU only.

Required product scope (user clarification)

• Desktop UI is primary; semantic setup/indexing/search must require no terminal.
• Google EmbeddingGemma 2 is the required production model, matching DigUp.
• Local document, image, audio and video search are required product milestones.
• A separate CLI-only build supports headless Linux without desktop dependencies.
• Production automatically uses available hardware acceleration with CPU fallback; this Windows VM is tested in forced-CPU mode only.
• Installer ships the native worker. First launch automatically downloads/verifies missing model files and manages indexing in the UI; no terminal setup is required.
• Earlier MiniLM proof is an experiment and does not satisfy the model requirement.
• First interaction must be responsive: prepare indexes and a real Google query in the background, retain ready sessions, and show immediate keyword results. Model download and initialization must have honest progress and retry states.

1. Search foundation

• Rust workspace, reusable engine, CLI, versioned SQLite storage.
• Filename/path and ranked full-text retrieval with strict filters.
• Safe folder discovery, exclusions, bounded extraction, passage locations.
• Incremental updates, rename/deletion reconciliation, crash recovery.
• Persistence and filesystem lifecycle integration tests (Windows CPU).

2. Document and semantic retrieval

• Text, Markdown, code, PDF and DOCX extraction with real fixtures (Windows CPU).
• Validate actual embedding model artifacts, license, CPU runtime and quantization.
• Versioned embeddings, ANN index, exact-match priority and hybrid fusion.
• English/German relevance corpus and measured top-10 recall.
• Atomic model/index replacement and resumable persistent work queues.

3. Desktop

• Tauri shell, accessible keyboard search, progressive results and previews (shell built; UI interaction verification pending).
• Folder selection, exclusion controls, resource budgets and indexing status.
• Global shortcut, opening/revealing source files, PDF page navigation.
• Watchers and reconciliation across Windows, macOS and Linux.

4. Media

• Genuine Google image, audio and video embeddings integrated with the shared search service and CPU-tested FFmpeg decoding; music tags and duration filters.
• Frozen independent media corpus with retrieval, timestamp, negative-query and indexing-cost measurements (see MEDIA-QUALITY.md).
• Improve precise spoken-passage localization, music hard negatives and calibrated handling of queries without a relevant match.
• Package and verify decoders on every release OS; test production acceleration.
• CPU OCR, image retrieval, audio transcription, video segment locations.
• Isolated extraction workers with timeouts and resource ceilings.

5. Release gates

• Shared monotonic Basic/Debug search metrics, granular bounded engine traces, actual worker query diagnostics, and headless JSON performance output.
• Verify detailed performance UI interaction on native desktop platforms; repeat overhead measurements on representative libraries and hardware.
• Formatting, lint, tests and security/dependency review.
• CPU benchmarks: filename p95 <50 ms/1M records; text <150 ms/100K documents; semantic <500 ms on declared hardware.
• Measure cold start, indexing throughput, RAM, disk and idle CPU.
• Record the very first foreground query and distinct unseen queries after production background preparation; report initialization time separately. Warm p95 alone does not satisfy the first-use requirement.
• Test migrations, interrupted work, invalid documents, inaccessible/offline roots and cloud placeholders.
• Automated native build matrix and documented supported cross-compilation paths.
• Execute Linux/macOS platform tests; create Windows/macOS/Linux installers.
• Signing/notarization configuration, upgrade/rollback validation, release instructions.

6. Local agents and optional answers

• Read-only MCP stdio server using the shared search service, persistent Google worker, structured filters, source citations and measured search stats.
• End-to-end MCP initialization, discovery, search, invalid-input and source-access tests; no protocol text mixed with diagnostic stdout.
• Installable agent skill and copyable MCP client configuration, with clear database scope, readiness and untrusted-source handling.
• Optional OpenAI-compatible summaries/questions, disabled by default; explicit selected-source action, endpoint disclosure, bounded payloads, credentials kept out of persistent settings, diagnostics and error messages.
• Verify real HTTP wire format against a local test endpoint; reject invalid settings, stale sources and unauthorized citations; preserve search on failure.
• Native UI interaction and chosen-provider quality/latency checks before claiming production-ready answers or compatibility with every provider.

7. EmbeddingGemma 2 efficiency

Audit and measured priorities: EMBEDDINGGEMMA2-AUDIT.md.

• Audit actual required model, task prefixes, pooling/normalization, native sequence processing and CPU/runtime policies against primary sources.
• Record real CPU 1/2/4-thread query/first-use/document measurements and paired repeated-query/filter-change results with unchanged retrieval evidence.
• Bounded provider-scoped actual-query vector LRU with accurate cache traces; preserve filtering, source freshness, failures and actual warmup probes.
• Reduce semantic settle delay to 120 ms and retain old ready generation during rebuilding; test initial-generation and activation readiness boundaries.
• Bound whole IPC write/read deadlines and recover from a nonreading helper.
• Declare/test x86 CPU minimum or baseline/AVX2 runtime selection.
• Incrementally reuse unchanged document/media vectors in atomic generations.
• Benchmark genuine token-budgeted multi-sequence batching and isolation.
• Implement and measure shared foreground/indexer resource budgets/priority.
• Attribute startup cost; measure smaller query context and selective encoders.
• Buffered bounded media IPC and linear UTF-8 source-window offset processing.
• Evaluate 256d MRL and content-specific vision budgets on independent data.
• Validate GPU precision, synchronization-aware timings, parity and fallback on actual accelerator hardware; CPU VM checks do not satisfy this gate.

Completion requires recorded evidence; targets are not measured results.

Completed foundation details

• SQLite FTS5 with exact filename priority, text snippets and strict filters.
• Default directory exclusions and protection against indexing the index itself.
• Metadata reuse, atomic root reconciliation and schema-v1 migration coverage.
• Actual CLI process integration tests and independent-reader concurrency test.
• TypeScript desktop frontend build and dependency audit passed; native validation underway.
• Initial README commit pushed to Gitea main via protected Infisical injection.
• Native desktop build and guarded opening tests passed on Windows.
• Genuine multilingual CPU model proof: 5/6 top-1, 6/6 recall@3; quality gate remains open.
• Native Rust CPU embedding adapter verifies pinned model/tokenizer assets; integration with the search index and platform runtime packaging remain open.
• PDF page and DOCX paragraph matches persist; corrupt/encrypted/over-limit documents remain filename-searchable and report extraction errors.
• CLI exposes folder/type/size/date filters and additional directory exclusions.

The partial features above do not close broader production milestones such as crash-resumable jobs, fuzzy retrieval, watcher reliability or performance targets.