searchi/docs/ROADMAP.md

Production readiness roadmap

Status: Windows and Linux v0.1.4 previews published; initial Windows v0.1.0 installed. Broader production gates remain open. Windows VM tests use CPU only.

Karl's release priority, 2026-10-13: publish a usable first Windows version and install it on DEV-WIN11 now. Extensive platform/GPU, clean-machine, signing and large-library checks are deferred follow-up, not blockers for this initial release. Keep their evidence and unchecked production gates intact.

Karl's current improvement priority: music relevance, precise spoken-passage timestamps and irrelevant results when the requested content is absent. The million-record lexical benchmark is deferred while this quality work proceeds. Google EmbeddingGemma 2 remains the main text/image/audio/video search model.

• Evaluate actual whitelisted music metadata as additional Google text evidence, retaining raw audio and real metadata source citations.
• Evaluate finer local ASR token/word timing and source-preserving phrase windows with Google text retrieval; preserve every competitor and reject invalid timestamps.
• Validate richer no-answer evidence on new independent positive/negative data; do not retune the failed universal cosine cutoff on old held-out results.
• Integrate accepted improvements into shared engine/UI/headless clients with verified setup, lifecycle and CPU performance evidence. Diagnostics alone do not raise the media quality rating.
• Publish v0.1.0 Windows installer/checksum, verify public download bytes, install on DEV-WIN11 and leave the app running with a desktop launcher.
• Publish Windows desktop v0.1.1 with single-instance handling; verify public bytes and isolated genuine Google CPU launch. Existing user installation is not automatically upgraded. See release evidence.
• Publish Windows desktop v0.1.2 with incomplete AI-answer rejection; verify public installer/checksum bytes and isolated genuine CPU package launch. See release evidence; user installation not auto-upgraded.
• Publish Windows desktop v0.1.3 with isolated document processing and bounded media previews; installer helper and isolated CPU startup verified. See evidence. Native UI playback and upgrade checks stay open.
• Publish the Linux x86-64 headless preview with current query-role runtime, verified public bytes and genuine document/code/audio CPU archive checks on Ubuntu 24.04 WSL. See headless evidence; desktop, GPU and broader distribution compatibility remain open.

Required product scope (user clarification)

• Desktop UI is primary; semantic setup/indexing/search must require no terminal.
• Google EmbeddingGemma 2 is the required production model, matching DigUp.
• Local document, image, audio and video search are required product milestones.
• A separate CLI-only build supports headless Linux without desktop dependencies.
• Production automatically uses available hardware acceleration with CPU fallback; this Windows VM is tested in forced-CPU mode only.
• Installer ships the native worker. First launch automatically downloads/verifies missing model files and manages indexing in the UI; no terminal setup is required.
• Earlier MiniLM proof is an experiment and does not satisfy the model requirement.
• First interaction must be responsive: prepare indexes and a real Google query in the background, retain ready sessions, and show immediate keyword results. Model download and initialization must have honest progress and retry states.

1. Search foundation

• Rust workspace, reusable engine, CLI, versioned SQLite storage.
• Filename/path and ranked full-text retrieval with strict filters.
• Safe folder discovery, exclusions, bounded extraction, passage locations.
• Incremental updates, rename/deletion reconciliation, crash recovery.
• Persistence and filesystem lifecycle integration tests (Windows CPU).

2. Document and semantic retrieval

• Text, Markdown, code, PDF and DOCX extraction with real fixtures (Windows CPU).
• Validate actual embedding model artifacts, license, CPU runtime and quantization.
• Versioned embeddings, ANN index, exact-match priority and hybrid fusion.
• English/German relevance corpus and measured top-10 recall.
• Atomic model/index replacement and resumable persistent work queues.
• Completed-source manifests with title/model/window policy keys and citation remapping; reuse only verified complete document/media groups.
• Durable generation tasks, crash resume and final corpus reconciliation; see incremental semantic design.

3. Desktop

• Image preview pixel ceilings use actual decoded format; a real PNG renamed .jpg cannot receive the larger JPEG allowance (Windows CPU regression).
• Replace lifetime-growing native media asset grants with bounded preview access, including range limits, stale-source handling and selection/reload tests. Do not use permanent forbidden patterns as revocation.
• Validate native custom-protocol image/audio/video playback and seeking on release platforms; structural Windows CPU tests alone do not close this gate. See preview contract and evidence.
  • Linux WSL native Rust preview tests (9/9) and full desktop suite (45/45) passed on the immutable implementation snapshot; actual playback remains open.
• Official single-instance transport with isolated explicit profiles; Windows CPU duplicate launch exits without a second query worker. See launch evidence; visual focus and other OS checks remain open.
• Persisted solid/glass/frosted appearance and bounded background tint; opaque reading surfaces, serialized settings updates and high-contrast fallback.
• Verify native glass visually on Windows 11 22H2+; macOS/Linux currently use an explicit opaque fallback. Native compositor performance remains open.
• Tauri shell, accessible keyboard search, progressive results and previews (shell built; UI interaction verification pending).
• Folder selection, exclusion controls, resource budgets and indexing status.
• Global shortcut, opening/revealing source files, PDF page navigation.
• Watchers and reconciliation across Windows, macOS and Linux.
  • Native notification coordinator, bounded targeted reconciliation, periodic verification and CPU-tested Windows/Linux create/update/rename/delete.
  • Durable same-stat media hint invalidation, guarded metadata writes and real Google reindex/reuse regression; owned decoder scratch avoids feedback.
  • Native Linux watcher-driven UI refresh, independent refresh timing, temporary selection restoration and explicit selection override.
  • Native macOS events, Windows/macOS watcher-driven UI refresh and representative storage failure/contention checks; see automatic indexing.

4. Media

• Genuine Google image, audio and video embeddings integrated with the shared search service and CPU-tested FFmpeg decoding; music tags and duration filters.
• Frozen independent media corpus with retrieval, timestamp, negative-query and indexing-cost measurements (see MEDIA-QUALITY.md).
• Improve precise spoken-passage localization, music hard negatives and calibrated handling of queries without a relevant match.
  • Prepare bounded CPU speech diagnostics and test process failures, timestamp units and frozen-label scoring; see speech proposal.
  • Build a pinned CPU VAD diagnostic with explicit one-thread inference, native error detection and full probabilities; actual Silero silence/input contracts passed. Media quality comparison remains open (see SPEECH-VAD.md).
  • Complete frozen five-source VAD/base-ASR/Google CPU diagnostic: component costs decreased, but localization stayed 4/8 point and 6/8 overlap; no production default selected. Independent held-out media quality remains open.
  • Run genuine local speech candidates, Google transcript retrieval and held-out English/German/music/no-speech checks before selecting a default.
• Package and verify decoders on every release OS; test production acceleration.
• Native Linux x86-64 pinned decoder provenance, startup, WAV probe and CPU decode; broader codec/ABI coverage and desktop packaging remain open.
• CPU OCR, image retrieval, audio transcription, video segment locations.
• Isolated extraction workers with timeouts and resource ceilings. See implementation subgoals and provenance contract.

5. Release gates

• Normal worker build/staging defaults to hardware-detecting auto with CPU fallback; missing Vulkan/glslc fails configuration rather than hiding GPU loss.
• Provision release-build GPU SDKs and verify one hardware-detecting package per OS/architecture, including missing-driver fallback and dependency closure.
• Windows-x64 official pinned SDK copied into workspace with verified cache provenance; universal CPU/Vulkan worker and embedded-frontend desktop built.
• Real Windows archive integrity, packaged decoder startup and auto-mode CPU inference with a controlled no-GPU-provider registry; Google CPU parity.
• Normal Windows x64 archive includes baseline/optimized CPU dispatch, OS AVX-state guards, shared app-local MSVC CRT and hashed DLL inventory; real optimized and baseline-only package inference passes in the CPU VM.
• Normal native Linux x86-64 CPU/Vulkan worker build, hashed 19-library staging and genuine optimized/baseline-only CPU code-query inference outside the build tree; desktop, ABI/older-hardware and GPU validation remain open.
• Actual Linux automatic CPU fallback with Vulkan loader, ICDs/drivers and GPU devices absent in a verified private namespace; no GPU provider loaded.
• Verify CPU dispatch on older physical CPUs and a clean Windows machine; dependency inspection and baseline execution on a modern VM are insufficient.
• Actual missing/incompatible GPU drivers, precision, device loss and memory pressure on release hardware; native GUI and clean-machine archive launch.
• Windows desktop archive packager includes explicit runtime/helper files and notices, checks architecture and records per-file checksums; excludes model caches and user data. Controlled packaging regression tests pass.
• Embedded-frontend Windows Release archive launches outside the source tree with isolated data and a ready real Google CPU semantic generation.
• Verify native interactions, clean-machine prerequisites, GPU dependencies, upgrades and signing.
• Preflight native license and the complete offline Google model pair before modifying staged resources; corrupted/missing assets preserve existing worker.
• Transactional resource staging, verified copied bytes, process locking, interrupted-copy/swap recovery and backup preservation on failed rollback.
• Normal Windows/Linux source-build provenance gate before staging and archive preparation; stale inventories preserve existing resources/packages. Actual normal builds verified. See runtime provenance.
• Prepare static macOS Metal inventory and mandatory normal-profile staging, with embedded-source/dependency/architecture metadata regressions.
• Execute native macOS runtime build, relocation and genuine CPU/Metal inference; fixture metadata tests do not satisfy this gate.
• Verify power-loss durability and release-tool interaction with resource directory swaps on each OS; fault-injected process interruption is narrower.
• Shared monotonic Basic/Debug search metrics, granular bounded engine traces, actual worker query diagnostics, and headless JSON performance output.
• Verify detailed performance UI interaction on native desktop platforms; repeat overhead measurements on representative libraries and hardware.
• Formatting, lint, tests and security/dependency review.
  • Inventory locked Rust/npm advisories; prepare byte-verified compatible upstream GLib iterator backport for the Linux desktop dependency graph.
  • Execute optimized Linux GLib backport regressions: three tests and the owner-lifetime compile-fail test passed; stored log hash and all 121 upstream source files reverified.
  • Native Linux 0.1.4 desktop build and packaged CPU startup use the selected local GLib backport; native graphical interaction remains unchecked.
  • Execute cargo-audit 0.22.2 against both current lockfiles with a fresh RustSec database and warnings denied; zero vulnerability-class findings, two tracked maintenance notices and exit 1. See DEPENDENCY-REVIEW.md.
  • Complete native graphical interaction and broader native runtime/source security review. No broad security gate is closed by these narrow checks.
• CPU benchmarks: filename p95 <50 ms/1M records; text <150 ms/100K documents; semantic <500 ms on declared hardware.
• Measure cold start, indexing throughput, RAM, disk and idle CPU.
• Record the very first foreground query and distinct unseen queries after production background preparation; report initialization time separately. Warm p95 alone does not satisfy the first-use requirement.
• Test migrations, interrupted work, invalid documents, inaccessible/offline roots and cloud placeholders.
• Automated native build matrix and documented supported cross-compilation paths.
• Native Linux headless workspace tests: 106 passed, 18 explicitly ignored.
• Native Linux Google text/code/media reference parity and service lifecycle/rebuild recovery: seven genuine CPU integration tests passed.
• Unsigned Linux x86-64 headless archive: verify extracted checksums/modes, mixed document/code/audio indexing, decoder discovery, semantic retrieval, line citations and strict filters outside the source/build directories.
• Execute native Linux video/codec coverage and Linux desktop/macOS platform tests; create Windows/macOS/Linux installers.
• Signing/notarization configuration, upgrade/rollback validation, release instructions.

6. Local agents and optional answers

• Read-only MCP stdio server using the shared search service, persistent Google worker, structured filters, source citations and measured search stats.
• End-to-end MCP initialization, discovery, search, invalid-input and source-access tests; no protocol text mixed with diagnostic stdout.
• Installable agent skill and copyable MCP client configuration, with clear database scope, readiness and untrusted-source handling.
• Optional OpenAI-compatible summaries/questions, disabled by default; explicit selected-source action, endpoint disclosure, bounded payloads, credentials kept out of persistent settings, diagnostics and error messages.
• Verify real HTTP wire format against a local test endpoint; reject invalid settings, stale sources and unauthorized citations; preserve search on failure.
• Native UI interaction and chosen-provider quality/latency checks before claiming production-ready answers or compatibility with every provider.

7. EmbeddingGemma 2 efficiency

Audit and measured priorities: EMBEDDINGGEMMA2-AUDIT.md.

• Audit actual required model, task prefixes, pooling/normalization, native sequence processing and CPU/runtime policies against primary sources.
• Record real CPU 1/2/4-thread query/first-use/document measurements and paired repeated-query/filter-change results with unchanged retrieval evidence.
• Bounded provider-scoped actual-query vector LRU with accurate cache traces; preserve filtering, source freshness, failures and actual warmup probes.
• Bounded shared prepared-ANN handoff before SQL activation; retain query model/LRU and readiness across publication. Actual CPU first new query has no foreground graph load; recovery/handoff evidence.
• Reduce semantic settle delay to 120 ms and retain old ready generation during rebuilding; test initial-generation and activation readiness boundaries.
• Bound whole IPC write/read deadlines and recover from a nonreading helper; CPU-only real helper-process regressions cover blocked writes and responses.
• Windows x64 baseline/optimized CPU runtime selection and OS AVX-state checks, verified with real packaged baseline and optimized inference.
• Validate CPU selection on older hardware and remaining architectures.
• Completed-source vector reuse in atomic schema-7 generations, verified with genuine Google CPU document/code/image/audio/video fixtures.
• Resume completed staging sources after failed publication; incomplete groups are purged under the OS build lock before retry.
• Pre-inference staging pruning with compatible rename remapping and 100,000-chunk storage boundary; late validation after ANN preparation.
• Exact parser-input SHA provenance and same-metadata overwrite detection on normal rescans; stale/missing agent excerpts fail closed.
• Actual Google CPU same-metadata text mutation before/during inference: reject publication, rescan without metadata workarounds and reuse unaffected vectors.
• Abrupt owned CLI termination after a committed partial audio chunk; resume completed documents and reencode unfinished media without duplicates. Process termination does not prove power-loss durability.
• Concurrent deletion/media mutation, component-policy reuse and indexing/foreground measurements. See INCREMENTAL-SEMANTIC.md.
• Benchmark genuine token-budgeted multi-sequence batching and isolation.
• Implement and measure shared foreground/indexer resource budgets/priority.
  • Shared inference admission, foreground priority between background encodes, total-thread settings migration and retained warm sessions on save failure; real Google CPU overlap regression passed on Windows.
  • Measure representative mixed-media contention, single-thread latency, extraction/decoder/ANN work and native UI p95 before closing this gate.
• Attribute startup cost; measure smaller query context and selective encoders.
  • Query-only 512-token context in the normal worker, desktop/CLI/MCP integration and actual Windows CPU parity/boundary/reference tests. Paired samples reduced private commitment by 183 MB; resident memory and cached startup were essentially unchanged. See QUERY-CONTEXT.md.
  • Native Linux query-context and CLI/MCP cited-result checks, with six paired CPU process measurements; resident/PSS memory effectively unchanged.
  • Verify native macOS, GPU parity and updated installed packages.
  • Evaluate text readiness before the larger media projector finishes downloading; preserve verified assets and honest media setup status.
    • Typed backbone/projector verification callbacks with full-pair setup semantics preserved; 16 asset tests passed, including gated media download.
    • Integrate explicit text/media coverage, retained query readiness and media retry status across engine/desktop; genuine CPU promotion reuses identical document vectors with zero document re-encodes. See early text setup.
    • Native Linux CPU first use during genuine delayed media download, failure/retry and promotion; actual first-input timing recorded separately. Focused activation retest preserves selection and query process, records fresh timing/current generation after idle. Windows/macOS checks stay open.
• Buffered bounded media IPC and linear UTF-8 source-window offset processing.
• Evaluate 256d MRL and content-specific vision budgets on independent data.
• Validate GPU precision, synchronization-aware timings, parity and fallback on actual accelerator hardware; CPU VM checks do not satisfy this gate.

Completion requires recorded evidence; targets are not measured results.

8. Source-code retrieval

• Shared code classification, strict lexical/ANN filtering before top-k, expanded source/build formats, bounded overlapping passages and actual lines.
• Desktop Code scope, CLI --code-only, read-only MCP code_only, immutable scope snapshots and code-format-preserving previews.
• Google's CodeRetrieval query instruction, explicit worker capabilities, task-scoped vector cache and genuine CPU prompt/debug parity.
• Exact original source windows when Google's tokenizer normalizes text; verified UTF-8 offsets, full tail coverage and actual token-budget checks.
• Schema-11 migration schedules unchanged source for re-extraction; tests preserve ordinary documents and filename availability through the upgrade.
• Frozen 9-query real Google CPU regression with lexical comparison, ordinary-document exclusion and exact MCP source excerpt verification.
• Evaluate independent real repositories, more languages, overloaded symbols, near-duplicate routines and queries with no relevant implementation.
  • Audit mixed-task routing and eligibility cache behavior; see routing audit.
  • Complete the unchanged 32-file/91-query real Google CPU mixed baseline: Code scope 14/14 top-1 versus All files 10/14; all 10 no-answer queries returned candidates. See mixed quality.
  • Measure task fusion and cache cost against independently reviewed held-out data, preserving the frozen diagnostic for regression comparison.
    • Freeze disjoint bilingual text/code calibration and held-out labels after independent AI review; verify exact Git-preserved source/label bytes.
    • Add a shared-engine calibration-only runner retaining raw cosine, lexical provenance, hybrid ranks and atomic incomplete evidence.
    • Execute the frozen calibration-only baseline with genuine Google CPU inference; preserve raw results and leave held-out requests unevaluated.
    • Freeze the proposed relevance policy, then evaluate held-out queries without tuning against their results.
  • Explicit English/German code intent vetoes exclusive audio routing; actual Google CPU regression preserves code among qualifying audio candidates.
• Validate code search UI interaction on native Windows/macOS/Linux and actual GPU hardware. Tiny CPU fixtures do not close those release gates.

Completed foundation details

• SQLite FTS5 with exact filename priority, text snippets and strict filters.
• Default directory exclusions and protection against indexing the index itself.
• Metadata reuse, atomic root reconciliation and schema-v1 migration coverage.
• Actual CLI process integration tests and independent-reader concurrency test.
• TypeScript desktop frontend build and dependency audit passed; native validation underway.
• Initial README commit pushed to Gitea main via protected Infisical injection.
• Native desktop build and guarded opening tests passed on Windows.
• Genuine multilingual CPU model proof: 5/6 top-1, 6/6 recall@3; quality gate remains open.
• Native Rust CPU embedding adapter verifies pinned model/tokenizer assets; integration with the search index and platform runtime packaging remain open.
• PDF page and DOCX paragraph matches persist; corrupt/encrypted/over-limit documents remain filename-searchable and report extraction errors.
• CLI exposes folder/type/size/date filters and additional directory exclusions.

The partial features above do not close broader production milestones such as crash-resumable jobs, fuzzy retrieval, watcher reliability or performance targets.

Document worker follow-up

• Shared PDF/DOCX process adapter, exact-byte post-parser validation, cancellation rollback and desktop resource discovery. See evidence.
• Build and require the extraction helper in Windows release packages; extracted installer helper passed genuine PDF indexing.
• Build, verify and publish updated Linux headless archive with its required helper; genuine relocated PDF/DOCX and Google CPU checks passed.

Current shared-code check

At source fd67906, Windows CPU-only validation passed strict cargo clippy --workspace --all-targets --locked -- -D warnings and workspace formatting. The public v0.1.3 tag resolves to the recorded application source 3bf95583533ce44021c8ecd9733f32062488976e.

A bounded Infrastructure Ops System-catalog query for macOS test availability returned no registered Mac workstation. No Mac network availability or native build was inferred from that lookup; macOS execution gates remain unchecked.

• Current native Linux desktop release compilation and local loader closure on Ubuntu WSL; evidence.
• Linux desktop relocated normal resource discovery and CPU query-worker startup under an owned software-display test; no native UI interaction claim.
• Publish Ubuntu 24.04 x86-64 Debian desktop preview; verify public bytes and extracted package CPU startup. See evidence.
• Linux desktop graphical interaction and broader distribution compatibility.

Rust audit and goal review, 2026-10-13

• Audit core/service and desktop/CLI Rust boundaries; avoid unnecessary naming/module churn.
• Correct valid pre-1970 filesystem timestamps across indexing, source eligibility and previews. Changed crates passed 100 tests (six ignored), desktop 47; strict Clippy and formatting passed. See audit.
• Publish Windows v0.1.4 with the timestamp correction; verify public bytes, packaged helper provenance and isolated real Google CPU query-worker startup. See evidence.
• Publish updated Linux headless and Debian packages with the timestamp correction; genuine archived-document CPU search, unchanged rescan and packaged query-worker startup passed. Public bytes verified; evidence.
• Protect mixed quality evidence from accidental report overwrite and partial JSON publication; eight evaluator contract tests passed.
• Finish the unchanged frozen mixed regression on the current 0.1.4 engine; independent score recomputation confirms one known routing improvement and no scored regression. All-files positive top-1 74/80, recall@5 80/80; no-answer failures remain 10/10. See comparison; independent quality gates stay open.

Initial Windows delivery is complete. Native interaction/upgrade, actual GPU, macOS execution and independent media-quality work remain real follow-up; the full production goal remains active. Preserve the detailed unchecked gates above.