232 lines
15 KiB
Markdown
232 lines
15 KiB
Markdown
# Production readiness roadmap
|
|
|
|
Status: implementation started; no production release yet. Windows VM tests use CPU only.
|
|
|
|
## Required product scope (user clarification)
|
|
|
|
- Desktop UI is primary; semantic setup/indexing/search must require no terminal.
|
|
- Google EmbeddingGemma 2 is the required production model, matching DigUp.
|
|
- Local document, image, audio and video search are required product milestones.
|
|
- A separate CLI-only build supports headless Linux without desktop dependencies.
|
|
- Production automatically uses available hardware acceleration with CPU fallback;
|
|
this Windows VM is tested in forced-CPU mode only.
|
|
- Installer ships the native worker. First launch automatically downloads/verifies
|
|
missing model files and manages indexing in the UI; no terminal setup is required.
|
|
- Earlier MiniLM proof is an experiment and does not satisfy the model requirement.
|
|
- First interaction must be responsive: prepare indexes and a real Google query
|
|
in the background, retain ready sessions, and show immediate keyword results.
|
|
Model download and initialization must have honest progress and retry states.
|
|
|
|
## 1. Search foundation
|
|
- [x] Rust workspace, reusable engine, CLI, versioned SQLite storage.
|
|
- [ ] Filename/path and ranked full-text retrieval with strict filters.
|
|
- [ ] Safe folder discovery, exclusions, bounded extraction, passage locations.
|
|
- [ ] Incremental updates, rename/deletion reconciliation, crash recovery.
|
|
- [x] Persistence and filesystem lifecycle integration tests (Windows CPU).
|
|
|
|
## 2. Document and semantic retrieval
|
|
- [x] Text, Markdown, code, PDF and DOCX extraction with real fixtures (Windows CPU).
|
|
- [ ] Validate actual embedding model artifacts, license, CPU runtime and quantization.
|
|
- [ ] Versioned embeddings, ANN index, exact-match priority and hybrid fusion.
|
|
- [ ] English/German relevance corpus and measured top-10 recall.
|
|
- [ ] Atomic model/index replacement and resumable persistent work queues.
|
|
- [x] Completed-source manifests with title/model/window policy keys and
|
|
citation remapping; reuse only verified complete document/media groups.
|
|
- [ ] Durable generation tasks, crash resume and final corpus reconciliation;
|
|
see [incremental semantic design](INCREMENTAL-SEMANTIC.md).
|
|
|
|
## 3. Desktop
|
|
- [ ] Tauri shell, accessible keyboard search, progressive results and previews (shell built; UI interaction verification pending).
|
|
- [ ] Folder selection, exclusion controls, resource budgets and indexing status.
|
|
- [ ] Global shortcut, opening/revealing source files, PDF page navigation.
|
|
- [ ] Watchers and reconciliation across Windows, macOS and Linux.
|
|
- [x] Native notification coordinator, bounded targeted reconciliation,
|
|
periodic verification and CPU-tested Windows/Linux create/update/rename/delete.
|
|
- [x] Durable same-stat media hint invalidation, guarded metadata writes and
|
|
real Google reindex/reuse regression; owned decoder scratch avoids feedback.
|
|
- [x] Native Linux watcher-driven UI refresh, independent refresh timing,
|
|
temporary selection restoration and explicit selection override.
|
|
- [ ] Native macOS events, Windows/macOS watcher-driven UI refresh and representative storage
|
|
failure/contention checks; see [automatic indexing](AUTOMATIC-INDEXING.md).
|
|
|
|
## 4. Media
|
|
- [x] Genuine Google image, audio and video embeddings integrated with the shared
|
|
search service and CPU-tested FFmpeg decoding; music tags and duration filters.
|
|
- [x] Frozen independent media corpus with retrieval, timestamp, negative-query
|
|
and indexing-cost measurements (see MEDIA-QUALITY.md).
|
|
- [ ] Improve precise spoken-passage localization, music hard negatives and
|
|
calibrated handling of queries without a relevant match.
|
|
- [ ] Package and verify decoders on every release OS; test production acceleration.
|
|
- [x] Native Linux x86-64 pinned decoder provenance, startup, WAV probe and CPU
|
|
decode; broader codec/ABI coverage and desktop packaging remain open.
|
|
- [ ] CPU OCR, image retrieval, audio transcription, video segment locations.
|
|
- [ ] Isolated extraction workers with timeouts and resource ceilings.
|
|
|
|
## 5. Release gates
|
|
- [x] Normal worker build/staging defaults to hardware-detecting auto with CPU
|
|
fallback; missing Vulkan/glslc fails configuration rather than hiding GPU loss.
|
|
- [ ] Provision release-build GPU SDKs and verify one hardware-detecting package
|
|
per OS/architecture, including missing-driver fallback and dependency closure.
|
|
- [x] Windows-x64 official pinned SDK copied into workspace with verified cache
|
|
provenance; universal CPU/Vulkan worker and embedded-frontend desktop built.
|
|
- [x] Real Windows archive integrity, packaged decoder startup and auto-mode
|
|
CPU inference with a controlled no-GPU-provider registry; Google CPU parity.
|
|
- [x] Normal Windows x64 archive includes baseline/optimized CPU dispatch,
|
|
OS AVX-state guards, shared app-local MSVC CRT and hashed DLL inventory;
|
|
real optimized and baseline-only package inference passes in the CPU VM.
|
|
- [x] Normal native Linux x86-64 CPU/Vulkan worker build, hashed 19-library
|
|
staging and genuine optimized/baseline-only CPU code-query inference outside
|
|
the build tree; desktop, ABI/older-hardware and GPU validation remain open.
|
|
- [x] Actual Linux automatic CPU fallback with Vulkan loader, ICDs/drivers and
|
|
GPU devices absent in a verified private namespace; no GPU provider loaded.
|
|
- [ ] Verify CPU dispatch on older physical CPUs and a clean Windows machine;
|
|
dependency inspection and baseline execution on a modern VM are insufficient.
|
|
- [ ] Actual missing/incompatible GPU drivers, precision, device loss and memory
|
|
pressure on release hardware; native GUI and clean-machine archive launch.
|
|
- [x] Windows desktop archive packager includes explicit runtime/helper files
|
|
and notices, checks architecture and records per-file checksums; excludes
|
|
model caches and user data. Controlled packaging regression tests pass.
|
|
- [x] Embedded-frontend Windows Release archive launches outside the source tree
|
|
with isolated data and a ready real Google CPU semantic generation.
|
|
- [ ] Verify native interactions, clean-machine prerequisites, GPU dependencies,
|
|
upgrades and signing.
|
|
- [x] Preflight native license and the complete offline Google model pair before
|
|
modifying staged resources; corrupted/missing assets preserve existing worker.
|
|
- [x] Transactional resource staging, verified copied bytes, process locking,
|
|
interrupted-copy/swap recovery and backup preservation on failed rollback.
|
|
- [ ] Verify power-loss durability and release-tool interaction with resource
|
|
directory swaps on each OS; fault-injected process interruption is narrower.
|
|
- [x] Shared monotonic Basic/Debug search metrics, granular bounded engine traces,
|
|
actual worker query diagnostics, and headless JSON performance output.
|
|
- [ ] Verify detailed performance UI interaction on native desktop platforms;
|
|
repeat overhead measurements on representative libraries and hardware.
|
|
- [ ] Formatting, lint, tests and security/dependency review.
|
|
- [ ] CPU benchmarks: filename p95 <50 ms/1M records; text <150 ms/100K documents; semantic <500 ms on declared hardware.
|
|
- [ ] Measure cold start, indexing throughput, RAM, disk and idle CPU.
|
|
- [ ] Record the very first foreground query and distinct unseen queries after
|
|
production background preparation; report initialization time separately.
|
|
Warm p95 alone does not satisfy the first-use requirement.
|
|
- [ ] Test migrations, interrupted work, invalid documents, inaccessible/offline roots and cloud placeholders.
|
|
- [ ] Automated native build matrix and documented supported cross-compilation paths.
|
|
- [x] Native Linux headless workspace tests: 106 passed, 18 explicitly ignored.
|
|
- [x] Native Linux Google text/code/media reference parity and service
|
|
lifecycle/rebuild recovery: seven genuine CPU integration tests passed.
|
|
- [x] Unsigned Linux x86-64 headless archive: verify extracted checksums/modes,
|
|
mixed document/code/audio indexing, decoder discovery, semantic retrieval,
|
|
line citations and strict filters outside the source/build directories.
|
|
- [ ] Execute native Linux video/codec coverage and Linux desktop/macOS
|
|
platform tests; create Windows/macOS/Linux installers.
|
|
- [ ] Signing/notarization configuration, upgrade/rollback validation, release instructions.
|
|
|
|
## 6. Local agents and optional answers
|
|
|
|
- [x] Read-only MCP stdio server using the shared search service, persistent
|
|
Google worker, structured filters, source citations and measured search stats.
|
|
- [x] End-to-end MCP initialization, discovery, search, invalid-input and
|
|
source-access tests; no protocol text mixed with diagnostic stdout.
|
|
- [x] Installable agent skill and copyable MCP client configuration, with clear
|
|
database scope, readiness and untrusted-source handling.
|
|
- [x] Optional OpenAI-compatible summaries/questions, disabled by default;
|
|
explicit selected-source action, endpoint disclosure, bounded payloads,
|
|
credentials kept out of persistent settings, diagnostics and error messages.
|
|
- [x] Verify real HTTP wire format against a local test endpoint; reject invalid
|
|
settings, stale sources and unauthorized citations; preserve search on failure.
|
|
- [ ] Native UI interaction and chosen-provider quality/latency checks before
|
|
claiming production-ready answers or compatibility with every provider.
|
|
|
|
## 7. EmbeddingGemma 2 efficiency
|
|
|
|
Audit and measured priorities: [EMBEDDINGGEMMA2-AUDIT.md](EMBEDDINGGEMMA2-AUDIT.md).
|
|
|
|
- [x] Audit actual required model, task prefixes, pooling/normalization, native
|
|
sequence processing and CPU/runtime policies against primary sources.
|
|
- [x] Record real CPU 1/2/4-thread query/first-use/document measurements and
|
|
paired repeated-query/filter-change results with unchanged retrieval evidence.
|
|
- [x] Bounded provider-scoped actual-query vector LRU with accurate cache traces;
|
|
preserve filtering, source freshness, failures and actual warmup probes.
|
|
- [x] Bounded shared prepared-ANN handoff before SQL activation; retain query
|
|
model/LRU and readiness across publication. Actual CPU first new query has
|
|
no foreground graph load; [recovery/handoff evidence](RECOVERY-AND-HANDOFF.md).
|
|
- [x] Reduce semantic settle delay to 120 ms and retain old ready generation
|
|
during rebuilding; test initial-generation and activation readiness boundaries.
|
|
- [x] Bound whole IPC write/read deadlines and recover from a nonreading helper;
|
|
CPU-only real helper-process regressions cover blocked writes and responses.
|
|
- [x] Windows x64 baseline/optimized CPU runtime selection and OS AVX-state
|
|
checks, verified with real packaged baseline and optimized inference.
|
|
- [ ] Validate CPU selection on older hardware and remaining architectures.
|
|
- [x] Completed-source vector reuse in atomic schema-7 generations, verified
|
|
with genuine Google CPU document/code/image/audio/video fixtures.
|
|
- [x] Resume completed staging sources after failed publication; incomplete
|
|
groups are purged under the OS build lock before retry.
|
|
- [x] Pre-inference staging pruning with compatible rename remapping and
|
|
100,000-chunk storage boundary; late validation after ANN preparation.
|
|
- [x] Exact parser-input SHA provenance and same-metadata overwrite detection
|
|
on normal rescans; stale/missing agent excerpts fail closed.
|
|
- [x] Actual Google CPU same-metadata text mutation before/during inference:
|
|
reject publication, rescan without metadata workarounds and reuse unaffected vectors.
|
|
- [x] Abrupt owned CLI termination after a committed partial audio chunk;
|
|
resume completed documents and reencode unfinished media without duplicates.
|
|
Process termination does not prove power-loss durability.
|
|
- [ ] Concurrent deletion/media mutation, component-policy reuse and
|
|
indexing/foreground measurements.
|
|
See [INCREMENTAL-SEMANTIC.md](INCREMENTAL-SEMANTIC.md).
|
|
- [ ] Benchmark genuine token-budgeted multi-sequence batching and isolation.
|
|
- [ ] Implement and measure shared foreground/indexer resource budgets/priority.
|
|
- [x] Shared inference admission, foreground priority between background
|
|
encodes, total-thread settings migration and retained warm sessions on save
|
|
failure; real Google CPU overlap regression passed on Windows.
|
|
- [ ] Measure representative mixed-media contention, single-thread latency,
|
|
extraction/decoder/ANN work and native UI p95 before closing this gate.
|
|
- [ ] Attribute startup cost; measure smaller query context and selective encoders.
|
|
- [x] Query-only 512-token context in the normal worker, desktop/CLI/MCP
|
|
integration and actual Windows CPU parity/boundary/reference tests.
|
|
Paired samples reduced private commitment by 183 MB; resident memory and
|
|
cached startup were essentially unchanged. See [QUERY-CONTEXT.md](QUERY-CONTEXT.md).
|
|
- [ ] Verify native Linux/macOS, GPU parity and updated installed packages.
|
|
- [ ] Evaluate text readiness before the larger media projector finishes
|
|
downloading; preserve verified assets and honest media setup status.
|
|
- [ ] Buffered bounded media IPC and linear UTF-8 source-window offset processing.
|
|
- [ ] Evaluate 256d MRL and content-specific vision budgets on independent data.
|
|
- [ ] Validate GPU precision, synchronization-aware timings, parity and fallback
|
|
on actual accelerator hardware; CPU VM checks do not satisfy this gate.
|
|
|
|
Completion requires recorded evidence; targets are not measured results.
|
|
|
|
## 8. Source-code retrieval
|
|
|
|
- [x] Shared code classification, strict lexical/ANN filtering before top-k,
|
|
expanded source/build formats, bounded overlapping passages and actual lines.
|
|
- [x] Desktop Code scope, CLI `--code-only`, read-only MCP `code_only`, immutable
|
|
scope snapshots and code-format-preserving previews.
|
|
- [x] Google's CodeRetrieval query instruction, explicit worker capabilities,
|
|
task-scoped vector cache and genuine CPU prompt/debug parity.
|
|
- [x] Exact original source windows when Google's tokenizer normalizes text;
|
|
verified UTF-8 offsets, full tail coverage and actual token-budget checks.
|
|
- [x] Schema-11 migration schedules unchanged source for re-extraction; tests
|
|
preserve ordinary documents and filename availability through the upgrade.
|
|
- [x] Frozen 9-query real Google CPU regression with lexical comparison,
|
|
ordinary-document exclusion and exact MCP source excerpt verification.
|
|
- [ ] Evaluate independent real repositories, more languages, overloaded
|
|
symbols, near-duplicate routines and queries with no relevant implementation.
|
|
- [ ] Validate code search UI interaction on native Windows/macOS/Linux and
|
|
actual GPU hardware. Tiny CPU fixtures do not close those release gates.
|
|
|
|
## Completed foundation details
|
|
|
|
- SQLite FTS5 with exact filename priority, text snippets and strict filters.
|
|
- Default directory exclusions and protection against indexing the index itself.
|
|
- Metadata reuse, atomic root reconciliation and schema-v1 migration coverage.
|
|
- Actual CLI process integration tests and independent-reader concurrency test.
|
|
- TypeScript desktop frontend build and dependency audit passed; native validation underway.
|
|
- Initial README commit pushed to Gitea main via protected Infisical injection.
|
|
- Native desktop build and guarded opening tests passed on Windows.
|
|
- Genuine multilingual CPU model proof: 5/6 top-1, 6/6 recall@3; quality gate remains open.
|
|
- Native Rust CPU embedding adapter verifies pinned model/tokenizer assets;
|
|
integration with the search index and platform runtime packaging remain open.
|
|
- PDF page and DOCX paragraph matches persist; corrupt/encrypted/over-limit
|
|
documents remain filename-searchable and report extraction errors.
|
|
- CLI exposes folder/type/size/date filters and additional directory exclusions.
|
|
|
|
The partial features above do not close broader production milestones such as
|
|
crash-resumable jobs, fuzzy retrieval, watcher reliability or performance targets.
|