karl
f77112db97
2.1 KiB
Architecture and boundaries
The Rust engine owns storage and search. Its callers do not construct SQL. The CLI and desktop shell share the engine; the Tauri bridge runs filesystem work outside the UI thread using independent database connections.
The current lexical index is a foundation. It uses SQLite WAL and FTS5 with exact-name priority and source snippets. Reconciliation is transactional: incomplete traversal must not delete previously indexed files. A root that cannot be read produces an error, not an empty successful reconciliation.
The next storage design separates metadata discovery, persistent extraction jobs and passage publication. Bounded workers publish small batches so search can see newly indexed names before a complete root finishes. A generation marker only removes stale entries after a successful walk. Watchers enqueue changes; reconciliation catches lost events. Overlapping roots need explicit ownership before root removal is implemented.
Semantic inference is a replaceable CPU adapter with explicit model artifact download, pinned checksums, normalization and versioned chunk vectors. Search must never download a model implicitly. No synthetic embeddings qualify as semantic search. An exact lexical match keeps priority during hybrid fusion.
Document extractors eventually run in isolated workers with input size, timeout and output limits. PDF pages, code lines and media timestamps are stored as source locations, not inferred by the frontend.
Indexes contain private extracted text. No telemetry or remote search is enabled. Model downloads and release checks must be explicit operations.
Known foundation limits
- Substring filename search scans the metadata table; the million-file target requires an additional substring index and measured query planning.
- Initial indexing publishes after the root transaction finishes.
- File extraction currently covers bounded UTF-8 text, not PDF/DOCX/media.
- No automatic watcher, persistent extraction queue or ANN index yet.
- Native CI and signed package validation require actual platform runners.