searchi/crates/searchi-core/README.md

Searchi core

The core has no desktop or inference dependencies. Its current retrieval engine combines literal filename matching with SQLite FTS5 BM25 keyword search. It does not implement semantic search.

SearchIndex::open creates or migrates schema version 2. Current-schema opens do not acquire a write lock. WAL allows independent readers to use the committed index while reconciliation runs. All indexing changes are committed atomically; a traversal or supported-file read error rolls back the entire reconciliation.

index_folder applies IndexOptions::default(): directories named .git, node_modules, and target are pruned. index_folder_with_options lets callers replace those names. Matching directory names are ASCII case insensitive. An explicit root is always traversed, even if its name is excluded. Symlinks are not followed. The index's own database, WAL, and shared-memory files are excluded.

Metadata includes byte size and modification time at nanosecond resolution. Unchanged files bypass extraction. IndexReport.indexed counts newly extracted or updated records; unchanged counts reused records; skipped counts new or changed records whose content could not be extracted because of format, encoding, or size. Unsupported files remain searchable by filename. Content reads are limited to 2 MiB. Reconciliation tracks seen files in SQLite rather than retaining every path in process memory.

search_with_options supports canonical folder boundaries, extension, inclusive size bounds, and inclusive Unix millisecond modification-time bounds. Filters apply before limiting results. search retains the unfiltered API. Query text is escaped as literal FTS terms, not interpreted as operators. Results are capped at 1,000 and queries at 4,096 bytes.

Known limits

• Filename substring lookup uses instr and scans metadata. A dedicated substring index and benchmarks are needed before claiming million-file latency targets.
• Files changed without changing size or modification time require a future forced refresh/content-fingerprint option. Timestamp precision depends on the filesystem. Indexing does not yet run a filesystem watcher.
• Metadata modification times before the Unix epoch or beyond the nanosecond integer range are explicit errors. Unicode filename case folding is currently limited by SQLite's built-in lower function.
• Lossy path conversion does not preserve non-Unicode native filenames. This must be addressed before Linux production support for arbitrary byte filenames.
• The files.root ownership model does not yet support overlapping selected roots independently. Use disjoint roots until root memberships are implemented.
• Extraction runs in the caller and a large reconciliation holds one transaction. Bounded worker isolation, persistent jobs, and crash-resumable scans remain production milestones.
• PDF, DOCX, OCR, media extraction, and real model inference remain unimplemented.