karl
f77112db97
Core CI / core (ubuntu-latest) (push) Successful in 1m0s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
50 lines
2.9 KiB
Markdown
50 lines
2.9 KiB
Markdown
# Searchi core
|
|
|
|
The core has no desktop or inference dependencies. Its current retrieval engine
|
|
combines literal filename matching with SQLite FTS5 BM25 keyword search. It does
|
|
not implement semantic search.
|
|
|
|
`SearchIndex::open` creates or migrates schema version 2. Current-schema opens do
|
|
not acquire a write lock. WAL allows independent readers to use the committed
|
|
index while reconciliation runs. All indexing changes are committed atomically;
|
|
a traversal or supported-file read error rolls back the entire reconciliation.
|
|
|
|
`index_folder` applies `IndexOptions::default()`: directories named `.git`,
|
|
`node_modules`, and `target` are pruned. `index_folder_with_options` lets callers
|
|
replace those names. Matching directory names are ASCII case insensitive. An
|
|
explicit root is always traversed, even if its name is excluded. Symlinks are not
|
|
followed. The index's own database, WAL, and shared-memory files are excluded.
|
|
|
|
Metadata includes byte size and modification time at nanosecond resolution.
|
|
Unchanged files bypass extraction. `IndexReport.indexed` counts newly extracted
|
|
or updated records; `unchanged` counts reused records; `skipped` counts new or
|
|
changed records whose content could not be extracted because of format, encoding,
|
|
or size. Unsupported files remain searchable by filename. Content reads are
|
|
limited to 2 MiB. Reconciliation tracks seen files in SQLite rather than retaining
|
|
every path in process memory.
|
|
|
|
`search_with_options` supports canonical folder boundaries, extension, inclusive
|
|
size bounds, and inclusive Unix millisecond modification-time bounds. Filters
|
|
apply before limiting results. `search` retains the unfiltered API. Query text is
|
|
escaped as literal FTS terms, not interpreted as operators. Results are capped at
|
|
1,000 and queries at 4,096 bytes.
|
|
|
|
## Known limits
|
|
|
|
- Filename substring lookup uses `instr` and scans metadata. A dedicated substring
|
|
index and benchmarks are needed before claiming million-file latency targets.
|
|
- Files changed without changing size or modification time require a future
|
|
forced refresh/content-fingerprint option. Timestamp precision depends on the
|
|
filesystem. Indexing does not yet run a filesystem watcher.
|
|
- Metadata modification times before the Unix epoch or beyond the nanosecond
|
|
integer range are explicit errors. Unicode filename case folding is currently
|
|
limited by SQLite's built-in `lower` function.
|
|
- Lossy path conversion does not preserve non-Unicode native filenames. This must
|
|
be addressed before Linux production support for arbitrary byte filenames.
|
|
- The `files.root` ownership model does not yet support overlapping selected
|
|
roots independently. Use disjoint roots until root memberships are implemented.
|
|
- Extraction runs in the caller and a large reconciliation holds one transaction.
|
|
Bounded worker isolation, persistent jobs, and crash-resumable scans remain
|
|
production milestones.
|
|
- PDF, DOCX, OCR, media extraction, and real model inference remain unimplemented.
|