feat: integrate Google semantic search and branded desktop launcher
Core CI / core (ubuntu-latest) (push) Failing after 3m27s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m45s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s

This commit is contained in:
karl committed 2026-10-12 17:44:56 +02:00
1 parent a90df15503
commit caf6a2f82e
80 files changed
+14971 -837

No files matched your search

+24 -10
View File
@@ -4,7 +4,7 @@ The core has no desktop or inference dependencies. Its current retrieval engine
combines literal filename matching with SQLite FTS5 BM25 keyword search. It does
not implement semantic search.
`SearchIndex::open` creates or migrates schema version 7. Current-schema opens do
`SearchIndex::open` creates or migrates schema version 9. Current-schema opens do
not acquire a write lock. WAL allows independent readers to use the committed
index while reconciliation runs. All indexing changes are committed atomically;
a traversal or plain-text file read error rolls back the entire reconciliation.
@@ -82,20 +82,33 @@ Exact filenames always precede other lanes. Ties use persistent file row IDs,
and keyword ties use relevance then row ID. Selected rows are deduplicated.
Filename/path substrings of at least three Unicode characters use an
external-content trigram FTS index. The rarest known query trigram supplies a
candidate superset; the whole literal substring is verified on the candidate.
external-content trigram FTS index. A positional anchor grows around the rarest
known query trigram, admitting only neighboring grams within four times its
document frequency (with a small floor) and at most 32 Unicode characters.
This rejects repeated-identifier false candidates inside FTS without extending
through dense common suffix postings. The anchor supplies a candidate superset;
the whole literal substring is still verified on each candidate.
At most 256 unique query trigrams are inspected. This lookup budget does not cap
candidate rows or hide matching results. All size/type/folder/date filters apply
before each lane's SQL LIMIT. Shorter substrings retain complete metadata scans
and can be slow. Broad queries under restrictive filters can also examine many
matching candidates; no universal filtered-query latency claim is made.
Trigram frequencies use the FTS vocabulary and a bounded 4,096-term cache on each
`SearchIndex`. Initial frequency lookups can be expensive; the cache is cleared
after reconciliation. Keep an independent reader engine alive across searches to
benefit from warm behavior. These frequency estimates affect candidate selection
cost, not correctness. A source query batches all finalists' indexed passage IDs
and selects actual matching passages before choosing one source per file.
`prewarm()` refreshes persisted trigram frequencies from the FTS vocabulary only
when names/path membership is dirty, then loads a bounded 4,096-term reader cache.
Foreground frequency misses use an indexed persisted table, not vocabulary
postings scans. Name/path insert/delete/change marks statistics dirty; content
changes alone do not. Statistics affect candidate cost, not match correctness.
Background preparation exercises actual first/last filenames, alphabetic name
tokens, and three words of the first document. UI callers should complete this
preparation before enabling interaction. Representative inputs are disclosed in
the benchmark report; preparation is not counted as foreground query time.
Unfiltered content retrieval materializes ranked row IDs before reading finalist
metadata and snippets. Filtered requests retain complete filter-before-LIMIT
ranking. Tests compare ordering, scores, snippets, and sources between both paths.
A source query batches all finalists' indexed passage IDs and chooses one actual
matching source per file. Keep an independent reader alive across searches.
Migration to the new indexes preserves file content and passage IDs and rebuilds
both filename/path and passage indexes atomically. Large migrations can take time
@@ -105,7 +118,8 @@ opens. The measured Windows synthetic fixture grew from about 295 MiB to 605 MiB
## Known limits
- Short substring queries and broad restrictive filters can still scan many
metadata candidates. Cold frequency-cache fills exceed the warm filename target.
metadata candidates. Statistics preparation may take seconds after large scans;
arbitrary unseen or OS-cold queries have no universal latency guarantee.
- Files changed without changing size or modification time require a future
forced refresh/content-fingerprint option. Timestamp precision depends on the
filesystem. Indexing does not yet run a filesystem watcher.