feat: integrate Google semantic search and branded desktop launcher
Core CI / core (ubuntu-latest) (push) Failing after 3m27s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m45s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
Core CI / core (ubuntu-latest) (push) Failing after 3m27s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m45s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
This commit is contained in:
1 parent
a90df15503
commit
caf6a2f82e
80 files changed
+14971
-837
No files matched your search
@@ -4,7 +4,7 @@ The core has no desktop or inference dependencies. Its current retrieval engine
|
||||
combines literal filename matching with SQLite FTS5 BM25 keyword search. It does
|
||||
not implement semantic search.
|
||||
|
||||
`SearchIndex::open` creates or migrates schema version 7. Current-schema opens do
|
||||
`SearchIndex::open` creates or migrates schema version 9. Current-schema opens do
|
||||
not acquire a write lock. WAL allows independent readers to use the committed
|
||||
index while reconciliation runs. All indexing changes are committed atomically;
|
||||
a traversal or plain-text file read error rolls back the entire reconciliation.
|
||||
@@ -82,20 +82,33 @@ Exact filenames always precede other lanes. Ties use persistent file row IDs,
|
||||
and keyword ties use relevance then row ID. Selected rows are deduplicated.
|
||||
|
||||
Filename/path substrings of at least three Unicode characters use an
|
||||
external-content trigram FTS index. The rarest known query trigram supplies a
|
||||
candidate superset; the whole literal substring is verified on the candidate.
|
||||
external-content trigram FTS index. A positional anchor grows around the rarest
|
||||
known query trigram, admitting only neighboring grams within four times its
|
||||
document frequency (with a small floor) and at most 32 Unicode characters.
|
||||
This rejects repeated-identifier false candidates inside FTS without extending
|
||||
through dense common suffix postings. The anchor supplies a candidate superset;
|
||||
the whole literal substring is still verified on each candidate.
|
||||
At most 256 unique query trigrams are inspected. This lookup budget does not cap
|
||||
candidate rows or hide matching results. All size/type/folder/date filters apply
|
||||
before each lane's SQL LIMIT. Shorter substrings retain complete metadata scans
|
||||
and can be slow. Broad queries under restrictive filters can also examine many
|
||||
matching candidates; no universal filtered-query latency claim is made.
|
||||
|
||||
Trigram frequencies use the FTS vocabulary and a bounded 4,096-term cache on each
|
||||
`SearchIndex`. Initial frequency lookups can be expensive; the cache is cleared
|
||||
after reconciliation. Keep an independent reader engine alive across searches to
|
||||
benefit from warm behavior. These frequency estimates affect candidate selection
|
||||
cost, not correctness. A source query batches all finalists' indexed passage IDs
|
||||
and selects actual matching passages before choosing one source per file.
|
||||
`prewarm()` refreshes persisted trigram frequencies from the FTS vocabulary only
|
||||
when names/path membership is dirty, then loads a bounded 4,096-term reader cache.
|
||||
Foreground frequency misses use an indexed persisted table, not vocabulary
|
||||
postings scans. Name/path insert/delete/change marks statistics dirty; content
|
||||
changes alone do not. Statistics affect candidate cost, not match correctness.
|
||||
Background preparation exercises actual first/last filenames, alphabetic name
|
||||
tokens, and three words of the first document. UI callers should complete this
|
||||
preparation before enabling interaction. Representative inputs are disclosed in
|
||||
the benchmark report; preparation is not counted as foreground query time.
|
||||
|
||||
Unfiltered content retrieval materializes ranked row IDs before reading finalist
|
||||
metadata and snippets. Filtered requests retain complete filter-before-LIMIT
|
||||
ranking. Tests compare ordering, scores, snippets, and sources between both paths.
|
||||
A source query batches all finalists' indexed passage IDs and chooses one actual
|
||||
matching source per file. Keep an independent reader alive across searches.
|
||||
|
||||
Migration to the new indexes preserves file content and passage IDs and rebuilds
|
||||
both filename/path and passage indexes atomically. Large migrations can take time
|
||||
@@ -105,7 +118,8 @@ opens. The measured Windows synthetic fixture grew from about 295 MiB to 605 MiB
|
||||
## Known limits
|
||||
|
||||
- Short substring queries and broad restrictive filters can still scan many
|
||||
metadata candidates. Cold frequency-cache fills exceed the warm filename target.
|
||||
metadata candidates. Statistics preparation may take seconds after large scans;
|
||||
arbitrary unseen or OS-cold queries have no universal latency guarantee.
|
||||
- Files changed without changing size or modification time require a future
|
||||
forced refresh/content-fingerprint option. Timestamp precision depends on the
|
||||
filesystem. Indexing does not yet run a filesystem watcher.
|
||||
|
||||
Reference in new issue
Block a user