karl
81846b3e98
4.0 KiB
Source-code relevance checkpoint
Recorded 2026-10-12 using the actual verified Google EmbeddingGemma 2 Q8_0 model, rebuilt native worker, shared hybrid service and MCP protocol. Windows VM, CPU only, one inference thread. No simulated provider or injected vectors.
Google describes local code retrieval in its
launch blog.
Its model card
specifies task: code retrieval | query: {query} for queries and
title: {filename} | text: {code} for indexed source. searchi uses that pairing
in Code scope. Google's published benchmark measures their full-precision
checkpoint; it is not a searchi benchmark.
Frozen small regression
The evaluator fixes the implementations,
queries and expected files before inference. Eight short source files use
generic filenames; a ninth ordinary document deliberately contains search
terms. All nine files are indexed, so code filtering must exclude that document
before ranking. The expected code excerpts are checked byte-for-byte through
MCP read_source, including line locations. Fixture hashes and every returned
rank are recorded in raw CPU evidence.
| Query intent | Source | Lexical rank | Hybrid rank |
|---|---|---|---|
| Streaming file SHA256 | Python | absent | 1 |
| Lower bound in sorted numbers | C++ | absent | 1 |
| Sum amounts by category | TypeScript | absent | 1 |
| Retry requests with exponential backoff | Python | absent | 1 |
| Release semaphore after async failure | C# | absent | 1 |
| Escape unsafe HTML characters | PHP | absent | 1 |
| Ten most expensive records | Python | absent | 1 |
| File verification without full memory load, German | Python | absent | 1 |
Exact refreshAccessToken symbol |
Rust | 1 | 1 |
Top-1 success: 9/9 hybrid, 1/9 lexical. Every expected hybrid source was verified and no ordinary document leaked into Code scope. Measured MCP roundtrips for these nine uncached code queries ranged from 63.88–228.21 ms, median 110.76 ms. Those timings exclude process/model startup and describe this tiny corpus only; they are not desktop first-use or large-project targets.
Defect discovered and fixed
The first actual indexing attempt failed on TypeScript: Google's tokenizer normalized source characters, invalidating a detokenization-based byte-offset assumption. The worker now retains original UTF-8 source, checks the complete prefixed token budget, and splits normalized long inputs on original character boundaries. It returns exact original slices with bounded overlap and complete tail coverage. Short code that fits uses a direct fast path. A separate actual CPU regression covers the failing TypeScript plus roughly 38 KB of repeated code with German, Japanese and emoji. Fixtures were not edited to hide failure.
Additional checks cover strict filtering before lexical/ANN candidate limits, named build files, overlapping routine context, schema upgrades, task-specific query caching, actual code/general vector separation and debug-vector parity.
Reproduce and limits
Build the CLI and provision the pinned CPU worker/model as documented in BUILDING.md and SEMANTIC.md, then run:
$env:SEARCHI_FORCE_CPU = '1'
python tools/code-eval/evaluate.py
The script uses the repository's generated .searchi/code-eval folder and
default local runtime/model paths. It does not index private project folders.
Outputs stay in that generated folder until deliberately reviewed and copied
into documentation evidence.
These are synthetic, short implementations with one intended answer per query. They do not prove general project accuracy, declaration-first ranking, call-graph understanding or reliable no-answer detection. Extraction is line-based, not an AST parser. Independent real repositories, difficult near-duplicates, more languages, native UI interaction, other operating systems and actual GPU execution remain open evaluation gates.