78 lines
4.0 KiB
Markdown
78 lines
4.0 KiB
Markdown
|
# Source-code relevance checkpoint
|
|||
|
|
|
|||
|
|
Recorded 2026-10-12 using the actual verified Google EmbeddingGemma 2 Q8_0
|
|||
|
|
model, rebuilt native worker, shared hybrid service and MCP protocol. Windows
|
|||
|
|
VM, CPU only, one inference thread. No simulated provider or injected vectors.
|
|||
|
|
|
|||
|
|
Google describes local code retrieval in its
|
|||
|
|
[launch blog](https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/).
|
|||
|
|
Its [model card](https://ai.google.dev/gemma/docs/embeddinggemma/model_card_2)
|
|||
|
|
specifies `task: code retrieval | query: {query}` for queries and
|
|||
|
|
`title: {filename} | text: {code}` for indexed source. searchi uses that pairing
|
|||
|
|
in Code scope. Google's published benchmark measures their full-precision
|
|||
|
|
checkpoint; it is not a searchi benchmark.
|
|||
|
|
|
|||
|
|
## Frozen small regression
|
|||
|
|
|
|||
|
|
[The evaluator](../tools/code-eval/evaluate.py) fixes the implementations,
|
|||
|
|
queries and expected files before inference. Eight short source files use
|
|||
|
|
generic filenames; a ninth ordinary document deliberately contains search
|
|||
|
|
terms. All nine files are indexed, so code filtering must exclude that document
|
|||
|
|
before ranking. The expected code excerpts are checked byte-for-byte through
|
|||
|
|
MCP `read_source`, including line locations. Fixture hashes and every returned
|
|||
|
|
rank are recorded in [raw CPU evidence](evidence/code-search-cpu.json).
|
|||
|
|
|
|||
|
|
| Query intent | Source | Lexical rank | Hybrid rank |
|
|||
|
|
| --- | --- | --- | --- |
|
|||
|
|
| Streaming file SHA256 | Python | absent | 1 |
|
|||
|
|
| Lower bound in sorted numbers | C++ | absent | 1 |
|
|||
|
|
| Sum amounts by category | TypeScript | absent | 1 |
|
|||
|
|
| Retry requests with exponential backoff | Python | absent | 1 |
|
|||
|
|
| Release semaphore after async failure | C# | absent | 1 |
|
|||
|
|
| Escape unsafe HTML characters | PHP | absent | 1 |
|
|||
|
|
| Ten most expensive records | Python | absent | 1 |
|
|||
|
|
| File verification without full memory load, German | Python | absent | 1 |
|
|||
|
|
| Exact `refreshAccessToken` symbol | Rust | 1 | 1 |
|
|||
|
|
|
|||
|
|
Top-1 success: **9/9 hybrid, 1/9 lexical**. Every expected hybrid source was
|
|||
|
|
verified and no ordinary document leaked into Code scope. Measured MCP
|
|||
|
|
roundtrips for these nine uncached code queries ranged from **63.88–228.21 ms**,
|
|||
|
|
median **110.76 ms**. Those timings exclude process/model startup and describe
|
|||
|
|
this tiny corpus only; they are not desktop first-use or large-project targets.
|
|||
|
|
|
|||
|
|
## Defect discovered and fixed
|
|||
|
|
|
|||
|
|
The first actual indexing attempt failed on TypeScript: Google's tokenizer
|
|||
|
|
normalized source characters, invalidating a detokenization-based byte-offset
|
|||
|
|
assumption. The worker now retains original UTF-8 source, checks the complete
|
|||
|
|
prefixed token budget, and splits normalized long inputs on original character
|
|||
|
|
boundaries. It returns exact original slices with bounded overlap and complete
|
|||
|
|
tail coverage. Short code that fits uses a direct fast path. A separate actual
|
|||
|
|
CPU regression covers the failing TypeScript plus roughly 38 KB of repeated
|
|||
|
|
code with German, Japanese and emoji. Fixtures were not edited to hide failure.
|
|||
|
|
|
|||
|
|
Additional checks cover strict filtering before lexical/ANN candidate limits,
|
|||
|
|
named build files, overlapping routine context, schema upgrades, task-specific
|
|||
|
|
query caching, actual code/general vector separation and debug-vector parity.
|
|||
|
|
|
|||
|
|
## Reproduce and limits
|
|||
|
|
|
|||
|
|
Build the CLI and provision the pinned CPU worker/model as documented in
|
|||
|
|
[BUILDING.md](BUILDING.md) and [SEMANTIC.md](SEMANTIC.md), then run:
|
|||
|
|
|
|||
|
|
```powershell
|
|||
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|||
|
|
python tools/code-eval/evaluate.py
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The script uses the repository's generated `.searchi/code-eval` folder and
|
|||
|
|
default local runtime/model paths. It does not index private project folders.
|
|||
|
|
Outputs stay in that generated folder until deliberately reviewed and copied
|
|||
|
|
into documentation evidence.
|
|||
|
|
|
|||
|
|
These are synthetic, short implementations with one intended answer per query.
|
|||
|
|
They do not prove general project accuracy, declaration-first ranking,
|
|||
|
|
call-graph understanding or reliable no-answer detection. Extraction is
|
|||
|
|
line-based, not an AST parser. Independent real repositories, difficult
|
|||
|
|
near-duplicates, more languages, native UI interaction, other operating systems
|
|||
|
|
and actual GPU execution remain open evaluation gates.
|