# Source-code relevance checkpoint

Recorded 2026-10-12 using the actual verified Google EmbeddingGemma 2 Q8_0
model, rebuilt native worker, shared hybrid service and MCP protocol. Windows
VM, CPU only, one inference thread. No simulated provider or injected vectors.

Google describes local code retrieval in its
[launch blog](https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/).
Its [model card](https://ai.google.dev/gemma/docs/embeddinggemma/model_card_2)
specifies `task: code retrieval | query: {query}` for queries and
`title: {filename} | text: {code}` for indexed source. searchi uses that pairing
in Code scope. Google's published benchmark measures their full-precision
checkpoint; it is not a searchi benchmark.

## Frozen small regression

[The evaluator](../tools/code-eval/evaluate.py) fixes the implementations,
queries and expected files before inference. Eight short source files use
generic filenames; a ninth ordinary document deliberately contains search
terms. All nine files are indexed, so code filtering must exclude that document
before ranking. The expected code excerpts are checked byte-for-byte through
MCP `read_source`, including line locations. Fixture hashes and every returned
rank are recorded in [raw CPU evidence](evidence/code-search-cpu.json).

| Query intent | Source | Lexical rank | Hybrid rank |
| --- | --- | --- | --- |
| Streaming file SHA256 | Python | absent | 1 |
| Lower bound in sorted numbers | C++ | absent | 1 |
| Sum amounts by category | TypeScript | absent | 1 |
| Retry requests with exponential backoff | Python | absent | 1 |
| Release semaphore after async failure | C# | absent | 1 |
| Escape unsafe HTML characters | PHP | absent | 1 |
| Ten most expensive records | Python | absent | 1 |
| File verification without full memory load, German | Python | absent | 1 |
| Exact `refreshAccessToken` symbol | Rust | 1 | 1 |

Top-1 success: **9/9 hybrid, 1/9 lexical**. Every expected hybrid source was
verified and no ordinary document leaked into Code scope. Measured MCP
roundtrips for these nine uncached code queries ranged from **63.88–228.21 ms**,
median **110.76 ms**. Those timings exclude process/model startup and describe
this tiny corpus only; they are not desktop first-use or large-project targets.

## Defect discovered and fixed

The first actual indexing attempt failed on TypeScript: Google's tokenizer
normalized source characters, invalidating a detokenization-based byte-offset
assumption. The worker now retains original UTF-8 source, checks the complete
prefixed token budget, and splits normalized long inputs on original character
boundaries. It returns exact original slices with bounded overlap and complete
tail coverage. Short code that fits uses a direct fast path. A separate actual
CPU regression covers the failing TypeScript plus roughly 38 KB of repeated
code with German, Japanese and emoji. Fixtures were not edited to hide failure.

Additional checks cover strict filtering before lexical/ANN candidate limits,
named build files, overlapping routine context, schema upgrades, task-specific
query caching, actual code/general vector separation and debug-vector parity.

## Reproduce and limits

Build the CLI and provision the pinned CPU worker/model as documented in
[BUILDING.md](BUILDING.md) and [SEMANTIC.md](SEMANTIC.md), then run:

```powershell
$env:SEARCHI_FORCE_CPU = '1'
python tools/code-eval/evaluate.py
```

The script uses the repository's generated `.searchi/code-eval` folder and
default local runtime/model paths. It does not index private project folders.
Outputs stay in that generated folder until deliberately reviewed and copied
into documentation evidence.

These are synthetic, short implementations with one intended answer per query.
They do not prove general project accuracy, declaration-first ranking,
call-graph understanding or reliable no-answer detection. Extraction is
line-based, not an AST parser. Independent real repositories, difficult
near-duplicates, more languages, native UI interaction, other operating systems
and actual GPU execution remain open evaluation gates.