karl
81846b3e98
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
Core CI / core (ubuntu-latest) (push) Canceled after 2m23s
79 lines
4.0 KiB
Markdown
79 lines
4.0 KiB
Markdown
# Source-code relevance checkpoint
|
||
|
||
Recorded 2026-10-12 using the actual verified Google EmbeddingGemma 2 Q8_0
|
||
model, rebuilt native worker, shared hybrid service and MCP protocol. Windows
|
||
VM, CPU only, one inference thread. No simulated provider or injected vectors.
|
||
|
||
Google describes local code retrieval in its
|
||
[launch blog](https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/).
|
||
Its [model card](https://ai.google.dev/gemma/docs/embeddinggemma/model_card_2)
|
||
specifies `task: code retrieval | query: {query}` for queries and
|
||
`title: {filename} | text: {code}` for indexed source. searchi uses that pairing
|
||
in Code scope. Google's published benchmark measures their full-precision
|
||
checkpoint; it is not a searchi benchmark.
|
||
|
||
## Frozen small regression
|
||
|
||
[The evaluator](../tools/code-eval/evaluate.py) fixes the implementations,
|
||
queries and expected files before inference. Eight short source files use
|
||
generic filenames; a ninth ordinary document deliberately contains search
|
||
terms. All nine files are indexed, so code filtering must exclude that document
|
||
before ranking. The expected code excerpts are checked byte-for-byte through
|
||
MCP `read_source`, including line locations. Fixture hashes and every returned
|
||
rank are recorded in [raw CPU evidence](evidence/code-search-cpu.json).
|
||
|
||
| Query intent | Source | Lexical rank | Hybrid rank |
|
||
| --- | --- | --- | --- |
|
||
| Streaming file SHA256 | Python | absent | 1 |
|
||
| Lower bound in sorted numbers | C++ | absent | 1 |
|
||
| Sum amounts by category | TypeScript | absent | 1 |
|
||
| Retry requests with exponential backoff | Python | absent | 1 |
|
||
| Release semaphore after async failure | C# | absent | 1 |
|
||
| Escape unsafe HTML characters | PHP | absent | 1 |
|
||
| Ten most expensive records | Python | absent | 1 |
|
||
| File verification without full memory load, German | Python | absent | 1 |
|
||
| Exact `refreshAccessToken` symbol | Rust | 1 | 1 |
|
||
|
||
Top-1 success: **9/9 hybrid, 1/9 lexical**. Every expected hybrid source was
|
||
verified and no ordinary document leaked into Code scope. Measured MCP
|
||
roundtrips for these nine uncached code queries ranged from **63.88–228.21 ms**,
|
||
median **110.76 ms**. Those timings exclude process/model startup and describe
|
||
this tiny corpus only; they are not desktop first-use or large-project targets.
|
||
|
||
## Defect discovered and fixed
|
||
|
||
The first actual indexing attempt failed on TypeScript: Google's tokenizer
|
||
normalized source characters, invalidating a detokenization-based byte-offset
|
||
assumption. The worker now retains original UTF-8 source, checks the complete
|
||
prefixed token budget, and splits normalized long inputs on original character
|
||
boundaries. It returns exact original slices with bounded overlap and complete
|
||
tail coverage. Short code that fits uses a direct fast path. A separate actual
|
||
CPU regression covers the failing TypeScript plus roughly 38 KB of repeated
|
||
code with German, Japanese and emoji. Fixtures were not edited to hide failure.
|
||
|
||
Additional checks cover strict filtering before lexical/ANN candidate limits,
|
||
named build files, overlapping routine context, schema upgrades, task-specific
|
||
query caching, actual code/general vector separation and debug-vector parity.
|
||
|
||
## Reproduce and limits
|
||
|
||
Build the CLI and provision the pinned CPU worker/model as documented in
|
||
[BUILDING.md](BUILDING.md) and [SEMANTIC.md](SEMANTIC.md), then run:
|
||
|
||
```powershell
|
||
$env:SEARCHI_FORCE_CPU = '1'
|
||
python tools/code-eval/evaluate.py
|
||
```
|
||
|
||
The script uses the repository's generated `.searchi/code-eval` folder and
|
||
default local runtime/model paths. It does not index private project folders.
|
||
Outputs stay in that generated folder until deliberately reviewed and copied
|
||
into documentation evidence.
|
||
|
||
These are synthetic, short implementations with one intended answer per query.
|
||
They do not prove general project accuracy, declaration-first ranking,
|
||
call-graph understanding or reliable no-answer detection. Extraction is
|
||
line-based, not an AST parser. Independent real repositories, difficult
|
||
near-duplicates, more languages, native UI interaction, other operating systems
|
||
and actual GPU execution remain open evaluation gates.
|