searchi/docs/EMBEDDINGGEMMA2-AUDIT.md

# EmbeddingGemma 2: efficiency and performance audit

Research and Windows CPU measurements, 2026-10-12. **The required Google model
is integrated correctly, but the surrounding pipeline still has substantial
efficiency and media-quality work.** This audit keeps fresh inference, cached
query vectors, indexing throughput and user-visible delays separate.

## What is correct

The pinned model is `google/embeddinggemma-2`, with Q8_0 backbone/projector
artifacts verified by size and SHA-256. It is not the older text-only
EmbeddingGemma or cloud Gemini Embedding 2. Our runtime uses its dedicated
architecture, bidirectional encoding, mean pooling and L2-normalized 768d output.
Text-only query workers retain the model; media projector initialization is lazy.
No language-model generation or network round trip is involved in search.

Our search-query and titled-document prefixes match Google's instructions.
Overlapping 2,048-token document windows are a deliberate runtime policy below
the model's 8,192-token limit; verified byte offsets retain the passage tail.
Audio is mono 16 kHz. Native runtime parity was independently checked earlier;
that does not establish broad retrieval quality.
[Google's model card and guidance](https://ai.google.dev/gemma/docs/embeddinggemma/model_card_2),
[local adapter](../crates/searchi-semantic/src/gemma.rs),
[native worker](../crates/searchi-semantic/native/worker.cpp).

## Actual CPU measurements

Host: Windows x86-64 VM, four exposed Ryzen 7 7800X3D logical CPUs and about
9.9 GiB RAM. No GPU inference, rendering or media decoding was used. Native
worker is a Release build; MCP profile is identified separately below.
OS file pages may already be warm. Tests used public/generated fixture content.

### Fresh model inference

Eight text queries, three repetitions each, with real Google vectors checked
for finite values, 768 dimensions and unit length. Each request still runs the
encoder; this table has no query cache.

| CPU threads | Warm p50 | Warm p95 | First query after worker starts |
| --- | ---: | ---: | ---: |
| 1 | 74.90 ms | 86.59 ms | 113.29 ms |
| 2 | 40.80 ms | 46.66 ms | 64.25 ms |
| 4 | 34.16 ms | 49.86 ms | 49.98 ms |

Includes Python NDJSON encode/decode and worker inference; excludes the Rust
adapter, ANN, UI and media encoders. Thread configurations ran sequentially,
not randomized. These 24 samples per setting are a diagnostic, not a production
p95 guarantee. Two threads had the best observed tail; four improved the median
and document throughput but did not improve this query tail.
[Raw samples and worker counters](evidence/embeddinggemma2-native-cpu.json).

Direct worker process startup took 653–816 ms; a separate verified backbone
checksum took 170 ms in Python. These are not the whole desktop startup.
Actual Release MCP startup through verified assets, shared service and real
warmup took **4.88 seconds**. Its first unseen foreground search took **81.95 ms**.
This exposes an additional startup gap to profile, without attributing it to an
unmeasured stage or weakening artifact verification.
[Release shared-service sample](evidence/embeddinggemma2-mcp-release.json).

### Repeated searches and filter changes

We added a 32-entry LRU of **actual query vectors**, scoped to the retained
provider/fingerprint. Exact trimmed queries reuse their vector. Filters, source
freshness, ANN retrieval and ranking run again; complete search results are not
cached. The cache is bounded, in memory only, and never seeded with the warmup
query to disguise first use. Debug traces distinguish hits from real inference.

Same seven-file corpus, native CPU worker, one thread and Rust Debug executable:

| Case | Before mean | After mean | Samples per run |
| --- | ---: | ---: | ---: |
| Repeat an already searched query | 71.19 ms | 1.20 ms | 8 |
| Change filters for that query | 70.94 ms | 1.07 ms | 4 |

All **20 compared result sets** retained identical filenames, source locations,
snippets, scores and order, including seven different descriptive queries.
The uncached first query still invoked Google: 82.99 ms before, 77.17 ms after;
that small difference is not an optimization claim. The later Release run
measured 0.75 ms mean for repeats and 0.70 ms for filter changes. There is no
paired old-Release run, so do not call this a Release before/after speedup.
[Before](evidence/embeddinggemma2-mcp-before.json),
[after](evidence/embeddinggemma2-mcp-after.json).

The old CLI binary hash was not captured; its source checkpoint was `9870df7`.
The new reports include executable hashes. No other inference benchmark was
intentionally run concurrently, but compilation/background VM work can contend.

### Indexing is a different workload

Eight short documents in one native IPC request took 3.64 s / 1.87 s / 1.24 s
with 1 / 2 / 4 threads. Eight separate requests took 3.84 s / 1.89 s / 1.24 s.
The single/batch vector cosine minimum was 0.999999997. **Our eight-document
request is serial inference, not genuine multi-sequence batching.** It mainly
reduces IPC overhead. These are single comparisons, not throughput distributions.

Earlier independent media indexing took 668 seconds for 20 assets/118 chunks on
one CPU thread. Frame/recording costs and quality cannot be inferred from fast
text queries. Precise spoken-topic localization, music confusion and no-answer
handling remain failed gates; the earlier experimental media rating is still
4/10. This cache change does not improve that score.
[Actual media evidence](MEDIA-QUALITY.md).

## Changes made from this audit

- Added the bounded real-vector cache, with failure/eviction/fingerprint,
  filtering/source lifecycle and trace tests.
- Reduced the intentional semantic UI delay from 350 to 120 ms; the separate
  60 ms lexical debounce and bounded request lanes remain. That removes 230 ms
  of configured waiting, not a measured 230 ms compositor improvement.
- Kept an existing ready semantic generation searchable during rebuilds.
  Initial indexing still needs a generation; activation/warmup still withdraws
  backend readiness. Foreground/indexer CPU contention is not solved by this.
- Updated capture validation to use an unseen query when recording actual
  worker-inference counters, rather than mislabeling a cache hit.

## Highest-impact next work

| Priority | Evidence / proposed work | Required acceptance evidence |
| --- | --- | --- |
| P0 | Bound the whole IPC exchange: synchronous writes currently precede the response timeout. | Nonreading-helper regression; large request deadline and recovery. |
| P0 | Decide the x86 CPU baseline: current static build enables AVX2 without runtime CPU variants. | Declared minimum ISA or tested baseline/AVX2 dispatch; older-CPU launch. |
| P1 | Reuse unchanged document/media vectors; replacement builds currently re-encode all sources. | One-file change embeds only affected chunks; atomic activation and stale/deletion tests. |
| P1 | Pack independent short sequences by token budget; the native worker currently encodes serially. | Vector isolation/parity, length-bucketed CPU/GPU throughput and memory comparisons. |
| P1 | Give interactive queries a shared resource budget/priority during indexing. | First-unseen and warm p95 with real background document/media jobs. |
| P1 | Profile startup stages and compare a smaller query-only context. | Verification/load/graph-warmup/RSS measured separately; no fake first-query cache. |
| P2 | Buffer bounded IPC reads, remove unnecessary stream tying, and make window offset work linear. | Protocol bounds, UTF-8 offsets/tails and real media/payload timing parity. |
| P2 | Consider modality-selective projector loading; current lazy combined projector loads both encoders. | Real text/image/audio RSS and load measurements on supported backends. |

[DigUp's pinned runtime](https://github.com/ARahim3/DigUp/blob/6faa29e4c315a3a591de1b41bd78751bdf39588d/Sources/LlamaRuntime/LlamaEmbedder.swift)
demonstrates independent-sequence packing and a smaller text-query context.
Those are useful patterns, not proof of an identical speedup on this VM.

Google supports re-normalized MRL truncation to 512/256/128d. A 256d experiment
could reduce vector storage threefold; it does **not** eliminate the model's
forward pass. Keep 768d until independent retrieval tests pass. Higher vision
budgets trade cost for detail, so retain content-specific quality experiments.
[Google's truncation and modality guidance](https://ai.google.dev/gemma/docs/embeddinggemma/model_card_2).

GPU execution remains untested here. Audit its precision against Google's
BF16/FP32 guidance, and move the inference timing boundary past asynchronous
embedding synchronization before interpreting GPU stage traces. CPU-only results
do not validate accelerator throughput, quality or failure recovery.

## Reproduce

The benchmark requires already verified model assets, a compiled CPU worker and
an explicitly selected fixture index. It does not download or index user files.
Read its scope fields and raw samples before comparing hardware or profiles.

```powershell
python tools/embeddinggemma2/benchmark.py --mode native --models .searchi/models/embeddinggemma2 --worker .searchi/runtime/cpu/Release/searchi-gemma-worker.exe --threads 1 2 4 --repeats 3 --output .searchi/native-performance.json
python tools/embeddinggemma2/benchmark.py --mode mcp --models .searchi/models/embeddinggemma2 --worker .searchi/runtime/cpu/Release/searchi-gemma-worker.exe --cli target/release/searchi.exe --db .searchi/desktop-capture/index.db --repeats 8 --rust-profile release --output .searchi/mcp-performance.json
```