karl
8b9b079469
9.4 KiB
EmbeddingGemma 2: efficiency and performance audit
Research and Windows CPU measurements, 2026-10-12. The required Google model is integrated correctly, but the surrounding pipeline still has substantial efficiency and media-quality work. This audit keeps fresh inference, cached query vectors, indexing throughput and user-visible delays separate.
What is correct
The pinned model is google/embeddinggemma-2, with Q8_0 backbone/projector
artifacts verified by size and SHA-256. It is not the older text-only
EmbeddingGemma or cloud Gemini Embedding 2. Our runtime uses its dedicated
architecture, bidirectional encoding, mean pooling and L2-normalized 768d output.
Text-only query workers retain the model; media projector initialization is lazy.
No language-model generation or network round trip is involved in search.
Our search-query and titled-document prefixes match Google's instructions. Overlapping 2,048-token document windows are a deliberate runtime policy below the model's 8,192-token limit; verified byte offsets retain the passage tail. Audio is mono 16 kHz. Native runtime parity was independently checked earlier; that does not establish broad retrieval quality. Google's model card and guidance, local adapter, native worker.
Actual CPU measurements
Host: Windows x86-64 VM, four exposed Ryzen 7 7800X3D logical CPUs and about 9.9 GiB RAM. No GPU inference, rendering or media decoding was used. Native worker is a Release build; MCP profile is identified separately below. OS file pages may already be warm. Tests used public/generated fixture content.
Fresh model inference
Eight text queries, three repetitions each, with real Google vectors checked for finite values, 768 dimensions and unit length. Each request still runs the encoder; this table has no query cache.
| CPU threads | Warm p50 | Warm p95 | First query after worker starts |
|---|---|---|---|
| 1 | 74.90 ms | 86.59 ms | 113.29 ms |
| 2 | 40.80 ms | 46.66 ms | 64.25 ms |
| 4 | 34.16 ms | 49.86 ms | 49.98 ms |
Includes Python NDJSON encode/decode and worker inference; excludes the Rust adapter, ANN, UI and media encoders. Thread configurations ran sequentially, not randomized. These 24 samples per setting are a diagnostic, not a production p95 guarantee. Two threads had the best observed tail; four improved the median and document throughput but did not improve this query tail. Raw samples and worker counters.
Direct worker process startup took 653–816 ms; a separate verified backbone checksum took 170 ms in Python. These are not the whole desktop startup. Actual Release MCP startup through verified assets, shared service and real warmup took 4.88 seconds. Its first unseen foreground search took 81.95 ms. This exposes an additional startup gap to profile, without attributing it to an unmeasured stage or weakening artifact verification. Release shared-service sample.
Repeated searches and filter changes
We added a 32-entry LRU of actual query vectors, scoped to the retained provider/fingerprint. Exact trimmed queries reuse their vector. Filters, source freshness, ANN retrieval and ranking run again; complete search results are not cached. The cache is bounded, in memory only, and never seeded with the warmup query to disguise first use. Debug traces distinguish hits from real inference.
Same seven-file corpus, native CPU worker, one thread and Rust Debug executable:
| Case | Before mean | After mean | Samples per run |
|---|---|---|---|
| Repeat an already searched query | 71.19 ms | 1.20 ms | 8 |
| Change filters for that query | 70.94 ms | 1.07 ms | 4 |
All 20 compared result sets retained identical filenames, source locations, snippets, scores and order, including seven different descriptive queries. The uncached first query still invoked Google: 82.99 ms before, 77.17 ms after; that small difference is not an optimization claim. The later Release run measured 0.75 ms mean for repeats and 0.70 ms for filter changes. There is no paired old-Release run, so do not call this a Release before/after speedup. Before, after.
The old CLI binary hash was not captured; its source checkpoint was 9870df7.
The new reports include executable hashes. No other inference benchmark was
intentionally run concurrently, but compilation/background VM work can contend.
Indexing is a different workload
Eight short documents in one native IPC request took 3.64 s / 1.87 s / 1.24 s with 1 / 2 / 4 threads. Eight separate requests took 3.84 s / 1.89 s / 1.24 s. The single/batch vector cosine minimum was 0.999999997. Our eight-document request is serial inference, not genuine multi-sequence batching. It mainly reduces IPC overhead. These are single comparisons, not throughput distributions.
Earlier independent media indexing took 668 seconds for 20 assets/118 chunks on one CPU thread. Frame/recording costs and quality cannot be inferred from fast text queries. Precise spoken-topic localization, music confusion and no-answer handling remain failed gates; the earlier experimental media rating is still 4/10. This cache change does not improve that score. Actual media evidence.
Changes made from this audit
- Added the bounded real-vector cache, with failure/eviction/fingerprint, filtering/source lifecycle and trace tests.
- Reduced the intentional semantic UI delay from 350 to 120 ms; the separate 60 ms lexical debounce and bounded request lanes remain. That removes 230 ms of configured waiting, not a measured 230 ms compositor improvement.
- Kept an existing ready semantic generation searchable during rebuilds. Initial indexing still needs a generation; activation/warmup still withdraws backend readiness. Foreground/indexer CPU contention is not solved by this.
- Updated capture validation to use an unseen query when recording actual worker-inference counters, rather than mislabeling a cache hit.
Highest-impact next work
| Priority | Evidence / proposed work | Required acceptance evidence |
|---|---|---|
| P0 | Bound the whole IPC exchange: synchronous writes currently precede the response timeout. | Nonreading-helper regression; large request deadline and recovery. |
| P0 | Decide the x86 CPU baseline: current static build enables AVX2 without runtime CPU variants. | Declared minimum ISA or tested baseline/AVX2 dispatch; older-CPU launch. |
| P1 | Reuse unchanged document/media vectors; replacement builds currently re-encode all sources. | One-file change embeds only affected chunks; atomic activation and stale/deletion tests. |
| P1 | Pack independent short sequences by token budget; the native worker currently encodes serially. | Vector isolation/parity, length-bucketed CPU/GPU throughput and memory comparisons. |
| P1 | Give interactive queries a shared resource budget/priority during indexing. | First-unseen and warm p95 with real background document/media jobs. |
| P1 | Profile startup stages and compare a smaller query-only context. | Verification/load/graph-warmup/RSS measured separately; no fake first-query cache. |
| P2 | Buffer bounded IPC reads, remove unnecessary stream tying, and make window offset work linear. | Protocol bounds, UTF-8 offsets/tails and real media/payload timing parity. |
| P2 | Consider modality-selective projector loading; current lazy combined projector loads both encoders. | Real text/image/audio RSS and load measurements on supported backends. |
DigUp's pinned runtime demonstrates independent-sequence packing and a smaller text-query context. Those are useful patterns, not proof of an identical speedup on this VM.
Google supports re-normalized MRL truncation to 512/256/128d. A 256d experiment could reduce vector storage threefold; it does not eliminate the model's forward pass. Keep 768d until independent retrieval tests pass. Higher vision budgets trade cost for detail, so retain content-specific quality experiments. Google's truncation and modality guidance.
GPU execution remains untested here. Audit its precision against Google's BF16/FP32 guidance, and move the inference timing boundary past asynchronous embedding synchronization before interpreting GPU stage traces. CPU-only results do not validate accelerator throughput, quality or failure recovery.
Reproduce
The benchmark requires already verified model assets, a compiled CPU worker and an explicitly selected fixture index. It does not download or index user files. Read its scope fields and raw samples before comparing hardware or profiles.
python tools/embeddinggemma2/benchmark.py --mode native --models .searchi/models/embeddinggemma2 --worker .searchi/runtime/cpu/Release/searchi-gemma-worker.exe --threads 1 2 4 --repeats 3 --output .searchi/native-performance.json
python tools/embeddinggemma2/benchmark.py --mode mcp --models .searchi/models/embeddinggemma2 --worker .searchi/runtime/cpu/Release/searchi-gemma-worker.exe --cli target/release/searchi.exe --db .searchi/desktop-capture/index.db --repeats 8 --rust-profile release --output .searchi/mcp-performance.json