searchi/docs/SEMANTIC.md

# Local semantic search

Searchi uses **Google EmbeddingGemma 2**, with the pinned Q8 GGUF text encoder
and llama.cpp revision used by DigUp. Production has no MiniLM, simulated-vector,
or remote embedding fallback. The rejected ONNX experiment is retained only
under the opt-in minilm-proof feature and tools/semantic-proof/HISTORICAL.md.

## Model and runtime

- Model: google/embeddinggemma-2, **768-dimensional** output.
- GGUF publisher: ggml-org/embeddinggemma-2-GGUF.
- Revision: bfcd298762cc34d0357ece5ebdd31791a3a374d8.
- llama.cpp: b11461, commit 4d756bc72bf00a4aacf410ae15a2d315f3db400d,
  with DigUp's no-logits memory patch.
- Query prompt: task: search result | query: TEXT.
- Document prompt: title: TITLE_OR_none | text: TEXT.
- Noncausal encoder, mean pooling, finite L2-normalized vectors.
- Query/document token budgets: 512/2048.

The persistent native helper uses bounded NDJSON over stdin/stdout. It has no
HTTP server or download code. Rust verifies the text model's length and SHA256
before starting it. Setup provisions assets; search never downloads them.

Production device mode is auto, with available compiled acceleration and CPU
fallback. Model/context initialization and returned inference/nonfinite-output
failures retry on CPU once. A lost GPU worker can be restarted as CPU for one
retry of that request. WorkerInfo and responses report the actual device.
**Windows VM tests remain CPU only**, via SEARCHI_FORCE_CPU=1 and Device::Cpu.
GPU execution/failure recovery has not been exercised here.

The Windows worker links a static MSVC CRT and static llama libraries, without
an external OpenMP runtime. Production CMake supports cpu, auto, metal, vulkan,
and cuda; CPU builds force GPU flags off. Runtime/provider parity and packaging
on other operating systems remain independent release checks.

| Artifact | Bytes | SHA256 |
| --- | ---: | --- |
| embeddinggemma-2-Q8_0.gguf | 309,855,456 | 2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135 |
| mmproj-embeddinggemma-2-Q8_0.gguf | 554,821,024 | c4a8a52691ecef40618438928bdf9e68379b854e24166f292592353db0aab64f |

Setup downloads both pinned files. The native worker implements genuine text, image pixels, and mono 16 kHz audio embeddings through the verified Google projector in the same 768-dimensional query space. The shared service now persists unified document, image, video-frame and soundtrack/audio/music vectors with source locations and metadata. Scoped desktop media previews and an independent retrieval evaluation are implemented; precise localization, broader quality, native playback QA and releases remain gates.

Weights use Apache-2.0; llama.cpp and DigUp use MIT; USearch and NumKong use
Apache-2.0. Retain notices with distributions. Five DigUp text reference vectors
are copied under MIT in crates/searchi-semantic/tests/fixtures with the upstream
license and source commit alongside them.

Primary sources:

- [Google developer guide](https://developers.googleblog.com/embeddinggemma-2-the-developer-guide/)
- [Model documentation](https://huggingface.co/docs/transformers/model_doc/embedding_gemma2)
- [Pinned GGUF publisher](https://huggingface.co/ggml-org/embeddinggemma-2-GGUF/tree/bfcd298762cc34d0357ece5ebdd31791a3a374d8)
- [Pinned DigUp reference](https://github.com/ARahim3/DigUp/tree/6faa29e4c315a3a591de1b41bd78751bdf39588d)
- [USearch HNSW and filtered retrieval](https://github.com/unum-cloud/USearch)

## Shared API and bounds

GemmaEmbedder owns actual tokenization, document/query prompts, and inference.
HybridSearch is the shared CLI/desktop service:

~~~rust
HybridSearch::open(db, model_dir, worker_executable, threads)
HybridSearch::embedding_status(db) // No model/runtime load or network
service.warmup()                  // Real query inference + ANN cache
service.index_embeddings() // Explicit document-only generation
service.index_embeddings_with_media(&tools, &MediaLimits::default()) // Unified generation
service.search(query, limit, &SearchOptions)
service.search_semantic(query, limit, &SearchOptions) // Diagnostics
service.worker_info()             // Actual model/runtime/device/threads
~~~

Threads are bounded to 1–8; VM tests use one. Document batches contain at most
eight windows. Request/response budgets are 16/4 MiB, startup has a 120-second
deadline, and each request has a 180-second deadline. Malformed or interrupted
IPC and invalid embeddings fail explicitly.

The service makes 64 KiB UTF-8 slices with 512-byte overlap before genuine model
token windows. It preserves global byte ranges and page/paragraph/line sources,
handles 2 MiB passages, caps each passage at 2048 windows, and bounds titles to
512 UTF-8 bytes. Generation fingerprints include this chunk policy as well as
model/runtime/prompts/token budgets/normalization.

Keep a prewarmed query service and a separate background builder. Readiness
requires actual inference, rather than merely finding verified model files.

## Persistence, ANN, and ranking

Semantic storage has an independent schema version within the shared SQLite
database. Chunks store generation, passage/path identity, original-source hash,
location, byte range, snippet, and vector. HNSW uses USearch 2.26.4, cosine/F32,
16 connections, construction expansion 128, and search expansion 64. NumKong
dispatches supported CPU SIMD kernels with portable fallback. The Cargo config
contains the Windows NK_DYNAMIC override required for static NumKong linkage.

Staging vectors and a real HNSW graph are persisted with an ANN blob SHA256.
Activation is atomic. Failed rebuilds preserve the active generation; the next
rebuild reclaims abandoned staging under a cross-process lock. Search uses a
consistent read snapshot.

Metadata filters and source hashes establish eligible keys before ANN top-k.
The cache is keyed by generation, corpus revision, and normalized filters.
Persistent dirty-path triggers invalidate affected files; unchanged warm
queries hash zero passages. Large change batches or expired history rebuild
eligibility. Default-filter prewarm includes that initial work.

Fusion uses reciprocal-rank fusion with k = 60. Exact filenames and literal
identifiers retain priority. Ordinary partial filenames receive lexical
weight 1.15 and can be outranked by keyword/semantic consensus. File results
retain source snippets and deterministic path tie breaks.

Current caps are 100,000 chunks and a 512 MiB serialized ANN blob per generation.
Full rebuilds still regenerate embeddings and the graph. Incremental reuse/jobs
and efficient ANN updates remain production work. One million filename records
does not imply one million semantic chunks under these defaults.

## Executed Windows CPU evidence

Actual Google CPU vectors match five pinned DigUp text references at cosine
**0.99979–0.99994**, exceeding the 0.999 gate. These cover English, Bengali,
Arabic, a short document, and a document beyond the sliding attention window.
Actual tests also cover English/German retrieval, persisted vectors, changed/
deleted-source suppression, verified long windows, and a tail **after 64 KiB**.

The immutable 20-document/24-query seed corpus produced:

| Lane | Top 1 | Recall at 3 |
| --- | ---: | ---: |
| Lexical | 5/24 | 5/24 |
| Google semantic | 24/24 | 24/24 |
| Hybrid HNSW | 24/24 | 24/24 |

With one CPU thread, fresh model/context startup took approximately **7 seconds
in the background**, actual query/ANN prewarm **128 ms**, and first foreground
natural-language search **99 ms** with the correct supplier document first.
Warm seed-corpus hybrid p50/p95 was **92/107 ms**. Model downloads are excluded.

A separate release-profile benchmark used 100,000 synthetic normalized 768-dimensional
vectors and 20 queries against exact nearest neighbors:

| Metric | Measurement |
| --- | ---: |
| Recall at 10 vs exact | 98.5% |
| HNSW query p95 | 1.37 ms |
| Warm service retrieval p95 | 6.40 ms |
| Warm passages rehashed | 0 |
| Initial eligibility work | 1.61 seconds |
| Graph construction/serialization | 57.95 seconds |
| Reported graph memory | 554 MB |
| Dispatched CPU kernel | skylake |

This is **index-only performance**, not neural quality across 100,000 documents.
The seed corpus and synthetic vectors do not certify production relevance.
Representative corpora, cold/selective-filter latency, concurrent indexing,
resource profiles, GPU parity/fallback, multimodal retrieval, and platform
releases remain validation gates.

## Reproduce

Run from the repository root. Provisioning uses the network explicitly:

~~~powershell
python tools/embeddinggemma2/download.py
python tools/embeddinggemma2/build_runtime.py --backend cpu
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_GEMMA_MODEL_DIR = (Resolve-Path .searchi/models/embeddinggemma2).Path
$env:SEARCHI_GEMMA_WORKER = (Resolve-Path .searchi/runtime/cpu/Release/searchi-gemma-worker.exe).Path
cargo test -p searchi-semantic --test gemma_cpu -- --ignored --nocapture
cargo test -p searchi-service --test google_cpu -- --ignored --nocapture
cargo test -p searchi-service --lib google_cpu_preserves -- --ignored --nocapture
cargo run -p searchi-service --example evaluate -- .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/google-relevance-ann.json
$env:SEARCHI_ANN_REPORT = (Join-Path (Get-Location) '.searchi/ann-100k.json')
cargo test --release -p searchi-service ann_100k -- --ignored --nocapture
~~~

The ignored ANN test deliberately requires release mode. Default tests require
no model/network and cover protocol bounds, invalid vectors, UTF-8 coverage,
fusion, lifecycle, checksums, pre-top-k filtering, and dirty-path caching.
Linux/macOS and GPU checks are not inferred from this Windows CPU run.

## Genuine image and audio adapter

`GemmaEmbedder::embed_images(&[&Path])` uses a provisional 280-token image
budget. `embed_images_with_budget(paths, budget)` supports 70 through 1120
Google visual tokens, including full-resolution model input. Use
`embed_images_with_metadata` for encoded dimensions and the chosen budget.
The aspect-preserving dimensions are multiples of 48; Rust performs bounded
PNG/JPEG/WebP/BMP decoding, applies EXIF orientation, and bicubic Catmull-Rom
resizing before sending RGB pixels. No captions or filenames produce vectors.
Bounds are eight paths per call, one image per inference operation, 64 MiB
input files, 64 Mi-pixel JPEG sources (scaled decoding above 16 Mi pixels),
16 Mi-pixel non-JPEG sources, and a 256 MiB decoding-buffer/output ceiling. Decoder dimensions
and output size are checked before pixel allocation. Tiny/extreme images still
receive a valid bounded grid. The same API embeds decoded video frame files.

`GemmaEmbedder::embed_audio_pcm(samples)` accepts normalized finite mono 16 kHz
PCM with 1–480,000 samples, at most 30 seconds, on a little-endian host. It
encodes actual sound through Google MTMD and the shared mean-pooled text model.
Callers must segment the entire recording/movie soundtrack and retain actual
window timestamps; this API never silently truncates audio. It provides no
transcript. Speech, music, and environmental sounds require independent quality
evaluation even though all use this genuine encoder.

The projector is size/SHA256 verified on first media use, then loaded lazily in
the retained native worker. Model initialization and returned inference failures
share the CPU fallback policy. Native video/FFmpeg spawning is disabled; the
media adapter owns extraction. Media generations must include
`GEMMA_MEDIA_FINGERPRINT` and image budget/window policy in their identity.
Text fingerprints remain unchanged. GPU media inference is untested here.

Actual Windows CPU parity with copied MIT DigUp fixtures passed for screenshot
280, photo 140, frame 70, and speech: cosine respectively **0.99994194,
0.99974583, 0.99921928, and 0.99983318**, all exceeding the 0.999 gate. The test
completed in 58.79 seconds including fresh model/projector startup. This proves
runtime parity on those inputs, not broad image/audio retrieval quality.
Run with the model/worker environment above:

~~~powershell
cargo test -p searchi-semantic --test gemma_media_cpu -- --ignored --nocapture
~~~
The large camera JPEG policy, established decoder tradeoffs, EXIF/resource test,
and memory-bound caveats are documented in [the semantic crate](../crates/searchi-semantic/README.md).

## Unified media indexing and setup

The desktop and default CLI `embed` command use `index_embeddings_with_media`.
Document-only indexing is an explicit `--text-only` choice. `searchi-media`
prepares actual image files, movie frames and mono 16 kHz audio windows with
verified caller-supplied FFmpeg/FFprobe executables. There is no transcription,
fabricated caption or model-derived filename substitution. Existing stream/file
artist/title metadata enters guarded keyword indexing.

Default media policy uses image 280/frame 70 visual tokens, 30-second primary
audio windows with two-second overlap, up to 240 distributed frames plus 120
scene candidates, up to 600 audio windows, a four-hour/16 GiB file ceiling and
45-second subprocess deadlines. Coverage includes the audio tail or reports an
explicit resource error; video sampling spans the movie without guaranteeing
every transient scene. Media files have stable content identity, persisted
vectors and sources (`image`, `timestamp_ms`); source changes/deletions suppress
stale results. Generations include preprocessing/window/sampling fingerprints.
Per-file decoding failures are reported and persisted rather than silently
counted as successful media indexing. Other filename/text matches stay useful.

Developer preparation from the repository root:

~~~powershell
python tools/embeddinggemma2/build_runtime.py --backend cpu --jobs 2
python tools/embeddinggemma2/stage_runtime.py --backend cpu
python tools/media-runtime/provision.py --stage
$env:SEARCHI_FORCE_CPU = '1'
$env:WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS = '--disable-gpu --disable-accelerated-video-decode'
~~~

The decoder script verifies a pinned FFmpeg 9.0.2 LGPL artifact and stages tools,
notices and the manifest without global installation. Only Windows x86-64 has
been executed; Linux/ARM64 entries require native checks, and macOS decoder
packaging is still missing. End-user packages must include the worker/decoders;
Google weights are provisioned automatically and verified locally. Search does
not perform setup downloads. Headless CLI helper discovery and explicit paths
are documented in [README](../README.md) and [BUILDING](BUILDING.md).

[Independent media evaluation](MEDIA-QUALITY.md) preserves the frozen corpus,
actual production pipeline, direct-harness failure, music confusions, speech
point-localization failures and unsuppressed no-answer results. Strong file
recall is insufficient for accurate spoken-topic navigation. A ten-second audio
experiment reduces some distances but does not close that gate; it is not a
new default. Quality policies must be validated on held-out data rather than
inventing thresholds from the evaluation corpus.