searchi/docs/MEDIA-QUALITY.md

# Media search quality

## Evaluation contract

The independent v1 corpus contains **20 real assets and 65 frozen queries**: 12 photos, four videos, two music performances, and two sound effects. Queries include English/German visual memories, visually similar hard negatives, timestamp-sensitive actions, spoken topics, exact artist/title metadata, strict filter constraints, and no-answer cases.

Labels were frozen before media inference with fixture SHA256 `b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393`. Original asset hashes and licenses are recorded in [fixture.json](../fixtures/media/fixture.json) and [provenance](../fixtures/media/README.md). All Windows VM evaluation uses the real Google worker on **CPU, one thread**, with CPU FFmpeg decoding. No simulated embeddings or inferred model captions enter the corpus labels.

The reference environment is Windows 11 Pro in a shared VM exposing four logical CPUs on an AMD Ryzen 7 7800X3D host, approximately 9.93 GiB visible memory. Inference deliberately uses one thread and `SEARCHI_FORCE_CPU=1`. [Recorded environment and worker hash](../fixtures/media/environment.json). Observed worker RAM varied around 0.9–1.2 GB during indexing; these snapshots are not a measured peak-memory guarantee.

The corpus has one assistant annotator. Actual image/video frames were visually inspected; speech timestamps use independently published NASA captions. Human audio listening and blinded second-annotator review remain unexecuted. Existing artist/title tags are retained and checked at the stream level, where Vorbis stores them. English/German pairs refer to the same assets and are correlated, not twice as many independent examples. There is only one spoken-topic video and two music files; category-perfect recall can therefore be easy and must not be extrapolated.

## Results

The first **direct real-model dense reference** completed on CPU with 94/95 candidate encodings. This is an embedding/retrieval diagnostic, not yet the production pipeline. Candidate filtering in that reference is a benchmark constraint; exact metadata was deliberately marked unsupported.

| Category | Queries | File recall@1 | File recall@5 | MRR | Precise timestamp hit@1 |
|---|---:|---:|---:|---:|---:|
| Photo memory | 24 | 100% | 100% | 1.000 | — |
| Video actions | 4 | 100% | 100% | 1.000 | 100% |
| Video scenes | 6 | 100% | 100% | 1.000 | 100% |
| Flower temporal states | 4 | 100% | 100% | 1.000 | 75% |
| Spoken topics in video | 8 | 100% | 100% | 1.000 | **25%** |
| Music memories | 4 | **50%** | **50%** | 0.500 | — |
| Sound effects | 4 | 100% | 100% | 1.000 | — |

Across 55 supported positive queries (including one reference extension constraint), recall@1/@5 and MRR were **96.36% / 96.36% / 0.964**. Across 22 temporal queries, segment overlap@1 was **95.45%**, but precise point hits were **68.18%**. [Full recorded summary and failures](../fixtures/media/reference-summary.json), [rankings](../fixtures/media/reference-results.json).

Important failures:

- **Music encoding harness:** the original jazz violin Vorbis decode briefly overshot normalized PCM range. The direct tool sent unclamped floats, and the worker correctly rejected that one candidate. The two associated music misses are an explicit preprocessing failure, not evidence that the model confuses piano with violin. The production decoder clamps its PCM and must be evaluated separately; preserve this failed reference rather than hiding it.
- **Speech localization:** all eight spoken-topic queries found the right file and an overlapping thirty-second audio segment, but six returned a segment start outside the labelled utterance. Correct-file retrieval therefore does **not** fulfill precise “jump to the bit where somebody mentions a topic.” Shorter local windows or another alignment mechanism need independent evaluation.
- **Temporal visual distinction:** one red-flower query selected the right clip but the wrong stage of opening.
- **No-answer behavior:** all six supported absent/negative queries returned nearest neighbors (**100% unsuppressed return rate**). No arbitrary cosine threshold was invented. This exposes missing abstention evidence, not a calibrated confidence score.
- **Exact metadata:** four identity/no-answer cases remain unsupported in the direct embedding lane. A separate actual FFprobe → guarded Searchi core FTS run passed all three frozen artist/title queries at rank 1 and returned no result for the absent artist: **recall@1/@5 = 100%, MRR = 1.0; absent-artist return rate = 0%**. [Metadata results](../fixtures/media/metadata-summary.json). These are three small lexical identity cases, not proof that hybrid fallback abstains or that all music collections work.

Encoding took **498.66 seconds** for the photo/frame/audio candidates in this CPU-one-thread run. Query p50/p95 were **92.78 / 122.93 ms** after encoding. Runtime reported `google/embeddinggemma-2`, 768 dimensions, CPU, one thread, `b11461-no-logits-v1`. Other concurrent VM work can affect timings; this is not a clean-room hardware benchmark.

### Actual production pipeline: 30-second audio windows

The real shared Searchi service, FFmpeg 9.0.2 decoder, metadata extraction, guarded core index, ANN retrieval and hybrid ranking completed **20/20 files and 118 chunks with zero encoding failures**. This rerun used the fixed decoder's exact sample trimming; the earlier direct harness failure remains recorded separately. Indexing took **668.04 seconds**, and warm query p50/p95 were **92.23 / 113.38 ms**, on one CPU thread. These serial evaluation runs can still share the VM with unrelated work.

Across 58 supported positive queries, actual file recall@1/@5 was **96.55% / 100%**, with **MRR 0.983**. EN recall@1 was 96.77% (31 positives), DE 96.30% (27 positives). Photo memories 24/24, video file retrieval 22/22, sound effects 4/4, artist/title metadata 3/3 and the positive extension filter 1/1 passed at rank 1. Music memories passed **2/4 at rank 1 and 4/4 in the first five**: both violin queries ranked piano first despite successful decoding, an actual music hard-negative confusion.

Temporal point accuracy was **12/22 (54.55%)**; segment overlap was 18/22 (81.82%). Video action/scenery point hits passed 10/10, flower state hits 2/4, and spoken-topic hits **0/8**. Two spoken orange-soil queries surfaced image/video-frame hits instead of audio. Remaining speech starts ranged from 0.539 to 15.777 seconds outside the labelled utterance. The 0/8 strict score does not mean every start is equally unusable; raw distance and wrong-channel failures are retained rather than tuning a tolerance to this test corpus.

All six supported no-answer cases returned results, including the absent artist through semantic fallback. The separate lexical-only metadata absence test passed, but that does **not** establish abstention for the hybrid product. A requested temporal-range negative was explicitly unsupported, rather than silently counted as a pass. [Production summary](../fixtures/media/pipeline-30s-summary.json), [complete actual rankings](../fixtures/media/pipeline-30s-results.json).

### Same-label shorter audio experiment

The same fixed decoder, files, queries and two-second overlap were rerun with ten-second audio windows. All 20 files indexed successfully into **154 chunks**, with zero failures. This is a predeclared policy comparison, not a new production default.

| Metric | 30-second windows | 10-second windows |
|---|---:|---:|
| CPU indexing time | 668.04 s | 671.51 s |
| Actual chunks | 118 | 154 |
| File recall@1 / @5 | 96.55% / 100% | 96.55% / 100% |
| MRR | 0.983 | 0.983 |
| Speech precise point hits | 0/8 | 1/8 |
| Speech segment overlap | 6/8 | 6/8 |
| Overall temporal point hits | 12/22 | 13/22 |
| Worst correct-channel timestamp distance | 15.777 s | 3.777 s |
| Warm query p50 / p95 | 92.23 / 113.38 ms | 90.65 / 110.42 ms |
| Music top-1 | 2/4 | 2/4 |
| No-answer returned results | 6/6 | 6/6 |

Shorter windows improve the worst raw localization distance, but only one strict utterance hit and unchanged wrong-channel failures are insufficient for the original spoken-topic navigation goal. Similar measured cost here does not establish cost equivalence for long recordings; window count, media mix and VM contention differ. The actual hybrid artist/title regression still passes all three identity cases. [Ten-second summary](../fixtures/media/pipeline-10s-summary.json), [complete rankings and policy fingerprint](../fixtures/media/pipeline-10s-results.json).

### Predeclared photo-budget comparison

The same 12 photos and 24 frozen EN/DE memories were encoded at 70, 280 and 1120 tokens. Query embeddings were computed once and shared; no labels or thresholds changed.

| Image tokens | Actual CPU batch encoding time | Recall@1 / @5 | MRR |
|---|---:|---:|---:|
| 70 | 47.61 s | 100% / 100% | 1.000 |
| 280 | 199.65 s | 100% / 100% | 1.000 |
| 1120 | 1493.32 s | 100% / 100% | 1.000 |

There was no retrieval improvement on this small corpus despite a substantial high-budget cost. The high-budget run overlapped independent native compilation/profiling in the shared CPU VM, so the table is an observed experiment, not a controlled hardware scaling law. A worker RAM snapshot reached approximately 1.5 GB; peak RAM was not instrumented. Equal recall here does not establish equivalent retrieval for small text, dense screenshots or subtle visual distinctions. Keep the existing production policy pending a broader quality/cost corpus; these observations do not justify making 1120 tokens the default. [Budget summary](../fixtures/media/image-budget-summary.json), [actual dimensions, rankings and worker report](../fixtures/media/image-budget-results.json).

Production defaults have not changed as part of these experiments. The current **self-rating is 4/10: experimental media retrieval**. Real visual retrieval is promising, but precise speech timestamps, music hard negatives and abstention remain essential gaps. This rating follows the rubric below rather than extrapolating the small corpus's high file-recall score. The production measurements above precede a subsequent spoken-cue channel-routing improvement; that change has not been rescored against this corpus and must not inherit a better measured score without a new run.

The existing **20-document/24-query synthetic regression** was rerun with the actual Google CPU worker and unchanged fixture SHA256 `3bab37fc0ec9c2e2e40684962b049172b5078358c6487c9033ed8697b8128dd7`:

| Lane | Top-1 | Recall@3 | Recall@10 |
|---|---:|---:|---:|
| Filename/keyword | 20.83% | 20.83% | 20.83% |
| Semantic | 100% | 100% | 100% |
| Hybrid | 100% | 100% | 100% |

Measured hybrid p95 was **146.83 ms** on this run. [Recorded output](../fixtures/media/document-baseline.json). This remains a synthetic seed, not independent production-quality evidence or a large-corpus performance claim.

## Scoring and release gates

Report recall@1/@5, MRR, language/type breakdowns, segment-overlap hits, precise timestamp point hits, and failure categories. No-answer return rate is distinct from positive recall. Do not invent an arbitrary cosine rejection threshold from these results; calibrate on a separate development set, then validate on held-out negatives.

Self-rating rubric is declared before scores: **1–2/10** broken core retrieval; **3–4/10** experimental retrieval with missing essential capabilities or unvalidated abstention/metadata; **5–6/10** useful limited beta after independent basic retrieval/filter/metadata checks; **7–8/10** strong pilot with high recall and timestamp quality on representative corpora; **9–10/10** requires production release evidence across all operating systems, large representative libraries, calibrated no-answer behavior, and complete lifecycle/security/resource verification. A small diagnostic alone cannot justify a production score.

Remaining gates: broader production metadata/filter enforcement beyond the four positive seed constraints, calibrated no-answer handling, reliable spoken-topic alignment and channel selection, realistic long videos, audio listening review, unseen music/accents, corrupt codec cases, missing files/renames, hardware resource bounds, and native Windows/Linux/macOS validation. The measured production pipeline and explicit failures above establish a reproducible baseline; they do not close these gates.