# Real-media quality evaluation This independent tool owns no production/UI code. It feeds real pixels and decoded PCM to the actual Google CPU worker, keeps frozen source labels outside inference input, and reports ranking/localization failures. ```powershell $env:SEARCHI_FORCE_CPU = '1' python tools/media-eval/download.py python tools/media-eval/prepare.py --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe cargo run --manifest-path tools/media-eval/Cargo.toml --bin searchi-media-eval -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe fixtures/media/reference-results.json python tools/media-eval/score.py fixtures/media/reference-results.json fixtures/media/reference-summary.json cargo run --manifest-path tools/media-eval/Cargo.toml --bin metadata -- fixtures/media .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/metadata-results.json python tools/media-eval/score.py fixtures/media/metadata-results.json fixtures/media/metadata-summary.json cargo run --manifest-path tools/media-eval/Cargo.toml --bin pipeline -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/pipeline-30s-results.json 30 cargo run --manifest-path tools/media-eval/Cargo.toml --bin pipeline -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/pipeline-10s-results.json 10 python tools/media-eval/score.py fixtures/media/pipeline-30s-results.json fixtures/media/pipeline-30s-summary.json python tools/media-eval/score.py fixtures/media/pipeline-10s-results.json fixtures/media/pipeline-10s-summary.json cargo run --manifest-path tools/media-eval/Cargo.toml --bin image_budgets -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe fixtures/media/image-budget-results.json python tools/media-eval/score_budgets.py fixtures/media/image-budget-results.json fixtures/media/image-budget-summary.json python -m unittest discover -s tools/media-eval -p test_*.py ``` The standalone Cargo workspace keeps evaluation dependencies separate. Native inference and FFmpeg decoding use CPU only and one thread. No test enables a GPU provider. The model/worker/decoder paths above refer to explicitly provisioned local developer artifacts; they are not end-user setup requirements. The dense reference samples video every two seconds and PCM in thirty-second windows. Photos use 140 image tokens and frames 70. The separate comparison fixes the same 12 photos/24 queries and varies only photo token budgets **70, 280, 1120**. It does not change labels or choose a threshold after observing scores. ## Interpretation - Recall@1/@5 and MRR rank distinct source files, choosing each file's best segment. Per-category/per-language metrics remain separate. - Segment overlap credits a returned segment that actually intersects the labelled event in the correct file. Point hits require the returned start timestamp itself to fall inside the event. A thirty-second speech window may pass overlap while opening far too early; both metrics are required. - No-answer queries remain separate from positive recall. The tool exposes returned neighbors and scores without inventing an abstention threshold. A reject threshold needs independent calibration and held-out validation. - Direct model candidate filters are **reference pool constraints**, not evidence that production filters work. Artist/title identity cannot be recovered reliably from a performance embedding; exact metadata rows are explicitly unsupported until evaluated through Searchi's real lexical metadata lane. - `metadata` tests the actual FFprobe/guarded core FTS lane; `pipeline` tests the actual unified Searchi media generation and hybrid ranking. The optional final argument varies audio window seconds (30 versus 10) while retaining two-second overlap, the same real files, decoder and frozen labels. It does not change production defaults. - Model agreement with another implementation, synthetic document-seed scores, and this small-media benchmark each answer different questions. None is production quality certification. Detailed provenance is in [the fixture README](../../fixtures/media/README.md). The final assessment belongs in [MEDIA-QUALITY.md](../../docs/MEDIA-QUALITY.md). ## Music metadata hypothesis diagnostic `music_metadata` compares raw audio with the per-file maximum over real audio and one Google-embedded metadata passage. It has no tuned weight or rejection threshold and does not change production. `prepare` freezes actual FFprobe tags, source hashes, default production audio windows, and all 65 existing queries before inference. Existing plans/reports cannot be overwritten. ```powershell cargo test --manifest-path tools/media-eval/Cargo.toml --bin music_metadata --locked cargo run --manifest-path tools/media-eval/Cargo.toml --bin music_metadata --locked -- prepare fixtures/media .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/music-metadata-v1-plan.json $env:SEARCHI_FORCE_CPU = '1' cargo run --manifest-path tools/media-eval/Cargo.toml --bin music_metadata --locked -- run fixtures/media .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/music-metadata-v1-plan.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/qa/music-metadata-v1-results.json ``` The whitelist is artist, title, album, genre and composer in that fixed order, bounded to 2048 UTF-8 bytes. Missing/blank tags produce no metadata vector; encoder, comment, URLs and copyright boilerplate are excluded. Literal misleading tags remain misleading evidence: this tool does not verify musical instruments or generate a description. Metadata evidence has no invented timestamp. This is the known two-music-file pool, not mixed production retrieval. Other-file positives are explicitly ineligible, filters remain explicitly unsupported, and all unfiltered negatives remain visible without abstention. Existing queries are diagnostics rather than new held-out evidence. Unseen music with missing and misleading tags needs independently frozen validation before deployment.