searchi/tools/media-eval/README.md

Real-media quality evaluation

This independent tool owns no production/UI code. It feeds real pixels and decoded PCM to the actual Google CPU worker, keeps frozen source labels outside inference input, and reports ranking/localization failures.

$env:SEARCHI_FORCE_CPU = '1'
python tools/media-eval/download.py
python tools/media-eval/prepare.py --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe
cargo run --manifest-path tools/media-eval/Cargo.toml --bin searchi-media-eval -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe fixtures/media/reference-results.json
python tools/media-eval/score.py fixtures/media/reference-results.json fixtures/media/reference-summary.json
cargo run --manifest-path tools/media-eval/Cargo.toml --bin metadata -- fixtures/media .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/metadata-results.json
python tools/media-eval/score.py fixtures/media/metadata-results.json fixtures/media/metadata-summary.json
cargo run --manifest-path tools/media-eval/Cargo.toml --bin pipeline -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/pipeline-30s-results.json 30
cargo run --manifest-path tools/media-eval/Cargo.toml --bin pipeline -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/pipeline-10s-results.json 10
python tools/media-eval/score.py fixtures/media/pipeline-30s-results.json fixtures/media/pipeline-30s-summary.json
python tools/media-eval/score.py fixtures/media/pipeline-10s-results.json fixtures/media/pipeline-10s-summary.json
cargo run --manifest-path tools/media-eval/Cargo.toml --bin image_budgets -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe fixtures/media/image-budget-results.json
python tools/media-eval/score_budgets.py fixtures/media/image-budget-results.json fixtures/media/image-budget-summary.json
python -m unittest discover -s tools/media-eval -p test_*.py

The standalone Cargo workspace keeps evaluation dependencies separate. Native inference and FFmpeg decoding use CPU only and one thread. No test enables a GPU provider. The model/worker/decoder paths above refer to explicitly provisioned local developer artifacts; they are not end-user setup requirements.

The dense reference samples video every two seconds and PCM in thirty-second windows. Photos use 140 image tokens and frames 70. The separate comparison fixes the same 12 photos/24 queries and varies only photo token budgets 70, 280, 1120. It does not change labels or choose a threshold after observing scores.

Interpretation

• Recall@1/@5 and MRR rank distinct source files, choosing each file's best segment. Per-category/per-language metrics remain separate.
• Segment overlap credits a returned segment that actually intersects the labelled event in the correct file. Point hits require the returned start timestamp itself to fall inside the event. A thirty-second speech window may pass overlap while opening far too early; both metrics are required.
• No-answer queries remain separate from positive recall. The tool exposes returned neighbors and scores without inventing an abstention threshold. A reject threshold needs independent calibration and held-out validation.
• Direct model candidate filters are reference pool constraints, not evidence that production filters work. Artist/title identity cannot be recovered reliably from a performance embedding; exact metadata rows are explicitly unsupported until evaluated through Searchi's real lexical metadata lane.
• metadata tests the actual FFprobe/guarded core FTS lane; pipeline tests the actual unified Searchi media generation and hybrid ranking. The optional final argument varies audio window seconds (30 versus 10) while retaining two-second overlap, the same real files, decoder and frozen labels. It does not change production defaults.
• Model agreement with another implementation, synthetic document-seed scores, and this small-media benchmark each answer different questions. None is production quality certification.

Detailed provenance is in the fixture README. The final assessment belongs in MEDIA-QUALITY.md.

Music metadata hypothesis diagnostic

music_metadata compares raw audio with the per-file maximum over real audio and one Google-embedded metadata passage. It has no tuned weight or rejection threshold and does not change production. prepare freezes actual FFprobe tags, source hashes, default production audio windows, and all 65 existing queries before inference. Existing plans/reports cannot be overwritten.

cargo test --manifest-path tools/media-eval/Cargo.toml --bin music_metadata --locked
cargo run --manifest-path tools/media-eval/Cargo.toml --bin music_metadata --locked -- prepare fixtures/media .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/music-metadata-v1-plan.json
$env:SEARCHI_FORCE_CPU = '1'
cargo run --manifest-path tools/media-eval/Cargo.toml --bin music_metadata --locked -- run fixtures/media .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/music-metadata-v1-plan.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/qa/music-metadata-v1-results.json

The whitelist is artist, title, album, genre and composer in that fixed order, bounded to 2048 UTF-8 bytes. Missing/blank tags produce no metadata vector; encoder, comment, URLs and copyright boilerplate are excluded. Literal misleading tags remain misleading evidence: this tool does not verify musical instruments or generate a description. Metadata evidence has no invented timestamp.

This is the known two-music-file pool, not mixed production retrieval. Other-file positives are explicitly ineligible, filters remain explicitly unsupported, and all unfiltered negatives remain visible without abstention. Existing queries are diagnostics rather than new held-out evidence. Unseen music with missing and misleading tags needs independently frozen validation before deployment.

music_fields is the subsequent diagnostic hypothesis: individual whitelisted fields use the neutral document title File metadata and a shared 2048-byte budget. It reuses per-query audio maxima verbatim from the exact preserved v1 report (SHA-256 checked), avoiding duplicate audio inference. All queries and negatives remain; scores for every field are retained, with no threshold/weights.

cargo test --manifest-path tools/media-eval/Cargo.toml --bin music_fields --locked
cargo run --manifest-path tools/media-eval/Cargo.toml --bin music_fields --locked -- prepare .searchi/qa/music-metadata-v1-results.json .searchi/qa/music-fields-v2-plan.json
$env:SEARCHI_FORCE_CPU = '1'
cargo run --manifest-path tools/media-eval/Cargo.toml --bin music_fields --locked -- run fixtures/media .searchi/qa/music-metadata-v1-results.json .searchi/qa/music-fields-v2-plan.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/qa/music-fields-v2-results.json

Prepare before inference, preserve the plan hash, and use fresh output paths. Runtime/worker and source hashes must match v1. This diagnostic intentionally depends on its exact baseline and remains a known-pool hypothesis comparison; it is not a general production scorer or new independent validation.