karl
b8400c4fbf
Core CI / core (ubuntu-latest) (push) Failing after 3m31s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m39s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
36 lines
4.5 KiB
Markdown
36 lines
4.5 KiB
Markdown
# Real-media quality evaluation
|
|
|
|
This independent tool owns no production/UI code. It feeds real pixels and decoded PCM to the actual Google CPU worker, keeps frozen source labels outside inference input, and reports ranking/localization failures.
|
|
|
|
```powershell
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
python tools/media-eval/download.py
|
|
python tools/media-eval/prepare.py --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe
|
|
cargo run --manifest-path tools/media-eval/Cargo.toml --bin searchi-media-eval -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe fixtures/media/reference-results.json
|
|
python tools/media-eval/score.py fixtures/media/reference-results.json fixtures/media/reference-summary.json
|
|
cargo run --manifest-path tools/media-eval/Cargo.toml --bin metadata -- fixtures/media .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/metadata-results.json
|
|
python tools/media-eval/score.py fixtures/media/metadata-results.json fixtures/media/metadata-summary.json
|
|
cargo run --manifest-path tools/media-eval/Cargo.toml --bin pipeline -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/pipeline-30s-results.json 30
|
|
cargo run --manifest-path tools/media-eval/Cargo.toml --bin pipeline -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe fixtures/media/pipeline-10s-results.json 10
|
|
python tools/media-eval/score.py fixtures/media/pipeline-30s-results.json fixtures/media/pipeline-30s-summary.json
|
|
python tools/media-eval/score.py fixtures/media/pipeline-10s-results.json fixtures/media/pipeline-10s-summary.json
|
|
cargo run --manifest-path tools/media-eval/Cargo.toml --bin image_budgets -- fixtures/media .searchi/models/embeddinggemma2 .searchi/runtime/cpu/Release/searchi-gemma-worker.exe fixtures/media/image-budget-results.json
|
|
python tools/media-eval/score_budgets.py fixtures/media/image-budget-results.json fixtures/media/image-budget-summary.json
|
|
python -m unittest discover -s tools/media-eval -p test_*.py
|
|
```
|
|
|
|
The standalone Cargo workspace keeps evaluation dependencies separate. Native inference and FFmpeg decoding use CPU only and one thread. No test enables a GPU provider. The model/worker/decoder paths above refer to explicitly provisioned local developer artifacts; they are not end-user setup requirements.
|
|
|
|
The dense reference samples video every two seconds and PCM in thirty-second windows. Photos use 140 image tokens and frames 70. The separate comparison fixes the same 12 photos/24 queries and varies only photo token budgets **70, 280, 1120**. It does not change labels or choose a threshold after observing scores.
|
|
|
|
## Interpretation
|
|
|
|
- Recall@1/@5 and MRR rank distinct source files, choosing each file's best segment. Per-category/per-language metrics remain separate.
|
|
- Segment overlap credits a returned segment that actually intersects the labelled event in the correct file. Point hits require the returned start timestamp itself to fall inside the event. A thirty-second speech window may pass overlap while opening far too early; both metrics are required.
|
|
- No-answer queries remain separate from positive recall. The tool exposes returned neighbors and scores without inventing an abstention threshold. A reject threshold needs independent calibration and held-out validation.
|
|
- Direct model candidate filters are **reference pool constraints**, not evidence that production filters work. Artist/title identity cannot be recovered reliably from a performance embedding; exact metadata rows are explicitly unsupported until evaluated through Searchi's real lexical metadata lane.
|
|
- `metadata` tests the actual FFprobe/guarded core FTS lane; `pipeline` tests the actual unified Searchi media generation and hybrid ranking. The optional final argument varies audio window seconds (30 versus 10) while retaining two-second overlap, the same real files, decoder and frozen labels. It does not change production defaults.
|
|
- Model agreement with another implementation, synthetic document-seed scores, and this small-media benchmark each answer different questions. None is production quality certification.
|
|
|
|
Detailed provenance is in [the fixture README](../../fixtures/media/README.md). The final assessment belongs in [MEDIA-QUALITY.md](../../docs/MEDIA-QUALITY.md).
|