searchi/fixtures/music-validation-v1/README.md

# Unseen music metadata stress validation v1

This small source pool and its 18 queries were frozen before any model inference.
It is independent of the two music files used to motivate the metadata-field
hypothesis. Labels come from publisher descriptions, not model outputs. There is
one assistant annotator; human listening and a blinded second annotator have not
been executed. These instrument demos and short excerpts do not represent a
large music collection or full-song recognition.

| Source | Publisher / attribution | License | Download bytes |
| --- | --- | --- | ---: |
| [Acoustic guitar sample](https://commons.wikimedia.org/wiki/File:AcousticGuitarSample.ogg) | RyGuy / Ryan | PD-self | 889069 |
| [Cello strings](https://commons.wikimedia.org/wiki/File:Cello_strings.ogg) | Georg Feitscher | CC BY-SA 3.0 | 296474 |
| [Drum cadence A](https://commons.wikimedia.org/wiki/File:Drum_-_Cadence_A.ogg) | United States Navy Band | PD-USGov | 1574547 |
| [Bucket mute trumpet demo](https://commons.wikimedia.org/wiki/File:Bucket_mute_trumpet_demo.ogg) | Ovinus Real | CC BY-SA 4.0 | 228972 |
| [After You've Gone, tenor saxophone excerpt](https://commons.wikimedia.org/wiki/File:Afteryouvegone-first8bars.ogg) | Julien Grandgagnage / Beachcomber | CC BY-SA 3.0 | 300001 |
| [Ain't She Sweet, soprano saxophone excerpt](https://commons.wikimedia.org/wiki/File:Aintshesweet8bars.ogg) | Julien Grandgagnage / Beachcomber | CC BY-SA 3.0 | 250846 |

Total acquisition: **3539909 bytes**, approximately 3.38 MiB. License URLs and
publisher attribution are preserved in `fixture.json`; source SHA-1 comes from
the official Commons imageinfo API, acquired bytes are also SHA-256 locked in
`inputs.lock.json`. Do not substitute similarly named files. Raw audio and
generated containers stay under ignored `.searchi/qa/music-validation-v1`.

## Preparation and stress variants

Preparation takes the first at most 30 seconds, resamples to mono 16 kHz and
encodes lossless FLAC. It produces actual containers with original whitelisted
tags, missing tags and cyclically transferred actual tags from the next source.
The exact decoded PCM SHA-256 must match across each source's three variants
before audio vectors can be reused. These are controlled tag-corruption cases,
not naturally mislabeled customer files. Generated excerpts/variants retain the
original source's license and attribution, with these modifications disclosed.

The drum source actually has no descriptive tags. The guitar has only artist
`Ryan`. Cello supplies `cello strings`; trumpet supplies `Bucket mute`; the sax
titles identify tenor/soprano. Empty tag input creates no metadata vector. A
cyclic transfer from the tagless drum remains tagless; this fact is retained,
not replaced with an invented instrument label. Comments and encoder strings
are not semantic metadata input.

Two existing photo and two sound-effect files are fixed mixed-media competitors:
`image01`, `image02`, `sound01`, `sound02` from the previously frozen media corpus.
They are distractors, not new validation positives. All competitors stay in each
lane. This does not cover the entire 32-file mixed regression or spoken videos.

Twelve instrument queries have English/German pairs. Broad saxophone queries
credit both sax recordings; they are not incorrectly declared absent. Four
absence queries ask for bagpipes, an absent singer/title, and an absent spoken
topic. They remain separate from positive recall; no threshold has been selected.
Specific tenor/soprano labels follow primary descriptions, with listening still
needed to independently verify perceptual specificity. Language pairs and
metadata variants share recordings and are correlated, not independent examples.

## Run after reserving the CPU slot

```powershell
python tools/media-eval/prepare_music_validation.py fixtures/music-validation-v1/fixture.json .searchi/qa/music-validation-v1-canonical .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe
cargo test --manifest-path tools/media-eval/Cargo.toml --bin metadata_validation --locked
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_METADATA_VALIDATION_PLAN = 'C:/Users/karl/searchi/.searchi/qa/music-validation-v1-canonical/prepared.json'
$env:SEARCHI_TEST_FFMPEG = 'C:/Users/karl/searchi/.searchi/media-runtime/bin/ffmpeg.exe'
$env:SEARCHI_TEST_FFPROBE = 'C:/Users/karl/searchi/.searchi/media-runtime/bin/ffprobe.exe'
cargo test --manifest-path tools/media-eval/Cargo.toml --bin metadata_validation --locked -- --include-ignored
cargo run --manifest-path tools/media-eval/Cargo.toml --bin metadata_validation --locked -- fixtures/music-validation-v1/fixture.json .searchi/qa/music-validation-v1-canonical/prepared.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/music-validation-v1-canonical/results.json
```

Preparation already completed without inference. Its frozen plan SHA-256 is
`3e874108914e8f0334da8f12b704261e83755bd25dce6f93e22e8ce22b8dee78`;
fixture SHA-256 is
`f9d7785ff937020b802e6c56252e64bbc3cd22e9a09709ddce9b82f9adc1b577`.
Use a fresh output directory for another preparation; existing plans and result
files are preserved. Decoder hashes and actual container tags are checked.
Five tests passed, including actual production decoder/probe parity on all
18 stress containers; strict Clippy passed. This preflight performed no model
inference. Adjust the local fixture/helper paths for another development machine.

The diagnostic uses actual Google EmbeddingGemma 2 on CPU, one thread. Each of
the eight distinct audio sources is encoded once and both photos at 280 tokens.
Metadata fields use neutral `File metadata`, the unchanged 2048-byte whitelist
policy and maximum over each file's real evidence. All field, source and variant
citations remain explicit; a metadata hit has no fabricated timestamp. Four
lanes compare media-only, original tags, missing tags and misleading tags without
tuned weights or cutoff. Report positive recall and no-answer failure separately.
This direct cosine diagnostic is not production ANN/hybrid validation.

The earlier prepared plan
`8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2`
was never inferred. Pre-inference review found that its verification decoded full
float PCM while the production decoder uses millisecond trimming and s16
normalization. The corrected plan above matches the actual production decoder
and compares only the whitelisted tag evidence; labels, source bytes, query texts
and ranking hypothesis are unchanged. The earlier plan is preserved under its
original ignored directory.

## Actual frozen CPU results

The corrected policy completed from source `cbf07b0` without changing labels,
competitors, weights or a cutoff. Independent read-only recomputation checked all
18 queries, all four lanes, ten distinct competitors per lane and sorted scores.
Missing-metadata rankings exactly equal the media-only rankings for every query.

| Direct file retrieval | Media only | Original tags | Missing tags | Misleading tags |
| --- | ---: | ---: | ---: | ---: |
| Correct first file, fourteen positives | 8/14 | 11/14 | 8/14 | 5/14 |
| Recall in first five | 14/14 | 14/14 | 14/14 | 14/14 |
| MRR | 0.726 | 0.869 | 0.726 | 0.570 |
| No-answer queries returning candidates, failure | 4/4 | 4/4 | 4/4 | 4/4 |

Original fields improve both cello and both soprano-sax queries. However, the
previously correct German acoustic-guitar query regresses from first to second:
an unrelated tenor-sax title scores 0.65889 against the correct guitar audio's
0.65581. Original tags leave both bucket-muted-trumpet queries incorrect. Misleading
fields additionally lose both previously correct tenor-sax first hits and preserve
the guitar regression. These losses are not hidden by aggregate gains or top-five
recall. Four absent queries still return all ten candidates; no abstention
improvement or rejection threshold follows.

Ten media encodes (eight audio sources and two photos) plus sixteen real tag-field
encodes took **88.09 seconds** after model loading. That timer includes actual
probing/decoding/source checks as well as inference; it excludes model startup and
query inference and cannot isolate metadata's additional cost. The Windows 11
10.0.26300 shared VM exposed eight logical CPUs and 16708012 KiB visible RAM on
an AMD Ryzen 7 7800X3D host. Actual Google inference used one CPU thread, with the
normal runtime forced to CPU. The installed app's warmed CPU query worker remained
idle; no root model requests ran concurrently in the reserved inference slot.
The native runtime continues to identify audio support as experimental.

The raw ignored report `.searchi/qa/music-validation-v1-canonical/results.json`
contains 267667 bytes, SHA-256
`69bcef5ffa58c0f23e5f34641256255bf15f54c836f75fb1aa878b4bd3df1309`.
[Recomputed summary](evaluation-summary.json) preserves every query's first file,
source kind/field, score and first relevant rank, plus worker/fixture hashes.

**Do not deploy the unconditional maximum policy as a universally better music
ranking.** It is promising with informative correct tags, but this independently
frozen sample exposes a real regression and sensitivity to misleading metadata.
No policy was retuned after seeing these scores. Production ANN/hybrid evidence,
more representative music, human listening, missing-content calibration and
source-specific confidence remain open. The experimental media rating is unchanged.