Files
searchi/fixtures/music-validation-v1

Unseen music metadata stress validation v1

This small source pool and its 18 queries were frozen before any model inference. It is independent of the two music files used to motivate the metadata-field hypothesis. Labels come from publisher descriptions, not model outputs. There is one assistant annotator; human listening and a blinded second annotator have not been executed. These instrument demos and short excerpts do not represent a large music collection or full-song recognition.

Source Publisher / attribution License Download bytes
Acoustic guitar sample RyGuy / Ryan PD-self 889069
Cello strings Georg Feitscher CC BY-SA 3.0 296474
Drum cadence A United States Navy Band PD-USGov 1574547
Bucket mute trumpet demo Ovinus Real CC BY-SA 4.0 228972
After You've Gone, tenor saxophone excerpt Julien Grandgagnage / Beachcomber CC BY-SA 3.0 300001
Ain't She Sweet, soprano saxophone excerpt Julien Grandgagnage / Beachcomber CC BY-SA 3.0 250846

Total acquisition: 3539909 bytes, approximately 3.38 MiB. License URLs and publisher attribution are preserved in fixture.json; source SHA-1 comes from the official Commons imageinfo API, acquired bytes are also SHA-256 locked in inputs.lock.json. Do not substitute similarly named files. Raw audio and generated containers stay under ignored .searchi/qa/music-validation-v1.

Preparation and stress variants

Preparation takes the first at most 30 seconds, resamples to mono 16 kHz and encodes lossless FLAC. It produces actual containers with original whitelisted tags, missing tags and cyclically transferred actual tags from the next source. The exact decoded PCM SHA-256 must match across each source's three variants before audio vectors can be reused. These are controlled tag-corruption cases, not naturally mislabeled customer files. Generated excerpts/variants retain the original source's license and attribution, with these modifications disclosed.

The drum source actually has no descriptive tags. The guitar has only artist Ryan. Cello supplies cello strings; trumpet supplies Bucket mute; the sax titles identify tenor/soprano. Empty tag input creates no metadata vector. A cyclic transfer from the tagless drum remains tagless; this fact is retained, not replaced with an invented instrument label. Comments and encoder strings are not semantic metadata input.

Two existing photo and two sound-effect files are fixed mixed-media competitors: image01, image02, sound01, sound02 from the previously frozen media corpus. They are distractors, not new validation positives. All competitors stay in each lane. This does not cover the entire 32-file mixed regression or spoken videos.

Twelve instrument queries have English/German pairs. Broad saxophone queries credit both sax recordings; they are not incorrectly declared absent. Four absence queries ask for bagpipes, an absent singer/title, and an absent spoken topic. They remain separate from positive recall; no threshold has been selected. Specific tenor/soprano labels follow primary descriptions, with listening still needed to independently verify perceptual specificity. Language pairs and metadata variants share recordings and are correlated, not independent examples.

Run after reserving the CPU slot

python tools/media-eval/prepare_music_validation.py fixtures/music-validation-v1/fixture.json .searchi/qa/music-validation-v1-canonical .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe
cargo test --manifest-path tools/media-eval/Cargo.toml --bin metadata_validation --locked
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_METADATA_VALIDATION_PLAN = 'C:/Users/karl/searchi/.searchi/qa/music-validation-v1-canonical/prepared.json'
$env:SEARCHI_TEST_FFMPEG = 'C:/Users/karl/searchi/.searchi/media-runtime/bin/ffmpeg.exe'
$env:SEARCHI_TEST_FFPROBE = 'C:/Users/karl/searchi/.searchi/media-runtime/bin/ffprobe.exe'
cargo test --manifest-path tools/media-eval/Cargo.toml --bin metadata_validation --locked -- --include-ignored
cargo run --manifest-path tools/media-eval/Cargo.toml --bin metadata_validation --locked -- fixtures/music-validation-v1/fixture.json .searchi/qa/music-validation-v1-canonical/prepared.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/music-validation-v1-canonical/results.json

Preparation already completed without inference. Its frozen plan SHA-256 is 3e874108914e8f0334da8f12b704261e83755bd25dce6f93e22e8ce22b8dee78; fixture SHA-256 is f9d7785ff937020b802e6c56252e64bbc3cd22e9a09709ddce9b82f9adc1b577. Use a fresh output directory for another preparation; existing plans and result files are preserved. Decoder hashes and actual container tags are checked. Five tests passed, including actual production decoder/probe parity on all 18 stress containers; strict Clippy passed. This preflight performed no model inference. Adjust the local fixture/helper paths for another development machine.

The diagnostic uses actual Google EmbeddingGemma 2 on CPU, one thread. Each of the eight distinct audio sources is encoded once and both photos at 280 tokens. Metadata fields use neutral File metadata, the unchanged 2048-byte whitelist policy and maximum over each file's real evidence. All field, source and variant citations remain explicit; a metadata hit has no fabricated timestamp. Four lanes compare media-only, original tags, missing tags and misleading tags without tuned weights or cutoff. Report positive recall and no-answer failure separately. This direct cosine diagnostic is not production ANN/hybrid validation.

The earlier prepared plan 8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2 was never inferred. Pre-inference review found that its verification decoded full float PCM while the production decoder uses millisecond trimming and s16 normalization. The corrected plan above matches the actual production decoder and compares only the whitelisted tag evidence; labels, source bytes, query texts and ranking hypothesis are unchanged. The earlier plan is preserved under its original ignored directory.

Actual frozen CPU results

The corrected policy completed from source cbf07b0 without changing labels, competitors, weights or a cutoff. Independent read-only recomputation checked all 18 queries, all four lanes, ten distinct competitors per lane and sorted scores. Missing-metadata rankings exactly equal the media-only rankings for every query.

Direct file retrieval Media only Original tags Missing tags Misleading tags
Correct first file, fourteen positives 8/14 11/14 8/14 5/14
Recall in first five 14/14 14/14 14/14 14/14
MRR 0.726 0.869 0.726 0.570
No-answer queries returning candidates, failure 4/4 4/4 4/4 4/4

Original fields improve both cello and both soprano-sax queries. However, the previously correct German acoustic-guitar query regresses from first to second: an unrelated tenor-sax title scores 0.65889 against the correct guitar audio's 0.65581. Original tags leave both bucket-muted-trumpet queries incorrect. Misleading fields additionally lose both previously correct tenor-sax first hits and preserve the guitar regression. These losses are not hidden by aggregate gains or top-five recall. Four absent queries still return all ten candidates; no abstention improvement or rejection threshold follows.

Ten media encodes (eight audio sources and two photos) plus sixteen real tag-field encodes took 88.09 seconds after model loading. That timer includes actual probing/decoding/source checks as well as inference; it excludes model startup and query inference and cannot isolate metadata's additional cost. The Windows 11 10.0.26300 shared VM exposed eight logical CPUs and 16708012 KiB visible RAM on an AMD Ryzen 7 7800X3D host. Actual Google inference used one CPU thread, with the normal runtime forced to CPU. The installed app's warmed CPU query worker remained idle; no root model requests ran concurrently in the reserved inference slot. The native runtime continues to identify audio support as experimental.

The raw ignored report .searchi/qa/music-validation-v1-canonical/results.json contains 267667 bytes, SHA-256 69bcef5ffa58c0f23e5f34641256255bf15f54c836f75fb1aa878b4bd3df1309. Recomputed summary preserves every query's first file, source kind/field, score and first relevant rank, plus worker/fixture hashes.

Do not deploy the unconditional maximum policy as a universally better music ranking. It is promising with informative correct tags, but this independently frozen sample exposes a real regression and sensitivity to misleading metadata. No policy was retuned after seeing these scores. Production ANN/hybrid evidence, more representative music, human listening, missing-content calibration and source-specific confidence remain open. The experimental media rating is unchanged.