Freeze unseen music stress corpus and independently verified gated DTW experiment

2026-10-13 18:51:50 +02:00
1 parent 5ab24518b4
commit d27be302c8
@@ -0,0 +1,78 @@
+# Unseen music metadata stress validation v1
+
+This small source pool and its 18 queries were frozen before any model inference.
+It is independent of the two music files used to motivate the metadata-field
+hypothesis. Labels come from publisher descriptions, not model outputs. There is
+one assistant annotator; human listening and a blinded second annotator have not
+been executed. These instrument demos and short excerpts do not represent a
+large music collection or full-song recognition.
+
+| Source | Publisher / attribution | License | Download bytes |
+| --- | --- | --- | ---: |
+| [Acoustic guitar sample](https://commons.wikimedia.org/wiki/File:AcousticGuitarSample.ogg) | RyGuy / Ryan | PD-self | 889069 |
+| [Cello strings](https://commons.wikimedia.org/wiki/File:Cello_strings.ogg) | Georg Feitscher | CC BY-SA 3.0 | 296474 |
+| [Drum cadence A](https://commons.wikimedia.org/wiki/File:Drum_-_Cadence_A.ogg) | United States Navy Band | PD-USGov | 1574547 |
+| [Bucket mute trumpet demo](https://commons.wikimedia.org/wiki/File:Bucket_mute_trumpet_demo.ogg) | Ovinus Real | CC BY-SA 4.0 | 228972 |
+| [After You've Gone, tenor saxophone excerpt](https://commons.wikimedia.org/wiki/File:Afteryouvegone-first8bars.ogg) | Julien Grandgagnage / Beachcomber | CC BY-SA 3.0 | 300001 |
+| [Ain't She Sweet, soprano saxophone excerpt](https://commons.wikimedia.org/wiki/File:Aintshesweet8bars.ogg) | Julien Grandgagnage / Beachcomber | CC BY-SA 3.0 | 250846 |
+
+Total acquisition: **3539909 bytes**, approximately 3.38 MiB. License URLs and
+publisher attribution are preserved in `fixture.json`; source SHA-1 comes from
+the official Commons imageinfo API, acquired bytes are also SHA-256 locked in
+`inputs.lock.json`. Do not substitute similarly named files. Raw audio and
+generated containers stay under ignored `.searchi/qa/music-validation-v1`.
+
+## Preparation and stress variants
+
+Preparation takes the first at most 30 seconds, resamples to mono 16 kHz and
+encodes lossless FLAC. It produces actual containers with original whitelisted
+tags, missing tags and cyclically transferred actual tags from the next source.
+The exact decoded PCM SHA-256 must match across each source's three variants
+before audio vectors can be reused. These are controlled tag-corruption cases,
+not naturally mislabeled customer files. Generated excerpts/variants retain the
+original source's license and attribution, with these modifications disclosed.
+
+The drum source actually has no descriptive tags. The guitar has only artist
+`Ryan`. Cello supplies `cello strings`; trumpet supplies `Bucket mute`; the sax
+titles identify tenor/soprano. Empty tag input creates no metadata vector. A
+cyclic transfer from the tagless drum remains tagless; this fact is retained,
+not replaced with an invented instrument label. Comments and encoder strings
+are not semantic metadata input.
+
+Two existing photo and two sound-effect files are fixed mixed-media competitors:
+`image01`, `image02`, `sound01`, `sound02` from the previously frozen media corpus.
+They are distractors, not new validation positives. All competitors stay in each
+lane. This does not cover the entire 32-file mixed regression or spoken videos.
+
+Twelve instrument queries have English/German pairs. Broad saxophone queries
+credit both sax recordings; they are not incorrectly declared absent. Four
+absence queries ask for bagpipes, an absent singer/title, and an absent spoken
+topic. They remain separate from positive recall; no threshold has been selected.
+Specific tenor/soprano labels follow primary descriptions, with listening still
+needed to independently verify perceptual specificity. Language pairs and
+metadata variants share recordings and are correlated, not independent examples.
+
+## Run after reserving the CPU slot
+
+```powershell
+python tools/media-eval/prepare_music_validation.py fixtures/music-validation-v1/fixture.json .searchi/qa/music-validation-v1 .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe
+cargo test --manifest-path tools/media-eval/Cargo.toml --bin metadata_validation --locked
+$env:SEARCHI_FORCE_CPU = '1'
+cargo run --manifest-path tools/media-eval/Cargo.toml --bin metadata_validation --locked -- fixtures/music-validation-v1/fixture.json .searchi/qa/music-validation-v1/prepared.json .searchi/models/embeddinggemma2 .searchi/runtime/auto/Release/searchi-gemma-worker.exe .searchi/media-runtime/bin/ffmpeg.exe .searchi/media-runtime/bin/ffprobe.exe .searchi/qa/music-validation-v1/results.json
+```
+
+Preparation already completed without inference. Its frozen plan SHA-256 is
+`8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2`;
+fixture SHA-256 is
+`f9d7785ff937020b802e6c56252e64bbc3cd22e9a09709ddce9b82f9adc1b577`.
+Use a fresh output directory for another preparation; existing plans and result
+files are preserved. Decoder hashes and actual container tags are checked.
+
+The diagnostic uses actual Google EmbeddingGemma 2 on CPU, one thread. Each of
+the eight distinct audio sources is encoded once and both photos at 280 tokens.
+Metadata fields use neutral `File metadata`, the unchanged 2048-byte whitelist
+policy and maximum over each file's real evidence. All field, source and variant
+citations remain explicit; a metadata hit has no fabricated timestamp. Four
+lanes compare media-only, original tags, missing tags and misleading tags without
+tuned weights or cutoff. Report positive recall and no-answer failure separately.
+This direct cosine diagnostic is not production ANN/hybrid validation.