karl/searchi
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases 10 Wiki Activity

Record unseen music regressions and actual local reranker fidelity and cost

Browse Source
This commit is contained in:
karl committed 2026-10-13 18:59:45 +02:00
1 parent cbf07b0575
commit ec73d453e3
11 files changed
+1665 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

fixtures/music-validation-v1/README.md
+54 -1
View File
@@ -84,4 +84,57 @@ lanes compare media-only, original tags, missing tags and misleading tags withou
tuned weights or cutoff. Report positive recall and no-answer failure separately.
This direct cosine diagnostic is not production ANN/hybrid validation.
The earlier prepared plan 8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2 was never inferred. Pre-inference review found that its verification decoded full float PCM while the production decoder uses millisecond trimming and s16 normalization. The corrected plan above matches the actual production decoder and compares only the whitelisted tag evidence; labels, source bytes, query texts and ranking hypothesis are unchanged. The earlier plan is preserved under its original ignored directory.
The earlier prepared plan
`8598312836f3c027b578f9f93ca1cd2d38d708bc0fd713fe174f10776934cbb2`
was never inferred. Pre-inference review found that its verification decoded full
float PCM while the production decoder uses millisecond trimming and s16
normalization. The corrected plan above matches the actual production decoder
and compares only the whitelisted tag evidence; labels, source bytes, query texts
and ranking hypothesis are unchanged. The earlier plan is preserved under its
original ignored directory.
## Actual frozen CPU results
The corrected policy completed from source `cbf07b0` without changing labels,
competitors, weights or a cutoff. Independent read-only recomputation checked all
18 queries, all four lanes, ten distinct competitors per lane and sorted scores.
Missing-metadata rankings exactly equal the media-only rankings for every query.
| Direct file retrieval | Media only | Original tags | Missing tags | Misleading tags |
| --- | ---: | ---: | ---: | ---: |
| Correct first file, fourteen positives | 8/14 | 11/14 | 8/14 | 5/14 |
| Recall in first five | 14/14 | 14/14 | 14/14 | 14/14 |
| MRR | 0.726 | 0.869 | 0.726 | 0.570 |
| No-answer queries returning candidates, failure | 4/4 | 4/4 | 4/4 | 4/4 |
Original fields improve both cello and both soprano-sax queries. However, the
previously correct German acoustic-guitar query regresses from first to second:
an unrelated tenor-sax title scores 0.65889 against the correct guitar audio's
0.65581. Original tags leave both bucket-muted-trumpet queries incorrect. Misleading
fields additionally lose both previously correct tenor-sax first hits and preserve
the guitar regression. These losses are not hidden by aggregate gains or top-five
recall. Four absent queries still return all ten candidates; no abstention
improvement or rejection threshold follows.
Ten media encodes (eight audio sources and two photos) plus sixteen real tag-field
encodes took **88.09 seconds** after model loading. That timer includes actual
probing/decoding/source checks as well as inference; it excludes model startup and
query inference and cannot isolate metadata's additional cost. The Windows 11
10.0.26300 shared VM exposed eight logical CPUs and 16708012 KiB visible RAM on
an AMD Ryzen 7 7800X3D host. Actual Google inference used one CPU thread, with the
normal runtime forced to CPU. The installed app's warmed CPU query worker remained
idle; no root model requests ran concurrently in the reserved inference slot.
The native runtime continues to identify audio support as experimental.
The raw ignored report `.searchi/qa/music-validation-v1-canonical/results.json`
contains 267667 bytes, SHA-256
`69bcef5ffa58c0f23e5f34641256255bf15f54c836f75fb1aa878b4bd3df1309`.
[Recomputed summary](evaluation-summary.json) preserves every query's first file,
source kind/field, score and first relevant rank, plus worker/fixture hashes.
**Do not deploy the unconditional maximum policy as a universally better music
ranking.** It is promising with informative correct tags, but this independently
frozen sample exposes a real regression and sensitivity to misleading metadata.
No policy was retuned after seeing these scores. Production ANN/hybrid evidence,
more representative music, human listening, missing-content calibration and
source-specific confidence remain open. The experimental media rating is unchanged.