searchi/fixtures/absence-v3/README.md

# New absence holdout, v3

These 36 queries were frozen before their model results: 16 answerable,
16 close-but-absent, and four broad/partial requests. Each intent has an English
and German version. The fixture SHA-256 is in `fixture.lock.json`.

This is a new query holdout over an existing source corpus, not a new corpus
holdout. No historical absence query or model output was read to author these
queries. The author inspected document/code bytes and the original independent
media descriptions. One AI annotator supplied the labels; independent human
review has not happened. In particular, audio/music labels rely on the original
publisher descriptions and tags, not fresh human listening. Report these
categories separately rather than implying universal ground truth.

The 36 declared inputs are all 14 `relevance-v2` source files, all 20 licensed
`media` files, and two owned speech PCM fixtures containing Article 1 of the
Universal Declaration of Human Rights. See each original source manifest and
README for license, attribution and derivation. Existing licenses remain in
force. Speech WAVs remain ignored local test assets; their exact hashes are
frozen here, without adding another binary copy to Git.

Before inference:

```powershell
python fixtures/absence-v3/verify.py
```

Index a dedicated copy of only the declared bytes, preserving basenames.
Document/code/photo/audio/video extraction failures and missing semantic
channels must be reported independently of retrieval. An empty result after
failure is not successful abstention. Record asset coverage before interpreting
the strict absent metric. The four partial requests are not strict absence
tests; keeping relevant partial evidence is acceptable.

The old relevance runner deliberately accepts only the frozen v2 fixture.
This fixture requires a separate runner or explicit schema adaptation. Do not
silently bypass its frozen-policy gate or overwrite historical reports.
`build_fixture.py` records authorship and refuses to overwrite frozen labels.
Do not rerun it to revise labels after scores become available.

## Frozen-policy CPU results

`evaluation-summary.json` records the completed report's hash, runtime/source
hashes, coverage, timings and per-language/category scores. All 36 assets had
active semantic channels, with no extraction, semantic or speech failures.
The 36 queries were evaluated without changing either fixture or policy.

Primary neighbors appeared for 4/16 absent requests, down from 16/16 under the
previous leading-cluster rule. The remaining errors ask for a customs export
declaration when only transit insurance exists, and an HTTP registry fetcher
when only a local version parser exists. Those related sources exceed the
display threshold despite lacking the requested facts. Similarity is not
entailment.

All 16 positive requests retained their expected sources within five
accessible results (mean recall@5 1.0). Primary mean recall@5 was 0.625:
11/16 queries retained at least one expected primary source; 10/16 had the
correct primary first result. Five positive queries required opening related
suggestions. Both performer/composition music requests placed the wrong music
file first in raw retrieval, while retaining the correct file within five.
This is a real tradeoff, not a claim that all unsupported requests disappear.

Initial full text/media/speech indexing took 894.9 seconds on the CPU VM.
Query-role startup took 1.017 seconds; the ordered warm raw query median was
94.4 ms, maximum 122.9 ms. These are fixture timings, not a general desktop
performance promise or independent cold-input measurements. The persistent
profile retains the real index so unchanged files need not be encoded again.