karl/searchi
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases 10 Wiki Activity

Distinguish main matches from related suggestions and ship 0.1.9

Browse Source
This commit is contained in:
karl committed 2026-10-14 12:45:32 +02:00
1 parent a8ba594d0e
commit b0bdfc7816
40 files changed
+2463 -185
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

fixtures/absence-v3/README.md
+67
View File
@@ -0,0 +1,67 @@
# New absence holdout, v3
These 36 queries were frozen before their model results: 16 answerable,
16 close-but-absent, and four broad/partial requests. Each intent has an English
and German version. The fixture SHA-256 is in `fixture.lock.json`.
This is a new query holdout over an existing source corpus, not a new corpus
holdout. No historical absence query or model output was read to author these
queries. The author inspected document/code bytes and the original independent
media descriptions. One AI annotator supplied the labels; independent human
review has not happened. In particular, audio/music labels rely on the original
publisher descriptions and tags, not fresh human listening. Report these
categories separately rather than implying universal ground truth.
The 36 declared inputs are all 14 `relevance-v2` source files, all 20 licensed
`media` files, and two owned speech PCM fixtures containing Article 1 of the
Universal Declaration of Human Rights. See each original source manifest and
README for license, attribution and derivation. Existing licenses remain in
force. Speech WAVs remain ignored local test assets; their exact hashes are
frozen here, without adding another binary copy to Git.
Before inference:
```powershell
python fixtures/absence-v3/verify.py
```
Index a dedicated copy of only the declared bytes, preserving basenames.
Document/code/photo/audio/video extraction failures and missing semantic
channels must be reported independently of retrieval. An empty result after
failure is not successful abstention. Record asset coverage before interpreting
the strict absent metric. The four partial requests are not strict absence
tests; keeping relevant partial evidence is acceptable.
The old relevance runner deliberately accepts only the frozen v2 fixture.
This fixture requires a separate runner or explicit schema adaptation. Do not
silently bypass its frozen-policy gate or overwrite historical reports.
`build_fixture.py` records authorship and refuses to overwrite frozen labels.
Do not rerun it to revise labels after scores become available.
## Frozen-policy CPU results
`evaluation-summary.json` records the completed report's hash, runtime/source
hashes, coverage, timings and per-language/category scores. All 36 assets had
active semantic channels, with no extraction, semantic or speech failures.
The 36 queries were evaluated without changing either fixture or policy.
Primary neighbors appeared for 4/16 absent requests, down from 16/16 under the
previous leading-cluster rule. The remaining errors ask for a customs export
declaration when only transit insurance exists, and an HTTP registry fetcher
when only a local version parser exists. Those related sources exceed the
display threshold despite lacking the requested facts. Similarity is not
entailment.
All 16 positive requests retained their expected sources within five
accessible results (mean recall@5 1.0). Primary mean recall@5 was 0.625:
11/16 queries retained at least one expected primary source; 10/16 had the
correct primary first result. Five positive queries required opening related
suggestions. Both performer/composition music requests placed the wrong music
file first in raw retrieval, while retaining the correct file within five.
This is a real tradeoff, not a claim that all unsupported requests disappear.
Initial full text/media/speech indexing took 894.9 seconds on the CPU VM.
Query-role startup took 1.017 seconds; the ordered warm raw query median was
94.4 ms, maximum 122.9 ms. These are fixture timings, not a general desktop
performance promise or independent cold-input measurements. The persistent
profile retains the real index so unchanged files need not be encoded again.
fixtures/absence-v3/build_fixture.py
+106
View File
@@ -0,0 +1,106 @@
"""Freeze source-based labels without loading inference reports or model scores."""
import hashlib
import json
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
OUT = Path(__file__).parent
def digest(path):
return hashlib.sha256(path.read_bytes()).hexdigest()
assets = []
for source in ("relevance-v2", "media"):
fixture_path = ROOT / "fixtures" / source / "fixture.json"
fixture = json.loads(fixture_path.read_text(encoding="utf-8-sig"))
for asset in fixture["assets"]:
path = fixture_path.parent / asset["path"]
assert digest(path) == asset["sha256"], path
assets.append({"id": asset["id"], "path": path.relative_to(ROOT).as_posix(),
"sha256": digest(path), "bytes": path.stat().st_size,
"kind": asset["kind"], "source_manifest": fixture_path.relative_to(ROOT).as_posix(),
"label_basis": asset.get("observation", "Exact source text inspected by label author")})
for language in ("en", "de"):
path = ROOT / ".searchi/qa/speech-service-validation-v1/core/corpus" / f"speech-{language}.wav"
assert path.is_file(), path
assets.append({"id": f"speech-{language}", "path": path.relative_to(ROOT).as_posix(),
"sha256": digest(path), "bytes": path.stat().st_size, "kind": "speech",
"source_manifest": "fixtures/speech-validation-v1/fixture.json",
"label_basis": "Publisher Article 1 text; derived PCM with 5 second prefix; no model transcript used"})
queries = []
def pair(category, en, de, expected, rationale, scope="all", label="answerable"):
for language, text in (("en", en), ("de", de)):
queries.append({"id": f"a3-{len(queries)+1:03}", "text": text,
"language": language, "category": category, "scope": scope,
"expected_ids": expected, "label": label, "rationale": rationale,
"split": "new_blind_holdout"})
pair("document", "instructions for proving when a tenant stopped consuming power",
"Wie belege ich den Stromverbrauch beim Auszug eines Mieters?", ["meter-a"],
"Moving-day photograph, identifier and handover date support departing resident final bill.")
pair("document", "the form for transferring a tenant's electricity direct debit to the next resident",
"Formular zur Übertragung des Stromlastschriftmandats auf den Nachmieter", [],
"Meter reading document exists but no direct-debit transfer form or procedure.", label="absent")
pair("document", "proof required before collecting the fragile exhibit on loan",
"Welchen Nachweis braucht man vor Abholung des zerbrechlichen Leihobjekts?", ["museum-a"],
"Insurance certificate before collection of ceramic bowl.")
pair("document", "export customs declaration for taking the borrowed museum bowl overseas",
"Zollausfuhranmeldung für die geliehene Museumsschale ins Ausland", [],
"Transit insurance exists; neither customs/export declaration nor overseas loan rule exists.", label="absent")
pair("code", "parser that separates the plus suffix from prerelease identifiers",
"Parser, der den Plus-Suffix von Vorabversionskennungen trennt", ["code-version"],
"Semver Version parser splits '-' prerelease and '+' build identifiers.", scope="code")
pair("code", "version parser that fetches the newest release from an HTTP package registry",
"Versionsparser, der die neueste Version aus einer HTTP-Paketregistrierung abruft", [],
"Semver parsing exists; no network registry retrieval code in fixed two source files.", scope="code", label="absent")
pair("image", "photo of tiny peaks repeating upside down across quiet water",
"Foto von Bergspitzen, die sich kopfüber in ruhigem Wasser wiederholen", ["image03"],
"Mountain and tree reflections in daylight lake.")
pair("image", "photo of an alpine lake with a red canoe and two paddlers",
"Foto eines Bergsees mit rotem Kanu und zwei Paddlern", [],
"Lake photos exist but no canoe or two paddlers in any photo.", label="absent")
pair("video", "video where a feline keeps reaching up at a room's exit",
"Video, in dem eine Katze wiederholt am Ausgang eines Zimmers hochspringt", ["clip01"],
"Orange tabby twice jumps at white door lever.")
pair("video", "video of a cat squeezing through a flap into a snowy garden",
"Video einer Katze, die durch eine Klappe in einen verschneiten Garten geht", [],
"Door jumping cat exists; no flap, snowy garden or passage through it.", label="absent")
pair("music", "music by Kimiko Ishizaka with the thirteenth Goldberg variation",
"Musik von Kimiko Ishizaka mit der dreizehnten Goldberg-Variation", ["music01"],
"Original Vorbis tags plus publisher description identify performer and composition.")
pair("music", "music by Kimiko Ishizaka playing a Goldberg variation on a violin",
"Musik von Kimiko Ishizaka mit einer Goldberg-Variation auf der Geige", [],
"Kimiko piano and separate jazz violin exist; requested performer/instrument/composition conjunction does not.", label="absent")
pair("sound", "audio of a household pet warning someone with repeated barks",
"Audio von einem Haustier, das jemanden mit Bellen warnt", ["sound01"],
"Publisher dog bark description supports barking; warning intent is colloquial query, not a labelled mood.")
pair("sound", "audio of a dog barking while church bells ring behind it",
"Audio eines bellenden Hundes mit Kirchenglocken im Hintergrund", [],
"Dog bark recording exists; no independently described church bells in any audio.", label="absent")
pair("speech", "spoken passage saying people possess an inner moral judgement",
"Gesprochene Passage darüber, dass Menschen ein inneres moralisches Urteilsvermögen haben",
["speech-en", "speech-de"], "Article 1 states endowed with reason and conscience.")
pair("speech", "spoken explanation of the right to paid holidays and limited working hours",
"Gesprochene Erklärung des Rechts auf bezahlten Urlaub und begrenzte Arbeitszeit", [],
"Only Article 1 is in speech files; requested Article 24 topic absent from corpus.", label="absent")
pair("ambiguous_partial", "pictures of somewhere peaceful near water",
"Bilder von einem ruhigen Ort am Wasser", ["image03", "image05", "image11"],
"Subjective broad request: lake and beach both plausible; excluded from strict no-answer metric.", label="partial")
pair("ambiguous_partial", "what do we know about keeping the loaned exhibit safe",
"Was wissen wir darüber, das geliehene Ausstellungsstück sicher zu halten?", ["museum-a"],
"Insurance is relevant partial evidence, but no physical packing/security instruction; do not score as absent.", label="partial")
fixture = {"schema": 1, "frozen_before_inference": True,
"label_policy": "New queries authored without opening inference outputs. Existing source content/independent media descriptions only. One AI annotator, not independent human truth. Never revise labels after outputs. Historical absence20 not reused. Partial queries scored separately. Extraction/indexing failures must be reported separately, not successful abstentions.",
"corpus_policy": "All 36 declared files form fixed distractor corpus. Copy only declared source bytes into a dedicated corpus; no repository-wide index. Speech PCM stays ignored and is referenced by exact hash, not added to Git.",
"assets": assets, "queries": queries}
target = OUT / "fixture.json"
assert not target.exists(), "Frozen fixture must not be overwritten"
target.write_text(json.dumps(fixture, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
lock = {"fixture_sha256": digest(target), "assets": len(assets), "queries": len(queries),
"strict_answerable": sum(q["label"] == "answerable" for q in queries),
"strict_absent": sum(q["label"] == "absent" for q in queries),
"partial": sum(q["label"] == "partial" for q in queries)}
(OUT / "fixture.lock.json").write_text(json.dumps(lock, indent=2) + "\n", encoding="utf-8")
print(json.dumps(lock))
fixtures/absence-v3/evaluation-summary.json
+233
View File
@@ -0,0 +1,233 @@
{
"report_sha256": "3a032695607a5035016602cbd995d9c4ef028891e5310eb585a9356f2404b628",
"report_path": ".searchi/qa/absence-v3-results.json",
"fixture_sha256": "12354a55c1e5f1efe3437e653c074d247374852a420329db8d500771c4a24f53",
"policy_sha256": "d3d4e437a54d1d62a70bb09c2882db6362a2566525a54f92d5fdc7fdfb91e26c",
"complete": true,
"worker_info": {
"code_query_supported": true,
"context_tokens": 512,
"device": "cpu",
"dimensions": 768,
"media_supported": true,
"model": "google/embeddinggemma-2",
"ready": true,
"role": "query",
"runtime": "b11461-no-logits-v1",
"threads": 1
},
"runtime_hashes": {
"ffmpeg": "b778ca5786ee4b0b4f37f34478617093d2fdb9a532d281fb76cda69d1d579d46",
"ffprobe": "4e1b0afd27bf02d79739f27b2d650137470925afb19d54b5e89a708b7a389236",
"model_manifest": "fad82f171f4976d8eeb89c11d6a79fc46d92afb73c10a34f140d5b56e377397f",
"runner": "ca649639de16b8aea021d6cf6766c4240a7dfe54cc173f9ed379538b9c5a3057",
"speech_base": "60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe",
"speech_vad": "2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987",
"speech_worker": "9c34aa2b12a4c2ab876a1573045ef9a31e19e7dfff76a2b34efdf45555ad97aa",
"worker": "c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73"
},
"source_hashes": {
"relevance": "de1c031592af13929ddf061da247343e76ecf941b46dfb3821bfce15b9413769",
"routing": "6a7f20ad312517acaf7c9e6120a4eb55edc47f36e19267d54e7053506e01e755",
"service_lib": "12e4fd01149a7755b587b5d7ed9162fbe673d8869241a38feb8b33639fa4ccdc"
},
"asset_count": 36,
"covered_asset_count": 36,
"extraction_errors": [],
"semantic_failures": [],
"speech_failures": [],
"active_chunk_count": 172,
"lexical_ms": 40.328900000000004,
"initial_semantic_index_ms": 894883.7687,
"query_startup_ms": 1017.2243000000001,
"raw_query_median_ms": 94.3792,
"raw_query_max_ms": 122.8708,
"query_order_note": "lexical then raw then hybrid; hybrid reuses query vector. Replay baseline has no separate runtime timing. Scores not confidence; coverage failures must be separated.",
"summary": {
"all": {
"absent_count": 16,
"accessible_absent_nonempty": 16,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 16,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 4,
"positive_count": 16,
"primary_absent_nonempty": 4,
"primary_positive_recall5": 0.625,
"raw_absent_nonempty": 16,
"raw_positive_recall5": 1.0
},
"category:ambiguous_partial": {
"absent_count": 0,
"accessible_absent_nonempty": 0,
"accessible_positive_recall5": null,
"baseline_replay_absent_nonempty": 0,
"baseline_replay_positive_recall5": null,
"partial_count": 4,
"positive_count": 0,
"primary_absent_nonempty": 0,
"primary_positive_recall5": null,
"raw_absent_nonempty": 0,
"raw_positive_recall5": null
},
"category:code": {
"absent_count": 2,
"accessible_absent_nonempty": 2,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 2,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 0,
"positive_count": 2,
"primary_absent_nonempty": 2,
"primary_positive_recall5": 1.0,
"raw_absent_nonempty": 2,
"raw_positive_recall5": 1.0
},
"category:document": {
"absent_count": 4,
"accessible_absent_nonempty": 4,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 4,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 0,
"positive_count": 4,
"primary_absent_nonempty": 2,
"primary_positive_recall5": 0.5,
"raw_absent_nonempty": 4,
"raw_positive_recall5": 1.0
},
"category:image": {
"absent_count": 2,
"accessible_absent_nonempty": 2,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 2,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 0,
"positive_count": 2,
"primary_absent_nonempty": 0,
"primary_positive_recall5": 0.5,
"raw_absent_nonempty": 2,
"raw_positive_recall5": 1.0
},
"category:music": {
"absent_count": 2,
"accessible_absent_nonempty": 2,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 2,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 0,
"positive_count": 2,
"primary_absent_nonempty": 0,
"primary_positive_recall5": 0.0,
"raw_absent_nonempty": 2,
"raw_positive_recall5": 1.0
},
"category:sound": {
"absent_count": 2,
"accessible_absent_nonempty": 2,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 2,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 0,
"positive_count": 2,
"primary_absent_nonempty": 0,
"primary_positive_recall5": 1.0,
"raw_absent_nonempty": 2,
"raw_positive_recall5": 1.0
},
"category:speech": {
"absent_count": 2,
"accessible_absent_nonempty": 2,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 2,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 0,
"positive_count": 2,
"primary_absent_nonempty": 0,
"primary_positive_recall5": 0.5,
"raw_absent_nonempty": 2,
"raw_positive_recall5": 1.0
},
"category:video": {
"absent_count": 2,
"accessible_absent_nonempty": 2,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 2,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 0,
"positive_count": 2,
"primary_absent_nonempty": 0,
"primary_positive_recall5": 1.0,
"raw_absent_nonempty": 2,
"raw_positive_recall5": 1.0
},
"language:de": {
"absent_count": 8,
"accessible_absent_nonempty": 8,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 8,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 2,
"positive_count": 8,
"primary_absent_nonempty": 2,
"primary_positive_recall5": 0.5625,
"raw_absent_nonempty": 8,
"raw_positive_recall5": 1.0
},
"language:en": {
"absent_count": 8,
"accessible_absent_nonempty": 8,
"accessible_positive_recall5": 1.0,
"baseline_replay_absent_nonempty": 8,
"baseline_replay_positive_recall5": 1.0,
"partial_count": 2,
"positive_count": 8,
"primary_absent_nonempty": 2,
"primary_positive_recall5": 0.6875,
"raw_absent_nonempty": 8,
"raw_positive_recall5": 1.0
}
},
"positive_primary_any_relevant_at5": 11,
"positive_primary_top1_correct": 10,
"positive_accessible_top1_correct": 13,
"residual_false_primary": [
{
"id": "a3-007",
"category": "document",
"language": "en",
"primary_asset_ids": [
"museum-a"
],
"top_raw_cosine": 0.7194200754165649
},
{
"id": "a3-008",
"category": "document",
"language": "de",
"primary_asset_ids": [
"museum-a"
],
"top_raw_cosine": 0.7089265584945679
},
{
"id": "a3-011",
"category": "code",
"language": "en",
"primary_asset_ids": [
"code-version"
],
"top_raw_cosine": 0.7125321626663208
},
{
"id": "a3-012",
"category": "code",
"language": "de",
"primary_asset_ids": [
"code-version"
],
"top_raw_cosine": 0.7140055894851685
}
],
"interpretation": "Primary false-neighbor display improved from 16/16 to 4/16 absent queries. Accessible recall@5 stays 1.0; primary mean recall@5 falls to 0.625. Similarity alone cannot prove requested facts exist. No policy parameter was tuned after this holdout. Publisher-based music/sound labels and one AI annotator limit generalization."
}
fixtures/absence-v3/fixture.json
+776
View File
@@ -0,0 +1,776 @@
{
"schema": 1,
"frozen_before_inference": true,
"label_policy": "New queries authored without opening inference outputs. Existing source content/independent media descriptions only. One AI annotator, not independent human truth. Never revise labels after outputs. Historical absence20 not reused. Partial queries scored separately. Extraction/indexing failures must be reported separately, not successful abstentions.",
"corpus_policy": "All 36 declared files form fixed distractor corpus. Copy only declared source bytes into a dedicated corpus; no repository-wide index. Speech PCM stays ignored and is referenced by exact hash, not added to Git.",
"assets": [
{
"id": "code-pattern",
"path": "fixtures/relevance-v2/sources/BurntSushi__aho-corasick__src__packed__api.rs",
"sha256": "7bcb23b99bbf87e41348ef8d4db72a88dff744de503dcb684ac9d40659656f96",
"bytes": 23091,
"kind": "code",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "code-version",
"path": "fixtures/relevance-v2/sources/dtolnay__semver__src__parse.rs",
"sha256": "2d9cb28a72a813968c16da29a9690f8e53f1ce00727a1dbe67aec3ddb5ed59ed",
"bytes": 12187,
"kind": "code",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "bicycle-a",
"path": "fixtures/relevance-v2/sources/bicycle-a.md",
"sha256": "2d77a47673d5900784efa2eb7943efb29abcf36b19ef81e4291ce52dc47f1f70",
"bytes": 274,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "bicycle-b",
"path": "fixtures/relevance-v2/sources/bicycle-b.md",
"sha256": "6302d4a3832b65d25a46e8502cfa5dcaeb6256ef22cae9b7f6eda9d01e5d3375",
"bytes": 247,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "bakery-a",
"path": "fixtures/relevance-v2/sources/bakery-a.md",
"sha256": "158ee5091f8c6cdb6775da7244441731869da3645059f10048c975504003484d",
"bytes": 267,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "bakery-b",
"path": "fixtures/relevance-v2/sources/bakery-b.md",
"sha256": "c17ce0846c41b46b4346ed7e83815677dddd678cc6268297d865bca912df4920",
"bytes": 270,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "meter-a",
"path": "fixtures/relevance-v2/sources/meter-a.md",
"sha256": "ab41afc2781a045ab2eac305a68074830c9878cd951a8ee4fcd65335b4a91aab",
"bytes": 307,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "meter-b",
"path": "fixtures/relevance-v2/sources/meter-b.md",
"sha256": "6044b38f140d5302619f450fa6a7ad5ae08133e2d748a4279b325ca42b28fbb9",
"bytes": 255,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "museum-a",
"path": "fixtures/relevance-v2/sources/museum-a.md",
"sha256": "4e88213ad17ac4eb63f8acd13d9842ededebd6f211226876bd817bbd296246d0",
"bytes": 278,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "museum-b",
"path": "fixtures/relevance-v2/sources/museum-b.md",
"sha256": "51072d9e67336305851f74100583e748a4b9cbc817a83a4ec3575d5bdb792a15",
"bytes": 260,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "irrigation-a",
"path": "fixtures/relevance-v2/sources/irrigation-a.md",
"sha256": "e57ab03ca777a20f8a281ea700f1117d56b25439a12e05f08af5d3a928e78488",
"bytes": 306,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "irrigation-b",
"path": "fixtures/relevance-v2/sources/irrigation-b.md",
"sha256": "9a9b28cc09bc8f1468c6933f1d0969909e7f58478084fcbbda51b96dabb83c6c",
"bytes": 251,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "ferry-a",
"path": "fixtures/relevance-v2/sources/ferry-a.md",
"sha256": "c88a26ac20a896b2f815861e2fc10d6ccee8a572ff3ee637b4eecdcac4adce86",
"bytes": 261,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "ferry-b",
"path": "fixtures/relevance-v2/sources/ferry-b.md",
"sha256": "47c14d1b88e3f38167778347ec1978b4346555b66f81f321bf18e5d7ce868a41",
"bytes": 244,
"kind": "document",
"source_manifest": "fixtures/relevance-v2/fixture.json",
"label_basis": "Exact source text inspected by label author"
},
{
"id": "image01",
"path": "fixtures/media/assets/image01.jpg",
"sha256": "c0b2ebddafcff4cd60ae1d2f1b8e53d4e0f11643ac53d0113605f333523b21ed",
"bytes": 27390,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Long-haired tabby cat lying flat indoors, eyes toward camera."
},
{
"id": "image02",
"path": "fixtures/media/assets/image02.jpg",
"sha256": "72b46006e7f55dcad5bec5af977171a369917030cf2ff99e544806af939ffec1",
"bytes": 68905,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Fruit bowl with raspberries, mango, blackberries, strawberries and almonds, top down."
},
{
"id": "image03",
"path": "fixtures/media/assets/image03.jpg",
"sha256": "c4abbb061e6fcbcef25d36d7bb7f24018c0c768b23176fb2a77a9eb8f5b86f21",
"bytes": 34309,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Jagged mountains and fir trees reflected in a daylight lake."
},
{
"id": "image04",
"path": "fixtures/media/assets/image04.jpg",
"sha256": "8999f22131c22ee1c29d79dda352c2654131eb1b4f61ba7f8b8d21dbbc5cc791",
"bytes": 37735,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Dense blue star field and Milky Way above a dark mountain landscape."
},
{
"id": "image05",
"path": "fixtures/media/assets/image05.jpg",
"sha256": "1cfd9d9c3dea4ec4e1733c55424eef93983bce1167f60777768cae6c549fd889",
"bytes": 51106,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Wooden pier into still lake at night under a bright moon."
},
{
"id": "image06",
"path": "fixtures/media/assets/image06.jpg",
"sha256": "91947e87067c923db2e560f67cdd0c1edec55c9bb0fbab6a83ee7c7adf88860d",
"bytes": 27259,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Layered purple mountains under a yellow-orange sunset, tiny person on ridge."
},
{
"id": "image07",
"path": "fixtures/media/assets/image07.jpg",
"sha256": "0b0cc8c211680d19ebdaf2b797af430120c4e65f951143bbe06544eaa0691942",
"bytes": 50633,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Cows grazing in green rolling hills."
},
{
"id": "image08",
"path": "fixtures/media/assets/image08.jpg",
"sha256": "12b5fd662c054d1337e4adf733751da3e32c1fd40f88dfccc0bd73af4f620369",
"bytes": 65362,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Red sports coupe on a wooded road, motion-blurred background."
},
{
"id": "image09",
"path": "fixtures/media/assets/image09.jpg",
"sha256": "6112af5f9ed170d91c7f8e1e5de95142c0190e2e2f8fbbbccf2f106553371817",
"bytes": 68469,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Bowl of quinoa and vegetables, sliced carrots and zucchini on pale table."
},
{
"id": "image10",
"path": "fixtures/media/assets/image10.jpg",
"sha256": "632926d3bd0f4691ae7fd0a8e2cb8b455e0252511eebf322e91c64c17cdf0e2a",
"bytes": 23609,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Milk pouring from metal jug into cup of coffee with latte art."
},
{
"id": "image11",
"path": "fixtures/media/assets/image11.jpg",
"sha256": "c02ad4a8bbe8091cbc3a9a0abc602a905cb46056efa94141950f3af1c749c038",
"bytes": 27716,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Turquoise tropical sea and white beach fringed by green vegetation."
},
{
"id": "image12",
"path": "fixtures/media/assets/image12.jpg",
"sha256": "dd3bc409128a8702522bac7b953a5eb691662eebf21ac3ba0b3b15c70f86d54b",
"bytes": 14256,
"kind": "image",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Black and white silhouette of a person reaching a hand toward camera."
},
{
"id": "clip01",
"path": "fixtures/media/assets/clip01.webm",
"sha256": "1a1f420559b306574ba8a04aee0175c38c2e20815b1bd46a45aae86d17f3ad53",
"bytes": 2253125,
"kind": "video",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Orange tabby cat twice jumps against white door near its lever, then crouches."
},
{
"id": "clip02",
"path": "fixtures/media/assets/clip02.webm",
"sha256": "5652e6490bea85dfb7a1da05e00fc7f3d8cc2cbd54790c03a48223f2256bc3d1",
"bytes": 6001205,
"kind": "video",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Tall waterfall descending misty rocky hillside; a tiny person appears near its base."
},
{
"id": "clip03",
"path": "fixtures/media/assets/clip03.webm",
"sha256": "c6f8a348953395598a9a73b9bab1676436410797bce9f398f4be1531d6e76dda",
"bytes": 554058,
"kind": "video",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Timelapse of closed red flower bud opening among green purple-edged leaves."
},
{
"id": "clip04",
"path": "fixtures/media/assets/clip04.mp4",
"sha256": "5fa4af30f984f4d7d53b50990d1e21e41df50b62a8b3ec3a2a1d44f30431f0f1",
"bytes": 16710310,
"kind": "video",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Artemis Origins montage: Apollo astronauts, lunar surface/rover, a speech and future-space imagery. Speech intervals independently labelled from NASA publisher SRT."
},
{
"id": "music01",
"path": "fixtures/media/assets/music01.ogg",
"sha256": "29fcd20414ee19c2a5aa4339eaf6375079dc87959051ab0aea32c76df52f34dc",
"bytes": 4697594,
"kind": "music",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Soft solo piano performance of Bach Goldberg Variation13, original Vorbis artist/title tags retained."
},
{
"id": "music02",
"path": "fixtures/media/assets/music02.ogg",
"sha256": "9ce9c5d07f941c07cab575e58016d6ad1b09219a36477652118d8e85c3401dd8",
"bytes": 198070,
"kind": "music",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Jazz violin solo, original Vorbis tags identify artist Violinist WTD and title Jazz Violin Solo."
},
{
"id": "sound01",
"path": "fixtures/media/assets/sound01.wav",
"sha256": "bbd0f908b3514dd3bd7d2bc04dcf64f8d360a161e7f43cac5d6761e7add79451",
"bytes": 179848,
"kind": "sound",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Dog bark recording from independently described source; real waveform retained."
},
{
"id": "sound02",
"path": "fixtures/media/assets/sound02.mp3",
"sha256": "41191d0727073bf848bcc8f0bd851d71a0b0058e901abb1c1b236ad327bda52e",
"bytes": 39868,
"kind": "sound",
"source_manifest": "fixtures/media/fixture.json",
"label_basis": "Roaring creature sound effect from independently described CC0 source; real waveform retained."
},
{
"id": "speech-en",
"path": ".searchi/qa/speech-service-validation-v1/core/corpus/speech-en.wav",
"sha256": "0aa1fa97d367d987bfb3bb01d7eca11db77a544552ccb8dc0a78ad74d0b496ea",
"bytes": 720428,
"kind": "speech",
"source_manifest": "fixtures/speech-validation-v1/fixture.json",
"label_basis": "Publisher Article 1 text; derived PCM with 5 second prefix; no model transcript used"
},
{
"id": "speech-de",
"path": ".searchi/qa/speech-service-validation-v1/core/corpus/speech-de.wav",
"sha256": "4af09fe7d80056600e4c288161697a094108f343b069ffd581ccabc06a048afe",
"bytes": 519052,
"kind": "speech",
"source_manifest": "fixtures/speech-validation-v1/fixture.json",
"label_basis": "Publisher Article 1 text; derived PCM with 5 second prefix; no model transcript used"
}
],
"queries": [
{
"id": "a3-001",
"text": "instructions for proving when a tenant stopped consuming power",
"language": "en",
"category": "document",
"scope": "all",
"expected_ids": [
"meter-a"
],
"label": "answerable",
"rationale": "Moving-day photograph, identifier and handover date support departing resident final bill.",
"split": "new_blind_holdout"
},
{
"id": "a3-002",
"text": "Wie belege ich den Stromverbrauch beim Auszug eines Mieters?",
"language": "de",
"category": "document",
"scope": "all",
"expected_ids": [
"meter-a"
],
"label": "answerable",
"rationale": "Moving-day photograph, identifier and handover date support departing resident final bill.",
"split": "new_blind_holdout"
},
{
"id": "a3-003",
"text": "the form for transferring a tenant's electricity direct debit to the next resident",
"language": "en",
"category": "document",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Meter reading document exists but no direct-debit transfer form or procedure.",
"split": "new_blind_holdout"
},
{
"id": "a3-004",
"text": "Formular zur Übertragung des Stromlastschriftmandats auf den Nachmieter",
"language": "de",
"category": "document",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Meter reading document exists but no direct-debit transfer form or procedure.",
"split": "new_blind_holdout"
},
{
"id": "a3-005",
"text": "proof required before collecting the fragile exhibit on loan",
"language": "en",
"category": "document",
"scope": "all",
"expected_ids": [
"museum-a"
],
"label": "answerable",
"rationale": "Insurance certificate before collection of ceramic bowl.",
"split": "new_blind_holdout"
},
{
"id": "a3-006",
"text": "Welchen Nachweis braucht man vor Abholung des zerbrechlichen Leihobjekts?",
"language": "de",
"category": "document",
"scope": "all",
"expected_ids": [
"museum-a"
],
"label": "answerable",
"rationale": "Insurance certificate before collection of ceramic bowl.",
"split": "new_blind_holdout"
},
{
"id": "a3-007",
"text": "export customs declaration for taking the borrowed museum bowl overseas",
"language": "en",
"category": "document",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Transit insurance exists; neither customs/export declaration nor overseas loan rule exists.",
"split": "new_blind_holdout"
},
{
"id": "a3-008",
"text": "Zollausfuhranmeldung für die geliehene Museumsschale ins Ausland",
"language": "de",
"category": "document",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Transit insurance exists; neither customs/export declaration nor overseas loan rule exists.",
"split": "new_blind_holdout"
},
{
"id": "a3-009",
"text": "parser that separates the plus suffix from prerelease identifiers",
"language": "en",
"category": "code",
"scope": "code",
"expected_ids": [
"code-version"
],
"label": "answerable",
"rationale": "Semver Version parser splits '-' prerelease and '+' build identifiers.",
"split": "new_blind_holdout"
},
{
"id": "a3-010",
"text": "Parser, der den Plus-Suffix von Vorabversionskennungen trennt",
"language": "de",
"category": "code",
"scope": "code",
"expected_ids": [
"code-version"
],
"label": "answerable",
"rationale": "Semver Version parser splits '-' prerelease and '+' build identifiers.",
"split": "new_blind_holdout"
},
{
"id": "a3-011",
"text": "version parser that fetches the newest release from an HTTP package registry",
"language": "en",
"category": "code",
"scope": "code",
"expected_ids": [],
"label": "absent",
"rationale": "Semver parsing exists; no network registry retrieval code in fixed two source files.",
"split": "new_blind_holdout"
},
{
"id": "a3-012",
"text": "Versionsparser, der die neueste Version aus einer HTTP-Paketregistrierung abruft",
"language": "de",
"category": "code",
"scope": "code",
"expected_ids": [],
"label": "absent",
"rationale": "Semver parsing exists; no network registry retrieval code in fixed two source files.",
"split": "new_blind_holdout"
},
{
"id": "a3-013",
"text": "photo of tiny peaks repeating upside down across quiet water",
"language": "en",
"category": "image",
"scope": "all",
"expected_ids": [
"image03"
],
"label": "answerable",
"rationale": "Mountain and tree reflections in daylight lake.",
"split": "new_blind_holdout"
},
{
"id": "a3-014",
"text": "Foto von Bergspitzen, die sich kopfüber in ruhigem Wasser wiederholen",
"language": "de",
"category": "image",
"scope": "all",
"expected_ids": [
"image03"
],
"label": "answerable",
"rationale": "Mountain and tree reflections in daylight lake.",
"split": "new_blind_holdout"
},
{
"id": "a3-015",
"text": "photo of an alpine lake with a red canoe and two paddlers",
"language": "en",
"category": "image",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Lake photos exist but no canoe or two paddlers in any photo.",
"split": "new_blind_holdout"
},
{
"id": "a3-016",
"text": "Foto eines Bergsees mit rotem Kanu und zwei Paddlern",
"language": "de",
"category": "image",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Lake photos exist but no canoe or two paddlers in any photo.",
"split": "new_blind_holdout"
},
{
"id": "a3-017",
"text": "video where a feline keeps reaching up at a room's exit",
"language": "en",
"category": "video",
"scope": "all",
"expected_ids": [
"clip01"
],
"label": "answerable",
"rationale": "Orange tabby twice jumps at white door lever.",
"split": "new_blind_holdout"
},
{
"id": "a3-018",
"text": "Video, in dem eine Katze wiederholt am Ausgang eines Zimmers hochspringt",
"language": "de",
"category": "video",
"scope": "all",
"expected_ids": [
"clip01"
],
"label": "answerable",
"rationale": "Orange tabby twice jumps at white door lever.",
"split": "new_blind_holdout"
},
{
"id": "a3-019",
"text": "video of a cat squeezing through a flap into a snowy garden",
"language": "en",
"category": "video",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Door jumping cat exists; no flap, snowy garden or passage through it.",
"split": "new_blind_holdout"
},
{
"id": "a3-020",
"text": "Video einer Katze, die durch eine Klappe in einen verschneiten Garten geht",
"language": "de",
"category": "video",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Door jumping cat exists; no flap, snowy garden or passage through it.",
"split": "new_blind_holdout"
},
{
"id": "a3-021",
"text": "music by Kimiko Ishizaka with the thirteenth Goldberg variation",
"language": "en",
"category": "music",
"scope": "all",
"expected_ids": [
"music01"
],
"label": "answerable",
"rationale": "Original Vorbis tags plus publisher description identify performer and composition.",
"split": "new_blind_holdout"
},
{
"id": "a3-022",
"text": "Musik von Kimiko Ishizaka mit der dreizehnten Goldberg-Variation",
"language": "de",
"category": "music",
"scope": "all",
"expected_ids": [
"music01"
],
"label": "answerable",
"rationale": "Original Vorbis tags plus publisher description identify performer and composition.",
"split": "new_blind_holdout"
},
{
"id": "a3-023",
"text": "music by Kimiko Ishizaka playing a Goldberg variation on a violin",
"language": "en",
"category": "music",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Kimiko piano and separate jazz violin exist; requested performer/instrument/composition conjunction does not.",
"split": "new_blind_holdout"
},
{
"id": "a3-024",
"text": "Musik von Kimiko Ishizaka mit einer Goldberg-Variation auf der Geige",
"language": "de",
"category": "music",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Kimiko piano and separate jazz violin exist; requested performer/instrument/composition conjunction does not.",
"split": "new_blind_holdout"
},
{
"id": "a3-025",
"text": "audio of a household pet warning someone with repeated barks",
"language": "en",
"category": "sound",
"scope": "all",
"expected_ids": [
"sound01"
],
"label": "answerable",
"rationale": "Publisher dog bark description supports barking; warning intent is colloquial query, not a labelled mood.",
"split": "new_blind_holdout"
},
{
"id": "a3-026",
"text": "Audio von einem Haustier, das jemanden mit Bellen warnt",
"language": "de",
"category": "sound",
"scope": "all",
"expected_ids": [
"sound01"
],
"label": "answerable",
"rationale": "Publisher dog bark description supports barking; warning intent is colloquial query, not a labelled mood.",
"split": "new_blind_holdout"
},
{
"id": "a3-027",
"text": "audio of a dog barking while church bells ring behind it",
"language": "en",
"category": "sound",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Dog bark recording exists; no independently described church bells in any audio.",
"split": "new_blind_holdout"
},
{
"id": "a3-028",
"text": "Audio eines bellenden Hundes mit Kirchenglocken im Hintergrund",
"language": "de",
"category": "sound",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Dog bark recording exists; no independently described church bells in any audio.",
"split": "new_blind_holdout"
},
{
"id": "a3-029",
"text": "spoken passage saying people possess an inner moral judgement",
"language": "en",
"category": "speech",
"scope": "all",
"expected_ids": [
"speech-en",
"speech-de"
],
"label": "answerable",
"rationale": "Article 1 states endowed with reason and conscience.",
"split": "new_blind_holdout"
},
{
"id": "a3-030",
"text": "Gesprochene Passage darüber, dass Menschen ein inneres moralisches Urteilsvermögen haben",
"language": "de",
"category": "speech",
"scope": "all",
"expected_ids": [
"speech-en",
"speech-de"
],
"label": "answerable",
"rationale": "Article 1 states endowed with reason and conscience.",
"split": "new_blind_holdout"
},
{
"id": "a3-031",
"text": "spoken explanation of the right to paid holidays and limited working hours",
"language": "en",
"category": "speech",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Only Article 1 is in speech files; requested Article 24 topic absent from corpus.",
"split": "new_blind_holdout"
},
{
"id": "a3-032",
"text": "Gesprochene Erklärung des Rechts auf bezahlten Urlaub und begrenzte Arbeitszeit",
"language": "de",
"category": "speech",
"scope": "all",
"expected_ids": [],
"label": "absent",
"rationale": "Only Article 1 is in speech files; requested Article 24 topic absent from corpus.",
"split": "new_blind_holdout"
},
{
"id": "a3-033",
"text": "pictures of somewhere peaceful near water",
"language": "en",
"category": "ambiguous_partial",
"scope": "all",
"expected_ids": [
"image03",
"image05",
"image11"
],
"label": "partial",
"rationale": "Subjective broad request: lake and beach both plausible; excluded from strict no-answer metric.",
"split": "new_blind_holdout"
},
{
"id": "a3-034",
"text": "Bilder von einem ruhigen Ort am Wasser",
"language": "de",
"category": "ambiguous_partial",
"scope": "all",
"expected_ids": [
"image03",
"image05",
"image11"
],
"label": "partial",
"rationale": "Subjective broad request: lake and beach both plausible; excluded from strict no-answer metric.",
"split": "new_blind_holdout"
},
{
"id": "a3-035",
"text": "what do we know about keeping the loaned exhibit safe",
"language": "en",
"category": "ambiguous_partial",
"scope": "all",
"expected_ids": [
"museum-a"
],
"label": "partial",
"rationale": "Insurance is relevant partial evidence, but no physical packing/security instruction; do not score as absent.",
"split": "new_blind_holdout"
},
{
"id": "a3-036",
"text": "Was wissen wir darüber, das geliehene Ausstellungsstück sicher zu halten?",
"language": "de",
"category": "ambiguous_partial",
"scope": "all",
"expected_ids": [
"museum-a"
],
"label": "partial",
"rationale": "Insurance is relevant partial evidence, but no physical packing/security instruction; do not score as absent.",
"split": "new_blind_holdout"
}
]
}
fixtures/absence-v3/fixture.lock.json
+8
View File
@@ -0,0 +1,8 @@
{
"fixture_sha256": "12354a55c1e5f1efe3437e653c074d247374852a420329db8d500771c4a24f53",
"assets": 36,
"queries": 36,
"strict_answerable": 16,
"strict_absent": 16,
"partial": 4
}
fixtures/absence-v3/verify.py
+39
View File
@@ -0,0 +1,39 @@
"""Verify frozen labels and every declared corpus input before inference."""
import hashlib
import json
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
DIRECTORY = Path(__file__).parent
def verify():
target = DIRECTORY / "fixture.json"
fixture = json.loads(target.read_text(encoding="utf-8"))
lock = json.loads((DIRECTORY / "fixture.lock.json").read_text())
assert hashlib.sha256(target.read_bytes()).hexdigest() == lock["fixture_sha256"]
assert fixture["frozen_before_inference"] is True
assert len(fixture["assets"]) == lock["assets"] == 36
assert len(fixture["queries"]) == lock["queries"] == 36
ids = {asset["id"] for asset in fixture["assets"]}
assert len(ids) == 36
for asset in fixture["assets"]:
source = (ROOT / asset["path"]).resolve()
assert source.is_relative_to(ROOT)
assert source.stat().st_size == asset["bytes"]
assert hashlib.sha256(source.read_bytes()).hexdigest() == asset["sha256"], source
query_ids = set()
for query in fixture["queries"]:
assert query["id"] not in query_ids
query_ids.add(query["id"])
assert query["text"] and query["rationale"]
assert query["language"] in ("en", "de")
assert query["scope"] in ("all", "code")
assert query["split"] == "new_blind_holdout"
assert set(query["expected_ids"]) <= ids
assert (query["label"] == "absent") == (not query["expected_ids"])
for label, count_key in (("answerable", "strict_answerable"), ("absent", "strict_absent"), ("partial", "partial")):
assert sum(query["label"] == label for query in fixture["queries"]) == lock[count_key]
return lock
if __name__ == "__main__":
print(json.dumps(verify()))