feat: add multimodal search and granular performance diagnostics
Core CI / core (ubuntu-latest) (push) Failing after 3m31s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m39s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
Core CI / core (ubuntu-latest) (push) Failing after 3m31s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m39s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
This commit is contained in:
karl committed
1 parent
d3bb8c7618
commit
b8400c4fbf
110 files changed
+79395
-250
No files matched your search
@@ -0,0 +1,3 @@
|
||||
derived/
|
||||
inspection/
|
||||
*.sqlite*
|
||||
@@ -0,0 +1,22 @@
|
||||
# Independently labelled media diagnostic v1
|
||||
|
||||
`fixture.json` and `labels.lock.json` were frozen before media inference. The fixture contains 20 original public/licensed files and 65 English/German queries. Neutral filenames prevent filename semantics from giving away visual answers. Do not edit v1 labels after seeing rankings; corrections require a new version and another pre-scoring lock.
|
||||
|
||||
The assistant inspected every photo and sampled real video frames through `view_image`. Cat-action/flower-state intervals came from these frames. Spoken-topic intervals came from NASA's publisher-provided SRT, not a model transcript. Music/sound labels came from independent publisher descriptions and original embedded tags; human listening verification is **not** claimed. One annotator authored these labels, so a blinded second human review remains necessary.
|
||||
|
||||
## Provenance and redistribution
|
||||
|
||||
Every asset has its exact source URL, description page, license, byte count, SHA256, and modification record in the fixture. The assets are a small educational search benchmark, not a stock-media product.
|
||||
|
||||
- Twelve 640px photos are publisher-generated derivatives under the [Pexels License](https://www.pexels.com/license/); no local pixel modifications were made. Pexels IDs identify the sources; attribution is not required by that license. Creator names are not guessed.
|
||||
- `clip01.webm`: **Cat opening door**, Astropulse, [source](https://commons.wikimedia.org/wiki/File:Cat_opening_door.webm), [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). The original is unmodified. Reproduced frames are derivatives and retain that license and attribution.
|
||||
- `clip02.webm`: **Water flowing strem**, BhotaRohit, [source](https://commons.wikimedia.org/wiki/File:Water_flowing_strem.webm), [CC0](https://creativecommons.org/publicdomain/zero/1.0/). Visual inspection reveals a misty waterfall; labels deliberately follow the pixels rather than the filename.
|
||||
- `clip03.webm` and `sound02.mp3`: MDN contributors, original [interactive-example assets](https://github.com/mdn/interactive-examples), repository [CC0 notice](https://raw.githubusercontent.com/mdn/interactive-examples/main/LICENSE). The files are the real flower timelapse and roaring-creature effect.
|
||||
- `clip04.mp4`: NASA **Artemis Origins**, [source](https://images.nasa.gov/details/Artemis%20Origins), used for educational/informational evaluation under [NASA media usage permission](https://www.nasa.gov/nasa-brand-center/images-and-media/). No endorsement is implied; NASA insignia/trademarks are not used as Searchi branding. The footage is unmodified.
|
||||
- `music01.ogg`: **Variatio 13 a 2 Clav.**, Kimiko Ishizaka, [source](https://commons.wikimedia.org/wiki/File:Kimiko_Ishizaka_-_14_-_Variatio_13_a_2_Clav.ogg), [CC0](https://creativecommons.org/publicdomain/zero/1.0/). The project's primary [release description](https://github.com/robertDouglass/opengoldbergvariations.org/blob/master/config.toml) confirms the public-domain release, independently of the Commons pending-review banner. Original audio and Vorbis metadata are retained.
|
||||
- `music02.ogg`: **Jazz Violin Solo**, VWD, [source](https://commons.wikimedia.org/wiki/File:00_Jazz_Violin_Solo.ogg), author's worldwide public-domain dedication on that page. Original tags identify the artist as **Violinist WTD**, which is intentionally the strict metadata query rather than a guessed identity.
|
||||
- `sound01.wav`: **Dog barking mono**, Brandon Morris, uploaded by HaelDB, [source](https://opengameart.org/content/dog-barking-mono), [CC0](https://creativecommons.org/publicdomain/zero/1.0/). Original waveform retained.
|
||||
|
||||
All source licenses remain attached to their files. Root repository MIT licensing does not replace these third-party media terms. `derived/` and `inspection/` are ignored; generated derivatives carry their source licenses when redistributed.
|
||||
|
||||
This collection is deliberately tiny. Twenty public assets do not establish quality on a user's complete photo library, music collection, recorded meetings, or long movies.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 64 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,852 @@
|
||||
{
|
||||
"background_prewarm_ms": 133.0573,
|
||||
"device": "cpu",
|
||||
"documents": 20,
|
||||
"embedding_report": {
|
||||
"chunks": 20,
|
||||
"generation": 1,
|
||||
"model_fingerprint": "google/embeddinggemma-2:bfcd298762cc34d0357ece5ebdd31791a3a374d8:q8_0:llama-b11461-no-logits-v1:mean:l2:query512:doc2048:windows-overlap128:prompts-v1:byte64k-overlap512-title512-v1",
|
||||
"passages": 20
|
||||
},
|
||||
"first_foreground_ms": 108.6225,
|
||||
"first_foreground_top1": "supplier-agreement.txt",
|
||||
"hybrid": {
|
||||
"recall_at10": 1.0,
|
||||
"recall_at3": 1.0,
|
||||
"top1_accuracy": 1.0
|
||||
},
|
||||
"hybrid_p50_ms": 107.3451,
|
||||
"hybrid_p95_ms": 146.82819999999998,
|
||||
"indexing_ms": 4840.859100000001,
|
||||
"lexical": {
|
||||
"recall_at10": 0.20833333333333334,
|
||||
"recall_at3": 0.20833333333333334,
|
||||
"top1_accuracy": 0.20833333333333334
|
||||
},
|
||||
"limitation": "Synthetic seed corpus only; no production quality or large-corpus latency certification",
|
||||
"load_ms": 8644.3299,
|
||||
"model": "google/embeddinggemma-2",
|
||||
"queries": 24,
|
||||
"rankings": [
|
||||
{
|
||||
"expected": "supplier-contract",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "supplier-agreement.txt",
|
||||
"match_kind": "exact_filename",
|
||||
"source": null
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "exact",
|
||||
"lexical_rank": 1,
|
||||
"query": "supplier-agreement.txt",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "invoice",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "filename",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "network-incident.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "identifier",
|
||||
"lexical_rank": 1,
|
||||
"query": "INV-2026-1042",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "error-code",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "support-error.txt",
|
||||
"match_kind": "exact_identifier",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "search-engine-design.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "identifier",
|
||||
"lexical_rank": 1,
|
||||
"query": "E_SEARCH_071",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "supplier-contract",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "supplier-agreement.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "retry-design.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "How can we end our relationship with the supplier?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "supplier-contract",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "supplier-agreement.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "mietvertrag.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "de-crosslingual",
|
||||
"lexical_rank": null,
|
||||
"query": "Wie kündigen wir die Zusammenarbeit mit unserem Lieferanten?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "apartment-lease",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "mietvertrag.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "support-error.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "urlaubsrichtlinie.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en-crosslingual",
|
||||
"lexical_rank": null,
|
||||
"query": "Can I keep a pet in the rented apartment?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "apartment-lease",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "mietvertrag.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "reisekosten.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "monitor-warranty.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "de",
|
||||
"lexical_rank": null,
|
||||
"query": "Wo steht etwas über die Kaution für die Wohnung?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "travel-expenses",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "reisekosten.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "conference-agenda.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en-crosslingual",
|
||||
"lexical_rank": null,
|
||||
"query": "How do I claim my train and hotel costs?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "conference-agenda",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "conference-agenda.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "support-error.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "search-engine-design.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "de-crosslingual",
|
||||
"lexical_rank": null,
|
||||
"query": "Wann ist der Workshop zur Datenbankleistung?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "deployment",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "deployment-runbook.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "product-launch.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "support-error.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "What should we do if the new release fails its health check?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "backup-restore",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "backup-recovery.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "network-incident.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "de-crosslingual",
|
||||
"lexical_rank": null,
|
||||
"query": "Ich habe Dateien gelöscht und brauche sie aus der Sicherung zurück",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "api-retry",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "retry-design.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "deployment-runbook.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "network-incident.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "How should the app retry a temporary server failure?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "oauth",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "authentication-notes.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "retry-design.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "backup-recovery.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "How do we obtain another access token after it expires?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "network-outage",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "network-incident.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "backup-recovery.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "de-crosslingual",
|
||||
"lexical_rank": null,
|
||||
"query": "Warum war der Netzwerkswitch nicht erreichbar?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "launch-plan",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "product-launch.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "research-budget.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "deployment-runbook.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "The plan for announcing the new product after beta testing",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "research-budget",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "research-budget.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "retry-design.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "Money allocated for interviews and prototype usability studies",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "vacation-policy",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "urlaubsrichtlinie.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en-crosslingual",
|
||||
"lexical_rank": null,
|
||||
"query": "Who approves my time off and who covers urgent work?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "recipe",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "tomato-soup.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "gartenplanung.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "A soup made with roasted tomatoes and basil",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "renovation",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "renovation-estimate.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "Does the kitchen renovation quote include new flooring?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "warranty",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "monitor-warranty.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "network-incident.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "backup-recovery.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "The document about replacing a defective display under warranty",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "search-architecture",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "search-engine-design.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "support-error.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "backup-recovery.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": null,
|
||||
"query": "How can keyword results and embeddings be combined for file search?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "garden",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "gartenplanung.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "renovation-estimate.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en-crosslingual",
|
||||
"lexical_rank": null,
|
||||
"query": "Where is the plan for the raised vegetable bed and rainwater barrel?",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "cancellation-email",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "hybrid",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "urlaubsrichtlinie.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "conference-agenda.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "en",
|
||||
"lexical_rank": 1,
|
||||
"query": "Cancel tomorrow's planning meeting",
|
||||
"semantic_rank": 1
|
||||
},
|
||||
{
|
||||
"expected": "invoice",
|
||||
"hybrid_rank": 1,
|
||||
"hybrid_top3": [
|
||||
{
|
||||
"filename": "INV-2026-1042.txt",
|
||||
"match_kind": "exact_identifier",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "meeting-cancellation.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"filename": "search-engine-design.txt",
|
||||
"match_kind": "semantic",
|
||||
"source": {
|
||||
"kind": "line",
|
||||
"number": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"language": "identifier",
|
||||
"lexical_rank": 1,
|
||||
"query": "OFFICE-CHAIRS-42",
|
||||
"semantic_rank": 1
|
||||
}
|
||||
],
|
||||
"semantic": {
|
||||
"recall_at10": 1.0,
|
||||
"recall_at3": 1.0,
|
||||
"top1_accuracy": 1.0
|
||||
},
|
||||
"startup_init_ms": 8286.3466,
|
||||
"threads": 1
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"date": "2026-10-12",
|
||||
"os": "Microsoft Windows 11 Pro",
|
||||
"os_version": "10.0.26300",
|
||||
"visible_memory_kib": 10416556,
|
||||
"cpu_name": "AMD Ryzen 7 7800X3D 8-Core Processor ",
|
||||
"vm_vcpus": 4,
|
||||
"model_threads": 1,
|
||||
"device": "cpu",
|
||||
"force_cpu": "1",
|
||||
"ffmpeg_version": "9.0.2",
|
||||
"worker_sha256": "f2ad020307469371519b848c9fa92ae4c51ee8ae8a0ce917f8f408f5756e9086",
|
||||
"scope": "Shared Windows VM; timing affected by other work; RAM observations are not measured peak"
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"fixture_sha256": "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393",
|
||||
"worker": {
|
||||
"device": "cpu",
|
||||
"dimensions": 768,
|
||||
"media_supported": true,
|
||||
"model": "google/embeddinggemma-2",
|
||||
"ready": true,
|
||||
"runtime": "b11461-no-logits-v1",
|
||||
"threads": 1
|
||||
},
|
||||
"photos": 12,
|
||||
"queries": 24,
|
||||
"query_embedding_ms": 2262.7884,
|
||||
"budgets": [
|
||||
{
|
||||
"token_budget": 70,
|
||||
"encoding_ms": 47612.655699999996,
|
||||
"metrics": {
|
||||
"positive_queries": 24,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"token_budget": 280,
|
||||
"encoding_ms": 199652.08899999998,
|
||||
"metrics": {
|
||||
"positive_queries": 24,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"token_budget": 1120,
|
||||
"encoding_ms": 1493321.6608,
|
||||
"metrics": {
|
||||
"positive_queries": 24,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"interpretation": "Same frozen 12 photos and 24 correlated EN/DE queries. Query embeddings shared across budgets. Encoding time is CPU batch cost; ranking-only time excludes inference. Equal small-corpus recall does not prove equal quality on larger or more difficult images."
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"fixture_sha256": "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393",
|
||||
"frozen_before_media_scoring": true,
|
||||
"document_seed_sha256": "3bab37fc0ec9c2e2e40684962b049172b5078358c6487c9033ed8697b8128dd7",
|
||||
"query_count": 65,
|
||||
"asset_count": 20
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"encoding_failures": [],
|
||||
"encoding_ms": 1863.0235000000002,
|
||||
"fixture_sha256": "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393",
|
||||
"rows": [
|
||||
{
|
||||
"file_rankings": [
|
||||
{
|
||||
"candidate": {
|
||||
"asset": "music01",
|
||||
"match_kind": "media_metadata",
|
||||
"modality": "metadata"
|
||||
},
|
||||
"score": 2.847686714371868
|
||||
}
|
||||
],
|
||||
"query": {
|
||||
"category": "exact-metadata",
|
||||
"expected": [
|
||||
"music01"
|
||||
],
|
||||
"id": "q055",
|
||||
"language": "en",
|
||||
"metadata": {
|
||||
"artist": "Kimiko Ishizaka"
|
||||
},
|
||||
"text": "Kimiko Ishizaka"
|
||||
},
|
||||
"query_ms": 1.0931,
|
||||
"segment_top5": []
|
||||
},
|
||||
{
|
||||
"file_rankings": [
|
||||
{
|
||||
"candidate": {
|
||||
"asset": "music02",
|
||||
"match_kind": "media_metadata",
|
||||
"modality": "metadata"
|
||||
},
|
||||
"score": 4.671020217612865
|
||||
}
|
||||
],
|
||||
"query": {
|
||||
"category": "exact-metadata",
|
||||
"expected": [
|
||||
"music02"
|
||||
],
|
||||
"id": "q056",
|
||||
"language": "en",
|
||||
"metadata": {
|
||||
"title": "Jazz Violin Solo"
|
||||
},
|
||||
"text": "Jazz Violin Solo"
|
||||
},
|
||||
"query_ms": 0.7408,
|
||||
"segment_top5": []
|
||||
},
|
||||
{
|
||||
"file_rankings": [
|
||||
{
|
||||
"candidate": {
|
||||
"asset": "music02",
|
||||
"match_kind": "media_metadata",
|
||||
"modality": "metadata"
|
||||
},
|
||||
"score": 3.6759070488571206
|
||||
}
|
||||
],
|
||||
"query": {
|
||||
"category": "exact-metadata",
|
||||
"expected": [
|
||||
"music02"
|
||||
],
|
||||
"id": "q057",
|
||||
"language": "en",
|
||||
"metadata": {
|
||||
"artist": "Violinist WTD"
|
||||
},
|
||||
"text": "Violinist WTD"
|
||||
},
|
||||
"query_ms": 0.6757,
|
||||
"segment_top5": []
|
||||
},
|
||||
{
|
||||
"file_rankings": [],
|
||||
"query": {
|
||||
"category": "metadata-no-answer",
|
||||
"expected": [],
|
||||
"id": "q065",
|
||||
"language": "en",
|
||||
"metadata": {
|
||||
"artist": "Adele"
|
||||
},
|
||||
"text": "Adele"
|
||||
},
|
||||
"query_ms": 0.4148,
|
||||
"segment_top5": []
|
||||
}
|
||||
],
|
||||
"scope": "Actual FFprobe stream/format metadata + guarded Searchi core FTS; lexical identity queries, no embedding fallback; encoding_ms measures metadata extraction, not model inference",
|
||||
"worker": null
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"fixture_sha256": "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393",
|
||||
"worker": null,
|
||||
"overall": {
|
||||
"positive_queries": 3,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 0.0
|
||||
},
|
||||
"by_category": {
|
||||
"exact-metadata": {
|
||||
"positive_queries": 3,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"metadata-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 0.0
|
||||
}
|
||||
},
|
||||
"by_language": {
|
||||
"en": {
|
||||
"positive_queries": 3,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 0.0
|
||||
}
|
||||
},
|
||||
"encoding_failures": [],
|
||||
"encoding_ms": 1863.0235000000002,
|
||||
"query_p50_ms": 0.7408,
|
||||
"query_p95_ms": 1.0931,
|
||||
"failures": [],
|
||||
"source_scope": "Actual FFprobe stream/format metadata + guarded Searchi core FTS; lexical identity queries, no embedding fallback; encoding_ms measures metadata extraction, not model inference",
|
||||
"interpretation": "Small independently labelled public-media diagnostic; source_scope identifies the measured lane. No-answer return rate measures unsuppressed output, not a calibrated confidence policy. Audio interval overlap can pass while the returned start timestamp is too early; point hits expose this. This corpus alone cannot certify production quality."
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,561 @@
|
||||
{
|
||||
"fixture_sha256": "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393",
|
||||
"worker": {
|
||||
"device": "cpu",
|
||||
"dimensions": 768,
|
||||
"media_supported": true,
|
||||
"model": "google/embeddinggemma-2",
|
||||
"ready": true,
|
||||
"runtime": "b11461-no-logits-v1",
|
||||
"threads": 1
|
||||
},
|
||||
"overall": {
|
||||
"positive_queries": 58,
|
||||
"unsupported_queries": 1,
|
||||
"recall_at1": 0.9655172413793104,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 0.9827586206896551,
|
||||
"temporal_queries": 22,
|
||||
"segment_overlap_hit_at1": 0.8181818181818182,
|
||||
"timestamp_point_hit_at1": 0.5909090909090909,
|
||||
"segment_overlap_hit_at5": 0.8181818181818182,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 3.777000000000001,
|
||||
"no_answer_queries": 6,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"by_category": {
|
||||
"content-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"exact-filter": {
|
||||
"positive_queries": 1,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"exact-metadata": {
|
||||
"positive_queries": 3,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"filter-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"image-memory": {
|
||||
"positive_queries": 24,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"metadata-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"music-memory": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 0.5,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 0.75,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"sound-memory": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"speech-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"temporal-negative": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 1,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-action": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 4,
|
||||
"segment_overlap_hit_at1": 1.0,
|
||||
"timestamp_point_hit_at1": 1.0,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 0,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-scene": {
|
||||
"positive_queries": 6,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 6,
|
||||
"segment_overlap_hit_at1": 1.0,
|
||||
"timestamp_point_hit_at1": 1.0,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 0,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-speech": {
|
||||
"positive_queries": 8,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 8,
|
||||
"segment_overlap_hit_at1": 0.75,
|
||||
"timestamp_point_hit_at1": 0.125,
|
||||
"segment_overlap_hit_at5": 0.75,
|
||||
"timestamp_distance_median_s": 1.551,
|
||||
"timestamp_distance_max_s": 3.777000000000001,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-temporal": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 4,
|
||||
"segment_overlap_hit_at1": 0.5,
|
||||
"timestamp_point_hit_at1": 0.5,
|
||||
"segment_overlap_hit_at5": 0.5,
|
||||
"timestamp_distance_median_s": 1.005,
|
||||
"timestamp_distance_max_s": 1.005,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
}
|
||||
},
|
||||
"by_language": {
|
||||
"de": {
|
||||
"positive_queries": 27,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 0.9629629629629629,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 0.9814814814814815,
|
||||
"temporal_queries": 11,
|
||||
"segment_overlap_hit_at1": 0.8181818181818182,
|
||||
"timestamp_point_hit_at1": 0.6363636363636364,
|
||||
"segment_overlap_hit_at5": 0.8181818181818182,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 1.551,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"en": {
|
||||
"positive_queries": 31,
|
||||
"unsupported_queries": 1,
|
||||
"recall_at1": 0.967741935483871,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 0.9838709677419355,
|
||||
"temporal_queries": 11,
|
||||
"segment_overlap_hit_at1": 0.8181818181818182,
|
||||
"timestamp_point_hit_at1": 0.5454545454545454,
|
||||
"segment_overlap_hit_at5": 0.8181818181818182,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 3.777000000000001,
|
||||
"no_answer_queries": 4,
|
||||
"no_answer_return_rate": 1.0
|
||||
}
|
||||
},
|
||||
"encoding_failures": [],
|
||||
"encoding_ms": 671510.0948000001,
|
||||
"query_p50_ms": 90.6467,
|
||||
"query_p95_ms": 110.4152,
|
||||
"failures": [
|
||||
{
|
||||
"query": "q029",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip03",
|
||||
"end_s": 4.005,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_video",
|
||||
"modality": "image",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 2005
|
||||
},
|
||||
"start_s": 2.005
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip03": [
|
||||
[
|
||||
0,
|
||||
1.0
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.005
|
||||
},
|
||||
{
|
||||
"query": "q030",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip03",
|
||||
"end_s": 4.005,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_video",
|
||||
"modality": "image",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 2005
|
||||
},
|
||||
"start_s": 2.005
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip03": [
|
||||
[
|
||||
0,
|
||||
1.0
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.005
|
||||
},
|
||||
{
|
||||
"query": "q039",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 10.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 0
|
||||
},
|
||||
"start_s": 0.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
1.551,
|
||||
3.603
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.551
|
||||
},
|
||||
{
|
||||
"query": "q040",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 10.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 0
|
||||
},
|
||||
"start_s": 0.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
1.551,
|
||||
3.603
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.551
|
||||
},
|
||||
{
|
||||
"query": "q041",
|
||||
"category": "wrong-media-channel",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 26.167,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_video",
|
||||
"modality": "image",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 24167
|
||||
},
|
||||
"start_s": 24.167
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
13.129,
|
||||
16.316
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": null
|
||||
},
|
||||
{
|
||||
"query": "q042",
|
||||
"category": "wrong-media-channel",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 14.467,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_video",
|
||||
"modality": "image",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 12467
|
||||
},
|
||||
"start_s": 12.467
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
13.129,
|
||||
16.316
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": null
|
||||
},
|
||||
{
|
||||
"query": "q043",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 50.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 40000
|
||||
},
|
||||
"start_s": 40.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
43.777,
|
||||
52.419
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 3.777000000000001
|
||||
},
|
||||
{
|
||||
"query": "q045",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 66.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 56000
|
||||
},
|
||||
"start_s": 56.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
56.539,
|
||||
63.496
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 0.5390000000000015
|
||||
},
|
||||
{
|
||||
"query": "q046",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 66.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 56000
|
||||
},
|
||||
"start_s": 56.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
56.539,
|
||||
63.496
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 0.5390000000000015
|
||||
},
|
||||
{
|
||||
"query": "q049",
|
||||
"category": "hard-negative-confusion",
|
||||
"winner": "music01",
|
||||
"expected": [
|
||||
"music02"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query": "q050",
|
||||
"category": "hard-negative-confusion",
|
||||
"winner": "music01",
|
||||
"expected": [
|
||||
"music02"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query": "q059",
|
||||
"category": "unsupported-query-capability",
|
||||
"detail": "Production query API has no temporal-range filter yet"
|
||||
},
|
||||
{
|
||||
"query": "q060",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music01",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q061",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music01",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q062",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music01",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q063",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip04",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q064",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip04",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q065",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip04",
|
||||
"score": 0.01639344262295082
|
||||
}
|
||||
],
|
||||
"source_scope": "Actual Searchi hybrid pipeline and native extension filters; temporal-range filter unsupported; segment ends inferred from configured audio/frame policy, exact point timestamps come from actual result API",
|
||||
"interpretation": "Small independently labelled public-media diagnostic; source_scope identifies the measured lane. No-answer return rate measures unsuppressed output, not a calibrated confidence policy. Audio interval overlap can pass while the returned start timestamp is too early; point hits expose this. This corpus alone cannot certify production quality."
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,586 @@
|
||||
{
|
||||
"fixture_sha256": "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393",
|
||||
"worker": {
|
||||
"device": "cpu",
|
||||
"dimensions": 768,
|
||||
"media_supported": true,
|
||||
"model": "google/embeddinggemma-2",
|
||||
"ready": true,
|
||||
"runtime": "b11461-no-logits-v1",
|
||||
"threads": 1
|
||||
},
|
||||
"overall": {
|
||||
"positive_queries": 58,
|
||||
"unsupported_queries": 1,
|
||||
"recall_at1": 0.9655172413793104,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 0.9827586206896551,
|
||||
"temporal_queries": 22,
|
||||
"segment_overlap_hit_at1": 0.8181818181818182,
|
||||
"timestamp_point_hit_at1": 0.5454545454545454,
|
||||
"segment_overlap_hit_at5": 0.8181818181818182,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 15.777000000000001,
|
||||
"no_answer_queries": 6,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"by_category": {
|
||||
"content-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"exact-filter": {
|
||||
"positive_queries": 1,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"exact-metadata": {
|
||||
"positive_queries": 3,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"filter-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"image-memory": {
|
||||
"positive_queries": 24,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"metadata-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"music-memory": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 0.5,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 0.75,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"sound-memory": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"speech-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"temporal-negative": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 1,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-action": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 4,
|
||||
"segment_overlap_hit_at1": 1.0,
|
||||
"timestamp_point_hit_at1": 1.0,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 0,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-scene": {
|
||||
"positive_queries": 6,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 6,
|
||||
"segment_overlap_hit_at1": 1.0,
|
||||
"timestamp_point_hit_at1": 1.0,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 0,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-speech": {
|
||||
"positive_queries": 8,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 8,
|
||||
"segment_overlap_hit_at1": 0.75,
|
||||
"timestamp_point_hit_at1": 0.0,
|
||||
"segment_overlap_hit_at5": 0.75,
|
||||
"timestamp_distance_median_s": 1.551,
|
||||
"timestamp_distance_max_s": 15.777000000000001,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-temporal": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 4,
|
||||
"segment_overlap_hit_at1": 0.5,
|
||||
"timestamp_point_hit_at1": 0.5,
|
||||
"segment_overlap_hit_at5": 0.5,
|
||||
"timestamp_distance_median_s": 1.005,
|
||||
"timestamp_distance_max_s": 1.005,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
}
|
||||
},
|
||||
"by_language": {
|
||||
"de": {
|
||||
"positive_queries": 27,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 0.9629629629629629,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 0.9814814814814815,
|
||||
"temporal_queries": 11,
|
||||
"segment_overlap_hit_at1": 0.8181818181818182,
|
||||
"timestamp_point_hit_at1": 0.5454545454545454,
|
||||
"segment_overlap_hit_at5": 0.8181818181818182,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 15.777000000000001,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"en": {
|
||||
"positive_queries": 31,
|
||||
"unsupported_queries": 1,
|
||||
"recall_at1": 0.967741935483871,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 0.9838709677419355,
|
||||
"temporal_queries": 11,
|
||||
"segment_overlap_hit_at1": 0.8181818181818182,
|
||||
"timestamp_point_hit_at1": 0.5454545454545454,
|
||||
"segment_overlap_hit_at5": 0.8181818181818182,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 15.777000000000001,
|
||||
"no_answer_queries": 4,
|
||||
"no_answer_return_rate": 1.0
|
||||
}
|
||||
},
|
||||
"encoding_failures": [],
|
||||
"encoding_ms": 668040.7295,
|
||||
"query_p50_ms": 92.2329,
|
||||
"query_p95_ms": 113.3751,
|
||||
"failures": [
|
||||
{
|
||||
"query": "q029",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip03",
|
||||
"end_s": 4.005,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_video",
|
||||
"modality": "image",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 2005
|
||||
},
|
||||
"start_s": 2.005
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip03": [
|
||||
[
|
||||
0,
|
||||
1.0
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.005
|
||||
},
|
||||
{
|
||||
"query": "q030",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip03",
|
||||
"end_s": 4.005,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_video",
|
||||
"modality": "image",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 2005
|
||||
},
|
||||
"start_s": 2.005
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip03": [
|
||||
[
|
||||
0,
|
||||
1.0
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.005
|
||||
},
|
||||
{
|
||||
"query": "q039",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 30.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 0
|
||||
},
|
||||
"start_s": 0.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
1.551,
|
||||
3.603
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.551
|
||||
},
|
||||
{
|
||||
"query": "q040",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 30.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 0
|
||||
},
|
||||
"start_s": 0.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
1.551,
|
||||
3.603
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.551
|
||||
},
|
||||
{
|
||||
"query": "q041",
|
||||
"category": "wrong-media-channel",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 26.167,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_video",
|
||||
"modality": "image",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 24167
|
||||
},
|
||||
"start_s": 24.167
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
13.129,
|
||||
16.316
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": null
|
||||
},
|
||||
{
|
||||
"query": "q042",
|
||||
"category": "wrong-media-channel",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 14.467,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_video",
|
||||
"modality": "image",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 12467
|
||||
},
|
||||
"start_s": 12.467
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
13.129,
|
||||
16.316
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": null
|
||||
},
|
||||
{
|
||||
"query": "q043",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 58.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 28000
|
||||
},
|
||||
"start_s": 28.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
43.777,
|
||||
52.419
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 15.777000000000001
|
||||
},
|
||||
{
|
||||
"query": "q044",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 58.0,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 28000
|
||||
},
|
||||
"start_s": 28.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
43.777,
|
||||
52.419
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 15.777000000000001
|
||||
},
|
||||
{
|
||||
"query": "q045",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 80.63,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 56000
|
||||
},
|
||||
"start_s": 56.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
56.539,
|
||||
63.496
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 0.5390000000000015
|
||||
},
|
||||
{
|
||||
"query": "q046",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 80.63,
|
||||
"interval_note": "End inferred from configured pipeline window policy, actual API exposes only start timestamp",
|
||||
"match_kind": "semantic_audio",
|
||||
"modality": "audio",
|
||||
"source": {
|
||||
"kind": "timestamp_ms",
|
||||
"number": 56000
|
||||
},
|
||||
"start_s": 56.0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
56.539,
|
||||
63.496
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 0.5390000000000015
|
||||
},
|
||||
{
|
||||
"query": "q049",
|
||||
"category": "hard-negative-confusion",
|
||||
"winner": "music01",
|
||||
"expected": [
|
||||
"music02"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query": "q050",
|
||||
"category": "hard-negative-confusion",
|
||||
"winner": "music01",
|
||||
"expected": [
|
||||
"music02"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query": "q059",
|
||||
"category": "unsupported-query-capability",
|
||||
"detail": "Production query API has no temporal-range filter yet"
|
||||
},
|
||||
{
|
||||
"query": "q060",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music01",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q061",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music02",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q062",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music02",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q063",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip04",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q064",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip04",
|
||||
"score": 0.01639344262295082
|
||||
},
|
||||
{
|
||||
"query": "q065",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip04",
|
||||
"score": 0.01639344262295082
|
||||
}
|
||||
],
|
||||
"source_scope": "Actual Searchi hybrid pipeline and native extension filters; temporal-range filter unsupported; segment ends inferred from configured audio/frame policy, exact point timestamps come from actual result API",
|
||||
"interpretation": "Small independently labelled public-media diagnostic; source_scope identifies the measured lane. No-answer return rate measures unsuppressed output, not a calibrated confidence policy. Audio interval overlap can pass while the returned start timestamp is too early; point hits expose this. This corpus alone cannot certify production quality."
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,502 @@
|
||||
{
|
||||
"fixture_sha256": "b4e465a0eb9582b826eebd2a20797e1680e4907e336d02d0ef2194e2ba0af393",
|
||||
"worker": {
|
||||
"device": "cpu",
|
||||
"dimensions": 768,
|
||||
"model": "google/embeddinggemma-2",
|
||||
"ready": true,
|
||||
"runtime": "b11461-no-logits-v1",
|
||||
"threads": 1
|
||||
},
|
||||
"overall": {
|
||||
"positive_queries": 55,
|
||||
"unsupported_queries": 4,
|
||||
"recall_at1": 0.9636363636363636,
|
||||
"recall_at5": 0.9636363636363636,
|
||||
"mrr": 0.9636363636363636,
|
||||
"temporal_queries": 22,
|
||||
"segment_overlap_hit_at1": 0.9545454545454546,
|
||||
"timestamp_point_hit_at1": 0.6818181818181818,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 13.777000000000001,
|
||||
"no_answer_queries": 6,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"by_category": {
|
||||
"content-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"exact-filter": {
|
||||
"positive_queries": 1,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"exact-metadata": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 3,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"filter-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"image-memory": {
|
||||
"positive_queries": 24,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"metadata-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 1,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"music-memory": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 0.5,
|
||||
"recall_at5": 0.5,
|
||||
"mrr": 0.5,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"sound-memory": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"speech-no-answer": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"temporal-negative": {
|
||||
"positive_queries": 0,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": null,
|
||||
"recall_at5": null,
|
||||
"mrr": null,
|
||||
"temporal_queries": 0,
|
||||
"segment_overlap_hit_at1": null,
|
||||
"timestamp_point_hit_at1": null,
|
||||
"segment_overlap_hit_at5": null,
|
||||
"timestamp_distance_median_s": null,
|
||||
"timestamp_distance_max_s": null,
|
||||
"no_answer_queries": 1,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"video-action": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 4,
|
||||
"segment_overlap_hit_at1": 1.0,
|
||||
"timestamp_point_hit_at1": 1.0,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 0,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-scene": {
|
||||
"positive_queries": 6,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 6,
|
||||
"segment_overlap_hit_at1": 1.0,
|
||||
"timestamp_point_hit_at1": 1.0,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 0,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-speech": {
|
||||
"positive_queries": 8,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 8,
|
||||
"segment_overlap_hit_at1": 1.0,
|
||||
"timestamp_point_hit_at1": 0.25,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 13.129,
|
||||
"timestamp_distance_max_s": 13.777000000000001,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
},
|
||||
"video-temporal": {
|
||||
"positive_queries": 4,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 1.0,
|
||||
"recall_at5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"temporal_queries": 4,
|
||||
"segment_overlap_hit_at1": 0.75,
|
||||
"timestamp_point_hit_at1": 0.75,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 1.0,
|
||||
"no_answer_queries": 0,
|
||||
"no_answer_return_rate": null
|
||||
}
|
||||
},
|
||||
"by_language": {
|
||||
"de": {
|
||||
"positive_queries": 27,
|
||||
"unsupported_queries": 0,
|
||||
"recall_at1": 0.9629629629629629,
|
||||
"recall_at5": 0.9629629629629629,
|
||||
"mrr": 0.9629629629629629,
|
||||
"temporal_queries": 11,
|
||||
"segment_overlap_hit_at1": 0.9090909090909091,
|
||||
"timestamp_point_hit_at1": 0.6363636363636364,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 13.777000000000001,
|
||||
"no_answer_queries": 2,
|
||||
"no_answer_return_rate": 1.0
|
||||
},
|
||||
"en": {
|
||||
"positive_queries": 28,
|
||||
"unsupported_queries": 4,
|
||||
"recall_at1": 0.9642857142857143,
|
||||
"recall_at5": 0.9642857142857143,
|
||||
"mrr": 0.9642857142857143,
|
||||
"temporal_queries": 11,
|
||||
"segment_overlap_hit_at1": 1.0,
|
||||
"timestamp_point_hit_at1": 0.7272727272727273,
|
||||
"segment_overlap_hit_at5": 1.0,
|
||||
"timestamp_distance_median_s": 0,
|
||||
"timestamp_distance_max_s": 13.777000000000001,
|
||||
"no_answer_queries": 4,
|
||||
"no_answer_return_rate": 1.0
|
||||
}
|
||||
},
|
||||
"encoding_failures": [
|
||||
{
|
||||
"candidate": {
|
||||
"asset": "music02",
|
||||
"end_s": 30,
|
||||
"modality": "audio",
|
||||
"path": "C:\\Users\\karl\\searchi\\fixtures\\media\\derived\\music02-audio-0000.f32",
|
||||
"start_s": 0
|
||||
},
|
||||
"error": "Invalid normalized audio PCM"
|
||||
}
|
||||
],
|
||||
"encoding_ms": 498657.0532,
|
||||
"query_p50_ms": 92.78150000000001,
|
||||
"query_p95_ms": 122.9277,
|
||||
"failures": [
|
||||
{
|
||||
"query": "q030",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip03",
|
||||
"end_s": 4,
|
||||
"modality": "image",
|
||||
"path": "C:\\Users\\karl\\searchi\\fixtures\\media\\derived\\clip03-frame-0002.png",
|
||||
"start_s": 2,
|
||||
"token_budget": 70
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip03": [
|
||||
[
|
||||
0,
|
||||
1.0
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.0
|
||||
},
|
||||
{
|
||||
"query": "q039",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 30,
|
||||
"modality": "audio",
|
||||
"path": "C:\\Users\\karl\\searchi\\fixtures\\media\\derived\\clip04-audio-0000.f32",
|
||||
"start_s": 0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
1.551,
|
||||
3.603
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.551
|
||||
},
|
||||
{
|
||||
"query": "q040",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 30,
|
||||
"modality": "audio",
|
||||
"path": "C:\\Users\\karl\\searchi\\fixtures\\media\\derived\\clip04-audio-0000.f32",
|
||||
"start_s": 0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
1.551,
|
||||
3.603
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 1.551
|
||||
},
|
||||
{
|
||||
"query": "q041",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 30,
|
||||
"modality": "audio",
|
||||
"path": "C:\\Users\\karl\\searchi\\fixtures\\media\\derived\\clip04-audio-0000.f32",
|
||||
"start_s": 0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
13.129,
|
||||
16.316
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 13.129
|
||||
},
|
||||
{
|
||||
"query": "q042",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 30,
|
||||
"modality": "audio",
|
||||
"path": "C:\\Users\\karl\\searchi\\fixtures\\media\\derived\\clip04-audio-0000.f32",
|
||||
"start_s": 0
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
13.129,
|
||||
16.316
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 13.129
|
||||
},
|
||||
{
|
||||
"query": "q043",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 60,
|
||||
"modality": "audio",
|
||||
"path": "C:\\Users\\karl\\searchi\\fixtures\\media\\derived\\clip04-audio-0030.f32",
|
||||
"start_s": 30
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
43.777,
|
||||
52.419
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 13.777000000000001
|
||||
},
|
||||
{
|
||||
"query": "q044",
|
||||
"category": "timestamp-localization-miss",
|
||||
"candidate": {
|
||||
"asset": "clip04",
|
||||
"end_s": 60,
|
||||
"modality": "audio",
|
||||
"path": "C:\\Users\\karl\\searchi\\fixtures\\media\\derived\\clip04-audio-0030.f32",
|
||||
"start_s": 30
|
||||
},
|
||||
"expected_intervals": {
|
||||
"clip04": [
|
||||
[
|
||||
43.777,
|
||||
52.419
|
||||
]
|
||||
]
|
||||
},
|
||||
"timestamp_distance_s": 13.777000000000001
|
||||
},
|
||||
{
|
||||
"query": "q049",
|
||||
"category": "hard-negative-confusion",
|
||||
"winner": "music01",
|
||||
"expected": [
|
||||
"music02"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query": "q050",
|
||||
"category": "hard-negative-confusion",
|
||||
"winner": "music01",
|
||||
"expected": [
|
||||
"music02"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query": "q055",
|
||||
"category": "unsupported-query-capability",
|
||||
"detail": "Exact metadata requires production lexical lane; direct audio embeddings cannot verify artist/title identities"
|
||||
},
|
||||
{
|
||||
"query": "q056",
|
||||
"category": "unsupported-query-capability",
|
||||
"detail": "Exact metadata requires production lexical lane; direct audio embeddings cannot verify artist/title identities"
|
||||
},
|
||||
{
|
||||
"query": "q057",
|
||||
"category": "unsupported-query-capability",
|
||||
"detail": "Exact metadata requires production lexical lane; direct audio embeddings cannot verify artist/title identities"
|
||||
},
|
||||
{
|
||||
"query": "q059",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip01",
|
||||
"score": 0.750250822387441
|
||||
},
|
||||
{
|
||||
"query": "q060",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music01",
|
||||
"score": 0.5716409940327322
|
||||
},
|
||||
{
|
||||
"query": "q061",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music01",
|
||||
"score": 0.6223961635527826
|
||||
},
|
||||
{
|
||||
"query": "q062",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "music01",
|
||||
"score": 0.6181057666050869
|
||||
},
|
||||
{
|
||||
"query": "q063",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip04",
|
||||
"score": 0.6229413326862353
|
||||
},
|
||||
{
|
||||
"query": "q064",
|
||||
"category": "no-answer-nearest-neighbor-return",
|
||||
"winner": "clip03",
|
||||
"score": 0.5715326739972073
|
||||
},
|
||||
{
|
||||
"query": "q065",
|
||||
"category": "unsupported-query-capability",
|
||||
"detail": "Exact metadata requires production lexical lane; direct audio embeddings cannot verify artist/title identities"
|
||||
}
|
||||
],
|
||||
"source_scope": "Direct real-model dense reference baseline; candidate filters here are benchmark reference constraints, not production-filter certification; no rejection threshold is tuned",
|
||||
"interpretation": "Small independently labelled public-media diagnostic; source_scope identifies the measured lane. No-answer return rate measures unsuppressed output, not a calibrated confidence policy. Audio interval overlap can pass while the returned start timestamp is too early; point hits expose this. This corpus alone cannot certify production quality."
|
||||
}
|
||||
Reference in new issue
Block a user