66 lines
4.0 KiB
Markdown
66 lines
4.0 KiB
Markdown
|
# Semantic result evidence
|
||
|
|
|
||
|
|
Google EmbeddingGemma 2 retrieves the closest eligible sources. A closest source
|
||
|
|
can still be unrelated when the requested content is missing. Similarity and
|
||
|
|
reciprocal-rank-fusion scores are not probabilities or proof of presence.
|
||
|
|
|
||
|
|
The shared hybrid engine now distinguishes primary results from related
|
||
|
|
suggestions using the [frozen presentation policy](../tools/relevance-eval/policies/evidence-v1.json):
|
||
|
|
semantic-only primary candidates require raw cosine **at least 0.70** and membership
|
||
|
|
in the existing leading semantic cluster. Weaker candidates and the separated
|
||
|
|
tail remain available with a `_related` match-kind suffix. This is a presentation
|
||
|
|
boundary, not hard deletion or a claim that no matching artifact exists.
|
||
|
|
|
||
|
|
Independent filename and keyword results preserve their existing priority and
|
||
|
|
match kind, even when the same path also occurs among related semantic neighbors.
|
||
|
|
Primary results precede related suggestions within the caller's total limit.
|
||
|
|
Strict filters and source validation still run before classification. Raw
|
||
|
|
`search_semantic` diagnostics are unchanged.
|
||
|
|
|
||
|
|
Clients use `searchi_service::is_related_result` to identify suggestions rather
|
||
|
|
than interpreting fused scores as confidence. Debug tracing records
|
||
|
|
`hybrid.semantic_shortlist` with `main_candidates`, `related_candidates`,
|
||
|
|
`candidates_before` and `candidates_after`; no eligible source is silently
|
||
|
|
discarded by this classification stage.
|
||
|
|
|
||
|
|
The policy was frozen before new blind-query inference. Existing development
|
||
|
|
evidence predicts useful precision/visibility tradeoffs but is not independent
|
||
|
|
validation: 31/32 text/code positive tops meet the display floor, with 0/16 absent
|
||
|
|
tops; repeated Code-scope rows are 8/8 and 0/4. Historical mixed-media top-score
|
||
|
|
promotion is 49/55 positives and 1/6 absent queries, with strongest-file correctness
|
||
|
|
evaluated separately. Temporal near-misses can remain strong similarities.
|
||
|
|
Related suggestions preserve access to weak genuine results. These counts do
|
||
|
|
not justify a universal confidence claim or perfect absence detection.
|
||
|
|
|
||
|
|
## Frozen blind evaluation for 0.1.9
|
||
|
|
|
||
|
|
The completed [absence-v3 summary](../fixtures/absence-v3/evaluation-summary.json)
|
||
|
|
records actual Google CPU inference on 36 covered assets and 36 frozen queries:
|
||
|
|
16 positive, 16 absent and four partial requests. No extraction, semantic or
|
||
|
|
speech failure removed a required source or channel. The policy was not retuned
|
||
|
|
after evaluation.
|
||
|
|
|
||
|
|
- Absent queries showing primary neighbors fell from **16/16 to 4/16** compared
|
||
|
|
with a replay of the previously shipped leading-cluster rule using these same
|
||
|
|
actual vectors. This is not a timed run of the old binary.
|
||
|
|
- All **16/16 positive requests** retained expected sources within five accessible
|
||
|
|
results, including related suggestions. Primary first-result correctness was
|
||
|
|
**10/16**; primary mean recall@5 was **0.625**, with at least one correct primary
|
||
|
|
source for 11/16 requests. Five needed the related group.
|
||
|
|
- All 16 absent requests still had accessible related neighbors. The remaining
|
||
|
|
four primary errors were document/code near-misses, so absence is not solved.
|
||
|
|
|
||
|
|
Publisher-based media labels, one independent AI annotator and the small corpus
|
||
|
|
limit generalization. The report SHA-256 is
|
||
|
|
`3a032695607a5035016602cbd995d9c4ef028891e5310eb585a9356f2404b628`.
|
||
|
|
Initial indexing took 894.9 seconds, query startup 1.017 seconds and ordered warm
|
||
|
|
raw queries a median 94.4 ms on the shared CPU VM. Hybrid queries reused vectors
|
||
|
|
from the preceding raw calls; these are not cold first-input or UI benchmarks.
|
||
|
|
|
||
|
|
The current browser-rendered production UI capture verifies that related items
|
||
|
|
start collapsed, stay out of arrow-key navigation until expanded, remain
|
||
|
|
accessible when expanded, and restore a visible primary selection when collapsed.
|
||
|
|
Its exact source/build hashes are in screenshots/capture-manifest.json. Native
|
||
|
|
WebView interaction and OS integration are separate, unexecuted checks.
|
||
|
|
Windows/Linux desktop and headless/CLI 0.1.9 packages passed their actual CPU
|
||
|
|
startup/search checks. Publication verification is recorded in ROADMAP.md.
|