Prepare bounded speech diagnostics and document mixed routing risks

This commit is contained in:
karl committed 2026-10-13 08:26:01 +02:00
1 parent 5d9cfc4ea0
commit af7a40c183
7 files changed
+428

No files matched your search

+57
View File
@@ -0,0 +1,57 @@
# Mixed query routing audit
Read-only implementation audit, 2026-10-13, source revision `5d9cfc4`.
These findings describe current behavior; they are not a retrieval-quality
measurement or an implemented ranking change.
## Current behavior
`HybridSearch::query_vector` selects Google's CodeRetrieval task only when
`SearchOptions::code_only` is true. All-files searches use the general task,
including natural-language queries asking for an implementation. The bounded
query-vector LRU distinguishes tasks, so vectors cannot be reused across them
accidentally. See `crates/searchi-service/src/lib.rs`.
The audio preference in `crates/searchi-service/src/routing.rs` recognizes words
such as speech, sound and music. Document and image cues veto that preference;
code cues currently do not. If the preferred audio search returns any qualifying
candidate, it becomes the exclusive semantic lane. Consequently a query such as
"code that detects speech" can exclude semantic code candidates in All files.
Lexical candidates still participate in hybrid fusion, and explicit Code scope
bypasses this audio preference. This is a risk to evaluate, not proof that every
such query fails.
## Cost of adding lanes
`crates/searchi-service/src/store.rs` retains one eligibility-cache entry.
Its key includes the generation, user filters, source type and code scope.
Changing a filter replaces the entry and scans/hashes matching passage metadata
again. Alternating general and code lanes would therefore perform repeated
eligibility work even when query vectors and the ANN graph are warm. The existing
empty-audio fallback can also alternate audio and general filters.
Incremental dirty-path updates apply when the cached filter matches and the
revision is still within the dirty-history floor. A replacement cache design
must preserve those freshness checks, generation invalidation, strict filtering
before candidate limits, and the total allocation budget.
## Next experiment
First complete the unchanged frozen mixed baseline described in
[MIXED-QUALITY.md](MIXED-QUALITY.md). Keep its labels and source bytes unchanged.
Then compare soft audio preference and task-specific rank fusion against that
baseline, retaining exact filename/symbol priority. Different task scores must
not be treated as calibrated probabilities or compared blindly.
Before introducing multiple lanes, measure a bounded multi-entry eligibility
cache against a shared validated snapshot with derived code/origin views.
Bound memory across all entries, rather than giving each the full chunk budget.
Reuse the existing worker and prepared ANN graph.
Report All-files and Code scope separately: top-1, recall@5, hard negatives and
no-answer behavior, plus query-vector cache misses, inference time, eligibility
scans/hash calls, memory and warm p95. Include mutation/deletion regressions.
The existing `semantic.eligibility` and
`semantic.eligibility_filtered_scan_and_hash` traces expose eligibility cost.
Use an independently reviewed held-out corpus for acceptance; the frozen
diagnostic alone must not become the optimization target.
+8
View File
@@ -56,6 +56,10 @@ Status: implementation started; no production release yet. Windows VM tests use
and indexing-cost measurements (see MEDIA-QUALITY.md).
- [ ] Improve precise spoken-passage localization, music hard negatives and
calibrated handling of queries without a relevant match.
- [x] Prepare bounded CPU speech diagnostics and test process failures,
timestamp units and frozen-label scoring; see [speech proposal](SPEECH-ALIGNMENT.md).
- [ ] Run genuine local speech candidates, Google transcript retrieval and
held-out English/German/music/no-speech checks before selecting a default.
- [ ] Package and verify decoders on every release OS; test production acceleration.
- [x] Native Linux x86-64 pinned decoder provenance, startup, WAV probe and CPU
decode; broader codec/ABI coverage and desktop packaging remain open.
@@ -208,6 +212,10 @@ Completion requires recorded evidence; targets are not measured results.
ordinary-document exclusion and exact MCP source excerpt verification.
- [ ] Evaluate independent real repositories, more languages, overloaded
symbols, near-duplicate routines and queries with no relevant implementation.
- [x] Audit mixed-task routing and eligibility cache behavior; see
[routing audit](RETRIEVAL-ROUTING.md).
- [ ] Complete the frozen mixed baseline, then measure task fusion and cache
cost without tuning solely against the diagnostic labels.
- [ ] Validate code search UI interaction on native Windows/macOS/Linux and
actual GPU hardware. Tiny CPU fixtures do not close those release gates.
+5
View File
@@ -4,6 +4,11 @@ Status: researched implementation proposal, 2026-10-13. No transcription worker,
download integration, or ASR quality result is implemented by this document.
Google EmbeddingGemma 2 remains the production semantic model.
A bounded [CPU diagnostic harness](../tools/speech-eval/README.md) is now prepared.
Seven structural tests passed on Windows, including actual owned subprocess
failure, timeout, output overflow and retained diagnostics. No ASR model has been
downloaded or run for this proof; these tests establish no speech quality.
## Why add a transcript lane
[The frozen media evaluation](MEDIA-QUALITY.md) found the correct spoken-topic