Prepare bounded speech diagnostics and document mixed routing risks
This commit is contained in:
1 parent
5d9cfc4ea0
commit
af7a40c183
7 files changed
+428
No files matched your search
@@ -0,0 +1,57 @@
|
||||
# Mixed query routing audit
|
||||
|
||||
Read-only implementation audit, 2026-10-13, source revision `5d9cfc4`.
|
||||
These findings describe current behavior; they are not a retrieval-quality
|
||||
measurement or an implemented ranking change.
|
||||
|
||||
## Current behavior
|
||||
|
||||
`HybridSearch::query_vector` selects Google's CodeRetrieval task only when
|
||||
`SearchOptions::code_only` is true. All-files searches use the general task,
|
||||
including natural-language queries asking for an implementation. The bounded
|
||||
query-vector LRU distinguishes tasks, so vectors cannot be reused across them
|
||||
accidentally. See `crates/searchi-service/src/lib.rs`.
|
||||
|
||||
The audio preference in `crates/searchi-service/src/routing.rs` recognizes words
|
||||
such as speech, sound and music. Document and image cues veto that preference;
|
||||
code cues currently do not. If the preferred audio search returns any qualifying
|
||||
candidate, it becomes the exclusive semantic lane. Consequently a query such as
|
||||
"code that detects speech" can exclude semantic code candidates in All files.
|
||||
Lexical candidates still participate in hybrid fusion, and explicit Code scope
|
||||
bypasses this audio preference. This is a risk to evaluate, not proof that every
|
||||
such query fails.
|
||||
|
||||
## Cost of adding lanes
|
||||
|
||||
`crates/searchi-service/src/store.rs` retains one eligibility-cache entry.
|
||||
Its key includes the generation, user filters, source type and code scope.
|
||||
Changing a filter replaces the entry and scans/hashes matching passage metadata
|
||||
again. Alternating general and code lanes would therefore perform repeated
|
||||
eligibility work even when query vectors and the ANN graph are warm. The existing
|
||||
empty-audio fallback can also alternate audio and general filters.
|
||||
|
||||
Incremental dirty-path updates apply when the cached filter matches and the
|
||||
revision is still within the dirty-history floor. A replacement cache design
|
||||
must preserve those freshness checks, generation invalidation, strict filtering
|
||||
before candidate limits, and the total allocation budget.
|
||||
|
||||
## Next experiment
|
||||
|
||||
First complete the unchanged frozen mixed baseline described in
|
||||
[MIXED-QUALITY.md](MIXED-QUALITY.md). Keep its labels and source bytes unchanged.
|
||||
Then compare soft audio preference and task-specific rank fusion against that
|
||||
baseline, retaining exact filename/symbol priority. Different task scores must
|
||||
not be treated as calibrated probabilities or compared blindly.
|
||||
|
||||
Before introducing multiple lanes, measure a bounded multi-entry eligibility
|
||||
cache against a shared validated snapshot with derived code/origin views.
|
||||
Bound memory across all entries, rather than giving each the full chunk budget.
|
||||
Reuse the existing worker and prepared ANN graph.
|
||||
|
||||
Report All-files and Code scope separately: top-1, recall@5, hard negatives and
|
||||
no-answer behavior, plus query-vector cache misses, inference time, eligibility
|
||||
scans/hash calls, memory and warm p95. Include mutation/deletion regressions.
|
||||
The existing `semantic.eligibility` and
|
||||
`semantic.eligibility_filtered_scan_and_hash` traces expose eligibility cost.
|
||||
Use an independently reviewed held-out corpus for acceptance; the frozen
|
||||
diagnostic alone must not become the optimization target.
|
||||
@@ -56,6 +56,10 @@ Status: implementation started; no production release yet. Windows VM tests use
|
||||
and indexing-cost measurements (see MEDIA-QUALITY.md).
|
||||
- [ ] Improve precise spoken-passage localization, music hard negatives and
|
||||
calibrated handling of queries without a relevant match.
|
||||
- [x] Prepare bounded CPU speech diagnostics and test process failures,
|
||||
timestamp units and frozen-label scoring; see [speech proposal](SPEECH-ALIGNMENT.md).
|
||||
- [ ] Run genuine local speech candidates, Google transcript retrieval and
|
||||
held-out English/German/music/no-speech checks before selecting a default.
|
||||
- [ ] Package and verify decoders on every release OS; test production acceleration.
|
||||
- [x] Native Linux x86-64 pinned decoder provenance, startup, WAV probe and CPU
|
||||
decode; broader codec/ABI coverage and desktop packaging remain open.
|
||||
@@ -208,6 +212,10 @@ Completion requires recorded evidence; targets are not measured results.
|
||||
ordinary-document exclusion and exact MCP source excerpt verification.
|
||||
- [ ] Evaluate independent real repositories, more languages, overloaded
|
||||
symbols, near-duplicate routines and queries with no relevant implementation.
|
||||
- [x] Audit mixed-task routing and eligibility cache behavior; see
|
||||
[routing audit](RETRIEVAL-ROUTING.md).
|
||||
- [ ] Complete the frozen mixed baseline, then measure task fusion and cache
|
||||
cost without tuning solely against the diagnostic labels.
|
||||
- [ ] Validate code search UI interaction on native Windows/macOS/Linux and
|
||||
actual GPU hardware. Tiny CPU fixtures do not close those release gates.
|
||||
|
||||
|
||||
@@ -4,6 +4,11 @@ Status: researched implementation proposal, 2026-10-13. No transcription worker,
|
||||
download integration, or ASR quality result is implemented by this document.
|
||||
Google EmbeddingGemma 2 remains the production semantic model.
|
||||
|
||||
A bounded [CPU diagnostic harness](../tools/speech-eval/README.md) is now prepared.
|
||||
Seven structural tests passed on Windows, including actual owned subprocess
|
||||
failure, timeout, output overflow and retained diagnostics. No ASR model has been
|
||||
downloaded or run for this proof; these tests establish no speech quality.
|
||||
|
||||
## Why add a transcript lane
|
||||
|
||||
[The frozen media evaluation](MEDIA-QUALITY.md) found the correct spoken-topic
|
||||
|
||||
Reference in new issue
Block a user