2026-10-13 07:06:39 +02:00
|
|
|
# Mixed retrieval evaluator
|
|
|
|
|
|
|
|
|
|
`evaluate.py` verifies the frozen labels and every asset, copies only those
|
|
|
|
|
assets into an owned temporary corpus, and indexes it with the actual CLI.
|
|
|
|
|
Semantic indexing uses the real shared service and explicit Google worker/model
|
|
|
|
|
and CPU FFmpeg paths. Querying uses the same public read-only MCP search tool
|
|
|
|
|
that local agents use. Lexical and hybrid modes are reported separately; code
|
|
|
|
|
queries compare All files against Code scope. A semantic fallback is an error,
|
|
|
|
|
not a lexical result mislabeled as model evidence. Model files are not downloaded.
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
|
|
|
python tools/mixed-eval/evaluate.py --verify-only
|
|
|
|
|
python -m unittest discover -s tools/mixed-eval -p 'test_*.py'
|
|
|
|
|
python tools/mixed-eval/evaluate.py --cli target/release/searchi.exe --models .searchi/models/embeddinggemma2 --worker .searchi/runtime/auto/Release/searchi-gemma-worker.exe --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe --output .searchi/qa/mixed-v1.json
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Use actual absolute runtime paths for your platform if these development paths
|
|
|
|
|
differ. Inference runs must be coordinated with other CPU measurements. Expect
|
|
|
|
|
media indexing to take minutes; its explicit one-hour bound is a safety ceiling,
|
|
|
|
|
not a performance target. Indexing and query errors remain visible. Temporary
|
|
|
|
|
corpora/indexes are cleaned after the owned sessions close. Partial query results
|
2026-10-13 07:09:47 +02:00
|
|
|
are retained in the requested output if a later request fails. Indexing stage
|
|
|
|
|
stdout/stderr, JSON reports, failures, timeouts and elapsed time are also retained.
|
|
|
|
|
`complete: true` means evaluation finished, never that release quality passed.
|
|
|
|
|
The query rows retain native basic metrics separately from MCP roundtrip time.
|
2026-10-13 07:06:39 +02:00
|
|
|
|
|
|
|
|
Reported metrics are file-level top1, reciprocal rank and recall@5 after file
|
|
|
|
|
identity deduplication. Hard negatives in the first five and no-answer queries
|
|
|
|
|
returning any result are reported explicitly. This does **not** calibrate a
|
|
|
|
|
rejection threshold. Timestamp labels are retained but scored only for file
|
|
|
|
|
identity: use the separate media evaluator for segment localization. Unsupported
|
|
|
|
|
temporal filters are recorded as skipped, never counted as passed.
|
|
|
|
|
|
|
|
|
|
The raw report retains queries, expected IDs, ordered hits, source locations,
|
|
|
|
|
scores, modes/scopes/languages and MCP roundtrip times. Those times exclude
|
|
|
|
|
process/model initialization and do not represent native UI latency. Keep raw
|
|
|
|
|
reports under ignored `.searchi`; review deliberate evidence before publishing.
|
