2.5 KiB
Mixed retrieval evaluator
evaluate.py verifies the frozen labels and every asset, copies only those
assets into an owned temporary corpus, and indexes it with the actual CLI.
Semantic indexing uses the real shared service and explicit Google worker/model
and CPU FFmpeg paths. Querying uses the same public read-only MCP search tool
that local agents use. Lexical and hybrid modes are reported separately; code
queries compare All files against Code scope. A semantic fallback is an error,
not a lexical result mislabeled as model evidence. Model files are not downloaded.
$env:SEARCHI_FORCE_CPU = '1'
python tools/mixed-eval/evaluate.py --verify-only
python -m unittest discover -s tools/mixed-eval -p 'test_*.py'
python tools/mixed-eval/evaluate.py --cli target/release/searchi.exe --models .searchi/models/embeddinggemma2 --worker .searchi/runtime/auto/Release/searchi-gemma-worker.exe --ffmpeg .searchi/media-runtime/bin/ffmpeg.exe --ffprobe .searchi/media-runtime/bin/ffprobe.exe --output .searchi/qa/mixed-v1.json
Use actual absolute runtime paths for your platform if these development paths
differ. Inference runs must be coordinated with other CPU measurements. Expect
media indexing to take minutes; its explicit one-hour bound is a safety ceiling,
not a performance target. Indexing and query errors remain visible. Temporary
corpora/indexes are cleaned after the owned sessions close. Partial query results
are retained in the requested output if a later request fails. Indexing stage
stdout/stderr, JSON reports, failures, timeouts and elapsed time are also retained.
complete: true means evaluation finished, never that release quality passed.
The query rows retain native basic metrics separately from MCP roundtrip time.
Reported metrics are file-level top1, reciprocal rank and recall@5 after file identity deduplication. Hard negatives in the first five and no-answer queries returning any result are reported explicitly. This does not calibrate a rejection threshold. Timestamp labels are retained but scored only for file identity: use the separate media evaluator for segment localization. Unsupported temporal filters are recorded as skipped, never counted as passed.
The raw report retains queries, expected IDs, ordered hits, source locations,
scores, modes/scopes/languages and MCP roundtrip times. Those times exclude
process/model initialization and do not represent native UI latency. Keep raw
reports under ignored .searchi; review deliberate evidence before publishing.