Keep real model benchmark validity checks active under optimization

This commit is contained in:
karl committed 2026-10-13 13:54:33 +02:00
1 parent d568bdb2b1
commit 66bf7d7b47
4 files changed
+79 -27

No files matched your search

+13
View File
@@ -122,3 +122,16 @@ The query-role memory observation after first inference was 239,878,144 bytes
private commitment and 274,141,184 bytes resident working set for repeat zero;
these are different metrics and must not be summed. Native desktop first-input
and complete search latency remain separate open checks.
Benchmark contract hardening then replaced Python assertions with explicit
errors in both shared benchmark and query-context tooling. Model hashes,
CPU/device state, role/context, normalized vectors, source-window preservation,
MCP errors/filter behavior and batch parity remain checked under `python -O`.
Both tools refuse existing output reports. Four targeted invalid-vector/device
tests passed in normal and optimized modes. A genuine one-pair Google CPU
query/indexing run also passed under optimization, retaining twenty warmed
requests per role; this is a contract smoke rather than an additional latency
acceptance sample. Ignored report SHA-256:
`e611fd3c705a94a6c06ab6ae6a934e6f604665aacb17d0e86d6d5b162ac59e46`
(`.searchi/qa/query-context-optimized-contract.json`). Earlier recorded timing
reports were run without optimization and are unaffected by this correction.