Validate shared-engine lexical performance harness on Windows
This commit is contained in:
karl committed
1 parent
fb8b45dcc9
commit
4e277a107f
6 files changed
+1534
No files matched your search
@@ -0,0 +1,121 @@
|
||||
# Large-index lexical performance protocol
|
||||
|
||||
Status: **1,000-record smoke executed; large run deferred**. No production
|
||||
release gate is passed by this document.
|
||||
|
||||
`tools/scale-eval` uses the actual shared Rust `SearchIndex` engine, with no
|
||||
desktop, model worker or semantic inference. The intended scale is **1 million
|
||||
file records and 100,000 text documents**, with 100 measured requests per case
|
||||
and mode. A 1,000-record smoke run must pass first.
|
||||
|
||||
## What the fixture represents
|
||||
|
||||
The engine opens and migrates the database before bounded 1,000-row transactions
|
||||
insert synthetic metadata and one short passage per text document. Existing
|
||||
engine triggers populate filename trigram, full-text and passage indexes; all
|
||||
four FTS tables receive integrity checks, followed by SQLite
|
||||
integrity checking. No FTS schema or engine retrieval API is replaced.
|
||||
The intentionally sparse `content_search` receives internal integrity checking;
|
||||
the other three additionally compare external content. Requiring every empty
|
||||
file row in the sparse index would falsely report corruption.
|
||||
|
||||
Only two empty directories are created so production folder canonicalization
|
||||
and component-boundary filtering operate normally. The neighboring directories
|
||||
`chosen` and `chosen-other` test accidental prefix inclusion. No million-file
|
||||
filesystem corpus is created. Text/code records alternate extensions; a filtered
|
||||
common-term query combines folder, extension, code, size and modification time.
|
||||
Every request checks positive/absent outcomes; exact filenames must retain first
|
||||
place and unique content markers must identify their corresponding document.
|
||||
|
||||
This bypasses discovery, permissions, extraction, hashing source input, watcher
|
||||
traffic and real source access. Synthetic records have no verified extraction
|
||||
provenance and must never be used for semantic indexing or agent excerpts.
|
||||
Short repeated text and similarly shaped names are deliberately reproducible
|
||||
workloads, not a representative customer library. The database SHA identifies
|
||||
the actual run; absolute temporary-root bytes make it machine/run dependent.
|
||||
The fixed seed and record-generation algorithm identify the logical dataset.
|
||||
|
||||
## Run and preserve evidence
|
||||
|
||||
From the repository root, once no other performance/inference task owns CPU:
|
||||
|
||||
```powershell
|
||||
$env:SEARCHI_FORCE_CPU = '1'
|
||||
cargo build --release --manifest-path tools/scale-eval/Cargo.toml --locked -j 1
|
||||
tools/scale-eval/target/release/searchi-scale-eval.exe 1000 100 20 .searchi/qa/scale-smoke.json
|
||||
tools/scale-eval/target/release/searchi-scale-eval.exe 1000000 100000 100 .searchi/qa/scale-million.json
|
||||
python tools/scale-eval/summarize.py .searchi/qa/scale-million.json
|
||||
```
|
||||
|
||||
Linux/macOS use the same binary path without `.exe`. Build once, then retain
|
||||
the committed diagnostic `Cargo.lock` and its SHA alongside evidence; this
|
||||
standalone diagnostic intentionally has its own dependency lock and is outside
|
||||
the product workspace. Reports use `create_new`, refusing to overwrite earlier samples.
|
||||
The owned temporary database is removed automatically; callers' directories
|
||||
are never recursively deleted. A forcibly killed process may leave its owned
|
||||
`searchi-scale-*` directory, which should be inspected before explicit cleanup.
|
||||
|
||||
Record source commit, clean/dirty status, compiler (`rustc -Vv`), dependency lock
|
||||
SHA, CPU model, VM/physical status, RAM, storage, OS version, power mode and any
|
||||
competing jobs. Report embeds executable SHA, OS/architecture/logical CPUs,
|
||||
schema version, database SHA/size and seed time. SQLite queries run synchronously
|
||||
on one caller thread; logical CPU count does not describe a bounded whole-app
|
||||
resource budget. Monitor peak memory/storage externally during the large run.
|
||||
|
||||
## Distinct measurements
|
||||
|
||||
- Reader open, first unprepared input and background `prewarm` are separate.
|
||||
- Per-case first input after production prewarm precedes that case's repeated
|
||||
measurements. Prewarm itself searches representative names/content; this is
|
||||
not a wholly unseen cold query.
|
||||
- Warm exact/unique requests vary deterministic IDs; common name/keyword,
|
||||
strict filters and absent queries repeat. Basic/Debug are paired AB/BA to
|
||||
reduce order bias. Each mode retains every wall-clock sample and actual engine
|
||||
metrics. Debug retains full bounded traces; their nested durations are not
|
||||
summed. Result assertions run after measured retrieval and outside wall time.
|
||||
- Reopened-reader open and first keyword request are separate after warm tests.
|
||||
These **do not flush OS/SQLite storage caches** and are not cold-storage tests.
|
||||
- `summarize.py` uses nearest-rank p50/p95 and retains negative paired overhead
|
||||
caused by measurement noise. It never emits a production pass verdict.
|
||||
- The recorded exact-lookup SQL plan is a narrow schema diagnostic, not evidence
|
||||
that every dynamic production retrieval lane uses that plan. Inspect actual
|
||||
debug lane timings/candidate counters when investigating regressions.
|
||||
|
||||
UI input/paint, semantic inference/ANN, concurrent indexing, long documents,
|
||||
Unicode-heavy names, skewed customer content, network/removable storage,
|
||||
filesystem lifecycle and cold-machine tests remain separate release gates.
|
||||
The original 50 ms filename and 150 ms keyword p95 goals may be compared with
|
||||
these samples, but reaching them here proves only this synthetic lexical case.
|
||||
|
||||
## Windows VM smoke, 2026-10-13
|
||||
|
||||
Actual optimized shared-engine run: 1,000 metadata records, 100 short text
|
||||
documents, 20 paired samples per mode per case. All expected identities,
|
||||
combined filters and absent outcomes passed. No inference or GPU was used;
|
||||
the installed desktop remained running, so these are not isolated performance
|
||||
measurements. Rust 1.99.0, x86_64-pc-windows-msvc.
|
||||
|
||||
| Case | Basic warm p95 (ms) | Debug warm p95 (ms) |
|
||||
| --- | --- | --- |
|
||||
| Exact filename | 0.8893 | 1.3266 |
|
||||
| Unique keyword | 1.3823 | 2.2248 |
|
||||
| Common keyword | 1.3285 | 1.4759 |
|
||||
| Common filename | 0.6131 | 0.6961 |
|
||||
| Combined filters | 0.9019 | 1.1783 |
|
||||
| Absent keyword | 0.4442 | 0.6847 |
|
||||
|
||||
Reader open: 4.6814 ms; prewarm: 62.4835 ms; reopened reader: 3.1992 ms.
|
||||
Raw ignored evidence `.searchi/qa/scale-smoke-20261013.json`, SHA-256
|
||||
`79f58066f8e42ee3c249eeb341a8ce934b619e3c308038081845c8d24a4543de`.
|
||||
Executable SHA-256
|
||||
`75fd6d4053b0ffe636b298f3c514222a9a5adf9a316d211ebabb8ccb58c2a89a`;
|
||||
diagnostic lock SHA-256
|
||||
`4b426c7b38b765f1ae7e6947dcb52e21dc26a02ed4d50ba51354819a918a83c8`.
|
||||
|
||||
The initial diagnostic failures exposed an invalid external-content check for
|
||||
the sparse index and milliseconds supplied to nanosecond storage. Both were
|
||||
corrected in the harness; production engine behavior was unchanged. Statistics
|
||||
tests passed in normal and optimized Python modes; formatting and strict
|
||||
release all-target Clippy passed. A subsequent Clippy-only parity-expression
|
||||
cleanup did not change the recorded smoke executable. The million-record run,
|
||||
representative libraries and UI/semantic timings remain open.
|
||||
Reference in new issue
Block a user