122 lines
6.7 KiB
Markdown
122 lines
6.7 KiB
Markdown
# Large-index lexical performance protocol
|
|
|
|
Status: **1,000-record smoke executed; large run deferred**. No production
|
|
release gate is passed by this document.
|
|
|
|
`tools/scale-eval` uses the actual shared Rust `SearchIndex` engine, with no
|
|
desktop, model worker or semantic inference. The intended scale is **1 million
|
|
file records and 100,000 text documents**, with 100 measured requests per case
|
|
and mode. A 1,000-record smoke run must pass first.
|
|
|
|
## What the fixture represents
|
|
|
|
The engine opens and migrates the database before bounded 1,000-row transactions
|
|
insert synthetic metadata and one short passage per text document. Existing
|
|
engine triggers populate filename trigram, full-text and passage indexes; all
|
|
four FTS tables receive integrity checks, followed by SQLite
|
|
integrity checking. No FTS schema or engine retrieval API is replaced.
|
|
The intentionally sparse `content_search` receives internal integrity checking;
|
|
the other three additionally compare external content. Requiring every empty
|
|
file row in the sparse index would falsely report corruption.
|
|
|
|
Only two empty directories are created so production folder canonicalization
|
|
and component-boundary filtering operate normally. The neighboring directories
|
|
`chosen` and `chosen-other` test accidental prefix inclusion. No million-file
|
|
filesystem corpus is created. Text/code records alternate extensions; a filtered
|
|
common-term query combines folder, extension, code, size and modification time.
|
|
Every request checks positive/absent outcomes; exact filenames must retain first
|
|
place and unique content markers must identify their corresponding document.
|
|
|
|
This bypasses discovery, permissions, extraction, hashing source input, watcher
|
|
traffic and real source access. Synthetic records have no verified extraction
|
|
provenance and must never be used for semantic indexing or agent excerpts.
|
|
Short repeated text and similarly shaped names are deliberately reproducible
|
|
workloads, not a representative customer library. The database SHA identifies
|
|
the actual run; absolute temporary-root bytes make it machine/run dependent.
|
|
The fixed seed and record-generation algorithm identify the logical dataset.
|
|
|
|
## Run and preserve evidence
|
|
|
|
From the repository root, once no other performance/inference task owns CPU:
|
|
|
|
```powershell
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
cargo build --release --manifest-path tools/scale-eval/Cargo.toml --locked -j 1
|
|
tools/scale-eval/target/release/searchi-scale-eval.exe 1000 100 20 .searchi/qa/scale-smoke.json
|
|
tools/scale-eval/target/release/searchi-scale-eval.exe 1000000 100000 100 .searchi/qa/scale-million.json
|
|
python tools/scale-eval/summarize.py .searchi/qa/scale-million.json
|
|
```
|
|
|
|
Linux/macOS use the same binary path without `.exe`. Build once, then retain
|
|
the committed diagnostic `Cargo.lock` and its SHA alongside evidence; this
|
|
standalone diagnostic intentionally has its own dependency lock and is outside
|
|
the product workspace. Reports use `create_new`, refusing to overwrite earlier samples.
|
|
The owned temporary database is removed automatically; callers' directories
|
|
are never recursively deleted. A forcibly killed process may leave its owned
|
|
`searchi-scale-*` directory, which should be inspected before explicit cleanup.
|
|
|
|
Record source commit, clean/dirty status, compiler (`rustc -Vv`), dependency lock
|
|
SHA, CPU model, VM/physical status, RAM, storage, OS version, power mode and any
|
|
competing jobs. Report embeds executable SHA, OS/architecture/logical CPUs,
|
|
schema version, database SHA/size and seed time. SQLite queries run synchronously
|
|
on one caller thread; logical CPU count does not describe a bounded whole-app
|
|
resource budget. Monitor peak memory/storage externally during the large run.
|
|
|
|
## Distinct measurements
|
|
|
|
- Reader open, first unprepared input and background `prewarm` are separate.
|
|
- Per-case first input after production prewarm precedes that case's repeated
|
|
measurements. Prewarm itself searches representative names/content; this is
|
|
not a wholly unseen cold query.
|
|
- Warm exact/unique requests vary deterministic IDs; common name/keyword,
|
|
strict filters and absent queries repeat. Basic/Debug are paired AB/BA to
|
|
reduce order bias. Each mode retains every wall-clock sample and actual engine
|
|
metrics. Debug retains full bounded traces; their nested durations are not
|
|
summed. Result assertions run after measured retrieval and outside wall time.
|
|
- Reopened-reader open and first keyword request are separate after warm tests.
|
|
These **do not flush OS/SQLite storage caches** and are not cold-storage tests.
|
|
- `summarize.py` uses nearest-rank p50/p95 and retains negative paired overhead
|
|
caused by measurement noise. It never emits a production pass verdict.
|
|
- The recorded exact-lookup SQL plan is a narrow schema diagnostic, not evidence
|
|
that every dynamic production retrieval lane uses that plan. Inspect actual
|
|
debug lane timings/candidate counters when investigating regressions.
|
|
|
|
UI input/paint, semantic inference/ANN, concurrent indexing, long documents,
|
|
Unicode-heavy names, skewed customer content, network/removable storage,
|
|
filesystem lifecycle and cold-machine tests remain separate release gates.
|
|
The original 50 ms filename and 150 ms keyword p95 goals may be compared with
|
|
these samples, but reaching them here proves only this synthetic lexical case.
|
|
|
|
## Windows VM smoke, 2026-10-13
|
|
|
|
Actual optimized shared-engine run: 1,000 metadata records, 100 short text
|
|
documents, 20 paired samples per mode per case. All expected identities,
|
|
combined filters and absent outcomes passed. No inference or GPU was used;
|
|
the installed desktop remained running, so these are not isolated performance
|
|
measurements. Rust 1.99.0, x86_64-pc-windows-msvc.
|
|
|
|
| Case | Basic warm p95 (ms) | Debug warm p95 (ms) |
|
|
| --- | --- | --- |
|
|
| Exact filename | 0.8893 | 1.3266 |
|
|
| Unique keyword | 1.3823 | 2.2248 |
|
|
| Common keyword | 1.3285 | 1.4759 |
|
|
| Common filename | 0.6131 | 0.6961 |
|
|
| Combined filters | 0.9019 | 1.1783 |
|
|
| Absent keyword | 0.4442 | 0.6847 |
|
|
|
|
Reader open: 4.6814 ms; prewarm: 62.4835 ms; reopened reader: 3.1992 ms.
|
|
Raw ignored evidence `.searchi/qa/scale-smoke-20261013.json`, SHA-256
|
|
`79f58066f8e42ee3c249eeb341a8ce934b619e3c308038081845c8d24a4543de`.
|
|
Executable SHA-256
|
|
`75fd6d4053b0ffe636b298f3c514222a9a5adf9a316d211ebabb8ccb58c2a89a`;
|
|
diagnostic lock SHA-256
|
|
`4b426c7b38b765f1ae7e6947dcb52e21dc26a02ed4d50ba51354819a918a83c8`.
|
|
|
|
The initial diagnostic failures exposed an invalid external-content check for
|
|
the sparse index and milliseconds supplied to nanosecond storage. Both were
|
|
corrected in the harness; production engine behavior was unchanged. Statistics
|
|
tests passed in normal and optimized Python modes; formatting and strict
|
|
release all-target Clippy passed. A subsequent Clippy-only parity-expression
|
|
cleanup did not change the recorded smoke executable. The million-record run,
|
|
representative libraries and UI/semantic timings remain open.
|