155 lines
8.4 KiB
Markdown
155 lines
8.4 KiB
Markdown
# Large-index lexical performance protocol
|
|
|
|
Status: **1,000-record smoke and 100,000-record intermediate run executed;
|
|
million-record run deferred**. No production
|
|
release gate is passed by this document.
|
|
|
|
`tools/scale-eval` uses the actual shared Rust `SearchIndex` engine, with no
|
|
desktop, model worker or semantic inference. The intended scale is **1 million
|
|
file records and 100,000 text documents**, with 100 measured requests per case
|
|
and mode. A 1,000-record smoke run must pass first.
|
|
|
|
## What the fixture represents
|
|
|
|
The engine opens and migrates the database before bounded 1,000-row transactions
|
|
insert synthetic metadata and one short passage per text document. Existing
|
|
engine triggers populate filename trigram, full-text and passage indexes; all
|
|
four FTS tables receive integrity checks, followed by SQLite
|
|
integrity checking. No FTS schema or engine retrieval API is replaced.
|
|
The intentionally sparse `content_search` receives internal integrity checking;
|
|
the other three additionally compare external content. Requiring every empty
|
|
file row in the sparse index would falsely report corruption.
|
|
|
|
Only two empty directories are created so production folder canonicalization
|
|
and component-boundary filtering operate normally. The neighboring directories
|
|
`chosen` and `chosen-other` test accidental prefix inclusion. No million-file
|
|
filesystem corpus is created. Text/code records alternate extensions; a filtered
|
|
common-term query combines folder, extension, code, size and modification time.
|
|
Every request checks positive/absent outcomes; exact filenames must retain first
|
|
place and unique content markers must identify their corresponding document.
|
|
|
|
This bypasses discovery, permissions, extraction, hashing source input, watcher
|
|
traffic and real source access. Synthetic records have no verified extraction
|
|
provenance and must never be used for semantic indexing or agent excerpts.
|
|
Short repeated text and similarly shaped names are deliberately reproducible
|
|
workloads, not a representative customer library. The database SHA identifies
|
|
the actual run; absolute temporary-root bytes make it machine/run dependent.
|
|
The fixed seed and record-generation algorithm identify the logical dataset.
|
|
|
|
## Run and preserve evidence
|
|
|
|
From the repository root, once no other performance/inference task owns CPU:
|
|
|
|
```powershell
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
cargo build --release --manifest-path tools/scale-eval/Cargo.toml --locked -j 1
|
|
tools/scale-eval/target/release/searchi-scale-eval.exe 1000 100 20 .searchi/qa/scale-smoke.json
|
|
tools/scale-eval/target/release/searchi-scale-eval.exe 1000000 100000 100 .searchi/qa/scale-million.json
|
|
python tools/scale-eval/summarize.py .searchi/qa/scale-million.json
|
|
```
|
|
|
|
Linux/macOS use the same binary path without `.exe`. Build once, then retain
|
|
the committed diagnostic `Cargo.lock` and its SHA alongside evidence; this
|
|
standalone diagnostic intentionally has its own dependency lock and is outside
|
|
the product workspace. Reports use `create_new`, refusing to overwrite earlier samples.
|
|
The owned temporary database is removed automatically; callers' directories
|
|
are never recursively deleted. A forcibly killed process may leave its owned
|
|
`searchi-scale-*` directory, which should be inspected before explicit cleanup.
|
|
|
|
Record source commit, clean/dirty status, compiler (`rustc -Vv`), dependency lock
|
|
SHA, CPU model, VM/physical status, RAM, storage, OS version, power mode and any
|
|
competing jobs. Report embeds executable SHA, OS/architecture/logical CPUs,
|
|
schema version, database SHA/size and seed time. SQLite queries run synchronously
|
|
on one caller thread; logical CPU count does not describe a bounded whole-app
|
|
resource budget. Monitor peak memory/storage externally during the large run.
|
|
|
|
## Distinct measurements
|
|
|
|
- Reader open, first unprepared input and background `prewarm` are separate.
|
|
- Per-case first input after production prewarm precedes that case's repeated
|
|
measurements. Prewarm itself searches representative names/content; this is
|
|
not a wholly unseen cold query.
|
|
- Warm exact/unique requests vary deterministic IDs; common name/keyword,
|
|
strict filters and absent queries repeat. Basic/Debug are paired AB/BA to
|
|
reduce order bias. Each mode retains every wall-clock sample and actual engine
|
|
metrics. Debug retains full bounded traces; their nested durations are not
|
|
summed. Result assertions run after measured retrieval and outside wall time.
|
|
- Reopened-reader open and first keyword request are separate after warm tests.
|
|
These **do not flush OS/SQLite storage caches** and are not cold-storage tests.
|
|
- `summarize.py` uses nearest-rank p50/p95 and retains negative paired overhead
|
|
caused by measurement noise. It never emits a production pass verdict.
|
|
- The recorded exact-lookup SQL plan is a narrow schema diagnostic, not evidence
|
|
that every dynamic production retrieval lane uses that plan. Inspect actual
|
|
debug lane timings/candidate counters when investigating regressions.
|
|
|
|
UI input/paint, semantic inference/ANN, concurrent indexing, long documents,
|
|
Unicode-heavy names, skewed customer content, network/removable storage,
|
|
filesystem lifecycle and cold-machine tests remain separate release gates.
|
|
The original 50 ms filename and 150 ms keyword p95 goals may be compared with
|
|
these samples, but reaching them here proves only this synthetic lexical case.
|
|
|
|
## Windows VM smoke, 2026-10-13
|
|
|
|
Actual optimized shared-engine run: 1,000 metadata records, 100 short text
|
|
documents, 20 paired samples per mode per case. All expected identities,
|
|
combined filters and absent outcomes passed. No inference or GPU was used;
|
|
the installed desktop remained running, so these are not isolated performance
|
|
measurements. Rust 1.99.0, x86_64-pc-windows-msvc.
|
|
|
|
| Case | Basic warm p95 (ms) | Debug warm p95 (ms) |
|
|
| --- | --- | --- |
|
|
| Exact filename | 0.8893 | 1.3266 |
|
|
| Unique keyword | 1.3823 | 2.2248 |
|
|
| Common keyword | 1.3285 | 1.4759 |
|
|
| Common filename | 0.6131 | 0.6961 |
|
|
| Combined filters | 0.9019 | 1.1783 |
|
|
| Absent keyword | 0.4442 | 0.6847 |
|
|
|
|
Reader open: 4.6814 ms; prewarm: 62.4835 ms; reopened reader: 3.1992 ms.
|
|
Raw ignored evidence `.searchi/qa/scale-smoke-20261013.json`, SHA-256
|
|
`79f58066f8e42ee3c249eeb341a8ce934b619e3c308038081845c8d24a4543de`.
|
|
Executable SHA-256
|
|
`75fd6d4053b0ffe636b298f3c514222a9a5adf9a316d211ebabb8ccb58c2a89a`;
|
|
diagnostic lock SHA-256
|
|
`4b426c7b38b765f1ae7e6947dcb52e21dc26a02ed4d50ba51354819a918a83c8`.
|
|
|
|
The initial diagnostic failures exposed an invalid external-content check for
|
|
the sparse index and milliseconds supplied to nanosecond storage. Both were
|
|
corrected in the harness; production engine behavior was unchanged. Statistics
|
|
tests passed in normal and optimized Python modes; formatting and strict
|
|
release all-target Clippy passed. A subsequent Clippy-only parity-expression
|
|
cleanup did not change the recorded smoke executable. The million-record run,
|
|
representative libraries and UI/semantic timings remain open.
|
|
|
|
## Windows VM intermediate scale, 2026-10-13
|
|
|
|
Actual release diagnostic at source `59908ad`, clean worktree, used 100,000
|
|
file records, 10,000 short text documents and 30 requests per case per mode.
|
|
All assertions and integrity checks passed. Report:
|
|
`.searchi/qa/scale-100k.json`, SHA-256
|
|
`25a300f4912d33c07963eb712a9214879c326e8c1a16eeca130855798c561b71`.
|
|
Executable SHA-256:
|
|
`1c3a8774195ff13e2ffcbb4586446d1de68e2ac927452fd8c004ca4e7d09fd82`.
|
|
Dependency lock is unchanged from the smoke. Rust 1.99.0, LLVM 23.1.1,
|
|
Windows x64 MSVC; Q35 VM exposes Ryzen 7 7800X3D and eight logical CPUs,
|
|
17,109,004,288 bytes RAM. Storage/power configuration was not independently
|
|
measured. The installed desktop and its query worker remained running; no
|
|
benchmark-owned inference was launched. Background activity was not controlled.
|
|
|
|
| Case | Basic warm p95 ms | Debug warm p95 ms |
|
|
| --- | ---: | ---: |
|
|
| Exact name | 1.2085 | 1.4841 |
|
|
| Unique keyword | 0.7019 | 1.1082 |
|
|
| Common keyword | 9.9522 | 9.7389 |
|
|
| Common filename | 0.6121 | 0.7390 |
|
|
| Strict combined filters | 6.9529 | 7.2662 |
|
|
| Absent query | 0.5178 | 0.8753 |
|
|
|
|
Seed insertion took 28,224.5 ms and produced a 110,481,408-byte database.
|
|
Reader opening took 3.83 ms; first unprepared input 78.02 ms; production prewarm
|
|
283.56 ms. Reopening took 3.99 ms without flushing storage caches. A live
|
|
process observation reported 15,540,224-byte peak working set at that moment;
|
|
it is not a complete lifetime peak-memory measurement. This intermediate
|
|
synthetic result does not close the million-record, real-library, concurrent
|
|
indexing, semantic or UI gates and does not alter product ranking.
|