diff --git a/docs/SCALE-PERFORMANCE.md b/docs/SCALE-PERFORMANCE.md new file mode 100644 index 0000000..c2726f1 --- /dev/null +++ b/docs/SCALE-PERFORMANCE.md @@ -0,0 +1,121 @@ +# Large-index lexical performance protocol + +Status: **1,000-record smoke executed; large run deferred**. No production +release gate is passed by this document. + +`tools/scale-eval` uses the actual shared Rust `SearchIndex` engine, with no +desktop, model worker or semantic inference. The intended scale is **1 million +file records and 100,000 text documents**, with 100 measured requests per case +and mode. A 1,000-record smoke run must pass first. + +## What the fixture represents + +The engine opens and migrates the database before bounded 1,000-row transactions +insert synthetic metadata and one short passage per text document. Existing +engine triggers populate filename trigram, full-text and passage indexes; all +four FTS tables receive integrity checks, followed by SQLite +integrity checking. No FTS schema or engine retrieval API is replaced. +The intentionally sparse `content_search` receives internal integrity checking; +the other three additionally compare external content. Requiring every empty +file row in the sparse index would falsely report corruption. + +Only two empty directories are created so production folder canonicalization +and component-boundary filtering operate normally. The neighboring directories +`chosen` and `chosen-other` test accidental prefix inclusion. No million-file +filesystem corpus is created. Text/code records alternate extensions; a filtered +common-term query combines folder, extension, code, size and modification time. +Every request checks positive/absent outcomes; exact filenames must retain first +place and unique content markers must identify their corresponding document. + +This bypasses discovery, permissions, extraction, hashing source input, watcher +traffic and real source access. Synthetic records have no verified extraction +provenance and must never be used for semantic indexing or agent excerpts. +Short repeated text and similarly shaped names are deliberately reproducible +workloads, not a representative customer library. The database SHA identifies +the actual run; absolute temporary-root bytes make it machine/run dependent. +The fixed seed and record-generation algorithm identify the logical dataset. + +## Run and preserve evidence + +From the repository root, once no other performance/inference task owns CPU: + +```powershell +$env:SEARCHI_FORCE_CPU = '1' +cargo build --release --manifest-path tools/scale-eval/Cargo.toml --locked -j 1 +tools/scale-eval/target/release/searchi-scale-eval.exe 1000 100 20 .searchi/qa/scale-smoke.json +tools/scale-eval/target/release/searchi-scale-eval.exe 1000000 100000 100 .searchi/qa/scale-million.json +python tools/scale-eval/summarize.py .searchi/qa/scale-million.json +``` + +Linux/macOS use the same binary path without `.exe`. Build once, then retain +the committed diagnostic `Cargo.lock` and its SHA alongside evidence; this +standalone diagnostic intentionally has its own dependency lock and is outside +the product workspace. Reports use `create_new`, refusing to overwrite earlier samples. +The owned temporary database is removed automatically; callers' directories +are never recursively deleted. A forcibly killed process may leave its owned +`searchi-scale-*` directory, which should be inspected before explicit cleanup. + +Record source commit, clean/dirty status, compiler (`rustc -Vv`), dependency lock +SHA, CPU model, VM/physical status, RAM, storage, OS version, power mode and any +competing jobs. Report embeds executable SHA, OS/architecture/logical CPUs, +schema version, database SHA/size and seed time. SQLite queries run synchronously +on one caller thread; logical CPU count does not describe a bounded whole-app +resource budget. Monitor peak memory/storage externally during the large run. + +## Distinct measurements + +- Reader open, first unprepared input and background `prewarm` are separate. +- Per-case first input after production prewarm precedes that case's repeated + measurements. Prewarm itself searches representative names/content; this is + not a wholly unseen cold query. +- Warm exact/unique requests vary deterministic IDs; common name/keyword, + strict filters and absent queries repeat. Basic/Debug are paired AB/BA to + reduce order bias. Each mode retains every wall-clock sample and actual engine + metrics. Debug retains full bounded traces; their nested durations are not + summed. Result assertions run after measured retrieval and outside wall time. +- Reopened-reader open and first keyword request are separate after warm tests. + These **do not flush OS/SQLite storage caches** and are not cold-storage tests. +- `summarize.py` uses nearest-rank p50/p95 and retains negative paired overhead + caused by measurement noise. It never emits a production pass verdict. +- The recorded exact-lookup SQL plan is a narrow schema diagnostic, not evidence + that every dynamic production retrieval lane uses that plan. Inspect actual + debug lane timings/candidate counters when investigating regressions. + +UI input/paint, semantic inference/ANN, concurrent indexing, long documents, +Unicode-heavy names, skewed customer content, network/removable storage, +filesystem lifecycle and cold-machine tests remain separate release gates. +The original 50 ms filename and 150 ms keyword p95 goals may be compared with +these samples, but reaching them here proves only this synthetic lexical case. + +## Windows VM smoke, 2026-10-13 + +Actual optimized shared-engine run: 1,000 metadata records, 100 short text +documents, 20 paired samples per mode per case. All expected identities, +combined filters and absent outcomes passed. No inference or GPU was used; +the installed desktop remained running, so these are not isolated performance +measurements. Rust 1.99.0, x86_64-pc-windows-msvc. + +| Case | Basic warm p95 (ms) | Debug warm p95 (ms) | +| --- | --- | --- | +| Exact filename | 0.8893 | 1.3266 | +| Unique keyword | 1.3823 | 2.2248 | +| Common keyword | 1.3285 | 1.4759 | +| Common filename | 0.6131 | 0.6961 | +| Combined filters | 0.9019 | 1.1783 | +| Absent keyword | 0.4442 | 0.6847 | + +Reader open: 4.6814 ms; prewarm: 62.4835 ms; reopened reader: 3.1992 ms. +Raw ignored evidence `.searchi/qa/scale-smoke-20261013.json`, SHA-256 +`79f58066f8e42ee3c249eeb341a8ce934b619e3c308038081845c8d24a4543de`. +Executable SHA-256 +`75fd6d4053b0ffe636b298f3c514222a9a5adf9a316d211ebabb8ccb58c2a89a`; +diagnostic lock SHA-256 +`4b426c7b38b765f1ae7e6947dcb52e21dc26a02ed4d50ba51354819a918a83c8`. + +The initial diagnostic failures exposed an invalid external-content check for +the sparse index and milliseconds supplied to nanosecond storage. Both were +corrected in the harness; production engine behavior was unchanged. Statistics +tests passed in normal and optimized Python modes; formatting and strict +release all-target Clippy passed. A subsequent Clippy-only parity-expression +cleanup did not change the recorded smoke executable. The million-record run, +representative libraries and UI/semantic timings remain open. diff --git a/tools/scale-eval/Cargo.lock b/tools/scale-eval/Cargo.lock new file mode 100644 index 0000000..df6d1d8 --- /dev/null +++ b/tools/scale-eval/Cargo.lock @@ -0,0 +1,1072 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "aes" +version = "0.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35f0f96ce78e38c3dc6d8948aa8163d06385be74000f3c7a95bf1eef35d3ea32" +dependencies = [ + "cipher", + "cpubits", + "cpufeatures 0.3.1", +] + +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "once_cell", + "version_check", + "zerocopy", +] + +[[package]] +name = "alloc-no-stdlib" +version = "2.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" + +[[package]] +name = "alloc-stdlib" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195" +dependencies = [ + "alloc-no-stdlib", +] + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "bitflags" +version = "2.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "block-padding" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "710f1dd022ef4e93f8a438b4ba958de7f64308434fa6a87104481645cc30068b" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "brotli-decompressor" +version = "5.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytemuck" +version = "1.25.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95832e849adfb21180ccb6826a99da14e5d266ae5c2e668e1602cf234f153797" + +[[package]] +name = "byteorder-lite" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f1fe948ff07f4bd06c30984e69f5b4899c516a3ef74f34df92a2df2ab535495" + +[[package]] +name = "cbc" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce2dc9ee5f88d11e0beb842c88b33c8a5cf0d1329c4b19494af42b07dbfe8896" +dependencies = [ + "cipher", +] + +[[package]] +name = "cc" +version = "1.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6651c9ed80effdc7db0ff72512157f901af5e3549e341e24b1dd4887d836d838" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" + +[[package]] +name = "chacha20" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.1", + "rand_core", +] + +[[package]] +name = "cipher" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8cf2a2c93cd704877c0858356ed03480ff301ee950b43f1cbe4573b088bfa6c" +dependencies = [ + "crypto-common 0.2.2", + "inout", +] + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "core_detect" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f8f80099a98041a3d1622845c271458a2d73e688351bf3cb999266764b81d48" + +[[package]] +name = "cpubits" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15b85f9c39137c3a891689859392b1bd49812121d0d61c9caf00d46ed5ce06ae" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566" +dependencies = [ + "libc", +] + +[[package]] +name = "crc32fast" +version = "1.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer 0.10.4", + "crypto-common 0.1.7", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer 0.12.1", + "const-oid", + "crypto-common 0.2.2", +] + +[[package]] +name = "ecb" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26f2a8b3e564eba0877223dc343703ad0385794e882e6d13f3a4dd5c6b1f41ac" +dependencies = [ + "cipher", +] + +[[package]] +name = "encoding_rs" +version = "0.8.42" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e985e0451871ad22fb8d2b6b076e2028a502a0d3950998c2c5c0a4f9b5d9679" +dependencies = [ + "cfg-if", + "core_detect", + "multiversion_no_op", + "rustversion", + "scopeguard", + "simdutf8", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "fallible-iterator" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649" + +[[package]] +name = "fallible-streaming-iterator" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a" + +[[package]] +name = "fastrand" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" + +[[package]] +name = "fdeflate" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e6853b52649d4ac5c0bd02320cddc5ba956bdb407c4b75a2c6b75bf51500f8c" +dependencies = [ + "simd-adler32", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aedcfb3409746eddb02b9e19ebda1c3394f759a152e48ee875a0844d1b955484" + +[[package]] +name = "flate2" +version = "1.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb" +dependencies = [ + "crc32fast", + "miniz_oxide 0.9.1", + "zlib-rs", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", + "rand_core", +] + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" +dependencies = [ + "ahash", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "hashlink" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af" +dependencies = [ + "hashbrown 0.14.5", +] + +[[package]] +name = "hybrid-array" +version = "0.4.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27f864f10dfb56725ce5ce5472bc52252c8f93a4ab86327122cebf62c5f59a17" +dependencies = [ + "typenum", +] + +[[package]] +name = "image" +version = "0.25.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85ab80394333c02fe689eaf900ab500fbd0c2213da414687ebf995a65d5a6104" +dependencies = [ + "bytemuck", + "byteorder-lite", + "image-webp", + "moxcms", + "num-traits", + "png", + "zune-core", + "zune-jpeg", +] + +[[package]] +name = "image-webp" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "525e9ff3e1a4be2fbea1fdf0e98686a6d98b4d8f937e1bf7402245af1909e8c3" +dependencies = [ + "byteorder-lite", + "quick-error", +] + +[[package]] +name = "indexmap" +version = "2.14.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", +] + +[[package]] +name = "inout" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4250ce6452e92010fdf7268ccc5d14faa80bb12fc741938534c58f16804e03c7" +dependencies = [ + "block-padding", + "hybrid-array", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "libc" +version = "0.2.190" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce5d3ddc6d3fa000eb1536d85e147bfe31aacaba692ed6a876f95cb7c855be78" + +[[package]] +name = "libsqlite3-sys" +version = "0.30.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e99fb7a497b1e3339bc746195567ed8d3e24945ecd636e3619d20b9de9e9149" +dependencies = [ + "cc", + "pkg-config", + "vcpkg", +] + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "log" +version = "0.4.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" + +[[package]] +name = "lopdf" +version = "0.45.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfffda0fe1ab0157e1a13c14bebd3f28671f2fccb7922f0722ec53926e6922d3" +dependencies = [ + "aes", + "bitflags", + "brotli-decompressor", + "cbc", + "ecb", + "encoding_rs", + "flate2", + "getrandom", + "indexmap", + "itoa", + "log", + "md-5", + "nom", + "rand", + "rangemap", + "sha2 0.11.1", + "stringprep", + "thiserror", + "weezl", +] + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "miniz_oxide" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "moxcms" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb85c154ba489f01b25c0d36ae69a87e4a1c73a72631fc6c0eb6dde34a73e44b" +dependencies = [ + "num-traits", + "pxfm", +] + +[[package]] +name = "multiversion_no_op" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "743fb55ba31b18fb1ecef6bdc9aa2743314978ac084044301a7eee33fb99a20d" + +[[package]] +name = "nom" +version = "8.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405" +dependencies = [ + "memchr", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "pkg-config" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548" + +[[package]] +name = "png" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60769b8b31b2a9f263dae2776c37b1b28ae246943cf719eb6946a1db05128a61" +dependencies = [ + "bitflags", + "crc32fast", + "fdeflate", + "flate2", + "miniz_oxide 0.8.9", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "pxfm" +version = "0.1.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d55d956fa96f5ec02be2e13af0e20391a5aa83d6a074e3ad368959d0fab299ea" + +[[package]] +name = "quick-error" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a993555f31e5a609f617c12db6250dedcac1b0a85076912c436e6fc9b2c8e6a3" + +[[package]] +name = "quick-xml" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41b1177fdf999d2321d3fb46ff47159d9c1fb9ad66a4879f8c50a0b504615e9b" +dependencies = [ + "memchr", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af" +dependencies = [ + "chacha20", + "getrandom", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "rangemap" +version = "1.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a611d15b50743feb4c76b7d03edcb0e64f399c26961e4efe6975bc398be6aa3d" + +[[package]] +name = "rusqlite" +version = "0.32.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7753b721174eb8ff87a9a0e799e2d7bc3749323e773db92e0984debb00019d6e" +dependencies = [ + "bitflags", + "fallible-iterator", + "fallible-streaming-iterator", + "hashlink", + "libsqlite3-sys", + "smallvec", +] + +[[package]] +name = "rustix" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "searchi-core" +version = "0.1.0" +dependencies = [ + "anyhow", + "lopdf", + "quick-xml", + "rusqlite", + "searchi-media", + "serde", + "sha2 0.10.9", + "walkdir", + "zip", +] + +[[package]] +name = "searchi-media" +version = "0.1.0" +dependencies = [ + "anyhow", + "image", + "serde", + "serde_json", + "sha2 0.10.9", + "tempfile", +] + +[[package]] +name = "searchi-scale-eval" +version = "0.1.0" +dependencies = [ + "anyhow", + "rusqlite", + "searchi-core", + "serde_json", + "sha2 0.10.9", + "tempfile", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.9", +] + +[[package]] +name = "serde_json" +version = "1.0.154" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.10.7", +] + +[[package]] +name = "sha2" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d7069beb7d6ac7b9acd1039986e73443f24234f41074da099d6f994ac9ad19" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.1", + "digest 0.11.3", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-adler32" +version = "0.3.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea" + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + +[[package]] +name = "smallvec" +version = "1.16.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b3dc8af474f516a851ff4bd12db780f948b9250ad37211e4eec0bccea54e01b" + +[[package]] +name = "stringprep" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b4df3d392d81bd458a8a621b8bffbd2302a12ffe288a9d931670948749463b1" +dependencies = [ + "unicode-bidi", + "unicode-normalization", + "unicode-properties", +] + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d78c8dee4c7bf0e14673097256fed6142ce9d3b85a408189d07482442145823b" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom", + "once_cell", + "rustix", + "windows-sys", +] + +[[package]] +name = "thiserror" +version = "2.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.9", +] + +[[package]] +name = "tinyvec" +version = "1.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee" + +[[package]] +name = "typed-path" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e" + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-bidi" +version = "0.3.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" + +[[package]] +name = "unicode-ident" +version = "1.0.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563" + +[[package]] +name = "unicode-normalization" +version = "0.1.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" +dependencies = [ + "tinyvec", +] + +[[package]] +name = "unicode-properties" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" + +[[package]] +name = "vcpkg" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "weezl" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4ca08e5ef825b65b056d9efbd95c8750683f0a6d0466d02e96dc2e4e360f3d2" + +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "zerocopy" +version = "0.8.62" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "86502bf56ac7c77571a32e2647bb2a15894565e981fb2a48d7bde2d91c965a9d" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.62" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5457206954b06561e2608c7e19cf58b1926586d999c246eebe4502f7e2039d1a" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zip" +version = "8.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d04a6b5381502aa6087c94c669499eb1602eb9c5e8198e534de571f7154809b" +dependencies = [ + "crc32fast", + "flate2", + "indexmap", + "memchr", + "typed-path", + "zopfli", +] + +[[package]] +name = "zlib-rs" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b268e58e7c693d7c271f93ffc4ba3b380412554231c85bf61ca7af91042a4112" + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" + +[[package]] +name = "zopfli" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aaf7fc5d30c28483d93805c4a5e12b05bbb52407fa67c5f8bd552374cd01fb11" +dependencies = [ + "bumpalo", + "crc32fast", + "log", + "simd-adler32", +] + +[[package]] +name = "zune-core" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d56377fd46368984a170bc5aac5567e52ca5da874caa60bea39fcbca78fb658b" + +[[package]] +name = "zune-jpeg" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27bc9d5b815bc103f142aa054f561d9187d191692ec7c2d1e2b4737f8dbd7296" +dependencies = [ + "zune-core", +] diff --git a/tools/scale-eval/Cargo.toml b/tools/scale-eval/Cargo.toml new file mode 100644 index 0000000..a76aa69 --- /dev/null +++ b/tools/scale-eval/Cargo.toml @@ -0,0 +1,16 @@ +[package] +name = "searchi-scale-eval" +version = "0.1.0" +edition = "2021" +publish = false + +# Diagnostic only: intentionally outside the product workspace. +[workspace] + +[dependencies] +anyhow = "1" +rusqlite = { version = "0.32", features = ["bundled"] } +searchi-core = { path = "../../crates/searchi-core" } +serde_json = "1" +sha2 = "0.10" +tempfile = "3" diff --git a/tools/scale-eval/src/main.rs b/tools/scale-eval/src/main.rs new file mode 100644 index 0000000..437e7a9 --- /dev/null +++ b/tools/scale-eval/src/main.rs @@ -0,0 +1,250 @@ +//! Synthetic index performance only; no physical corpus, extraction or inference. +use anyhow::{bail, Context, Result}; +use rusqlite::{params, Connection}; +use searchi_core::{SearchIndex, SearchOptions}; +use serde_json::{json, Value}; +use sha2::{Digest, Sha256}; +use std::{env, fs, io::Read, path::Path, time::Instant}; + +const SEED: u64 = 20261013; +const LIMIT: usize = 20; + +fn hash(path: &Path) -> Result { + let mut file = fs::File::open(path)?; + let mut digest = Sha256::new(); + let mut buffer = [0u8; 65536]; + loop { + let count = file.read(&mut buffer)?; + if count == 0 { + break; + } + digest.update(&buffer[..count]); + } + Ok(format!("{:x}", digest.finalize())) +} + +fn seed(db: &Path, root: &Path, files: usize, docs: usize) -> Result<()> { + // Let the actual engine create/migrate its schema and all FTS triggers. + drop(SearchIndex::open(db)?); + let mut connection = Connection::open(db)?; + for start in (0..files).step_by(1000) { + let transaction = connection.transaction()?; + { + let mut insert = transaction.prepare("INSERT INTO files(path,root,filename,content,size,mtime,extension) VALUES(?1,?2,?3,?4,?5,?6,?7)")?; + let mut passage = transaction.prepare( + "INSERT INTO passages(text,file_path,kind,number) VALUES(?1,?2,'line',1)", + )?; + for id in start..(start + 1000).min(files) { + let extension = if id.is_multiple_of(2) { "txt" } else { "rs" }; + let filename = format!("record_{id:07}.{extension}"); + let path = root + .join(if id % 3 == 0 { + "chosen" + } else { + "chosen-other" + }) + .join(&filename); + let path = path.to_string_lossy(); + let content = if id < docs { + format!("Synthetic document uniquemarker{id:07} cancellation renewal notice category{} seed{SEED}", id % 97) + } else { + String::new() + }; + insert.execute(params![ + path, + root.to_string_lossy(), + filename, + content, + (id % 8192 + 100) as i64, + (1700000000000i64 + id as i64) * 1_000_000, + extension + ])?; + if !content.is_empty() { + passage.execute(params![content, path])?; + } + } + } + transaction.commit()?; + } + for table in ["search", "names", "content_search", "passage_search"] { + // content_search intentionally omits empty files; external-content + // comparison would incorrectly require those omitted row IDs. + let rank = if table == "content_search" { 0 } else { 1 }; + connection + .execute_batch(&format!( + "INSERT INTO {table}({table},rank) VALUES('integrity-check',{rank});" + )) + .with_context(|| format!("FTS integrity check for {table}"))?; + } + let integrity: String = connection.query_row("PRAGMA integrity_check", [], |r| r.get(0))?; + if integrity != "ok" { + bail!("SQLite integrity check: {integrity}"); + } + let counts: (usize, usize, usize) = connection.query_row( + "SELECT (SELECT count(*) FROM files),(SELECT count(*) FROM files WHERE content!=''),(SELECT count(*) FROM passages)", + [], |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + )?; + if counts != (files, docs, docs) { + bail!("seed row counts differ: {counts:?}"); + } + connection.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);")?; + Ok(()) +} + +fn run(index: &SearchIndex, case: &str, id: usize, root: &Path, debug: bool) -> Result { + let mut options = SearchOptions::default(); + let query = match case { + "exact" => format!( + "record_{id:07}.{}", + if id.is_multiple_of(2) { "txt" } else { "rs" } + ), + "keyword_unique" => format!("uniquemarker{id:07}"), + "common_keyword" => "cancellation".into(), + "common_name" => "record_".into(), + "strict_filters" => { + options.folder = Some(root.join("chosen")); + options.extension = Some("rs".into()); + options.code_only = true; + options.min_size = Some(110); + options.max_size = Some(119); + options.modified_after_unix_ms = Some(1700000000010); + options.modified_before_unix_ms = Some(1700000000019); + "cancellation".into() + } + "absent" => "nonexistentzzxyartifact".into(), + _ => bail!("unknown case"), + }; + let start = Instant::now(); + let response = index.search_with_options_measured(&query, LIMIT, &options, debug)?; + let wall_ms = start.elapsed().as_secs_f64() * 1000.0; + if case == "absent" { + if !response.results.is_empty() { + bail!("absent query returned results"); + } + } else { + let first = response + .results + .first() + .with_context(|| format!("positive {case} query returned nothing: {query}"))?; + if case == "exact" && (first.filename != query || first.match_kind != "exact_filename") { + bail!("exact filename priority failed"); + } + if case == "keyword_unique" && !first.filename.starts_with(&format!("record_{id:07}.")) { + bail!("unique keyword identity failed"); + } + if case == "strict_filters" + && (response.results.len() != 1 || first.filename != "record_0000015.rs") + { + bail!("strict filter expected exactly the seeded eligible record 15"); + } + for result in &response.results { + if case == "strict_filters" + && (!Path::new(&result.path).starts_with(root.join("chosen")) + || !result.filename.ends_with(".rs")) + { + bail!("strict folder/code/extension filter failed"); + } + } + } + if debug != response.metrics.trace.is_some() { + bail!("trace mode mismatch"); + } + Ok(json!({"wall_ms":wall_ms,"metrics":response.metrics})) +} + +fn main() -> Result<()> { + let args: Vec = env::args().skip(1).collect(); + if args.len() != 4 { + bail!("usage: searchi-scale-eval FILES DOCUMENTS SAMPLES REPORT.json"); + } + let files: usize = args[0].parse()?; + let docs: usize = args[1].parse()?; + let samples: usize = args[2].parse()?; + if !(100..=1000000).contains(&files) + || !(20..=100000).contains(&docs) + || docs > files + || !(20..=1000).contains(&samples) + { + bail!("require 100..1M files, 20..100K docs <= files, 20..1000 samples"); + } + // TempDir owns exactly this directory. No caller directory is recursively removed. + let owned = tempfile::Builder::new() + .prefix("searchi-scale-") + .tempdir()?; + let root = owned.path().join("corpus"); + for folder in ["chosen", "chosen-other"] { + fs::create_dir_all(root.join(folder))?; + } + let root = fs::canonicalize(root)?; + let db = owned.path().join("index.sqlite"); + let seeding = Instant::now(); + seed(&db, &root, files, docs)?; + let seed_ms = seeding.elapsed().as_secs_f64() * 1000.0; + let first_open = Instant::now(); + let mut index = SearchIndex::open(&db)?; + let open_ms = first_open.elapsed().as_secs_f64() * 1000.0; + let unprepared_first = run(&index, "exact", files / 2, &root, false)?; + let prepare = Instant::now(); + let cached_frequencies = index.prewarm()?; + let prepare_ms = prepare.elapsed().as_secs_f64() * 1000.0; + let mut cases = Vec::new(); + for case in [ + "exact", + "keyword_unique", + "common_keyword", + "common_name", + "strict_filters", + "absent", + ] { + let first = run(&index, case, docs / 2 + 1, &root, false)?; + let mut basic = Vec::with_capacity(samples); + let mut debug = Vec::with_capacity(samples); + for sample in 0..samples { + // Paired AB/BA to avoid systematically favoring one trace mode. + let id = (sample * 7919 + SEED as usize) % docs; + for detailed in if sample.is_multiple_of(2) { + [false, true] + } else { + [true, false] + } { + let value = run(&index, case, id, &root, detailed)?; + if detailed { + debug.push(value); + } else { + basic.push(value); + } + } + } + cases.push(json!({"case":case,"first_after_prewarm":first,"basic":basic,"debug":debug})); + } + drop(index); + let restart = Instant::now(); + let reopened = SearchIndex::open(&db)?; + let reopened_open_ms = restart.elapsed().as_secs_f64() * 1000.0; + let reopened_first = run(&reopened, "keyword_unique", docs - 1, &root, false)?; + drop(reopened); + let connection = Connection::open(&db)?; + let schema: i64 = connection.query_row("PRAGMA user_version", [], |r| r.get(0))?; + let plan: Vec = connection + .prepare( + "EXPLAIN QUERY PLAN SELECT id FROM files WHERE lower(filename)='record_0000001.rs'", + )? + .query_map([], |r| r.get(3))? + .collect::>()?; + connection.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);")?; + drop(connection); + let report = json!({"kind":"synthetic_lexical_index_only","seed":SEED,"files":files,"documents":docs,"schema":schema, + "os":env::consts::OS,"arch":env::consts::ARCH,"logical_cpus":std::thread::available_parallelism()?.get(), + "executable_sha256":hash(&env::current_exe()?)?,"seed_ms":seed_ms,"database_bytes":fs::metadata(&db)?.len(), + "database_sha256":hash(&db)?,"first_reader_open_ms":open_ms,"unprepared_first_input":unprepared_first, + "prewarm_ms":prepare_ms,"cached_frequencies":cached_frequencies,"reopened_open_ms":reopened_open_ms, + "reopened_first_input":reopened_first,"exact_lookup_plan":plan,"cases":cases, + "cold_storage_tested":false,"physical_files_created":0,"inference":false}); + // create_new prevents accidental replacement of previous evidence. + let output = fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(&args[3])?; + serde_json::to_writer_pretty(output, &report)?; + Ok(()) +} diff --git a/tools/scale-eval/summarize.py b/tools/scale-eval/summarize.py new file mode 100644 index 0000000..fe818ba --- /dev/null +++ b/tools/scale-eval/summarize.py @@ -0,0 +1,45 @@ +"""Summarize retained raw samples; nearest-rank p95, no gate claims.""" +import json +import math +import sys + + +def percentile(values, probability): + if not values or not 0 < probability <= 1: + raise ValueError("nonempty samples and probability in (0, 1] required") + if any(not math.isfinite(value) or value < 0 for value in values): + raise ValueError("samples must be finite nonnegative milliseconds") + ordered = sorted(values) + return ordered[math.ceil(len(ordered) * probability) - 1] + + +def summarize(report): + if report.get("kind") != "synthetic_lexical_index_only": + raise ValueError("unsupported report") + rows = [] + for case in report["cases"]: + if len(case["basic"]) != len(case["debug"]) or not case["basic"]: + raise ValueError("paired basic/debug samples required") + row = {"case": case["case"], "samples_per_mode": len(case["basic"])} + for mode in ("basic", "debug"): + values = [sample["wall_ms"] for sample in case[mode]] + row[mode] = { + "p50_ms": percentile(values, 0.5), + "p95_ms": percentile(values, 0.95), + "maximum_ms": max(values), + "engine_p95_ms": percentile( + [sample["metrics"]["total_ms"] for sample in case[mode]], 0.95 + ), + } + # Pair order follows the same deterministic request IDs in both arrays. + row["mean_paired_debug_minus_basic_ms"] = sum( + debug["wall_ms"] - basic["wall_ms"] + for basic, debug in zip(case["basic"], case["debug"]) + ) / len(case["basic"]) + rows.append(row) + return {"kind": report["kind"], "production_gate_pass": None, "cases": rows} + + +if __name__ == "__main__": + with open(sys.argv[1], encoding="utf-8") as source: + print(json.dumps(summarize(json.load(source)), indent=2)) diff --git a/tools/scale-eval/test_summary.py b/tools/scale-eval/test_summary.py new file mode 100644 index 0000000..2d10317 --- /dev/null +++ b/tools/scale-eval/test_summary.py @@ -0,0 +1,30 @@ +import unittest +from summarize import percentile, summarize + + +class SummaryTests(unittest.TestCase): + def test_nearest_rank_and_unsorted_input(self): + self.assertEqual(percentile(list(range(20, 0, -1)), 0.95), 19) + self.assertEqual(percentile([3, 1, 2], 0.5), 2) + + def test_reject_invalid_samples(self): + for values in ([], [-1], [float("nan")], [float("inf")]): + with self.assertRaises(ValueError): + percentile(values, 0.95) + + def test_paired_overhead_retains_negative_noise(self): + sample = lambda ms: {"wall_ms": ms, "metrics": {"total_ms": ms / 2}} + report = {"kind": "synthetic_lexical_index_only", "cases": [{ + "case": "exact", "basic": [sample(4), sample(6)], + "debug": [sample(3), sample(5)], + }]} + result = summarize(report) + self.assertIsNone(result["production_gate_pass"]) + self.assertEqual(result["cases"][0]["mean_paired_debug_minus_basic_ms"], -1) + report["cases"][0]["debug"].pop() + with self.assertRaises(ValueError): + summarize(report) + + +if __name__ == "__main__": + unittest.main()