Validate shared-engine lexical performance harness on Windows

docs/SCALE-PERFORMANCE.md

@@ -0,0 +1,121 @@
+# Large-index lexical performance protocol
+
+Status: **1,000-record smoke executed; large run deferred**. No production
+release gate is passed by this document.
+
+`tools/scale-eval` uses the actual shared Rust `SearchIndex` engine, with no
+desktop, model worker or semantic inference. The intended scale is **1 million
+file records and 100,000 text documents**, with 100 measured requests per case
+and mode. A 1,000-record smoke run must pass first.
+
+## What the fixture represents
+
+The engine opens and migrates the database before bounded 1,000-row transactions
+insert synthetic metadata and one short passage per text document. Existing
+engine triggers populate filename trigram, full-text and passage indexes; all
+four FTS tables receive integrity checks, followed by SQLite
+integrity checking. No FTS schema or engine retrieval API is replaced.
+The intentionally sparse `content_search` receives internal integrity checking;
+the other three additionally compare external content. Requiring every empty
+file row in the sparse index would falsely report corruption.
+
+Only two empty directories are created so production folder canonicalization
+and component-boundary filtering operate normally. The neighboring directories
+`chosen` and `chosen-other` test accidental prefix inclusion. No million-file
+filesystem corpus is created. Text/code records alternate extensions; a filtered
+common-term query combines folder, extension, code, size and modification time.
+Every request checks positive/absent outcomes; exact filenames must retain first
+place and unique content markers must identify their corresponding document.
+
+This bypasses discovery, permissions, extraction, hashing source input, watcher
+traffic and real source access. Synthetic records have no verified extraction
+provenance and must never be used for semantic indexing or agent excerpts.
+Short repeated text and similarly shaped names are deliberately reproducible
+workloads, not a representative customer library. The database SHA identifies
+the actual run; absolute temporary-root bytes make it machine/run dependent.
+The fixed seed and record-generation algorithm identify the logical dataset.
+
+## Run and preserve evidence
+
+From the repository root, once no other performance/inference task owns CPU:
+
+```powershell
+$env:SEARCHI_FORCE_CPU = '1'
+cargo build --release --manifest-path tools/scale-eval/Cargo.toml --locked -j 1
+tools/scale-eval/target/release/searchi-scale-eval.exe 1000 100 20 .searchi/qa/scale-smoke.json
+tools/scale-eval/target/release/searchi-scale-eval.exe 1000000 100000 100 .searchi/qa/scale-million.json
+python tools/scale-eval/summarize.py .searchi/qa/scale-million.json
+```
+
+Linux/macOS use the same binary path without `.exe`. Build once, then retain
+the committed diagnostic `Cargo.lock` and its SHA alongside evidence; this
+standalone diagnostic intentionally has its own dependency lock and is outside
+the product workspace. Reports use `create_new`, refusing to overwrite earlier samples.
+The owned temporary database is removed automatically; callers' directories
+are never recursively deleted. A forcibly killed process may leave its owned
+`searchi-scale-*` directory, which should be inspected before explicit cleanup.
+
+Record source commit, clean/dirty status, compiler (`rustc -Vv`), dependency lock
+SHA, CPU model, VM/physical status, RAM, storage, OS version, power mode and any
+competing jobs. Report embeds executable SHA, OS/architecture/logical CPUs,
+schema version, database SHA/size and seed time. SQLite queries run synchronously
+on one caller thread; logical CPU count does not describe a bounded whole-app
+resource budget. Monitor peak memory/storage externally during the large run.
+
+## Distinct measurements
+
+- Reader open, first unprepared input and background `prewarm` are separate.
+- Per-case first input after production prewarm precedes that case's repeated
+  measurements. Prewarm itself searches representative names/content; this is
+  not a wholly unseen cold query.
+- Warm exact/unique requests vary deterministic IDs; common name/keyword,
+  strict filters and absent queries repeat. Basic/Debug are paired AB/BA to
+  reduce order bias. Each mode retains every wall-clock sample and actual engine
+  metrics. Debug retains full bounded traces; their nested durations are not
+  summed. Result assertions run after measured retrieval and outside wall time.
+- Reopened-reader open and first keyword request are separate after warm tests.
+  These **do not flush OS/SQLite storage caches** and are not cold-storage tests.
+- `summarize.py` uses nearest-rank p50/p95 and retains negative paired overhead
+  caused by measurement noise. It never emits a production pass verdict.
+- The recorded exact-lookup SQL plan is a narrow schema diagnostic, not evidence
+  that every dynamic production retrieval lane uses that plan. Inspect actual
+  debug lane timings/candidate counters when investigating regressions.
+
+UI input/paint, semantic inference/ANN, concurrent indexing, long documents,
+Unicode-heavy names, skewed customer content, network/removable storage,
+filesystem lifecycle and cold-machine tests remain separate release gates.
+The original 50 ms filename and 150 ms keyword p95 goals may be compared with
+these samples, but reaching them here proves only this synthetic lexical case.
+
+## Windows VM smoke, 2026-10-13
+
+Actual optimized shared-engine run: 1,000 metadata records, 100 short text
+documents, 20 paired samples per mode per case. All expected identities,
+combined filters and absent outcomes passed. No inference or GPU was used;
+the installed desktop remained running, so these are not isolated performance
+measurements. Rust 1.99.0, x86_64-pc-windows-msvc.
+
+| Case | Basic warm p95 (ms) | Debug warm p95 (ms) |
+| --- | --- | --- |
+| Exact filename | 0.8893 | 1.3266 |
+| Unique keyword | 1.3823 | 2.2248 |
+| Common keyword | 1.3285 | 1.4759 |
+| Common filename | 0.6131 | 0.6961 |
+| Combined filters | 0.9019 | 1.1783 |
+| Absent keyword | 0.4442 | 0.6847 |
+
+Reader open: 4.6814 ms; prewarm: 62.4835 ms; reopened reader: 3.1992 ms.
+Raw ignored evidence `.searchi/qa/scale-smoke-20261013.json`, SHA-256
+`79f58066f8e42ee3c249eeb341a8ce934b619e3c308038081845c8d24a4543de`.
+Executable SHA-256
+`75fd6d4053b0ffe636b298f3c514222a9a5adf9a316d211ebabb8ccb58c2a89a`;
+diagnostic lock SHA-256
+`4b426c7b38b765f1ae7e6947dcb52e21dc26a02ed4d50ba51354819a918a83c8`.
+
+The initial diagnostic failures exposed an invalid external-content check for
+the sparse index and milliseconds supplied to nanosecond storage. Both were
+corrected in the harness; production engine behavior was unchanged. Statistics
+tests passed in normal and optimized Python modes; formatting and strict
+release all-target Clippy passed. A subsequent Clippy-only parity-expression
+cleanup did not change the recorded smoke executable. The million-record run,
+representative libraries and UI/semantic timings remain open.

tools/scale-eval/Cargo.toml

@@ -0,0 +1,16 @@
+[package]
+name = "searchi-scale-eval"
+version = "0.1.0"
+edition = "2021"
+publish = false
+
+# Diagnostic only: intentionally outside the product workspace.
+[workspace]
+
+[dependencies]
+anyhow = "1"
+rusqlite = { version = "0.32", features = ["bundled"] }
+searchi-core = { path = "../../crates/searchi-core" }
+serde_json = "1"
+sha2 = "0.10"
+tempfile = "3"

tools/scale-eval/src/main.rs

@@ -0,0 +1,250 @@
+//! Synthetic index performance only; no physical corpus, extraction or inference.
+use anyhow::{bail, Context, Result};
+use rusqlite::{params, Connection};
+use searchi_core::{SearchIndex, SearchOptions};
+use serde_json::{json, Value};
+use sha2::{Digest, Sha256};
+use std::{env, fs, io::Read, path::Path, time::Instant};
+
+const SEED: u64 = 20261013;
+const LIMIT: usize = 20;
+
+fn hash(path: &Path) -> Result<String> {
+    let mut file = fs::File::open(path)?;
+    let mut digest = Sha256::new();
+    let mut buffer = [0u8; 65536];
+    loop {
+        let count = file.read(&mut buffer)?;
+        if count == 0 {
+            break;
+        }
+        digest.update(&buffer[..count]);
+    }
+    Ok(format!("{:x}", digest.finalize()))
+}
+
+fn seed(db: &Path, root: &Path, files: usize, docs: usize) -> Result<()> {
+    // Let the actual engine create/migrate its schema and all FTS triggers.
+    drop(SearchIndex::open(db)?);
+    let mut connection = Connection::open(db)?;
+    for start in (0..files).step_by(1000) {
+        let transaction = connection.transaction()?;
+        {
+            let mut insert = transaction.prepare("INSERT INTO files(path,root,filename,content,size,mtime,extension) VALUES(?1,?2,?3,?4,?5,?6,?7)")?;
+            let mut passage = transaction.prepare(
+                "INSERT INTO passages(text,file_path,kind,number) VALUES(?1,?2,'line',1)",
+            )?;
+            for id in start..(start + 1000).min(files) {
+                let extension = if id.is_multiple_of(2) { "txt" } else { "rs" };
+                let filename = format!("record_{id:07}.{extension}");
+                let path = root
+                    .join(if id % 3 == 0 {
+                        "chosen"
+                    } else {
+                        "chosen-other"
+                    })
+                    .join(&filename);
+                let path = path.to_string_lossy();
+                let content = if id < docs {
+                    format!("Synthetic document uniquemarker{id:07} cancellation renewal notice category{} seed{SEED}", id % 97)
+                } else {
+                    String::new()
+                };
+                insert.execute(params![
+                    path,
+                    root.to_string_lossy(),
+                    filename,
+                    content,
+                    (id % 8192 + 100) as i64,
+                    (1700000000000i64 + id as i64) * 1_000_000,
+                    extension
+                ])?;
+                if !content.is_empty() {
+                    passage.execute(params![content, path])?;
+                }
+            }
+        }
+        transaction.commit()?;
+    }
+    for table in ["search", "names", "content_search", "passage_search"] {
+        // content_search intentionally omits empty files; external-content
+        // comparison would incorrectly require those omitted row IDs.
+        let rank = if table == "content_search" { 0 } else { 1 };
+        connection
+            .execute_batch(&format!(
+                "INSERT INTO {table}({table},rank) VALUES('integrity-check',{rank});"
+            ))
+            .with_context(|| format!("FTS integrity check for {table}"))?;
+    }
+    let integrity: String = connection.query_row("PRAGMA integrity_check", [], |r| r.get(0))?;
+    if integrity != "ok" {
+        bail!("SQLite integrity check: {integrity}");
+    }
+    let counts: (usize, usize, usize) = connection.query_row(
+        "SELECT (SELECT count(*) FROM files),(SELECT count(*) FROM files WHERE content!=''),(SELECT count(*) FROM passages)",
+        [], |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)),
+    )?;
+    if counts != (files, docs, docs) {
+        bail!("seed row counts differ: {counts:?}");
+    }
+    connection.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);")?;
+    Ok(())
+}
+
+fn run(index: &SearchIndex, case: &str, id: usize, root: &Path, debug: bool) -> Result<Value> {
+    let mut options = SearchOptions::default();
+    let query = match case {
+        "exact" => format!(
+            "record_{id:07}.{}",
+            if id.is_multiple_of(2) { "txt" } else { "rs" }
+        ),
+        "keyword_unique" => format!("uniquemarker{id:07}"),
+        "common_keyword" => "cancellation".into(),
+        "common_name" => "record_".into(),
+        "strict_filters" => {
+            options.folder = Some(root.join("chosen"));
+            options.extension = Some("rs".into());
+            options.code_only = true;
+            options.min_size = Some(110);
+            options.max_size = Some(119);
+            options.modified_after_unix_ms = Some(1700000000010);
+            options.modified_before_unix_ms = Some(1700000000019);
+            "cancellation".into()
+        }
+        "absent" => "nonexistentzzxyartifact".into(),
+        _ => bail!("unknown case"),
+    };
+    let start = Instant::now();
+    let response = index.search_with_options_measured(&query, LIMIT, &options, debug)?;
+    let wall_ms = start.elapsed().as_secs_f64() * 1000.0;
+    if case == "absent" {
+        if !response.results.is_empty() {
+            bail!("absent query returned results");
+        }
+    } else {
+        let first = response
+            .results
+            .first()
+            .with_context(|| format!("positive {case} query returned nothing: {query}"))?;
+        if case == "exact" && (first.filename != query || first.match_kind != "exact_filename") {
+            bail!("exact filename priority failed");
+        }
+        if case == "keyword_unique" && !first.filename.starts_with(&format!("record_{id:07}.")) {
+            bail!("unique keyword identity failed");
+        }
+        if case == "strict_filters"
+            && (response.results.len() != 1 || first.filename != "record_0000015.rs")
+        {
+            bail!("strict filter expected exactly the seeded eligible record 15");
+        }
+        for result in &response.results {
+            if case == "strict_filters"
+                && (!Path::new(&result.path).starts_with(root.join("chosen"))
+                    || !result.filename.ends_with(".rs"))
+            {
+                bail!("strict folder/code/extension filter failed");
+            }
+        }
+    }
+    if debug != response.metrics.trace.is_some() {
+        bail!("trace mode mismatch");
+    }
+    Ok(json!({"wall_ms":wall_ms,"metrics":response.metrics}))
+}
+
+fn main() -> Result<()> {
+    let args: Vec<String> = env::args().skip(1).collect();
+    if args.len() != 4 {
+        bail!("usage: searchi-scale-eval FILES DOCUMENTS SAMPLES REPORT.json");
+    }
+    let files: usize = args[0].parse()?;
+    let docs: usize = args[1].parse()?;
+    let samples: usize = args[2].parse()?;
+    if !(100..=1000000).contains(&files)
+        || !(20..=100000).contains(&docs)
+        || docs > files
+        || !(20..=1000).contains(&samples)
+    {
+        bail!("require 100..1M files, 20..100K docs <= files, 20..1000 samples");
+    }
+    // TempDir owns exactly this directory. No caller directory is recursively removed.
+    let owned = tempfile::Builder::new()
+        .prefix("searchi-scale-")
+        .tempdir()?;
+    let root = owned.path().join("corpus");
+    for folder in ["chosen", "chosen-other"] {
+        fs::create_dir_all(root.join(folder))?;
+    }
+    let root = fs::canonicalize(root)?;
+    let db = owned.path().join("index.sqlite");
+    let seeding = Instant::now();
+    seed(&db, &root, files, docs)?;
+    let seed_ms = seeding.elapsed().as_secs_f64() * 1000.0;
+    let first_open = Instant::now();
+    let mut index = SearchIndex::open(&db)?;
+    let open_ms = first_open.elapsed().as_secs_f64() * 1000.0;
+    let unprepared_first = run(&index, "exact", files / 2, &root, false)?;
+    let prepare = Instant::now();
+    let cached_frequencies = index.prewarm()?;
+    let prepare_ms = prepare.elapsed().as_secs_f64() * 1000.0;
+    let mut cases = Vec::new();
+    for case in [
+        "exact",
+        "keyword_unique",
+        "common_keyword",
+        "common_name",
+        "strict_filters",
+        "absent",
+    ] {
+        let first = run(&index, case, docs / 2 + 1, &root, false)?;
+        let mut basic = Vec::with_capacity(samples);
+        let mut debug = Vec::with_capacity(samples);
+        for sample in 0..samples {
+            // Paired AB/BA to avoid systematically favoring one trace mode.
+            let id = (sample * 7919 + SEED as usize) % docs;
+            for detailed in if sample.is_multiple_of(2) {
+                [false, true]
+            } else {
+                [true, false]
+            } {
+                let value = run(&index, case, id, &root, detailed)?;
+                if detailed {
+                    debug.push(value);
+                } else {
+                    basic.push(value);
+                }
+            }
+        }
+        cases.push(json!({"case":case,"first_after_prewarm":first,"basic":basic,"debug":debug}));
+    }
+    drop(index);
+    let restart = Instant::now();
+    let reopened = SearchIndex::open(&db)?;
+    let reopened_open_ms = restart.elapsed().as_secs_f64() * 1000.0;
+    let reopened_first = run(&reopened, "keyword_unique", docs - 1, &root, false)?;
+    drop(reopened);
+    let connection = Connection::open(&db)?;
+    let schema: i64 = connection.query_row("PRAGMA user_version", [], |r| r.get(0))?;
+    let plan: Vec<String> = connection
+        .prepare(
+            "EXPLAIN QUERY PLAN SELECT id FROM files WHERE lower(filename)='record_0000001.rs'",
+        )?
+        .query_map([], |r| r.get(3))?
+        .collect::<rusqlite::Result<_>>()?;
+    connection.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);")?;
+    drop(connection);
+    let report = json!({"kind":"synthetic_lexical_index_only","seed":SEED,"files":files,"documents":docs,"schema":schema,
+        "os":env::consts::OS,"arch":env::consts::ARCH,"logical_cpus":std::thread::available_parallelism()?.get(),
+        "executable_sha256":hash(&env::current_exe()?)?,"seed_ms":seed_ms,"database_bytes":fs::metadata(&db)?.len(),
+        "database_sha256":hash(&db)?,"first_reader_open_ms":open_ms,"unprepared_first_input":unprepared_first,
+        "prewarm_ms":prepare_ms,"cached_frequencies":cached_frequencies,"reopened_open_ms":reopened_open_ms,
+        "reopened_first_input":reopened_first,"exact_lookup_plan":plan,"cases":cases,
+        "cold_storage_tested":false,"physical_files_created":0,"inference":false});
+    // create_new prevents accidental replacement of previous evidence.
+    let output = fs::OpenOptions::new()
+        .write(true)
+        .create_new(true)
+        .open(&args[3])?;
+    serde_json::to_writer_pretty(output, &report)?;
+    Ok(())
+}

tools/scale-eval/summarize.py

@@ -0,0 +1,45 @@
+"""Summarize retained raw samples; nearest-rank p95, no gate claims."""
+import json
+import math
+import sys
+
+
+def percentile(values, probability):
+    if not values or not 0 < probability <= 1:
+        raise ValueError("nonempty samples and probability in (0, 1] required")
+    if any(not math.isfinite(value) or value < 0 for value in values):
+        raise ValueError("samples must be finite nonnegative milliseconds")
+    ordered = sorted(values)
+    return ordered[math.ceil(len(ordered) * probability) - 1]
+
+
+def summarize(report):
+    if report.get("kind") != "synthetic_lexical_index_only":
+        raise ValueError("unsupported report")
+    rows = []
+    for case in report["cases"]:
+        if len(case["basic"]) != len(case["debug"]) or not case["basic"]:
+            raise ValueError("paired basic/debug samples required")
+        row = {"case": case["case"], "samples_per_mode": len(case["basic"])}
+        for mode in ("basic", "debug"):
+            values = [sample["wall_ms"] for sample in case[mode]]
+            row[mode] = {
+                "p50_ms": percentile(values, 0.5),
+                "p95_ms": percentile(values, 0.95),
+                "maximum_ms": max(values),
+                "engine_p95_ms": percentile(
+                    [sample["metrics"]["total_ms"] for sample in case[mode]], 0.95
+                ),
+            }
+        # Pair order follows the same deterministic request IDs in both arrays.
+        row["mean_paired_debug_minus_basic_ms"] = sum(
+            debug["wall_ms"] - basic["wall_ms"]
+            for basic, debug in zip(case["basic"], case["debug"])
+        ) / len(case["basic"])
+        rows.append(row)
+    return {"kind": report["kind"], "production_gate_pass": None, "cases": rows}
+
+
+if __name__ == "__main__":
+    with open(sys.argv[1], encoding="utf-8") as source:
+        print(json.dumps(summarize(json.load(source)), indent=2))

tools/scale-eval/test_summary.py

@@ -0,0 +1,30 @@
+import unittest
+from summarize import percentile, summarize
+
+
+class SummaryTests(unittest.TestCase):
+    def test_nearest_rank_and_unsorted_input(self):
+        self.assertEqual(percentile(list(range(20, 0, -1)), 0.95), 19)
+        self.assertEqual(percentile([3, 1, 2], 0.5), 2)
+
+    def test_reject_invalid_samples(self):
+        for values in ([], [-1], [float("nan")], [float("inf")]):
+            with self.assertRaises(ValueError):
+                percentile(values, 0.95)
+
+    def test_paired_overhead_retains_negative_noise(self):
+        sample = lambda ms: {"wall_ms": ms, "metrics": {"total_ms": ms / 2}}
+        report = {"kind": "synthetic_lexical_index_only", "cases": [{
+            "case": "exact", "basic": [sample(4), sample(6)],
+            "debug": [sample(3), sample(5)],
+        }]}
+        result = summarize(report)
+        self.assertIsNone(result["production_gate_pass"])
+        self.assertEqual(result["cases"][0]["mean_paired_debug_minus_basic_ms"], -1)
+        report["cases"][0]["debug"].pop()
+        with self.assertRaises(ValueError):
+            summarize(report)
+
+
+if __name__ == "__main__":
+    unittest.main()