diff --git a/crates/searchi-core/README.md b/crates/searchi-core/README.md index def216e..1b8aa07 100644 --- a/crates/searchi-core/README.md +++ b/crates/searchi-core/README.md @@ -4,7 +4,7 @@ The core has no desktop or inference dependencies. Its current retrieval engine combines literal filename matching with SQLite FTS5 BM25 keyword search. It does not implement semantic search. -`SearchIndex::open` creates or migrates schema version 4. Current-schema opens do +`SearchIndex::open` creates or migrates schema version 7. Current-schema opens do not acquire a write lock. WAL allows independent readers to use the committed index while reconciliation runs. All indexing changes are committed atomically; a traversal or plain-text file read error rolls back the entire reconciliation. @@ -73,10 +73,39 @@ apply before limiting results. `search` retains the unfiltered API. Query text i escaped as literal FTS terms, not interpreted as operators. Results are capped at 1,000 and queries at 4,096 bytes. +Retrieval runs complete SQL-filtered lanes: indexed exact filename equality, +literal filename substring, literal path substring, and content-only BM25. +Multiword queries also have a tokenized name/path fallback, so `annual report` +can find `annual_report.md`. Single-token filename identifiers are matched +literally; the text tokenizer still handles punctuation inside document content. +Exact filenames always precede other lanes. Ties use persistent file row IDs, +and keyword ties use relevance then row ID. Selected rows are deduplicated. + +Filename/path substrings of at least three Unicode characters use an +external-content trigram FTS index. The rarest known query trigram supplies a +candidate superset; the whole literal substring is verified on the candidate. +At most 256 unique query trigrams are inspected. This lookup budget does not cap +candidate rows or hide matching results. All size/type/folder/date filters apply +before each lane's SQL LIMIT. Shorter substrings retain complete metadata scans +and can be slow. Broad queries under restrictive filters can also examine many +matching candidates; no universal filtered-query latency claim is made. + +Trigram frequencies use the FTS vocabulary and a bounded 4,096-term cache on each +`SearchIndex`. Initial frequency lookups can be expensive; the cache is cleared +after reconciliation. Keep an independent reader engine alive across searches to +benefit from warm behavior. These frequency estimates affect candidate selection +cost, not correctness. A source query batches all finalists' indexed passage IDs +and selects actual matching passages before choosing one source per file. + +Migration to the new indexes preserves file content and passage IDs and rebuilds +both filename/path and passage indexes atomically. Large migrations can take time +and increase database size; they must not be confused with normal current-schema +opens. The measured Windows synthetic fixture grew from about 295 MiB to 605 MiB. + ## Known limits -- Filename substring lookup uses `instr` and scans metadata. A dedicated substring - index and benchmarks are needed before claiming million-file latency targets. +- Short substring queries and broad restrictive filters can still scan many + metadata candidates. Cold frequency-cache fills exceed the warm filename target. - Files changed without changing size or modification time require a future forced refresh/content-fingerprint option. Timestamp precision depends on the filesystem. Indexing does not yet run a filesystem watcher. @@ -91,5 +120,5 @@ escaped as literal FTS terms, not interpreted as operators. Results are capped a Bounded worker isolation, persistent jobs, and crash-resumable scans remain production milestones. - OCR, media extraction, and real model inference belong to subsequent modules. -- Passage selection currently adds a bounded query per result; query batching and - passage ranking benchmarks remain necessary for large result lists. +- Linux/macOS measurements, realistic mixed-document corpora, cold startup, + memory budgets, and indexing performance remain unverified. diff --git a/crates/searchi-core/benchmarks/README.md b/crates/searchi-core/benchmarks/README.md new file mode 100644 index 0000000..dc891de --- /dev/null +++ b/crates/searchi-core/benchmarks/README.md @@ -0,0 +1,104 @@ +# CPU retrieval performance + +`examples/performance.rs` builds an explicitly synthetic SQLite corpus using the +real core schema and normal FTS triggers. It measures warm core retrieval only; +there is no filesystem scanning, document extraction, model inference, GUI +rendering, or GPU provider in this benchmark. + +The default smoke fixture contains 10,000 file records and 1,000 searchable +document/passages. The acceptance fixture contains 1,000,000 file records and +100,000 searchable documents, one paragraph per document. The remaining records +have filenames but no content. Names and documents are synthetic; they do not +represent a realistic corpus distribution, so passing these measurements alone +would not establish production performance. + +```powershell +cargo run --release -p searchi-core --example performance -- --files 1000000 --documents 100000 --samples 20 --db target/benchmarks/full.db +``` + +Use a new database path when seeding. The harness refuses to overwrite an +existing database. To compare engine versions on the same corpus, use `--reuse true`; +this validates file/document counts and synthetic root ownership before opening +and migrating the existing fixture. It records migration time separately and +reports zero seed time. It seeds at most 5,000 records per transaction, checkpoints +the WAL, then performs three warmups and 20 measured queries per case. Percentiles +use the nearest-rank convention; with 20 samples, p95 is the 19th sorted sample. +The first execution of each query is recorded separately before the remaining +two warmups; this is first use of that query on the current engine, not a cold +OS disk-cache test. Timing uses a monotonic clock. Results include committed database size and fixture +seed time, which must not be described as document extraction throughput. + +Cases cover an exact filename near the end of the corpus, a broad filename +substring, a unique content token, and a broad content token. All queries use the +public search API with limit 20, including actual source-passage selection. Target +p95 is below 50 ms for filenames and below 150 ms for keyword search. A result +above either target is explicitly marked as a failure. Database file bytes exclude +temporary WAL/SHM sidecars and process memory. + +## Reference environment + +- Date: 2026-10-12. +- Windows VM: Windows 11 Pro, version 10.0.26300. +- CPU name exposed by VM: AMD Ryzen 7 7800X3D 8-Core Processor. +- VM allocation: 4 logical processors, 10,666,553,344 bytes RAM (about 9.93 GiB). +- Rust release build, bundled SQLite, CPU only. +- Toolchain: rustc 1.99.0 (b940084d7 2026-09-28), cargo 1.99.0. +- Engine baseline: `a9b6592330d88745eadf2a748bb3a65f62830590`, schema 4. +- Storage device and host scheduling contention are not characterized. + +Run results are stored in `results/`. Linux and macOS measurements are pending. +No cross-platform acceptance claim can be inferred from the Windows VM results. + +## Baseline result + +The full acceptance fixture fails all four latency targets. This is a verified +performance gap, not a production-ready retrieval result. + +| Query case | p50 | p95 | p95 target | Result | +| --- | ---: | ---: | ---: | --- | +| Exact filename | 404.76 ms | 633.82 ms | <50 ms | Failed | +| Broad filename substring | 3,385.76 ms | 4,131.48 ms | <50 ms | Failed | +| Unique content token | 311.63 ms | 355.78 ms | <150 ms | Failed | +| Broad keyword | 2,186.24 ms | 2,746.91 ms | <150 ms | Failed | + +The 1M/100K synthetic fixture took 277.23 seconds to seed and its checkpointed +database occupies 309,338,112 bytes (about 295 MiB). Smaller 10K/1K smoke results +passed the same targets, showing why small-corpus tests must not be used as proof +of scalability. Neither seed time measures document extraction throughput. + +The baseline implementation scanned file metadata for substring matching and used +per-result passage lookups. Its measured behavior warranted a dedicated filename +substring index, candidate-first FTS retrieval, and batched passage selection. +The following section records the resulting implementation and measurements. + +## Optimized result, schema 7 + +The unchanged full fixture was migrated and reused. All four **warm** target +measurements pass in this Windows CPU run. This does not establish production +acceptance for all workloads or operating systems. + +| Query case | Baseline p95 | Optimized p50 | Optimized p95 | First use | Warm target | +| --- | ---: | ---: | ---: | ---: | --- | +| Exact filename | 633.82 ms | 11.66 ms | 13.17 ms | 279.73 ms | Passed | +| Broad filename substring | 4,131.48 ms | 0.39 ms | 1.68 ms | 3.34 ms | Passed | +| Unique content token | 355.78 ms | 0.28 ms | 0.39 ms | 50.08 ms | Passed | +| Broad keyword | 2,746.91 ms | 97.92 ms | 111.54 ms | 101.35 ms | Passed | + +The updated checkpointed database is 634,449,920 bytes (about 605 MiB). The +improvement trades additional indexes and disk space for retrieval latency. +The initial schema-4-to-5 backfill took 19.65 seconds on this fixture, as recorded +in the intermediate report. The final report's 3.84 ms `migration_seconds` is an +already-current-schema open, not a measurement of the initial index backfill. +Indexed equality, trigram candidate retrieval, content-only FTS, and a single +batched source lookup replaced full metadata joins and per-result FTS scans. +Rarest-trigram vocabulary counts are cached in the reader engine. The first-use +exact result exceeds the warm target and is explicitly retained as an outstanding +latency issue; opening a fresh reader for every query discards this cache. + +`windows-cpu-full-optimized-v7.json` is the final measurement. Earlier schema 5/6 +reports are intermediate experiments and retain failed exact-query measurements. +The final run did not overlap another benchmark process. Dedicated host isolation +is still not established. Query distribution and filters are not represented by +this four-case fixture; restrictive broad filters and sub-three-character searches +remain potentially slow, while correctness tests ensure late matching files are +not lost through arbitrary pre-filter caps. diff --git a/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized-v6.json b/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized-v6.json new file mode 100644 index 0000000..a3699ea --- /dev/null +++ b/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized-v6.json @@ -0,0 +1,47 @@ +{ + "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference", + "files": 1000000, + "documents": 100000, + "seed_seconds": 0.0, + "migration_seconds": 0.0055438, + "fixture_reused": true, + "database_bytes": 634449920, + "database_path": "target/benchmarks/full.db", + "architecture": "x86_64", + "os": "windows", + "cpu_only": true, + "queries": [ + { + "query": "invoice_0999999.txt", + "samples": 20, + "p50_ms": 120.81269999999999, + "p95_ms": 137.9349, + "target_p95_ms": 50.0, + "target_met": false + }, + { + "query": "invoice_", + "samples": 20, + "p50_ms": 0.5431, + "p95_ms": 0.7148, + "target_p95_ms": 50.0, + "target_met": true + }, + { + "query": "marker0099999", + "samples": 20, + "p50_ms": 0.5455, + "p95_ms": 0.6604, + "target_p95_ms": 150.0, + "target_met": true + }, + { + "query": "cancellation", + "samples": 20, + "p50_ms": 98.23469999999999, + "p95_ms": 103.2545, + "target_p95_ms": 150.0, + "target_met": true + } + ] +} diff --git a/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized-v7.json b/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized-v7.json new file mode 100644 index 0000000..863a5bb --- /dev/null +++ b/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized-v7.json @@ -0,0 +1,51 @@ +{ + "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference", + "files": 1000000, + "documents": 100000, + "seed_seconds": 0.0, + "migration_seconds": 0.0038446, + "fixture_reused": true, + "database_bytes": 634449920, + "database_path": "target/benchmarks/full.db", + "architecture": "x86_64", + "os": "windows", + "cpu_only": true, + "queries": [ + { + "query": "invoice_0999999.txt", + "samples": 20, + "p50_ms": 11.659600000000001, + "first_query_ms": 279.7262, + "p95_ms": 13.167399999999999, + "target_p95_ms": 50.0, + "target_met": true + }, + { + "query": "invoice_", + "samples": 20, + "p50_ms": 0.3923, + "first_query_ms": 3.3355, + "p95_ms": 1.6800000000000002, + "target_p95_ms": 50.0, + "target_met": true + }, + { + "query": "marker0099999", + "samples": 20, + "p50_ms": 0.2847, + "first_query_ms": 50.083000000000006, + "p95_ms": 0.3948, + "target_p95_ms": 150.0, + "target_met": true + }, + { + "query": "cancellation", + "samples": 20, + "p50_ms": 97.9169, + "first_query_ms": 101.3489, + "p95_ms": 111.5428, + "target_p95_ms": 150.0, + "target_met": true + } + ] +} diff --git a/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized.json b/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized.json new file mode 100644 index 0000000..7198b66 --- /dev/null +++ b/crates/searchi-core/benchmarks/results/windows-cpu-full-optimized.json @@ -0,0 +1,47 @@ +{ + "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference", + "files": 1000000, + "documents": 100000, + "seed_seconds": 0.0, + "migration_seconds": 19.651645, + "fixture_reused": true, + "database_bytes": 633917440, + "database_path": "target/benchmarks/full.db", + "architecture": "x86_64", + "os": "windows", + "cpu_only": true, + "queries": [ + { + "query": "invoice_0999999.txt", + "samples": 20, + "p50_ms": 115.7827, + "p95_ms": 127.39, + "target_p95_ms": 50.0, + "target_met": false + }, + { + "query": "invoice_", + "samples": 20, + "p50_ms": 0.4833, + "p95_ms": 1.734, + "target_p95_ms": 50.0, + "target_met": true + }, + { + "query": "marker0099999", + "samples": 20, + "p50_ms": 0.3963, + "p95_ms": 0.5996, + "target_p95_ms": 150.0, + "target_met": true + }, + { + "query": "cancellation", + "samples": 20, + "p50_ms": 105.1206, + "p95_ms": 112.6383, + "target_p95_ms": 150.0, + "target_met": true + } + ] +} diff --git a/crates/searchi-core/benchmarks/results/windows-cpu-full.json b/crates/searchi-core/benchmarks/results/windows-cpu-full.json new file mode 100644 index 0000000..d696825 --- /dev/null +++ b/crates/searchi-core/benchmarks/results/windows-cpu-full.json @@ -0,0 +1,45 @@ +{ + "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference", + "files": 1000000, + "documents": 100000, + "seed_seconds": 277.2314138, + "database_bytes": 309338112, + "database_path": "target/benchmarks/full.db", + "architecture": "x86_64", + "os": "windows", + "cpu_only": true, + "queries": [ + { + "query": "invoice_0999999.txt", + "samples": 20, + "p50_ms": 404.75899999999996, + "p95_ms": 633.8166, + "target_p95_ms": 50.0, + "target_met": false + }, + { + "query": "invoice_", + "samples": 20, + "p50_ms": 3385.7561, + "p95_ms": 4131.4819, + "target_p95_ms": 50.0, + "target_met": false + }, + { + "query": "marker0099999", + "samples": 20, + "p50_ms": 311.6295, + "p95_ms": 355.7767, + "target_p95_ms": 150.0, + "target_met": false + }, + { + "query": "cancellation", + "samples": 20, + "p50_ms": 2186.2423, + "p95_ms": 2746.9136, + "target_p95_ms": 150.0, + "target_met": false + } + ] +} diff --git a/crates/searchi-core/benchmarks/results/windows-cpu-small-optimized.json b/crates/searchi-core/benchmarks/results/windows-cpu-small-optimized.json new file mode 100644 index 0000000..4f130fb --- /dev/null +++ b/crates/searchi-core/benchmarks/results/windows-cpu-small-optimized.json @@ -0,0 +1,47 @@ +{ + "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference", + "files": 10000, + "documents": 1000, + "seed_seconds": 0.0, + "migration_seconds": 0.2795255, + "fixture_reused": true, + "database_bytes": 6320128, + "database_path": "target/benchmarks/small.db", + "architecture": "x86_64", + "os": "windows", + "cpu_only": true, + "queries": [ + { + "query": "invoice_0009999.txt", + "samples": 20, + "p50_ms": 1.9414, + "p95_ms": 2.3285, + "target_p95_ms": 50.0, + "target_met": true + }, + { + "query": "invoice_", + "samples": 20, + "p50_ms": 0.3281, + "p95_ms": 1.2427, + "target_p95_ms": 50.0, + "target_met": true + }, + { + "query": "marker0000999", + "samples": 20, + "p50_ms": 0.2561, + "p95_ms": 0.40259999999999996, + "target_p95_ms": 150.0, + "target_met": true + }, + { + "query": "cancellation", + "samples": 20, + "p50_ms": 1.2418, + "p95_ms": 1.7393, + "target_p95_ms": 150.0, + "target_met": true + } + ] +} diff --git a/crates/searchi-core/benchmarks/results/windows-cpu-small.json b/crates/searchi-core/benchmarks/results/windows-cpu-small.json new file mode 100644 index 0000000..2003a17 --- /dev/null +++ b/crates/searchi-core/benchmarks/results/windows-cpu-small.json @@ -0,0 +1,45 @@ +{ + "fixture": "synthetic SQLite metadata and text; no filesystem extraction or semantic inference", + "files": 10000, + "documents": 1000, + "seed_seconds": 0.3428201, + "database_bytes": 3207168, + "database_path": "target/benchmarks/small.db", + "architecture": "x86_64", + "os": "windows", + "cpu_only": true, + "queries": [ + { + "query": "invoice_0009999.txt", + "samples": 20, + "p50_ms": 3.0624, + "p95_ms": 3.8085, + "target_p95_ms": 50.0, + "target_met": true + }, + { + "query": "invoice_", + "samples": 20, + "p50_ms": 20.8847, + "p95_ms": 28.4805, + "target_p95_ms": 50.0, + "target_met": true + }, + { + "query": "marker0000999", + "samples": 20, + "p50_ms": 1.7667, + "p95_ms": 2.7321, + "target_p95_ms": 150.0, + "target_met": true + }, + { + "query": "cancellation", + "samples": 20, + "p50_ms": 12.2173, + "p95_ms": 16.5439, + "target_p95_ms": 150.0, + "target_met": true + } + ] +} diff --git a/crates/searchi-core/examples/performance.rs b/crates/searchi-core/examples/performance.rs new file mode 100644 index 0000000..dd2e028 --- /dev/null +++ b/crates/searchi-core/examples/performance.rs @@ -0,0 +1,200 @@ +//! CPU-only synthetic retrieval benchmark. This does not measure filesystem +//! scanning, extraction, embedding inference, or desktop rendering performance. +use anyhow::{bail, Context, Result}; +use rusqlite::{params, Connection}; +use searchi_core::SearchIndex; +use serde::Serialize; +use std::{env, fs, path::PathBuf, time::Instant}; + +#[derive(Serialize)] +struct Measurements { + query: String, + samples: usize, + p50_ms: f64, + first_query_ms: f64, + p95_ms: f64, + target_p95_ms: f64, + target_met: bool, +} + +#[derive(Serialize)] +struct Report { + fixture: &'static str, + files: usize, + documents: usize, + seed_seconds: f64, + migration_seconds: f64, + fixture_reused: bool, + database_bytes: u64, + database_path: String, + architecture: &'static str, + os: &'static str, + cpu_only: bool, + queries: Vec, +} + +fn main() -> Result<()> { + let arguments = env::args().skip(1).collect::>(); + let mut files = 10_000usize; + let mut documents = 1_000usize; + let mut samples = 20usize; + let mut database = PathBuf::from("target/benchmarks/synthetic.db"); + let mut reuse = false; + for pair in arguments.chunks(2) { + if pair.len() != 2 { + bail!("usage: performance --files N --documents N --samples N --db NEW_PATH"); + } + match pair[0].as_str() { + "--files" => files = pair[1].parse()?, + "--documents" => documents = pair[1].parse()?, + "--samples" => samples = pair[1].parse()?, + "--db" => database = PathBuf::from(&pair[1]), + "--reuse" => reuse = pair[1].parse()?, + option => bail!("unknown argument {option}"), + } + } + if files == 0 + || files > 2_000_000 + || documents == 0 + || documents > files + || !(5..=1000).contains(&samples) + { + bail!("files must be1..2000000, documents1..files, samples5..1000"); + } + if database.exists() && !reuse { + bail!( + "benchmark database must be a new path: {}", + database.display() + ); + } + if reuse && !database.exists() { + bail!("reuse requires an existing synthetic fixture"); + } + if let Some(parent) = database.parent() { + fs::create_dir_all(parent)?; + } + // Seed through the actual versioned schema and its normal indexing triggers. + if reuse { + let connection = Connection::open(&database)?; + let (actual_files, actual_documents, foreign_roots): (usize,usize,usize) = connection.query_row( + "SELECT count(*),count(NULLIF(content,'')),sum(root!='synthetic://benchmark') FROM files",[], + |row|Ok((row.get(0)?,row.get(1)?,row.get(2)?)))?; + if (actual_files, actual_documents, foreign_roots) != (files, documents, 0) { + bail!("existing fixture does not match configured synthetic corpus"); + } + } + let migration_started = Instant::now(); + let index = SearchIndex::open(&database)?; + drop(index); + let migration_seconds = migration_started.elapsed().as_secs_f64(); + let start = Instant::now(); + let mut connection = Connection::open(&database)?; + connection.execute_batch("PRAGMA synchronous=NORMAL;")?; + if !reuse { + for batch_start in (0..files).step_by(5000) { + let transaction = connection.transaction()?; + { + let mut statement = transaction.prepare( + "INSERT INTO files(path,root,filename,content,size,mtime,extension) + VALUES (?1,'synthetic://benchmark',?2,?3,?4,0,'txt')", + )?; + let mut passage = transaction.prepare( + "INSERT INTO passages(text,file_path,kind,number) VALUES (?1,?2,'paragraph',1)", + )?; + for number in batch_start..(batch_start + 5000).min(files) { + let filename = format!("invoice_{number:07}.txt"); + let path = format!( + "synthetic://benchmark/folder_{:04}/{filename}", + number % 1000 + ); + let content = if number < documents { + format!("Supplier termination cancellation clause budget analysis marker{number:07}") + } else { + String::new() + }; + statement.execute(params![path, filename, content, content.len() as i64])?; + if number < documents { + passage.execute(params![content, path])?; + } + } + } + transaction.commit()?; + if batch_start % 100_000 == 0 { + eprintln!( + "seeded {} / {files} synthetic file records", + (batch_start + 5000).min(files) + ); + } + } + } + connection.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);")?; + drop(connection); + let seed_seconds = if reuse { + 0.0 + } else { + start.elapsed().as_secs_f64() + }; + let database_bytes = fs::metadata(&database)?.len(); + let index = SearchIndex::open(&database)?; + let exact = format!("invoice_{:07}.txt", files - 1); + let unique_content = format!("marker{:07}", documents - 1); + let cases = [ + (exact, 50.0), + ("invoice_".into(), 50.0), + (unique_content, 150.0), + ("cancellation".into(), 150.0), + ]; + let mut queries = Vec::new(); + for (query, target) in cases { + let first_started = Instant::now(); + index.search(&query, 20)?; + let first_query_ms = first_started.elapsed().as_secs_f64() * 1000.0; + for _ in 0..2 { + index.search(&query, 20)?; + } + let mut durations = Vec::with_capacity(samples); + for _ in 0..samples { + let started = Instant::now(); + let results = index + .search(&query, 20) + .with_context(|| format!("benchmark query {query}"))?; + if results.is_empty() { + bail!("synthetic query {query} unexpectedly returned no results"); + } + std::hint::black_box(results); + durations.push(started.elapsed().as_secs_f64() * 1000.0); + } + durations.sort_by(f64::total_cmp); + let percentile = |fraction: f64| { + durations[((durations.len() as f64 * fraction).ceil() as usize).saturating_sub(1)] + }; + let p95_ms = percentile(0.95); + eprintln!("query {query:?}: p95 {p95_ms:.3} ms; target < {target:.0} ms"); + queries.push(Measurements { + query, + samples, + p50_ms: percentile(0.5), + first_query_ms, + p95_ms, + target_p95_ms: target, + target_met: p95_ms < target, + }); + } + let report = Report { + fixture: + "synthetic SQLite metadata and text; no filesystem extraction or semantic inference", + files, + documents, + seed_seconds, + migration_seconds, + fixture_reused: reuse, + database_bytes, + database_path: database.display().to_string(), + architecture: env::consts::ARCH, + os: env::consts::OS, + cpu_only: true, + queries, + }; + println!("{}", serde_json::to_string_pretty(&report)?); + Ok(()) +} diff --git a/crates/searchi-core/src/lib.rs b/crates/searchi-core/src/lib.rs index 512a259..14411e1 100644 --- a/crates/searchi-core/src/lib.rs +++ b/crates/searchi-core/src/lib.rs @@ -5,16 +5,18 @@ //! Worker isolation remains a production milestone. mod extraction; use anyhow::{bail, Context, Result}; -use rusqlite::{params, Connection, OptionalExtension}; +use rusqlite::{params, params_from_iter, types::Value, Connection, OptionalExtension}; use serde::Serialize; use std::{ + cell::RefCell, + collections::{BTreeSet, HashMap}, fs, path::{Path, PathBuf}, time::UNIX_EPOCH, }; use walkdir::WalkDir; -const SCHEMA_VERSION: i64 = 4; +const SCHEMA_VERSION: i64 = 7; const MAX_RESULTS: usize = 1000; #[derive(Debug, Default, Serialize)] @@ -92,6 +94,7 @@ pub struct IndexStats { pub struct SearchIndex { connection: Connection, database_path: Option, + gram_frequencies: RefCell>, } impl SearchIndex { @@ -159,16 +162,66 @@ impl SearchIndex { CREATE TRIGGER IF NOT EXISTS files_passages_delete AFTER DELETE ON files BEGIN DELETE FROM passages WHERE file_path=old.path; END;", )?; - if version != 4 { + if version > 0 && version < 4 { // Re-extract records from older schemas to populate genuine locations. initialization.execute("UPDATE files SET mtime=-1", [])?; } - initialization.execute_batch("PRAGMA user_version=4;")?; + if version < 5 { + initialization.execute_batch("CREATE TABLE passages_new( + id INTEGER PRIMARY KEY, text TEXT NOT NULL, file_path TEXT NOT NULL, + kind TEXT NOT NULL, number INTEGER NOT NULL); + INSERT INTO passages_new SELECT rowid,text,file_path,kind,number FROM passages; + DROP TRIGGER files_passages_delete; + DROP TABLE passages; + ALTER TABLE passages_new RENAME TO passages; + CREATE TRIGGER files_passages_delete AFTER DELETE ON files BEGIN + DELETE FROM passages WHERE file_path=old.path; END; + CREATE INDEX passages_file ON passages(file_path,id); + CREATE VIRTUAL TABLE passage_search USING fts5(text,content='passages',content_rowid='id',tokenize='unicode61'); + CREATE TRIGGER passages_insert AFTER INSERT ON passages BEGIN + INSERT INTO passage_search(rowid,text) VALUES(new.id,new.text); END; + CREATE TRIGGER passages_delete AFTER DELETE ON passages BEGIN + INSERT INTO passage_search(passage_search,rowid,text) VALUES('delete',old.id,old.text); END; + CREATE TRIGGER passages_update AFTER UPDATE ON passages BEGIN + INSERT INTO passage_search(passage_search,rowid,text) VALUES('delete',old.id,old.text); + INSERT INTO passage_search(rowid,text) VALUES(new.id,new.text); END; + INSERT INTO passage_search(passage_search) VALUES('rebuild'); + CREATE INDEX files_filename ON files(lower(filename),id); + CREATE VIRTUAL TABLE names USING fts5(filename,path,content='files',content_rowid='id',tokenize='trigram'); + CREATE TRIGGER files_names_insert AFTER INSERT ON files BEGIN + INSERT INTO names(rowid,filename,path) VALUES(new.id,new.filename,new.path); END; + CREATE TRIGGER files_names_delete AFTER DELETE ON files BEGIN + INSERT INTO names(names,rowid,filename,path) VALUES('delete',old.id,old.filename,old.path); END; + CREATE TRIGGER files_names_update AFTER UPDATE ON files BEGIN + INSERT INTO names(names,rowid,filename,path) VALUES('delete',old.id,old.filename,old.path); + INSERT INTO names(rowid,filename,path) VALUES(new.id,new.filename,new.path); END; + INSERT INTO names(names) VALUES('rebuild'); + PRAGMA user_version=5;")?; + } + if version < 6 { + initialization.execute_batch("CREATE VIRTUAL TABLE content_search USING fts5(content, + content='files',content_rowid='id',tokenize='unicode61'); + CREATE TRIGGER files_content_insert AFTER INSERT ON files WHEN new.content!='' BEGIN + INSERT INTO content_search(rowid,content) VALUES(new.id,new.content); END; + CREATE TRIGGER files_content_delete AFTER DELETE ON files WHEN old.content!='' BEGIN + INSERT INTO content_search(content_search,rowid,content) VALUES('delete',old.id,old.content); END; + CREATE TRIGGER files_content_update AFTER UPDATE ON files BEGIN + INSERT INTO content_search(content_search,rowid,content) + SELECT 'delete',old.id,old.content WHERE old.content!=''; + INSERT INTO content_search(rowid,content) SELECT new.id,new.content WHERE new.content!=''; END; + INSERT INTO content_search(rowid,content) SELECT id,content FROM files WHERE content!=''; + PRAGMA user_version=6;")?; + } + initialization.execute_batch( + "CREATE VIRTUAL TABLE IF NOT EXISTS names_vocab USING fts5vocab(names,'row'); + PRAGMA user_version=7;", + )?; initialization.commit()?; } Ok(Self { connection, database_path, + gram_frequencies: RefCell::new(HashMap::new()), }) } @@ -300,13 +353,14 @@ impl SearchIndex { )?; transaction.execute("DELETE FROM seen_paths", [])?; transaction.commit()?; + self.gram_frequencies.borrow_mut().clear(); Ok(report) } /// Literal input is escaped before FTS parsing; user text cannot inject operators. /// Exact filenames precede partial filenames, which precede content matches. - /// Filename substring matching currently scans metadata; a dedicated substring - /// index is required before claiming million-file latency targets. + /// Substrings of at least three Unicode characters use a trigram index. + /// Shorter substrings retain complete matching using a slower metadata scan. pub fn search(&self, query: &str, limit: usize) -> Result> { self.search_with_options(query, limit, &SearchOptions::default()) } @@ -370,65 +424,237 @@ impl SearchIndex { .map(|term| format!("\"{}\"", term.replace('"', "\"\""))) .collect::>() .join(" AND "); - let mut statement = self.connection.prepare( - "WITH hits AS ( - SELECT rowid AS id, bm25(search, 8.0, 2.0, 1.0) AS rank, - snippet(search, 2, '', '', ' … ', 24) AS excerpt - FROM search WHERE search MATCH ?1) - SELECT f.path, f.filename, COALESCE(h.excerpt, ''), COALESCE(h.rank, 0), - CASE WHEN lower(f.filename)=lower(?2) THEN 0 - WHEN instr(lower(f.filename), lower(?2)) > 0 THEN 1 ELSE 2 END AS tier, f.extraction_error - FROM files f LEFT JOIN hits h ON h.id=f.id - WHERE (h.id IS NOT NULL OR instr(lower(f.filename), lower(?2)) > 0) - AND (?4 IS NULL OR substr(f.path,1,length(?4))=?4) - AND (?5 IS NULL OR f.extension=?5) - AND (?6 IS NULL OR f.size>=?6) AND (?7 IS NULL OR f.size<=?7) - AND (?8 IS NULL OR f.mtime/1000000>=?8) AND (?9 IS NULL OR f.mtime/1000000<=?9) - ORDER BY tier, h.rank, f.path LIMIT ?3", - )?; - let mut results = statement - .query_map( - params![ - fts, - query, - limit.min(MAX_RESULTS) as i64, - folder_prefix, - extension, - min_size, - max_size, - options.modified_after_unix_ms, - options.modified_before_unix_ms - ], - |row| { - let tier: i64 = row.get(4)?; - Ok(SearchResult { - path: row.get(0)?, - filename: row.get(1)?, - snippet: row.get(2)?, - score: -row.get::<_, f64>(3)?, - match_kind: match tier { - 0 => "exact_filename", - 1 => "filename", - _ => "content", - } - .to_owned(), - source: None, - extraction_error: row.get(5)?, - }) + let limit = limit.min(MAX_RESULTS); + let filters = " AND (?3 IS NULL OR substr(f.path,1,length(?3))=?3) + AND (?4 IS NULL OR f.extension=?4) + AND (?5 IS NULL OR f.size>=?5) AND (?6 IS NULL OR f.size<=?6) + AND (?7 IS NULL OR f.mtime/1000000>=?7) AND (?8 IS NULL OR f.mtime/1000000<=?8)"; + let filter_values = [ + folder_prefix.map_or(Value::Null, Value::Text), + extension.map_or(Value::Null, Value::Text), + min_size.map_or(Value::Null, Value::Integer), + max_size.map_or(Value::Null, Value::Integer), + options + .modified_after_unix_ms + .map_or(Value::Null, Value::Integer), + options + .modified_before_unix_ms + .map_or(Value::Null, Value::Integer), + ]; + let mut results = Vec::new(); + let mut selected = Vec::::new(); + // Each lane filters its entire candidate set before LIMIT. There is no + // hidden pre-filter candidate cap that can hide later matching files. + let trigram = query.chars().count() >= 3; + // A long FTS phrase may walk common leading trigram postings even when + // later trigrams are rare. Use the rarest known trigram as a candidate + // superset, then verify the whole literal substring. The 256-term lookup + // budget can only enlarge candidates; it never caps results. Frequency + // lookups are cached because FTS vocabulary counts have a first-use cost. + let characters = query.chars().collect::>(); + let grams = characters + .windows(3) + .map(|window| window.iter().collect::()) + .collect::>() + .into_iter() + .take(256) + .collect::>(); + let mut vocabulary = self + .connection + .prepare("SELECT doc FROM names_vocab WHERE term=?1")?; + let mut cache = self.gram_frequencies.borrow_mut(); + if cache.len() + grams.len() > 4096 { + cache.clear(); + } + let mut rare_first = grams + .into_iter() + .map(|gram| { + let term = gram.to_lowercase(); + let frequency = if let Some(frequency) = cache.get(&term) { + *frequency + } else { + let frequency = vocabulary + .query_row([&term], |row| row.get::<_, i64>(0)) + .optional()? + .unwrap_or(0); + cache.insert(term, frequency); + frequency + }; + Ok((frequency, gram)) + }) + .collect::>>()?; + rare_first.sort(); + drop(cache); + let grams = rare_first + .into_iter() + .take(1) + .map(|(_, gram)| format!("\"{}\"", gram.replace('"', "\"\""))) + .collect::>() + .join(" AND "); + let mut lanes = vec![ + ( + "exact_filename", + "files f".to_owned(), + "lower(f.filename)=lower(?1)".to_owned(), + query.to_owned(), + "f.id", + ), + ( + "filename", + if trigram { + "names CROSS JOIN files f ON f.id=names.rowid".into() + } else { + "files f".into() }, - )? - .collect::>>()?; - let mut passages = self.connection.prepare("SELECT kind,number,snippet(passages,0,'','',' … ',24) - FROM passages WHERE passages MATCH ?1 AND file_path=?2 ORDER BY bm25(passages),rowid LIMIT 1")?; - for result in &mut results { - let passage: Option<(String, u32, String)> = passages - .query_row(params![fts, result.path], |row| { - Ok((row.get(0)?, row.get(1)?, row.get(2)?)) - }) - .optional()?; - if let Some((kind, number, snippet)) = passage { - result.source = Some(SourceLocation { kind, number }); - result.snippet = snippet; + if trigram { + "names MATCH ?1 AND instr(lower(f.filename),lower(?2))>0 AND lower(f.filename)!=lower(?2)".into() + } else { + "instr(lower(f.filename),lower(?1))>0 AND lower(f.filename)!=lower(?2)".into() + }, + if trigram { + format!("filename:({grams})") + } else { + query.into() + }, + if trigram { "names.rowid" } else { "f.id" }, + ), + ( + "path", + if trigram { + "names CROSS JOIN files f ON f.id=names.rowid".into() + } else { + "files f".into() + }, + if trigram { + "names MATCH ?1 AND instr(lower(f.path),lower(?2))>0 AND instr(lower(f.filename),lower(?2))=0".into() + } else { + "instr(lower(f.path),lower(?1))>0 AND instr(lower(f.filename),lower(?2))=0" + .into() + }, + if trigram { + format!("path:({grams})") + } else { + query.into() + }, + if trigram { "names.rowid" } else { "f.id" }, + ), + ( + "content", + "content_search CROSS JOIN files f ON f.id=content_search.rowid".into(), + "content_search MATCH ?1".into(), + fts.clone(), + "rank,f.id", + ), + ]; + // Single-token names/identifiers use literal substring matching. A + // multiword query can also match token-separated names such as annual_report. + if query.split_whitespace().count() > 1 { + lanes.push(( + "name_terms", + "search CROSS JOIN files f ON f.id=search.rowid".into(), + "search MATCH ?1 AND rank MATCH 'bm25(8.0,2.0,1.0)'".into(), + fts.clone(), + "rank,f.id", + )); + } + for (kind, from, condition, expression, ordering) in lanes { + if results.len() == limit { + break; + } + let exclusions = if selected.is_empty() { + String::new() + } else { + format!( + " AND f.id NOT IN ({})", + selected + .iter() + .map(i64::to_string) + .collect::>() + .join(",") + ) + }; + let rank = if matches!(kind, "content" | "name_terms") { + "rank" + } else { + "0.0" + }; + let snippet = if kind == "content" { + "snippet(content_search,0,'','',' … ',24)" + } else if kind == "name_terms" { + "snippet(search,2,'','',' … ',24)" + } else { + "''" + }; + let sql = format!( + "SELECT f.id,f.path,f.filename,{snippet},{rank},f.extraction_error FROM {from} + WHERE {condition} {filters} {exclusions} ORDER BY {ordering} LIMIT ?9" + ); + let mut values = vec![Value::Text(expression), Value::Text(query.into())]; + values.extend(filter_values.iter().cloned()); + values.push(Value::Integer((limit - results.len()) as i64)); + let mut statement = self.connection.prepare(&sql)?; + let matches = statement + .query_map(params_from_iter(values), |row| { + Ok(( + row.get::<_, i64>(0)?, + SearchResult { + path: row.get(1)?, + filename: row.get(2)?, + snippet: row.get(3)?, + score: -row.get::<_, f64>(4)?, + match_kind: kind.into(), + source: None, + extraction_error: row.get(5)?, + }, + )) + })? + .collect::>>()?; + for (id, result) in matches { + selected.push(id); + results.push(result); + } + } + if !results.is_empty() { + // Ownership has a B-tree index. The rowid IN constraint intersects + // the FTS postings with only the actual finalists' passage IDs. + let placeholders = (0..results.len()) + .map(|_| "?") + .collect::>() + .join(","); + let sql = format!("WITH matches AS MATERIALIZED ( + SELECT p.file_path,p.kind,p.number,p.id,snippet(passage_search,0,'','',' … ',24) AS excerpt, + bm25(passage_search) AS relevance + FROM passage_search JOIN passages p ON p.id=passage_search.rowid + WHERE passage_search MATCH ? AND passage_search.rowid IN ( + SELECT id FROM passages WHERE file_path IN ({placeholders}))) + SELECT file_path,kind,number,excerpt FROM ( + SELECT *,row_number() OVER(PARTITION BY file_path ORDER BY relevance,id) AS ordinal FROM matches) + WHERE ordinal=1"); + let values = std::iter::once(Value::Text(fts)).chain( + results + .iter() + .map(|result| Value::Text(result.path.clone())), + ); + let mut statement = self.connection.prepare(&sql)?; + let sources = statement + .query_map(params_from_iter(values), |row| { + Ok(( + row.get::<_, String>(0)?, + ( + SourceLocation { + kind: row.get(1)?, + number: row.get(2)?, + }, + row.get::<_, String>(3)?, + ), + )) + })? + .collect::>>()?; + for result in &mut results { + if let Some((source, snippet)) = sources.get(&result.path) { + result.source = Some(source.clone()); + result.snippet = snippet.clone(); + } } } Ok(results) diff --git a/crates/searchi-core/tests/retrieval_lanes.rs b/crates/searchi-core/tests/retrieval_lanes.rs new file mode 100644 index 0000000..01e8d23 --- /dev/null +++ b/crates/searchi-core/tests/retrieval_lanes.rs @@ -0,0 +1,125 @@ +use anyhow::Result; +use searchi_core::{SearchIndex, SearchOptions}; +use std::fs; + +#[test] +fn filters_do_not_hide_late_matches_in_any_lane() -> Result<()> { + let root = tempfile::tempdir()?; + let rejected = root.path().join("early"); + let wanted = root.path().join("late"); + fs::create_dir(&rejected)?; + fs::create_dir(&wanted)?; + let db = root.path().join("index.db"); + let mut index = SearchIndex::open(&db)?; + for number in 0..2000 { + fs::write( + rejected.join(format!("shared_{number:04}.txt")), + "shared retrieval term", + )?; + } + index.index_folder(&rejected)?; + fs::write( + wanted.join("shared_last.md"), + "shared retrieval term unique-evidence", + )?; + index.index_folder(&wanted)?; + let options = SearchOptions { + folder: Some(wanted.clone()), + extension: Some("md".into()), + ..Default::default() + }; + assert_eq!( + index.search_with_options("shared", 1, &options)?[0].filename, + "shared_last.md" + ); + assert_eq!( + index.search_with_options("retrieval", 1, &options)?[0].filename, + "shared_last.md" + ); + assert_eq!( + index.search_with_options("ha", 1, &options)?[0].filename, + "shared_last.md" + ); + let result = index.search_with_options("unique-evidence", 1, &options)?; + assert!(result[0].source.is_some()); + assert!(result[0].snippet.contains("unique-evidence")); + Ok(()) +} + +#[test] +fn exact_priority_literal_identifiers_and_index_lifecycle() -> Result<()> { + let root = tempfile::tempdir()?; + fs::create_dir(root.path().join("contracts"))?; + fs::write( + root.path().join("name_[%].txt"), + "identifier ABC-123 and term", + )?; + fs::write(root.path().join("name_[%].txt.backup"), "term")?; + fs::write( + root.path().join("contracts").join("other.md"), + "name identifier ABC-123", + )?; + fs::write(root.path().join("annual_report.md"), "unrelated text")?; + fs::write(root.path().join("abc_missing_xyz.txt"), "unrelated text")?; + let mut index = SearchIndex::open(":memory:")?; + index.index_folder(root.path())?; + let result = index.search("NAME_[%].TXT", 10)?; + assert_eq!(result[0].match_kind, "exact_filename"); + assert_eq!(result[1].match_kind, "filename"); + assert_eq!(index.search("contracts", 1)?[0].match_kind, "path"); + assert_eq!(index.search("ABC-123", 10)?.len(), 2); + assert_eq!( + index.search("annual report", 1)?[0].filename, + "annual_report.md" + ); + assert!(index.search("abc_xyz", 10)?.is_empty()); + let first = index + .search("name", 10)? + .into_iter() + .map(|row| row.path) + .collect::>(); + let second = index + .search("name", 10)? + .into_iter() + .map(|row| row.path) + .collect::>(); + assert_eq!(first, second); + fs::rename( + root.path().join("name_[%].txt"), + root.path().join("replacement.txt"), + )?; + fs::remove_file(root.path().join("name_[%].txt.backup"))?; + index.index_folder(root.path())?; + assert!(index.search("name_[%].txt", 10)?.is_empty()); + assert_eq!(index.search("placement", 1)?[0].filename, "replacement.txt"); + Ok(()) +} + +#[test] +fn v4_migration_backfills_names_and_preserves_passage_ids() -> Result<()> { + let root = tempfile::tempdir()?; + let db = root.path().join("old.db"); + let connection = rusqlite::Connection::open(&db)?; + connection.execute_batch("CREATE TABLE files(id INTEGER PRIMARY KEY,path TEXT NOT NULL UNIQUE, + root TEXT NOT NULL,filename TEXT NOT NULL,content TEXT NOT NULL,size INTEGER NOT NULL,mtime INTEGER NOT NULL, + extension TEXT NOT NULL,extraction_error TEXT); + CREATE VIRTUAL TABLE search USING fts5(filename,path,content,content='files',content_rowid='id',tokenize='unicode61'); + CREATE VIRTUAL TABLE passages USING fts5(text,file_path UNINDEXED,kind UNINDEXED,number UNINDEXED); + INSERT INTO files VALUES(41,'/fixture/report.pdf','/fixture','report.pdf','preserved evidence',18,123,'pdf',NULL); + INSERT INTO search(search) VALUES('rebuild'); + INSERT INTO passages(rowid,text,file_path,kind,number) VALUES(88,'preserved evidence','/fixture/report.pdf','page',7); + PRAGMA user_version=4;")?; + drop(connection); + let index = SearchIndex::open(&db)?; + assert_eq!(index.search("port.p", 1)?[0].filename, "report.pdf"); + assert_eq!( + index.search("evidence", 1)?[0] + .source + .as_ref() + .unwrap() + .number, + 7 + ); + assert_eq!(index.passage_batch(0, 1)?[0].id, 88); + Ok(()) +}