feat: add desktop document search and verified native CPU embeddings
Core CI / core (ubuntu-latest) (push) Failing after 1m22s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 3m46s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
Core CI / core (ubuntu-latest) (push) Failing after 1m22s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 3m46s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
This commit is contained in:
karl committed
1 parent
f77112db97
commit
a9b6592330
46 files changed
+10794
-77
No files matched your search
@@ -14,3 +14,29 @@ jobs:
|
||||
- run: cargo fmt --all -- --check
|
||||
- run: cargo clippy --workspace --all-targets -- -D warnings
|
||||
- run: cargo test --workspace --locked
|
||||
desktop:
|
||||
strategy:
|
||||
matrix:
|
||||
os: [windows-latest, ubuntu-latest, macos-latest]
|
||||
runs-on: ${{ matrix.os }}
|
||||
defaults:
|
||||
run:
|
||||
working-directory: apps/desktop
|
||||
env:
|
||||
WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS: --disable-gpu
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
components: rustfmt, clippy
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 22
|
||||
- if: runner.os == 'Linux'
|
||||
run: sudo apt-get update && sudo apt-get install -y libwebkit2gtk-4.1-dev build-essential libssl-dev librsvg2-dev libayatana-appindicator3-dev
|
||||
- run: npm ci
|
||||
- run: npm run format:check
|
||||
- run: npm run build
|
||||
- run: cargo fmt --manifest-path src-tauri/Cargo.toml -- --check
|
||||
- run: cargo clippy --manifest-path src-tauri/Cargo.toml --all-targets --locked -- -D warnings
|
||||
- run: cargo test --manifest-path src-tauri/Cargo.toml --locked
|
||||
@@ -14,3 +14,29 @@ jobs:
|
||||
- run: cargo fmt --all -- --check
|
||||
- run: cargo clippy --workspace --all-targets -- -D warnings
|
||||
- run: cargo test --workspace --locked
|
||||
desktop:
|
||||
strategy:
|
||||
matrix:
|
||||
os: [windows-latest, ubuntu-latest, macos-latest]
|
||||
runs-on: ${{ matrix.os }}
|
||||
defaults:
|
||||
run:
|
||||
working-directory: apps/desktop
|
||||
env:
|
||||
WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS: --disable-gpu
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
components: rustfmt, clippy
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 22
|
||||
- if: runner.os == 'Linux'
|
||||
run: sudo apt-get update && sudo apt-get install -y libwebkit2gtk-4.1-dev build-essential libssl-dev librsvg2-dev libayatana-appindicator3-dev
|
||||
- run: npm ci
|
||||
- run: npm run format:check
|
||||
- run: npm run build
|
||||
- run: cargo fmt --manifest-path src-tauri/Cargo.toml -- --check
|
||||
- run: cargo clippy --manifest-path src-tauri/Cargo.toml --all-targets --locked -- -D warnings
|
||||
- run: cargo test --manifest-path src-tauri/Cargo.toml --locked
|
||||
@@ -5,3 +5,4 @@ dist/
|
||||
*.db-shm
|
||||
*.db-wal
|
||||
.searchi/
|
||||
/apps/desktop/src-tauri/gen/
|
||||
Generated
+1156
-3
File diff suppressed because it is too large.
Load diff
+1
-1
@@ -1,5 +1,5 @@
|
||||
[workspace]
|
||||
members = ["crates/searchi-core", "crates/searchi-cli"]
|
||||
members = ["crates/searchi-core", "crates/searchi-cli", "crates/searchi-semantic"]
|
||||
resolver = "2"
|
||||
|
||||
[workspace.package]
|
||||
|
||||
@@ -7,8 +7,9 @@ production readiness is tracked in [the roadmap](docs/ROADMAP.md).
|
||||
|
||||
`crates/searchi-core` owns persistent indexing and retrieval. `crates/searchi-cli`
|
||||
provides a headless interface. `apps/desktop` hosts the Tauri desktop app.
|
||||
The initial implementation provides lexical search; semantic and media search
|
||||
are explicit upcoming milestones, not simulated features.
|
||||
The initial implementation provides lexical search over filenames, bounded text,
|
||||
PDF pages and DOCX paragraphs. Semantic and media search are explicit upcoming
|
||||
milestones; a genuine CPU model proof is included separately.
|
||||
|
||||
All testing in the Windows development VM uses CPU only. Indexed content stays
|
||||
local. Treat the index as sensitive: it contains extracted document text.
|
||||
@@ -25,6 +26,7 @@ cargo run -p searchi-cli -- --help
|
||||
```
|
||||
|
||||
Desktop development instructions live in `apps/desktop/README.md`.
|
||||
Platform and CI details live in [the build guide](docs/BUILDING.md).
|
||||
Native CI builds must pass on all supported systems before release. Cross-platform
|
||||
source support does not imply signed macOS installers can be built on Windows.
|
||||
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
# Searchi desktop
|
||||
|
||||
This Tauri 2 shell calls the same Rust core as the CLI. It provides real folder indexing, native folder selection, filename/content search, folder/extension/size/date filters, keyboard navigation, escaped text previews, and native open/reveal actions. There are no mocked results or embeddings. Semantic retrieval, a global shortcut, watchers, and release installers remain roadmap work.
|
||||
|
||||
## Develop
|
||||
|
||||
Install Node.js LTS, Rust stable, and the platform prerequisites documented at https://v2.tauri.app/start/prerequisites/. Windows requires MSVC build tools and WebView2; Linux requires WebKitGTK and system development packages; macOS requires Xcode command line tools.
|
||||
|
||||
```sh
|
||||
cd apps/desktop
|
||||
npm ci
|
||||
npm run build
|
||||
npm run tauri dev
|
||||
```
|
||||
|
||||
`npm run dev` previews layout in a browser but cannot index or search files. The app reports this limitation explicitly. Press Ctrl+K (Cmd+K on macOS) to focus search; arrow keys navigate results. Choose a folder through the native dialog or enter an absolute path. Open/reveal commands canonicalize the path, verify it is a current file, and require exact membership in the index. The webview has no general opener or filesystem capability. Page/paragraph source locations are displayed when supplied by the core; opening at that exact location is future work.
|
||||
|
||||
The desktop index is stored in the platform app-data directory under `dev.searchi.desktop`. Search opens independent core connections while indexing runs on a blocking worker; a guard limits this shell to one indexing job at a time. All model testing on the Windows VM must remain CPU only. This shell configures no GPU provider.
|
||||
|
||||
For Windows VM UI testing, disable WebView2 GPU rendering as well, before starting Tauri in PowerShell:
|
||||
|
||||
```powershell
|
||||
$env:WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS = '--disable-gpu'
|
||||
npm.cmd run tauri dev
|
||||
```
|
||||
|
||||
## Verification and packaging
|
||||
|
||||
`npm run build` checks strict TypeScript and creates production frontend assets. `npm run tauri build -- --no-bundle` compiles the native shell when platform prerequisites are installed. Installer bundling is intentionally disabled until release icons, signing and native platform checks are complete. Native Windows, macOS, and Linux checks must be recorded separately; a frontend build does not prove the native shell works.
|
||||
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1" />
|
||||
<title>Searchi</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script type="module" src="/src/main.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
Generated
+1442
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"name": "searchi-desktop",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite --host 127.0.0.1",
|
||||
"build": "tsc --noEmit && vite build",
|
||||
"tauri": "tauri",
|
||||
"format": "prettier --write src index.html vite.config.ts tsconfig.json src-tauri/tauri.conf.json src-tauri/capabilities/default.json",
|
||||
"format:check": "prettier --check src index.html vite.config.ts tsconfig.json src-tauri/tauri.conf.json src-tauri/capabilities/default.json"
|
||||
},
|
||||
"dependencies": {
|
||||
"@tauri-apps/api": "^2.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@tauri-apps/cli": "^2.0.0",
|
||||
"prettier": "3.6.2",
|
||||
"typescript": "^5.7.0",
|
||||
"vite": "^6.0.0"
|
||||
}
|
||||
}
|
||||
Generated
+5408
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,24 @@
|
||||
[package]
|
||||
name = "searchi-desktop"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
description = "Local Searchi desktop shell"
|
||||
license = "MIT"
|
||||
|
||||
[workspace]
|
||||
|
||||
[features]
|
||||
custom-protocol = ["tauri/custom-protocol"]
|
||||
|
||||
[build-dependencies]
|
||||
tauri-build = { version = "2", features = [] }
|
||||
|
||||
[dependencies]
|
||||
tauri = { version = "2", features = [] }
|
||||
tauri-plugin-dialog = "2"
|
||||
tauri-plugin-opener = "2"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
searchi-core = { path = "../../../crates/searchi-core" }
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
@@ -0,0 +1,3 @@
|
||||
fn main() {
|
||||
tauri_build::build()
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"$schema": "../gen/schemas/desktop-schema.json",
|
||||
"identifier": "main",
|
||||
"description": "Minimal main-window access; filesystem access is mediated by Searchi commands.",
|
||||
"windows": ["main"],
|
||||
"permissions": ["core:default"]
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1012 B |
Binary file not shown.
|
After Width: | Height: | Size: 990 B |
@@ -0,0 +1,220 @@
|
||||
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
|
||||
|
||||
use searchi_core::{IndexReport, IndexStats, SearchIndex, SearchOptions, SearchResult};
|
||||
use serde::Deserialize;
|
||||
use std::{
|
||||
path::PathBuf,
|
||||
sync::{
|
||||
atomic::{AtomicBool, Ordering},
|
||||
Arc,
|
||||
},
|
||||
};
|
||||
use tauri::{Manager, State};
|
||||
use tauri_plugin_dialog::DialogExt;
|
||||
use tauri_plugin_opener::OpenerExt;
|
||||
|
||||
struct AppState {
|
||||
database: PathBuf,
|
||||
indexing: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
#[derive(Default, Deserialize)]
|
||||
struct Filters {
|
||||
folder: Option<PathBuf>,
|
||||
extension: Option<String>,
|
||||
min_size: Option<u64>,
|
||||
max_size: Option<u64>,
|
||||
modified_after_unix_ms: Option<i64>,
|
||||
modified_before_unix_ms: Option<i64>,
|
||||
}
|
||||
|
||||
impl From<Filters> for SearchOptions {
|
||||
fn from(value: Filters) -> Self {
|
||||
Self {
|
||||
folder: value.folder,
|
||||
extension: value.extension,
|
||||
min_size: value.min_size,
|
||||
max_size: value.max_size,
|
||||
modified_after_unix_ms: value.modified_after_unix_ms,
|
||||
modified_before_unix_ms: value.modified_before_unix_ms,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Each job owns its connection. Slow indexing never holds a desktop mutex.
|
||||
#[tauri::command]
|
||||
async fn search(
|
||||
query: String,
|
||||
filters: Option<Filters>,
|
||||
state: State<'_, AppState>,
|
||||
) -> Result<Vec<SearchResult>, String> {
|
||||
let database = state.database.clone();
|
||||
tauri::async_runtime::spawn_blocking(move || {
|
||||
SearchIndex::open(database)
|
||||
.and_then(|index| {
|
||||
index.search_with_options(&query, 50, &filters.unwrap_or_default().into())
|
||||
})
|
||||
.map_err(|e| e.to_string())
|
||||
})
|
||||
.await
|
||||
.map_err(|e| e.to_string())?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn pick_folder(app: tauri::AppHandle) -> Result<Option<String>, String> {
|
||||
tauri::async_runtime::spawn_blocking(move || {
|
||||
app.dialog()
|
||||
.file()
|
||||
.set_title("Choose a folder to index")
|
||||
.blocking_pick_folder()
|
||||
.map(|path| {
|
||||
path.into_path()
|
||||
.map(|path| path.to_string_lossy().into_owned())
|
||||
.map_err(|e| e.to_string())
|
||||
})
|
||||
.transpose()
|
||||
})
|
||||
.await
|
||||
.map_err(|e| e.to_string())?
|
||||
}
|
||||
|
||||
// No broad opener permission is given to the webview. Every action is checked
|
||||
// against the index and the current filesystem before invoking a native opener.
|
||||
fn validated_result(index: &SearchIndex, path: &str) -> Result<PathBuf, String> {
|
||||
let canonical = std::fs::canonicalize(path).map_err(|e| format!("File is unavailable: {e}"))?;
|
||||
if !canonical.is_file() {
|
||||
return Err("Result is no longer a file.".into());
|
||||
}
|
||||
if !index
|
||||
.contains_path(&canonical.to_string_lossy())
|
||||
.map_err(|e| e.to_string())?
|
||||
{
|
||||
return Err("Only files in your search index can be opened.".into());
|
||||
}
|
||||
Ok(canonical)
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn open_result(
|
||||
path: String,
|
||||
reveal: bool,
|
||||
app: tauri::AppHandle,
|
||||
state: State<'_, AppState>,
|
||||
) -> Result<(), String> {
|
||||
let database = state.database.clone();
|
||||
tauri::async_runtime::spawn_blocking(move || {
|
||||
let index = SearchIndex::open(database).map_err(|e| e.to_string())?;
|
||||
let canonical = validated_result(&index, &path)?;
|
||||
if reveal {
|
||||
app.opener()
|
||||
.reveal_item_in_dir(canonical)
|
||||
.map_err(|e| e.to_string())
|
||||
} else {
|
||||
app.opener()
|
||||
.open_path(canonical.to_string_lossy().into_owned(), None::<&str>)
|
||||
.map_err(|e| e.to_string())
|
||||
}
|
||||
})
|
||||
.await
|
||||
.map_err(|e| e.to_string())?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn stats(state: State<'_, AppState>) -> Result<IndexStats, String> {
|
||||
let database = state.database.clone();
|
||||
tauri::async_runtime::spawn_blocking(move || {
|
||||
SearchIndex::open(database)
|
||||
.and_then(|index| index.stats())
|
||||
.map_err(|e| e.to_string())
|
||||
})
|
||||
.await
|
||||
.map_err(|e| e.to_string())?
|
||||
}
|
||||
|
||||
// Reset the gate on every exit, including extractor failures or task panics.
|
||||
struct IndexGuard(Arc<AtomicBool>);
|
||||
impl Drop for IndexGuard {
|
||||
fn drop(&mut self) {
|
||||
self.0.store(false, Ordering::Release);
|
||||
}
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn index_folder(root: String, state: State<'_, AppState>) -> Result<IndexReport, String> {
|
||||
if state
|
||||
.indexing
|
||||
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire)
|
||||
.is_err()
|
||||
{
|
||||
return Err("A folder is already being indexed. Wait for it to finish.".into());
|
||||
}
|
||||
let guard = IndexGuard(state.indexing.clone());
|
||||
let database = state.database.clone();
|
||||
tauri::async_runtime::spawn_blocking(move || {
|
||||
let _guard = guard;
|
||||
SearchIndex::open(database)
|
||||
.and_then(|mut index| index.index_folder(root))
|
||||
.map_err(|e| e.to_string())
|
||||
})
|
||||
.await
|
||||
.map_err(|e| e.to_string())?
|
||||
}
|
||||
|
||||
fn main() {
|
||||
tauri::Builder::default()
|
||||
.plugin(tauri_plugin_dialog::init())
|
||||
.plugin(tauri_plugin_opener::init())
|
||||
.setup(|app| {
|
||||
let directory = app.path().app_data_dir()?;
|
||||
std::fs::create_dir_all(&directory)?;
|
||||
let database = directory.join("searchi.sqlite");
|
||||
SearchIndex::open(&database)?;
|
||||
app.manage(AppState {
|
||||
database,
|
||||
indexing: Arc::new(AtomicBool::new(false)),
|
||||
});
|
||||
Ok(())
|
||||
})
|
||||
.invoke_handler(tauri::generate_handler![
|
||||
search,
|
||||
stats,
|
||||
index_folder,
|
||||
pick_folder,
|
||||
open_result
|
||||
])
|
||||
.run(tauri::generate_context!())
|
||||
.expect("Searchi desktop could not start");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn indexing_gate_is_released_when_worker_unwinds() {
|
||||
let indexing = Arc::new(AtomicBool::new(true));
|
||||
let gate = indexing.clone();
|
||||
let result = std::panic::catch_unwind(move || {
|
||||
let _guard = IndexGuard(gate);
|
||||
panic!("extractor failure");
|
||||
});
|
||||
assert!(result.is_err());
|
||||
assert!(!indexing.load(Ordering::Acquire));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn opener_rejects_unindexed_and_deleted_files() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let indexed = root.path().join("indexed.txt");
|
||||
std::fs::write(&indexed, "retrieval fixture").unwrap();
|
||||
let mut index = SearchIndex::open(":memory:").unwrap();
|
||||
index.index_folder(root.path()).unwrap();
|
||||
assert!(validated_result(&index, &indexed.to_string_lossy()).is_ok());
|
||||
let unindexed = root.path().join("new.txt");
|
||||
std::fs::write(&unindexed, "not indexed").unwrap();
|
||||
assert!(validated_result(&index, &unindexed.to_string_lossy()).is_err());
|
||||
std::fs::remove_file(&indexed).unwrap();
|
||||
assert!(validated_result(&index, &indexed.to_string_lossy()).is_err());
|
||||
assert!(validated_result(&index, &root.path().to_string_lossy()).is_err());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"$schema": "https://schema.tauri.app/config/2",
|
||||
"productName": "Searchi",
|
||||
"version": "0.1.0",
|
||||
"identifier": "dev.searchi.desktop",
|
||||
"build": {
|
||||
"beforeDevCommand": "npm run dev",
|
||||
"beforeBuildCommand": "npm run build",
|
||||
"devUrl": "http://localhost:1420",
|
||||
"frontendDist": "../dist"
|
||||
},
|
||||
"app": {
|
||||
"windows": [
|
||||
{
|
||||
"title": "Searchi",
|
||||
"width": 1080,
|
||||
"height": 760,
|
||||
"minWidth": 680,
|
||||
"minHeight": 540
|
||||
}
|
||||
],
|
||||
"security": {
|
||||
"csp": "default-src 'self'; connect-src ipc: http://ipc.localhost; style-src 'self' 'unsafe-inline'; img-src 'self' data:"
|
||||
}
|
||||
},
|
||||
"bundle": { "active": false }
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
import { invoke, isTauri } from "@tauri-apps/api/core";
|
||||
import "./style.css";
|
||||
|
||||
interface SearchResult {
|
||||
path: string;
|
||||
filename: string;
|
||||
snippet: string;
|
||||
score: number;
|
||||
match_kind: string;
|
||||
source?: { kind: string; number: number };
|
||||
extraction_error?: string;
|
||||
}
|
||||
interface IndexReport {
|
||||
discovered: number;
|
||||
indexed: number;
|
||||
removed: number;
|
||||
skipped: number;
|
||||
}
|
||||
interface IndexStats {
|
||||
files: number;
|
||||
content_files: number;
|
||||
}
|
||||
|
||||
document.querySelector<HTMLDivElement>("#app")!.innerHTML = `
|
||||
<header><div class="wordmark">searchi<span class="dot">.</span></div><span class="privacy">● Local · CPU only</span></header>
|
||||
<main><section class="intro"><p class="eyebrow">YOUR FILES, WITHIN REACH</p><h1>Find the thing you remember.</h1><p>Search filenames and document content in your selected folders.</p></section>
|
||||
<form id="search-form" role="search"><label class="sr-only" for="query">Search your files</label><span class="search-icon" aria-hidden="true">⌕</span><input id="query" type="search" placeholder="A filename or words from a document…" autocomplete="off" spellcheck="false" maxlength="1000"><kbd>Ctrl K</kbd></form>
|
||||
<div class="toolbar"><span id="index-status">Loading your index…</span><button id="add-folder" type="button">+ Index a folder</button></div>
|
||||
<details id="filters"><summary>Filter results</summary><div class="filters-grid"><label>Within folder<input id="filter-folder" placeholder="Absolute folder path"></label><label>Extension<input id="filter-extension" placeholder="pdf, md, txt…"></label><label>Minimum size (bytes)<input id="filter-min" type="number" min="0" step="1"></label><label>Maximum size (bytes)<input id="filter-max" type="number" min="0" step="1"></label><label>Modified from<input id="filter-after" type="date"></label><label>Modified through<input id="filter-before" type="date"></label></div></details>
|
||||
<form id="folder-form" hidden><label for="folder">Folder to index</label><div class="folder-row"><input id="folder" placeholder="Absolute folder path" required><button id="browse-folder" type="button">Browse…</button><button type="submit">Index folder</button><button id="cancel-folder" type="button">Cancel</button></div><p class="hint">Only choose folders you want searched. Large folders may take a while.</p></form>
|
||||
<p id="notice" role="status" aria-live="polite"></p><div class="workspace"><section aria-label="Search results"><p id="result-summary" class="eyebrow">READY WHEN YOU ARE</p><ul id="results" aria-label="Matching files"></ul><div id="empty"><span aria-hidden="true">⌕</span><h2>Your next discovery starts here.</h2><p>Index a folder, then search a filename or a phrase from a document.</p></div></section><aside id="preview" aria-label="Result preview" hidden><p class="eyebrow">CONTENT PREVIEW</p><h2 id="preview-title"></h2><p id="preview-path" class="path"></p><p id="preview-kind"></p><div class="preview-actions"><button id="open-result" type="button">Open file</button><button id="reveal-result" type="button">Show in folder</button></div><pre id="preview-text"></pre></aside></div>
|
||||
</main><footer>Offline by default<span>↑ ↓ Navigate · Enter Preview · Esc Clear</span></footer>`;
|
||||
|
||||
const element = <T extends HTMLElement>(id: string) =>
|
||||
document.getElementById(id) as T;
|
||||
const query = element<HTMLInputElement>("query");
|
||||
const list = element<HTMLUListElement>("results");
|
||||
const notice = element<HTMLParagraphElement>("notice");
|
||||
let results: SearchResult[] = [];
|
||||
let selected = -1;
|
||||
let requestId = 0;
|
||||
let timer: ReturnType<typeof setTimeout>;
|
||||
|
||||
function showError(error: unknown) {
|
||||
notice.textContent = String(error);
|
||||
notice.className = "error";
|
||||
}
|
||||
function select(index: number, focus = false) {
|
||||
if (!results.length) return;
|
||||
selected = Math.max(0, Math.min(index, results.length - 1));
|
||||
const result = results[selected];
|
||||
list.querySelectorAll<HTMLButtonElement>("button").forEach((button, i) => {
|
||||
button.classList.toggle("selected", i === selected);
|
||||
button.setAttribute("aria-pressed", String(i === selected));
|
||||
if (i === selected && focus) button.focus();
|
||||
});
|
||||
element("preview").hidden = false;
|
||||
element("preview-title").textContent = result.filename;
|
||||
element("preview-path").textContent = result.path;
|
||||
element("preview-kind").textContent =
|
||||
`${result.match_kind} match${result.source ? ` · ${result.source.kind} ${result.source.number}` : ""}`;
|
||||
element("preview-text").textContent =
|
||||
result.snippet ||
|
||||
(result.extraction_error
|
||||
? `Content could not be extracted: ${result.extraction_error}`
|
||||
: "No extracted text is available for this file.");
|
||||
}
|
||||
|
||||
function render() {
|
||||
list.replaceChildren();
|
||||
element("empty").hidden = results.length > 0;
|
||||
element("preview").hidden = true;
|
||||
results.forEach((result, i) => {
|
||||
const row = document.createElement("li");
|
||||
const button = document.createElement("button");
|
||||
button.className = "result";
|
||||
button.type = "button";
|
||||
const title = document.createElement("strong");
|
||||
title.textContent = result.filename;
|
||||
const path = document.createElement("span");
|
||||
path.className = "path";
|
||||
path.textContent = result.path;
|
||||
const snippet = document.createElement("span");
|
||||
snippet.className = "snippet";
|
||||
snippet.textContent = result.snippet;
|
||||
const kind = document.createElement("span");
|
||||
kind.className = "kind";
|
||||
kind.textContent = result.match_kind;
|
||||
button.append(title, kind, path, snippet);
|
||||
button.addEventListener("click", () => select(i));
|
||||
row.append(button);
|
||||
list.append(row);
|
||||
});
|
||||
if (results.length) select(0);
|
||||
}
|
||||
|
||||
async function search() {
|
||||
const id = ++requestId;
|
||||
const value = query.value.trim();
|
||||
if (!value) {
|
||||
results = [];
|
||||
render();
|
||||
element("result-summary").textContent = "READY WHEN YOU ARE";
|
||||
element("empty").querySelector("h2")!.textContent =
|
||||
"Your next discovery starts here.";
|
||||
element("empty").querySelector("p")!.textContent =
|
||||
"Index a folder, then search a filename or a phrase from a document.";
|
||||
return;
|
||||
}
|
||||
element("result-summary").textContent = "SEARCHING…";
|
||||
const started = performance.now();
|
||||
try {
|
||||
const input = (id: string) => element<HTMLInputElement>(id).value.trim();
|
||||
const size = (id: string) => {
|
||||
const text = input(id);
|
||||
if (!text) return null;
|
||||
const value = Number(text);
|
||||
if (!Number.isSafeInteger(value) || value < 0)
|
||||
throw new Error(
|
||||
"File size must be a nonnegative whole number of bytes.",
|
||||
);
|
||||
return value;
|
||||
};
|
||||
// Date controls represent the user's local calendar day, including its end.
|
||||
const date = (id: string, end: boolean) => {
|
||||
const text = input(id);
|
||||
if (!text) return null;
|
||||
const stamp = new Date(
|
||||
`${text}T${end ? "23:59:59.999" : "00:00:00"}`,
|
||||
).getTime();
|
||||
if (!Number.isFinite(stamp))
|
||||
throw new Error("Enter a valid modification date.");
|
||||
return stamp;
|
||||
};
|
||||
const found = await invoke<SearchResult[]>("search", {
|
||||
query: value,
|
||||
filters: {
|
||||
folder: input("filter-folder") || null,
|
||||
extension: input("filter-extension") || null,
|
||||
min_size: size("filter-min"),
|
||||
max_size: size("filter-max"),
|
||||
modified_after_unix_ms: date("filter-after", false),
|
||||
modified_before_unix_ms: date("filter-before", true),
|
||||
},
|
||||
});
|
||||
if (id !== requestId) return;
|
||||
results = found;
|
||||
selected = -1;
|
||||
render();
|
||||
element("result-summary").textContent =
|
||||
`${found.length} RESULTS · ${Math.round(performance.now() - started)} MS`;
|
||||
if (!found.length) {
|
||||
element("empty").querySelector("h2")!.textContent =
|
||||
"No matching files yet.";
|
||||
element("empty").querySelector("p")!.textContent =
|
||||
"Try a shorter phrase, an exact filename, or index another folder.";
|
||||
}
|
||||
} catch (error) {
|
||||
if (id === requestId) {
|
||||
showError(error);
|
||||
element("result-summary").textContent = "SEARCH FAILED";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshStats() {
|
||||
const stats = await invoke<IndexStats>("stats");
|
||||
element("index-status").textContent =
|
||||
`${stats.files.toLocaleString()} files · ${stats.content_files.toLocaleString()} with searchable content`;
|
||||
}
|
||||
|
||||
query.addEventListener("input", () => {
|
||||
++requestId;
|
||||
clearTimeout(timer);
|
||||
timer = setTimeout(search, 100);
|
||||
});
|
||||
element("search-form").addEventListener("submit", (event) => {
|
||||
event.preventDefault();
|
||||
clearTimeout(timer);
|
||||
void search();
|
||||
});
|
||||
element("add-folder").addEventListener("click", () => {
|
||||
element("folder-form").hidden = false;
|
||||
element("folder").focus();
|
||||
});
|
||||
element("cancel-folder").addEventListener("click", () => {
|
||||
element("folder-form").hidden = true;
|
||||
query.focus();
|
||||
});
|
||||
element("browse-folder").addEventListener("click", async () => {
|
||||
try {
|
||||
const folder = await invoke<string | null>("pick_folder");
|
||||
if (folder) element<HTMLInputElement>("folder").value = folder;
|
||||
} catch (error) {
|
||||
showError(error);
|
||||
}
|
||||
});
|
||||
element("filters").addEventListener("change", () => {
|
||||
++requestId;
|
||||
clearTimeout(timer);
|
||||
void search();
|
||||
});
|
||||
async function openSelected(reveal: boolean) {
|
||||
if (selected < 0 || !results[selected]) return;
|
||||
try {
|
||||
await invoke("open_result", { path: results[selected].path, reveal });
|
||||
} catch (error) {
|
||||
showError(error);
|
||||
}
|
||||
}
|
||||
element("open-result").addEventListener("click", () => {
|
||||
void openSelected(false);
|
||||
});
|
||||
element("reveal-result").addEventListener("click", () => {
|
||||
void openSelected(true);
|
||||
});
|
||||
element("folder-form").addEventListener("submit", async (event) => {
|
||||
event.preventDefault();
|
||||
const input = element<HTMLInputElement>("folder");
|
||||
const buttons =
|
||||
element("folder-form").querySelectorAll<HTMLButtonElement>("button");
|
||||
buttons.forEach((button) => {
|
||||
button.disabled = true;
|
||||
});
|
||||
element<HTMLButtonElement>("add-folder").disabled = true;
|
||||
notice.className = "";
|
||||
notice.textContent = "Indexing folder… Search remains available.";
|
||||
try {
|
||||
const report = await invoke<IndexReport>("index_folder", {
|
||||
root: input.value.trim(),
|
||||
});
|
||||
notice.textContent = `Indexed ${report.indexed} files. ${report.removed} removed; ${report.skipped} skipped.`;
|
||||
element("folder-form").hidden = true;
|
||||
await refreshStats();
|
||||
await search();
|
||||
} catch (error) {
|
||||
showError(error);
|
||||
} finally {
|
||||
buttons.forEach((button) => {
|
||||
button.disabled = false;
|
||||
});
|
||||
element<HTMLButtonElement>("add-folder").disabled = false;
|
||||
}
|
||||
});
|
||||
document.addEventListener("keydown", (event) => {
|
||||
if ((event.ctrlKey || event.metaKey) && event.key.toLowerCase() === "k") {
|
||||
event.preventDefault();
|
||||
query.focus();
|
||||
query.select();
|
||||
}
|
||||
if (event.key === "Escape") {
|
||||
++requestId;
|
||||
query.value = "";
|
||||
void search();
|
||||
query.focus();
|
||||
}
|
||||
if (event.target === query || list.contains(event.target as Node)) {
|
||||
if (event.key === "ArrowDown" || event.key === "ArrowUp") {
|
||||
event.preventDefault();
|
||||
select(selected + (event.key === "ArrowDown" ? 1 : -1), true);
|
||||
}
|
||||
}
|
||||
});
|
||||
if (isTauri()) {
|
||||
void refreshStats().catch(showError);
|
||||
query.focus();
|
||||
} else {
|
||||
element("index-status").textContent = "Desktop preview";
|
||||
showError("Run the Tauri desktop app to index and search real files.");
|
||||
}
|
||||
@@ -0,0 +1,343 @@
|
||||
:root {
|
||||
font-family:
|
||||
Inter,
|
||||
Segoe UI,
|
||||
system-ui,
|
||||
sans-serif;
|
||||
color: #20332e;
|
||||
background: #f6f7f2;
|
||||
font-synthesis: none;
|
||||
color-scheme: light;
|
||||
}
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
body {
|
||||
margin: 0;
|
||||
}
|
||||
button,
|
||||
input {
|
||||
font: inherit;
|
||||
}
|
||||
button {
|
||||
cursor: pointer;
|
||||
}
|
||||
button:disabled {
|
||||
cursor: wait;
|
||||
opacity: 0.6;
|
||||
}
|
||||
header,
|
||||
footer {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 24px 40px;
|
||||
}
|
||||
.wordmark {
|
||||
font-size: 28px;
|
||||
font-weight: 750;
|
||||
letter-spacing: -1.5px;
|
||||
}
|
||||
.dot {
|
||||
color: #87a77f;
|
||||
}
|
||||
.privacy {
|
||||
font-size: 12px;
|
||||
color: #58705c;
|
||||
}
|
||||
main {
|
||||
max-width: 1120px;
|
||||
margin: 0 auto;
|
||||
padding: 24px 40px;
|
||||
}
|
||||
.intro {
|
||||
margin: 12px 0 30px;
|
||||
}
|
||||
.eyebrow {
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
letter-spacing: 1.8px;
|
||||
color: #78887c;
|
||||
}
|
||||
.intro h1 {
|
||||
font-size: clamp(28px, 4vw, 42px);
|
||||
letter-spacing: -1.5px;
|
||||
margin: 12px 0;
|
||||
}
|
||||
.intro > p:last-child {
|
||||
color: #738077;
|
||||
margin: 0;
|
||||
}
|
||||
#search-form {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
background: white;
|
||||
border: 1px solid #d2dccf;
|
||||
border-radius: 14px;
|
||||
padding: 18px 22px;
|
||||
box-shadow: 0 8px 30px #243b2910;
|
||||
}
|
||||
#query {
|
||||
width: 100%;
|
||||
border: 0;
|
||||
background: transparent;
|
||||
font-size: 19px;
|
||||
color: inherit;
|
||||
outline: none;
|
||||
}
|
||||
#search-form:focus-within {
|
||||
border-color: #588663;
|
||||
box-shadow: 0 0 0 3px #65866922;
|
||||
}
|
||||
.search-icon {
|
||||
font-size: 30px;
|
||||
color: #6a8770;
|
||||
}
|
||||
kbd {
|
||||
white-space: nowrap;
|
||||
color: #829086;
|
||||
font-size: 11px;
|
||||
border: 1px solid #dce3d9;
|
||||
padding: 5px;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.toolbar {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin: 18px 0;
|
||||
font-size: 12px;
|
||||
color: #78847b;
|
||||
}
|
||||
.toolbar button,
|
||||
.folder-row button {
|
||||
background: #e8ede2;
|
||||
border: 1px solid #d9e0d2;
|
||||
padding: 9px 14px;
|
||||
border-radius: 7px;
|
||||
color: #36583c;
|
||||
}
|
||||
#folder-form {
|
||||
background: #eef2e9;
|
||||
padding: 18px;
|
||||
border-radius: 10px;
|
||||
}
|
||||
#folder-form label {
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
}
|
||||
.folder-row {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-top: 10px;
|
||||
}
|
||||
.folder-row input {
|
||||
min-width: 0;
|
||||
flex: 1;
|
||||
padding: 9px;
|
||||
border: 1px solid #c9d3c3;
|
||||
border-radius: 6px;
|
||||
}
|
||||
.hint {
|
||||
color: #74806e;
|
||||
font-size: 12px;
|
||||
}
|
||||
.workspace {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1.3fr) minmax(250px, 1fr);
|
||||
gap: 28px;
|
||||
border-top: 1px solid #e1e6db;
|
||||
padding-top: 20px;
|
||||
}
|
||||
.workspace:has(aside[hidden]) {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
#results {
|
||||
list-style: none;
|
||||
padding: 0;
|
||||
margin: 0;
|
||||
}
|
||||
.result {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr auto;
|
||||
text-align: left;
|
||||
width: 100%;
|
||||
padding: 16px;
|
||||
background: transparent;
|
||||
border: 1px solid transparent;
|
||||
border-radius: 10px;
|
||||
color: inherit;
|
||||
gap: 8px;
|
||||
}
|
||||
.result:hover {
|
||||
background: #edf1e7;
|
||||
}
|
||||
.result.selected {
|
||||
background: white;
|
||||
border-color: #d1dbc9;
|
||||
box-shadow: 0 3px 12px #314b2810;
|
||||
}
|
||||
.result strong {
|
||||
font-size: 14px;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
.path {
|
||||
font-size: 11px;
|
||||
color: #79847a;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
.result .path,
|
||||
.snippet {
|
||||
grid-column: 1/-1;
|
||||
}
|
||||
.snippet {
|
||||
font-size: 12px;
|
||||
color: #61705f;
|
||||
display: -webkit-box;
|
||||
-webkit-line-clamp: 2;
|
||||
-webkit-box-orient: vertical;
|
||||
overflow: hidden;
|
||||
}
|
||||
.kind {
|
||||
font-size: 10px;
|
||||
padding: 3px 7px;
|
||||
border-radius: 5px;
|
||||
background: #e6eddf;
|
||||
color: #638054;
|
||||
}
|
||||
aside {
|
||||
padding: 18px 22px;
|
||||
border-left: 1px solid #dce3d5;
|
||||
min-width: 0;
|
||||
}
|
||||
aside h2 {
|
||||
font-size: 20px;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
aside pre {
|
||||
font: 13px/1.8 inherit;
|
||||
white-space: pre-wrap;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
#preview-kind {
|
||||
font-size: 12px;
|
||||
color: #67875f;
|
||||
}
|
||||
#empty {
|
||||
text-align: center;
|
||||
padding: 38px 18px;
|
||||
color: #91a088;
|
||||
}
|
||||
#empty > span {
|
||||
font-size: 55px;
|
||||
}
|
||||
#empty h2 {
|
||||
font-size: 20px;
|
||||
font-weight: 500;
|
||||
color: #607358;
|
||||
}
|
||||
#empty p {
|
||||
font-size: 13px;
|
||||
line-height: 1.6;
|
||||
}
|
||||
#notice {
|
||||
min-height: 18px;
|
||||
font-size: 12px;
|
||||
color: #627a5c;
|
||||
}
|
||||
#notice.error {
|
||||
color: #a54639;
|
||||
}
|
||||
footer {
|
||||
color: #95a18e;
|
||||
font-size: 11px;
|
||||
border-top: 1px solid #e1e6db;
|
||||
margin-top: 32px;
|
||||
}
|
||||
button:focus-visible,
|
||||
input:focus-visible {
|
||||
outline: 2px solid #5b8357;
|
||||
outline-offset: 3px;
|
||||
}
|
||||
.sr-only {
|
||||
position: absolute;
|
||||
width: 1px;
|
||||
height: 1px;
|
||||
padding: 0;
|
||||
margin: -1px;
|
||||
overflow: hidden;
|
||||
clip: rect(0, 0, 0, 0);
|
||||
white-space: nowrap;
|
||||
border: 0;
|
||||
}
|
||||
[hidden] {
|
||||
display: none !important;
|
||||
}
|
||||
@media (max-width: 760px) {
|
||||
header,
|
||||
footer {
|
||||
padding: 20px;
|
||||
}
|
||||
main {
|
||||
padding: 12px 20px;
|
||||
}
|
||||
.workspace {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
aside {
|
||||
border-left: 0;
|
||||
border-top: 1px solid #dce3d5;
|
||||
}
|
||||
.folder-row {
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
.folder-row input {
|
||||
flex-basis: 100%;
|
||||
}
|
||||
footer {
|
||||
gap: 12px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
}
|
||||
.filters-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||||
gap: 12px;
|
||||
padding: 16px 0;
|
||||
}
|
||||
.filters-grid label {
|
||||
font-size: 11px;
|
||||
color: #68785f;
|
||||
}
|
||||
.filters-grid input {
|
||||
display: block;
|
||||
width: 100%;
|
||||
margin-top: 6px;
|
||||
padding: 8px;
|
||||
border: 1px solid #d5decd;
|
||||
border-radius: 6px;
|
||||
background: white;
|
||||
}
|
||||
details summary {
|
||||
font-size: 12px;
|
||||
color: #61795a;
|
||||
cursor: pointer;
|
||||
}
|
||||
.preview-actions {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
}
|
||||
.preview-actions button {
|
||||
font-size: 12px;
|
||||
padding: 8px 12px;
|
||||
border: 1px solid #d3decb;
|
||||
border-radius: 6px;
|
||||
background: #e9efdf;
|
||||
color: #36583c;
|
||||
}
|
||||
@media (max-width: 760px) {
|
||||
.filters-grid {
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"lib": ["ES2022", "DOM"],
|
||||
"strict": true,
|
||||
"noEmit": true,
|
||||
"skipLibCheck": true
|
||||
},
|
||||
"include": ["src"]
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
import { defineConfig } from "vite";
|
||||
export default defineConfig({
|
||||
server: { port: 1420, strictPort: true },
|
||||
clearScreen: false,
|
||||
});
|
||||
@@ -7,6 +7,8 @@ GPU setup. Index only folders you explicitly choose.
|
||||
cargo run -p searchi-cli -- index --db ./searchi.db --root ./documents
|
||||
cargo run -p searchi-cli -- search --db ./searchi.db "termination clause" --limit 10
|
||||
cargo run -p searchi-cli -- search --db ./searchi.db "invoice" --json
|
||||
cargo run -p searchi-cli -- search --db ./searchi.db "invoice" --extension pdf --min-size 1024 --json
|
||||
cargo run -p searchi-cli -- index --db ./searchi.db --root ./documents --exclude private
|
||||
cargo run -p searchi-cli -- stats --db ./searchi.db --json
|
||||
```
|
||||
|
||||
@@ -23,6 +25,12 @@ snippet, ranking score, and match kind. An empty search emits `[]`. `stats --jso
|
||||
emits an object with `files` and `content_files`. JSON is the scripting interface;
|
||||
the human-readable output may change as preview support develops.
|
||||
|
||||
The foundation CLI searches filenames and extracted text. Semantic inference,
|
||||
document formats beyond the core extractor, and desktop integration remain
|
||||
tracked in the project roadmap; this interface does not fabricate embeddings.
|
||||
The CLI searches filenames, bounded text, PDF text and DOCX paragraphs. Results
|
||||
include actual matching source locations when available and extraction errors
|
||||
for documents that could not be read. Semantic inference and media formats
|
||||
remain tracked in the roadmap; this interface does not fabricate embeddings.
|
||||
|
||||
Filters apply before result limits: `--folder`, `--extension`, `--min-size`,
|
||||
`--max-size`, `--modified-after-ms`, and `--modified-before-ms`. Dates are Unix
|
||||
milliseconds and inclusive. Repeated `index --exclude NAME` flags add directory
|
||||
names to the defaults (`.git`, `node_modules`, `target`).
|
||||
@@ -4,7 +4,7 @@ use std::path::PathBuf;
|
||||
|
||||
use anyhow::{bail, Context, Result};
|
||||
use clap::{Parser, Subcommand};
|
||||
use searchi_core::SearchIndex;
|
||||
use searchi_core::{IndexOptions, SearchIndex, SearchOptions};
|
||||
|
||||
#[derive(Debug, Parser)]
|
||||
#[command(name = "searchi", version, about = "Search your local files offline")]
|
||||
@@ -23,6 +23,9 @@ enum Command {
|
||||
/// Folder to index recursively.
|
||||
#[arg(long)]
|
||||
root: PathBuf,
|
||||
/// Additional directory names to exclude; repeat this flag as needed.
|
||||
#[arg(long)]
|
||||
exclude: Vec<String>,
|
||||
},
|
||||
/// Search filenames and indexed document content.
|
||||
Search {
|
||||
@@ -36,6 +39,21 @@ enum Command {
|
||||
/// Emit a JSON array for scripting.
|
||||
#[arg(long)]
|
||||
json: bool,
|
||||
/// Restrict results to this folder and descendants.
|
||||
#[arg(long)]
|
||||
folder: Option<PathBuf>,
|
||||
/// File extension, with or without the leading dot.
|
||||
#[arg(long)]
|
||||
extension: Option<String>,
|
||||
#[arg(long)]
|
||||
min_size: Option<u64>,
|
||||
#[arg(long)]
|
||||
max_size: Option<u64>,
|
||||
/// Inclusive modification time in Unix milliseconds.
|
||||
#[arg(long)]
|
||||
modified_after_ms: Option<i64>,
|
||||
#[arg(long)]
|
||||
modified_before_ms: Option<i64>,
|
||||
},
|
||||
/// Show the indexed file and content counts.
|
||||
Stats {
|
||||
@@ -48,15 +66,17 @@ enum Command {
|
||||
|
||||
fn main() -> Result<()> {
|
||||
match Cli::parse().command {
|
||||
Command::Index { db, root } => {
|
||||
Command::Index { db, root, exclude } => {
|
||||
let mut options = IndexOptions::default();
|
||||
options.excluded_directory_names.extend(exclude);
|
||||
let mut index = SearchIndex::open(&db)
|
||||
.with_context(|| format!("Cannot open index {}", db.display()))?;
|
||||
let report = index
|
||||
.index_folder(&root)
|
||||
.index_folder_with_options(&root, &options)
|
||||
.with_context(|| format!("Cannot index folder {}", root.display()))?;
|
||||
println!(
|
||||
"Discovered: {} Indexed: {} Removed: {} Skipped: {}",
|
||||
report.discovered, report.indexed, report.removed, report.skipped
|
||||
"Discovered: {} Indexed: {} Unchanged: {} Removed: {} Skipped: {} Extraction failures: {}",
|
||||
report.discovered, report.indexed, report.unchanged, report.removed, report.skipped, report.failures
|
||||
);
|
||||
}
|
||||
Command::Search {
|
||||
@@ -64,6 +84,12 @@ fn main() -> Result<()> {
|
||||
query,
|
||||
limit,
|
||||
json,
|
||||
folder,
|
||||
extension,
|
||||
min_size,
|
||||
max_size,
|
||||
modified_after_ms,
|
||||
modified_before_ms,
|
||||
} => {
|
||||
if query.trim().is_empty() {
|
||||
bail!("Search text cannot be empty");
|
||||
@@ -72,7 +98,18 @@ fn main() -> Result<()> {
|
||||
let index = SearchIndex::open(&db)
|
||||
.with_context(|| format!("Cannot open index {}", db.display()))?;
|
||||
let results = index
|
||||
.search(&query, usize::from(limit))
|
||||
.search_with_options(
|
||||
&query,
|
||||
usize::from(limit),
|
||||
&SearchOptions {
|
||||
folder,
|
||||
extension,
|
||||
min_size,
|
||||
max_size,
|
||||
modified_after_unix_ms: modified_after_ms,
|
||||
modified_before_unix_ms: modified_before_ms,
|
||||
},
|
||||
)
|
||||
.context("Search failed")?;
|
||||
if json {
|
||||
println!("{}", serde_json::to_string_pretty(&results)?);
|
||||
@@ -81,6 +118,12 @@ fn main() -> Result<()> {
|
||||
} else {
|
||||
for result in results {
|
||||
println!("{} [{}]", result.path, result.match_kind);
|
||||
if let Some(source) = result.source {
|
||||
println!(" {} {}", source.kind, source.number);
|
||||
}
|
||||
if let Some(error) = result.extraction_error {
|
||||
println!(" Text extraction unavailable: {error}");
|
||||
}
|
||||
if !result.snippet.is_empty() {
|
||||
println!(" {}", result.snippet.replace(['\r', '\n'], " "));
|
||||
}
|
||||
|
||||
@@ -88,3 +88,42 @@ fn errors_are_nonzero_and_missing_database_is_not_created() {
|
||||
assert!(!bad_root.status.success());
|
||||
assert!(String::from_utf8_lossy(&bad_root.stderr).contains("missing-folder"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exclusions_and_filters_change_real_results() {
|
||||
let fixture = Fixture::new();
|
||||
fs::create_dir_all(fixture.0.join("documents/private")).unwrap();
|
||||
fs::write(fixture.0.join("documents/report.txt"), "quarterly invoice").unwrap();
|
||||
fs::write(fixture.0.join("documents/report.md"), "quarterly invoice").unwrap();
|
||||
fs::write(fixture.0.join("documents/private/secret.txt"), "invoice").unwrap();
|
||||
let indexed = fixture.run(&[
|
||||
"index",
|
||||
"--db",
|
||||
"index.db",
|
||||
"--root",
|
||||
"documents",
|
||||
"--exclude",
|
||||
"private",
|
||||
]);
|
||||
assert!(indexed.status.success());
|
||||
let searched = fixture.run(&[
|
||||
"search",
|
||||
"--db",
|
||||
"index.db",
|
||||
"invoice",
|
||||
"--extension",
|
||||
"md",
|
||||
"--json",
|
||||
]);
|
||||
assert!(
|
||||
searched.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&searched.stderr)
|
||||
);
|
||||
let results: serde_json::Value = serde_json::from_slice(&searched.stdout).unwrap();
|
||||
assert_eq!(results.as_array().unwrap().len(), 1);
|
||||
assert_eq!(results[0]["filename"], "report.md");
|
||||
let stats = fixture.run(&["stats", "--db", "index.db", "--json"]);
|
||||
let stats: serde_json::Value = serde_json::from_slice(&stats.stdout).unwrap();
|
||||
assert_eq!(stats["files"], 2);
|
||||
}
|
||||
@@ -9,6 +9,10 @@ anyhow.workspace = true
|
||||
serde.workspace = true
|
||||
rusqlite = { version = "0.32", features = ["bundled"] }
|
||||
walkdir = "2"
|
||||
lopdf = { version = "0.45", default-features = false }
|
||||
zip = { version = "8", default-features = false, features = ["deflate"] }
|
||||
quick-xml = "0.42"
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
serde_json.workspace = true
|
||||
@@ -4,10 +4,14 @@ The core has no desktop or inference dependencies. Its current retrieval engine
|
||||
combines literal filename matching with SQLite FTS5 BM25 keyword search. It does
|
||||
not implement semantic search.
|
||||
|
||||
`SearchIndex::open` creates or migrates schema version 2. Current-schema opens do
|
||||
`SearchIndex::open` creates or migrates schema version 4. Current-schema opens do
|
||||
not acquire a write lock. WAL allows independent readers to use the committed
|
||||
index while reconciliation runs. All indexing changes are committed atomically;
|
||||
a traversal or supported-file read error rolls back the entire reconciliation.
|
||||
a traversal or plain-text file read error rolls back the entire reconciliation.
|
||||
PDF/DOCX parser or limit failures retain a filename record with a persisted
|
||||
`SearchResult.extraction_error`, clear stale content, and increment
|
||||
`IndexReport.failures`. This counts attempts in this reconciliation, so an
|
||||
unchanged failed record retains its reason without incrementing the count again.
|
||||
|
||||
`index_folder` applies `IndexOptions::default()`: directories named `.git`,
|
||||
`node_modules`, and `target` are pruned. `index_folder_with_options` lets callers
|
||||
@@ -23,6 +27,46 @@ or size. Unsupported files remain searchable by filename. Content reads are
|
||||
limited to 2 MiB. Reconciliation tracks seen files in SQLite rather than retaining
|
||||
every path in process memory.
|
||||
|
||||
PDF text extraction uses `lopdf` with strict parsing and bounded load/page/font
|
||||
stream decompression. DOCX uses `zip` and namespace-aware `quick-xml` to read the
|
||||
main document body, including table paragraphs. The extractor does not execute
|
||||
macros or follow external document relationships. It supports UTF-8 DOCX XML;
|
||||
UTF-16 document parts, nested paragraphs, and XML document types are explicit
|
||||
failures. Headers, footers, comments, drawings, and tracked-deletion semantics
|
||||
are not yet covered. Encrypted PDFs are explicitly unsupported. Scanned PDFs
|
||||
without a text layer remain filename searchable; OCR is not implemented.
|
||||
|
||||
| Limit | Value |
|
||||
| --- | --- |
|
||||
| PDF/DOCX file input, before parsing | 32 MiB |
|
||||
| PDF load stream / page content / font decoding | 4 MiB per stream or page budget |
|
||||
| PDF page count, after parsing | 1,000 |
|
||||
| DOCX main XML, including decompressed ZIP output | 8 MiB |
|
||||
| DOCX ZIP entries / paragraphs | 10,000 each |
|
||||
| DOCX XML depth | 128 |
|
||||
| Extracted text per document | 2 MiB |
|
||||
|
||||
Input reads and DOCX XML decompression are capped while reading. PDF's loader and
|
||||
text extractor use the library's bounded decoding APIs. Page/entry counts and
|
||||
total PDF extracted text are checked after the parser produces them. These are
|
||||
not global process memory or CPU-time limits: multiple PDF objects/fonts can each
|
||||
consume their budget, and parsing may still be expensive. Process isolation and
|
||||
timeouts remain required before handling arbitrary untrusted documents in a
|
||||
production release.
|
||||
|
||||
Actual page/paragraph text is stored separately in an FTS passage index. Search
|
||||
results use the best passage that matches all literal query terms and expose
|
||||
`source: { kind, number }`, with one-based PDF page, DOCX paragraph, or plain-text
|
||||
line-range start. Paragraph numbers include empty paragraphs. A filename-only
|
||||
match or a match spanning several passages can have no source; the engine does
|
||||
not invent a location. Snippets with a source are drawn from that passage.
|
||||
|
||||
`passage_batch(after_id, limit)` exposes at most 256 `IndexedPassage` records for
|
||||
real embedding runtimes. IDs represent current rows and can be reused after
|
||||
reconciliation; downstream embedding caches must validate content and model
|
||||
fingerprints. Consumers must tokenize long pages/paragraphs into overlapping
|
||||
windows rather than silently truncate them to a model token limit.
|
||||
|
||||
`search_with_options` supports canonical folder boundaries, extension, inclusive
|
||||
size bounds, and inclusive Unix millisecond modification-time bounds. Filters
|
||||
apply before limiting results. `search` retains the unfiltered API. Query text is
|
||||
@@ -46,4 +90,6 @@ escaped as literal FTS terms, not interpreted as operators. Results are capped a
|
||||
- Extraction runs in the caller and a large reconciliation holds one transaction.
|
||||
Bounded worker isolation, persistent jobs, and crash-resumable scans remain
|
||||
production milestones.
|
||||
- PDF, DOCX, OCR, media extraction, and real model inference remain unimplemented.
|
||||
- OCR, media extraction, and real model inference belong to subsequent modules.
|
||||
- Passage selection currently adds a bounded query per result; query batching and
|
||||
passage ranking benchmarks remain necessary for large result lists.
|
||||
@@ -0,0 +1,267 @@
|
||||
//! Bounded inputs and outputs; parser allocations and CPU are not process isolated.
|
||||
use crate::SourceLocation;
|
||||
use anyhow::{bail, Context, Result};
|
||||
use quick_xml::{events::Event, name::ResolveResult, NsReader, XmlVersion};
|
||||
use std::{
|
||||
fs::File,
|
||||
io::{Cursor, Read},
|
||||
path::Path,
|
||||
};
|
||||
|
||||
const MAX_TEXT: usize = 2 * 1024 * 1024;
|
||||
const MAX_DOCUMENT: usize = 32 * 1024 * 1024;
|
||||
const MAX_XML: usize = 8 * 1024 * 1024;
|
||||
const MAX_PDF_STREAM: usize = 4 * 1024 * 1024;
|
||||
const MAX_PASSAGES: usize = 10_000;
|
||||
|
||||
pub(crate) struct Passage {
|
||||
pub text: String,
|
||||
pub source: SourceLocation,
|
||||
}
|
||||
|
||||
pub(crate) fn extract(path: &Path) -> Result<Option<Vec<Passage>>> {
|
||||
let extension = path
|
||||
.extension()
|
||||
.and_then(|value| value.to_str())
|
||||
.unwrap_or("")
|
||||
.to_ascii_lowercase();
|
||||
if extension == "pdf" || extension == "docx" {
|
||||
let bytes = read_bounded(path, MAX_DOCUMENT)?;
|
||||
return if extension == "pdf" {
|
||||
pdf(&bytes).map(Some)
|
||||
} else {
|
||||
docx(&bytes).map(Some)
|
||||
};
|
||||
}
|
||||
if !matches!(
|
||||
extension.as_str(),
|
||||
"txt"
|
||||
| "md"
|
||||
| "rs"
|
||||
| "py"
|
||||
| "js"
|
||||
| "ts"
|
||||
| "tsx"
|
||||
| "jsx"
|
||||
| "json"
|
||||
| "toml"
|
||||
| "yaml"
|
||||
| "yml"
|
||||
| "csv"
|
||||
| "html"
|
||||
| "css"
|
||||
| "xml"
|
||||
| "log"
|
||||
| "sh"
|
||||
) {
|
||||
return Ok(None);
|
||||
}
|
||||
let file = File::open(path)?;
|
||||
if file.metadata()?.len() > MAX_TEXT as u64 {
|
||||
return Ok(None);
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
file.take(MAX_TEXT as u64 + 1).read_to_end(&mut bytes)?;
|
||||
if bytes.len() > MAX_TEXT || bytes.contains(&0) {
|
||||
return Ok(None);
|
||||
}
|
||||
let Ok(text) = String::from_utf8(bytes) else {
|
||||
return Ok(None);
|
||||
};
|
||||
let lines = text.split_inclusive('\n').collect::<Vec<_>>();
|
||||
Ok(Some(
|
||||
lines
|
||||
.chunks(64)
|
||||
.enumerate()
|
||||
.map(|(index, lines)| Passage {
|
||||
text: lines.concat(),
|
||||
source: SourceLocation {
|
||||
kind: "line".into(),
|
||||
number: (index * 64 + 1) as u32,
|
||||
},
|
||||
})
|
||||
.collect(),
|
||||
))
|
||||
}
|
||||
|
||||
fn read_bounded(path: &Path, maximum: usize) -> Result<Vec<u8>> {
|
||||
let file = File::open(path)?;
|
||||
if file.metadata()?.len() > maximum as u64 {
|
||||
bail!("document input exceeds {maximum} bytes");
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
file.take(maximum as u64 + 1).read_to_end(&mut bytes)?;
|
||||
if bytes.len() > maximum {
|
||||
bail!("document input exceeds {maximum} bytes");
|
||||
}
|
||||
Ok(bytes)
|
||||
}
|
||||
|
||||
fn pdf(bytes: &[u8]) -> Result<Vec<Passage>> {
|
||||
let document = lopdf::Document::load_mem_with_options(
|
||||
bytes,
|
||||
lopdf::LoadOptions {
|
||||
strict: true,
|
||||
max_decompressed_size: Some(MAX_PDF_STREAM),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.context("invalid, encrypted, or over-limit PDF")?;
|
||||
if document.was_encrypted() || document.is_encrypted() {
|
||||
bail!("encrypted PDF is unsupported");
|
||||
}
|
||||
let pages = document.get_pages();
|
||||
if pages.is_empty() {
|
||||
bail!("PDF has no pages");
|
||||
}
|
||||
if pages.len() > 1000 {
|
||||
bail!("PDF exceeds 1000 pages");
|
||||
}
|
||||
let mut passages = Vec::new();
|
||||
let mut total = 0;
|
||||
for number in pages.keys() {
|
||||
let text = document
|
||||
.extract_text_with_limit(&[*number], MAX_PDF_STREAM)
|
||||
.with_context(|| format!("PDF text extraction failed on page {number}"))?;
|
||||
total += text.len();
|
||||
if total > MAX_TEXT {
|
||||
bail!("PDF text exceeds {MAX_TEXT} bytes");
|
||||
}
|
||||
if !text.trim().is_empty() {
|
||||
passages.push(Passage {
|
||||
text,
|
||||
source: SourceLocation {
|
||||
kind: "page".into(),
|
||||
number: *number,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(passages)
|
||||
}
|
||||
|
||||
fn docx(bytes: &[u8]) -> Result<Vec<Passage>> {
|
||||
let mut archive =
|
||||
zip::ZipArchive::new(Cursor::new(bytes)).context("invalid DOCX ZIP container")?;
|
||||
if archive.len() > 10_000 {
|
||||
bail!("DOCX exceeds 10000 archive entries");
|
||||
}
|
||||
let mut xml = String::new();
|
||||
let part = archive
|
||||
.by_name("word/document.xml")
|
||||
.context("DOCX lacks word/document.xml")?;
|
||||
if part.size() > MAX_XML as u64 {
|
||||
bail!("DOCX document XML exceeds {MAX_XML} bytes");
|
||||
}
|
||||
part.take(MAX_XML as u64 + 1)
|
||||
.read_to_string(&mut xml)
|
||||
.context("DOCX XML must be valid UTF-8")?;
|
||||
if xml.len() > MAX_XML {
|
||||
bail!("DOCX document XML exceeds {MAX_XML} bytes");
|
||||
}
|
||||
let mut reader = NsReader::from_str(&xml);
|
||||
let mut passages = Vec::new();
|
||||
let mut paragraph = String::new();
|
||||
let mut number = 0;
|
||||
let mut in_paragraph = false;
|
||||
let mut in_text = false;
|
||||
let mut total = 0;
|
||||
let mut depth = 0usize;
|
||||
let mut saw_document = false;
|
||||
loop {
|
||||
let (namespace, event) = reader.read_resolved_event().context("invalid DOCX XML")?;
|
||||
let word_namespace = matches!(namespace, ResolveResult::Bound(value) if matches!(value.as_ref(),
|
||||
"http://schemas.openxmlformats.org/wordprocessingml/2006/main" | "http://purl.oclc.org/ooxml/wordprocessingml/main"));
|
||||
match &event {
|
||||
Event::Start(start) => {
|
||||
if depth == 0 {
|
||||
if saw_document || !word_namespace || start.local_name().as_ref() != "document"
|
||||
{
|
||||
bail!("invalid DOCX document root");
|
||||
}
|
||||
saw_document = true;
|
||||
}
|
||||
depth += 1;
|
||||
if depth > 128 {
|
||||
bail!("DOCX XML exceeds 128 nesting levels");
|
||||
}
|
||||
}
|
||||
Event::End(_) => depth = depth.checked_sub(1).context("invalid DOCX XML depth")?,
|
||||
Event::Empty(_) if depth == 0 => bail!("invalid DOCX document root"),
|
||||
_ => {}
|
||||
}
|
||||
match event {
|
||||
Event::Start(event) if word_namespace => match event.local_name().as_ref() {
|
||||
"p" => {
|
||||
if in_paragraph {
|
||||
bail!("nested DOCX paragraphs are unsupported");
|
||||
}
|
||||
number += 1;
|
||||
if number > MAX_PASSAGES {
|
||||
bail!("DOCX exceeds 10000 paragraphs");
|
||||
}
|
||||
in_paragraph = true;
|
||||
paragraph.clear();
|
||||
}
|
||||
"t" => in_text = in_paragraph,
|
||||
_ => {}
|
||||
},
|
||||
Event::End(event) if word_namespace => match event.local_name().as_ref() {
|
||||
"t" => in_text = false,
|
||||
"p" => {
|
||||
in_paragraph = false;
|
||||
in_text = false;
|
||||
total += paragraph.len();
|
||||
if total > MAX_TEXT {
|
||||
bail!("DOCX text exceeds {MAX_TEXT} bytes");
|
||||
}
|
||||
if !paragraph.trim().is_empty() {
|
||||
passages.push(Passage {
|
||||
text: std::mem::take(&mut paragraph),
|
||||
source: SourceLocation {
|
||||
kind: "paragraph".into(),
|
||||
number: number as u32,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
},
|
||||
Event::Empty(event) if word_namespace => match event.local_name().as_ref() {
|
||||
"tab" if in_paragraph => paragraph.push('\t'),
|
||||
"br" | "cr" if in_paragraph => paragraph.push('\n'),
|
||||
"p" => {
|
||||
number += 1;
|
||||
if number > MAX_PASSAGES {
|
||||
bail!("DOCX exceeds 10000 paragraphs");
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
},
|
||||
Event::Text(event) if in_text => {
|
||||
paragraph.push_str(&event.xml_content(XmlVersion::Implicit1_0))
|
||||
}
|
||||
Event::CData(event) if in_text => {
|
||||
paragraph.push_str(&event.xml_content(XmlVersion::Implicit1_0))
|
||||
}
|
||||
Event::GeneralRef(event) if in_text => {
|
||||
let reference = format!("&{};", event.as_ref());
|
||||
paragraph.push_str(
|
||||
&quick_xml::escape::unescape(&reference).context("unsupported DOCX entity")?,
|
||||
);
|
||||
}
|
||||
Event::DocType(_) => bail!("DOCX XML document types are unsupported"),
|
||||
Event::Eof => {
|
||||
if in_paragraph || depth != 0 || !saw_document {
|
||||
bail!("incomplete DOCX document");
|
||||
}
|
||||
break;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
if total + paragraph.len() > MAX_TEXT {
|
||||
bail!("DOCX text exceeds {MAX_TEXT} bytes");
|
||||
}
|
||||
}
|
||||
Ok(passages)
|
||||
}
|
||||
+120
-56
@@ -1,21 +1,20 @@
|
||||
//! Offline lexical search engine shared by the CLI and desktop shell.
|
||||
//!
|
||||
//! Reconciliation is atomic. Incomplete directory walks never delete records.
|
||||
//! Text extraction is deliberately bounded; binary document extractors belong in
|
||||
//! isolated workers in the next milestone, not in this indexing transaction.
|
||||
//! Text and document extraction have byte limits, but still run in this process.
|
||||
//! Worker isolation remains a production milestone.
|
||||
mod extraction;
|
||||
use anyhow::{bail, Context, Result};
|
||||
use rusqlite::{params, Connection, OptionalExtension};
|
||||
use serde::Serialize;
|
||||
use std::{
|
||||
fs,
|
||||
io::Read,
|
||||
path::{Path, PathBuf},
|
||||
time::UNIX_EPOCH,
|
||||
};
|
||||
use walkdir::WalkDir;
|
||||
|
||||
const SCHEMA_VERSION: i64 = 2;
|
||||
const MAX_TEXT_BYTES: u64 = 2 * 1024 * 1024;
|
||||
const SCHEMA_VERSION: i64 = 4;
|
||||
const MAX_RESULTS: usize = 1000;
|
||||
|
||||
#[derive(Debug, Default, Serialize)]
|
||||
@@ -25,6 +24,7 @@ pub struct IndexReport {
|
||||
pub removed: usize,
|
||||
pub skipped: usize,
|
||||
pub unchanged: usize,
|
||||
pub failures: usize,
|
||||
}
|
||||
|
||||
/// Names of directories to prune anywhere below an explicitly chosen root.
|
||||
@@ -60,6 +60,27 @@ pub struct SearchResult {
|
||||
pub snippet: String,
|
||||
pub score: f64,
|
||||
pub match_kind: String,
|
||||
pub source: Option<SourceLocation>,
|
||||
pub extraction_error: Option<String>,
|
||||
}
|
||||
|
||||
/// A verified matching passage location; numbers are one based.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct SourceLocation {
|
||||
pub kind: String,
|
||||
pub number: u32,
|
||||
}
|
||||
|
||||
/// A bounded corpus page for downstream real embedding runtimes. IDs identify
|
||||
/// current index rows, not immutable content; consumers must validate content
|
||||
/// fingerprints across reconciliations before reusing persisted embeddings.
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct IndexedPassage {
|
||||
pub id: i64,
|
||||
pub path: String,
|
||||
pub filename: String,
|
||||
pub text: String,
|
||||
pub source: SourceLocation,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
@@ -103,7 +124,7 @@ impl SearchIndex {
|
||||
id INTEGER PRIMARY KEY, path TEXT NOT NULL UNIQUE,
|
||||
root TEXT NOT NULL, filename TEXT NOT NULL, content TEXT NOT NULL,
|
||||
size INTEGER NOT NULL DEFAULT -1, mtime INTEGER NOT NULL DEFAULT -1,
|
||||
extension TEXT NOT NULL DEFAULT '');
|
||||
extension TEXT NOT NULL DEFAULT '', extraction_error TEXT);
|
||||
CREATE INDEX IF NOT EXISTS files_root ON files(root);
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS search USING fts5(filename, path, content,
|
||||
content='files', content_rowid='id', tokenize='unicode61');
|
||||
@@ -128,9 +149,21 @@ impl SearchIndex {
|
||||
ALTER TABLE files ADD COLUMN extension TEXT NOT NULL DEFAULT '';
|
||||
PRAGMA user_version=2;",
|
||||
)?;
|
||||
} else {
|
||||
initialization.execute_batch("PRAGMA user_version=2;")?;
|
||||
}
|
||||
if version > 0 && version < 4 {
|
||||
initialization.execute("ALTER TABLE files ADD COLUMN extraction_error TEXT", [])?;
|
||||
}
|
||||
initialization.execute_batch(
|
||||
"CREATE VIRTUAL TABLE IF NOT EXISTS passages USING fts5(
|
||||
text, file_path UNINDEXED, kind UNINDEXED, number UNINDEXED, tokenize='unicode61');
|
||||
CREATE TRIGGER IF NOT EXISTS files_passages_delete AFTER DELETE ON files BEGIN
|
||||
DELETE FROM passages WHERE file_path=old.path; END;",
|
||||
)?;
|
||||
if version != 4 {
|
||||
// Re-extract records from older schemas to populate genuine locations.
|
||||
initialization.execute("UPDATE files SET mtime=-1", [])?;
|
||||
}
|
||||
initialization.execute_batch("PRAGMA user_version=4;")?;
|
||||
initialization.commit()?;
|
||||
}
|
||||
Ok(Self {
|
||||
@@ -218,17 +251,46 @@ impl SearchIndex {
|
||||
report.unchanged += 1;
|
||||
continue;
|
||||
}
|
||||
let content = extract_text(entry.path()).with_context(|| format!("read {path}"))?;
|
||||
if content.is_none() {
|
||||
let (extracted, extraction_error) = match extraction::extract(entry.path()) {
|
||||
Ok(extracted) => (extracted, None),
|
||||
Err(error) if matches!(extension.as_str(), "pdf" | "docx") => {
|
||||
report.failures += 1;
|
||||
let reason: String = format!("{error:#}").chars().take(2048).collect();
|
||||
(None, Some(reason))
|
||||
}
|
||||
Err(error) => {
|
||||
return Err(error)
|
||||
.with_context(|| format!("read {path}; reconciliation rolled back"))
|
||||
}
|
||||
};
|
||||
if extracted.is_none() {
|
||||
report.skipped += 1;
|
||||
}
|
||||
let extracted = extracted.unwrap_or_default();
|
||||
let content = extracted
|
||||
.iter()
|
||||
.map(|passage| passage.text.as_str())
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
transaction.execute(
|
||||
"INSERT INTO files(path,root,filename,content,size,mtime,extension) VALUES (?1,?2,?3,?4,?5,?6,?7)
|
||||
"INSERT INTO files(path,root,filename,content,size,mtime,extension,extraction_error) VALUES (?1,?2,?3,?4,?5,?6,?7,?8)
|
||||
ON CONFLICT(path) DO UPDATE SET root=excluded.root, filename=excluded.filename,
|
||||
content=excluded.content, size=excluded.size, mtime=excluded.mtime,
|
||||
extension=excluded.extension",
|
||||
params![path, root_key, filename, content.unwrap_or_default(), size, mtime, extension],
|
||||
extension=excluded.extension, extraction_error=excluded.extraction_error",
|
||||
params![path, root_key, filename, content, size, mtime, extension, extraction_error],
|
||||
)?;
|
||||
transaction.execute("DELETE FROM passages WHERE file_path=?1", [&path])?;
|
||||
for passage in extracted {
|
||||
transaction.execute(
|
||||
"INSERT INTO passages(text,file_path,kind,number) VALUES (?1,?2,?3,?4)",
|
||||
params![
|
||||
passage.text,
|
||||
path,
|
||||
passage.source.kind,
|
||||
passage.source.number
|
||||
],
|
||||
)?;
|
||||
}
|
||||
report.indexed += 1;
|
||||
}
|
||||
report.removed = transaction.execute(
|
||||
@@ -315,7 +377,7 @@ impl SearchIndex {
|
||||
FROM search WHERE search MATCH ?1)
|
||||
SELECT f.path, f.filename, COALESCE(h.excerpt, ''), COALESCE(h.rank, 0),
|
||||
CASE WHEN lower(f.filename)=lower(?2) THEN 0
|
||||
WHEN instr(lower(f.filename), lower(?2)) > 0 THEN 1 ELSE 2 END AS tier
|
||||
WHEN instr(lower(f.filename), lower(?2)) > 0 THEN 1 ELSE 2 END AS tier, f.extraction_error
|
||||
FROM files f LEFT JOIN hits h ON h.id=f.id
|
||||
WHERE (h.id IS NOT NULL OR instr(lower(f.filename), lower(?2)) > 0)
|
||||
AND (?4 IS NULL OR substr(f.path,1,length(?4))=?4)
|
||||
@@ -324,7 +386,7 @@ impl SearchIndex {
|
||||
AND (?8 IS NULL OR f.mtime/1000000>=?8) AND (?9 IS NULL OR f.mtime/1000000<=?9)
|
||||
ORDER BY tier, h.rank, f.path LIMIT ?3",
|
||||
)?;
|
||||
let results = statement
|
||||
let mut results = statement
|
||||
.query_map(
|
||||
params![
|
||||
fts,
|
||||
@@ -350,13 +412,56 @@ impl SearchIndex {
|
||||
_ => "content",
|
||||
}
|
||||
.to_owned(),
|
||||
source: None,
|
||||
extraction_error: row.get(5)?,
|
||||
})
|
||||
},
|
||||
)?
|
||||
.collect::<rusqlite::Result<Vec<_>>>()?;
|
||||
let mut passages = self.connection.prepare("SELECT kind,number,snippet(passages,0,'','',' … ',24)
|
||||
FROM passages WHERE passages MATCH ?1 AND file_path=?2 ORDER BY bm25(passages),rowid LIMIT 1")?;
|
||||
for result in &mut results {
|
||||
let passage: Option<(String, u32, String)> = passages
|
||||
.query_row(params![fts, result.path], |row| {
|
||||
Ok((row.get(0)?, row.get(1)?, row.get(2)?))
|
||||
})
|
||||
.optional()?;
|
||||
if let Some((kind, number, snippet)) = passage {
|
||||
result.source = Some(SourceLocation { kind, number });
|
||||
result.snippet = snippet;
|
||||
}
|
||||
}
|
||||
Ok(results)
|
||||
}
|
||||
|
||||
pub fn contains_path(&self, path: &str) -> Result<bool> {
|
||||
Ok(self.connection.query_row(
|
||||
"SELECT EXISTS(SELECT 1 FROM files WHERE path=?1)",
|
||||
[path],
|
||||
|row| row.get(0),
|
||||
)?)
|
||||
}
|
||||
|
||||
pub fn passage_batch(&self, after_id: i64, limit: usize) -> Result<Vec<IndexedPassage>> {
|
||||
let mut statement = self.connection.prepare("SELECT p.rowid,f.path,f.filename,p.text,p.kind,p.number
|
||||
FROM passages p JOIN files f ON f.path=p.file_path WHERE p.rowid>?1 ORDER BY p.rowid LIMIT ?2")?;
|
||||
let passages = statement
|
||||
.query_map(params![after_id, limit.min(256) as i64], |row| {
|
||||
Ok(IndexedPassage {
|
||||
id: row.get(0)?,
|
||||
path: row.get(1)?,
|
||||
filename: row.get(2)?,
|
||||
text: row.get(3)?,
|
||||
source: SourceLocation {
|
||||
kind: row.get(4)?,
|
||||
number: row.get(5)?,
|
||||
},
|
||||
})
|
||||
})?
|
||||
.collect::<rusqlite::Result<Vec<_>>>()?;
|
||||
Ok(passages)
|
||||
}
|
||||
|
||||
pub fn stats(&self) -> Result<IndexStats> {
|
||||
Ok(self.connection.query_row(
|
||||
"SELECT count(*), count(NULLIF(content, '')) FROM files",
|
||||
@@ -371,47 +476,6 @@ impl SearchIndex {
|
||||
}
|
||||
}
|
||||
|
||||
fn extract_text(path: &Path) -> Result<Option<String>> {
|
||||
let extension = path
|
||||
.extension()
|
||||
.and_then(|value| value.to_str())
|
||||
.unwrap_or("")
|
||||
.to_ascii_lowercase();
|
||||
if !matches!(
|
||||
extension.as_str(),
|
||||
"txt"
|
||||
| "md"
|
||||
| "rs"
|
||||
| "py"
|
||||
| "js"
|
||||
| "ts"
|
||||
| "tsx"
|
||||
| "jsx"
|
||||
| "json"
|
||||
| "toml"
|
||||
| "yaml"
|
||||
| "yml"
|
||||
| "csv"
|
||||
| "html"
|
||||
| "css"
|
||||
| "xml"
|
||||
| "log"
|
||||
| "sh"
|
||||
) {
|
||||
return Ok(None);
|
||||
}
|
||||
let file = fs::File::open(path)?;
|
||||
if file.metadata()?.len() > MAX_TEXT_BYTES {
|
||||
return Ok(None);
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
file.take(MAX_TEXT_BYTES + 1).read_to_end(&mut bytes)?;
|
||||
if bytes.len() as u64 > MAX_TEXT_BYTES || bytes.contains(&0) {
|
||||
return Ok(None);
|
||||
}
|
||||
Ok(String::from_utf8(bytes).ok())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
use anyhow::Result;
|
||||
use lopdf::{
|
||||
content::{Content, Operation},
|
||||
dictionary, Document, Object, Stream,
|
||||
};
|
||||
use searchi_core::SearchIndex;
|
||||
use std::{fs, io::Write, path::Path};
|
||||
|
||||
fn pdf(path: &Path) -> Result<()> {
|
||||
pdf_with_text(
|
||||
path,
|
||||
&[
|
||||
"First introductory page",
|
||||
"Supplier cancellation clause on second page",
|
||||
],
|
||||
)
|
||||
}
|
||||
|
||||
fn pdf_with_text(path: &Path, texts: &[&str]) -> Result<()> {
|
||||
let mut document = Document::with_version("1.5");
|
||||
let pages_id = document.new_object_id();
|
||||
let font = document.add_object(dictionary! {"Type"=>"Font", "Subtype"=>"Type1", "BaseFont"=>"Helvetica", "Encoding"=>"WinAnsiEncoding"});
|
||||
let resources = document.add_object(dictionary! {"Font"=>dictionary! {"F1"=>font}});
|
||||
let mut pages = Vec::new();
|
||||
for text in texts {
|
||||
let content = Content {
|
||||
operations: vec![
|
||||
Operation::new("BT", vec![]),
|
||||
Operation::new("Tf", vec![Object::Name(b"F1".to_vec()), 12.into()]),
|
||||
Operation::new("Td", vec![50.into(), 700.into()]),
|
||||
Operation::new("Tj", vec![Object::string_literal(*text)]),
|
||||
Operation::new("ET", vec![]),
|
||||
],
|
||||
};
|
||||
let stream = document.add_object(Stream::new(dictionary! {}, content.encode()?));
|
||||
let page = document
|
||||
.add_object(dictionary! {"Type"=>"Page", "Parent"=>pages_id, "Contents"=>stream});
|
||||
pages.push(Object::Reference(page));
|
||||
}
|
||||
document.objects.insert(pages_id, Object::Dictionary(dictionary! {"Type"=>"Pages", "Kids"=>pages,
|
||||
"Count"=>texts.len() as i64, "Resources"=>resources, "MediaBox"=>vec![0.into(),0.into(),595.into(),842.into()]}));
|
||||
let catalog = document.add_object(dictionary! {"Type"=>"Catalog", "Pages"=>pages_id});
|
||||
document.trailer.set("Root", catalog);
|
||||
document.compress();
|
||||
document.save(path)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compressed_document_limits_are_enforced() -> Result<()> {
|
||||
let root = tempfile::tempdir()?;
|
||||
let text = "a".repeat(4 * 1024 * 1024 + 100);
|
||||
pdf_with_text(&root.path().join("bomb.pdf"), &[&text])?;
|
||||
let xml = format!(
|
||||
r#"<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body><w:p><w:r><w:t>{}</w:t></w:r></w:p></w:body></w:document>"#,
|
||||
"b".repeat(2 * 1024 * 1024 + 1)
|
||||
);
|
||||
docx(&root.path().join("over-limit.docx"), &xml)?;
|
||||
let mut index = SearchIndex::open(":memory:")?;
|
||||
assert_eq!(index.index_folder(root.path())?.failures, 2);
|
||||
assert_eq!(index.stats()?.content_files, 0);
|
||||
assert!(index.search("bomb.pdf", 10)?[0].extraction_error.is_some());
|
||||
assert!(index.search("over-limit.docx", 10)?[0]
|
||||
.extraction_error
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.contains("exceeds"));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn encrypted_pdf_is_explicitly_unsupported() -> Result<()> {
|
||||
let root = tempfile::tempdir()?;
|
||||
let path = root.path().join("protected.pdf");
|
||||
pdf(&path)?;
|
||||
let mut document = Document::load(&path)?;
|
||||
document.trailer.set(
|
||||
"ID",
|
||||
vec![
|
||||
Object::string_literal("fixture-id"),
|
||||
Object::string_literal("fixture-id"),
|
||||
],
|
||||
);
|
||||
let state = lopdf::EncryptionState::try_from(lopdf::EncryptionVersion::V1 {
|
||||
document: &document,
|
||||
owner_password: "owner",
|
||||
user_password: "reader",
|
||||
permissions: lopdf::Permissions::all(),
|
||||
})?;
|
||||
document.encrypt(&state)?;
|
||||
document.save(&path)?;
|
||||
let mut index = SearchIndex::open(":memory:")?;
|
||||
assert_eq!(index.index_folder(root.path())?.failures, 1);
|
||||
assert!(index.search("protected.pdf", 10)?[0]
|
||||
.extraction_error
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.contains("encrypted"));
|
||||
assert!(index.search("cancellation", 10)?.is_empty());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn docx(path: &Path, xml: &str) -> Result<()> {
|
||||
let mut writer = zip::ZipWriter::new(fs::File::create(path)?);
|
||||
let options = zip::write::SimpleFileOptions::default()
|
||||
.compression_method(zip::CompressionMethod::Deflated);
|
||||
for (name, content) in [
|
||||
(
|
||||
"[Content_Types].xml",
|
||||
r#"<?xml version="1.0"?><Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/><Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/></Types>"#,
|
||||
),
|
||||
(
|
||||
"_rels/.rels",
|
||||
r#"<?xml version="1.0"?><Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/></Relationships>"#,
|
||||
),
|
||||
("word/document.xml", xml),
|
||||
] {
|
||||
writer.start_file(name, options)?;
|
||||
writer.write_all(content.as_bytes())?;
|
||||
}
|
||||
writer.finish()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn real_pdf_and_docx_return_matching_locations_and_persist() -> Result<()> {
|
||||
let root = tempfile::tempdir()?;
|
||||
pdf(&root.path().join("agreement.pdf"))?;
|
||||
docx(
|
||||
&root.path().join("notes.docx"),
|
||||
r#"<?xml version="1.0" encoding="UTF-8"?><w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body><w:p><w:r><w:t>Introduction</w:t></w:r></w:p><w:p/><w:p><w:r><w:t>Research & </w:t></w:r><w:r><w:t>development budget</w:t></w:r></w:p></w:body></w:document>"#,
|
||||
)?;
|
||||
let db = root.path().join("index.db");
|
||||
let mut index = SearchIndex::open(&db)?;
|
||||
assert_eq!(index.index_folder(root.path())?.indexed, 2);
|
||||
let result = index.search("cancellation", 10)?;
|
||||
assert_eq!(result[0].source.as_ref().unwrap().kind, "page");
|
||||
assert_eq!(result[0].source.as_ref().unwrap().number, 2);
|
||||
assert!(result[0].snippet.contains("cancellation"));
|
||||
let result = index.search("development", 10)?;
|
||||
assert_eq!(result[0].source.as_ref().unwrap().kind, "paragraph");
|
||||
assert_eq!(result[0].source.as_ref().unwrap().number, 3);
|
||||
assert!(result[0].snippet.contains("Research & development"));
|
||||
let batch = index.passage_batch(0, 2)?;
|
||||
assert_eq!(batch.len(), 2);
|
||||
let remainder = index.passage_batch(batch[1].id, 256)?;
|
||||
assert_eq!(remainder.len(), 2);
|
||||
assert!(remainder
|
||||
.iter()
|
||||
.any(|passage| passage.source.kind == "paragraph" && passage.source.number == 3));
|
||||
assert!(index.contains_path(&result[0].path)?);
|
||||
assert!(!index.contains_path("unknown")?);
|
||||
drop(index);
|
||||
let index = SearchIndex::open(&db)?;
|
||||
assert_eq!(
|
||||
index.search("budget", 10)?[0]
|
||||
.source
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.number,
|
||||
3
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn corrupt_document_reports_failure_and_preserves_filename_search() -> Result<()> {
|
||||
let root = tempfile::tempdir()?;
|
||||
fs::write(root.path().join("notes.txt"), "persistent evidence")?;
|
||||
let mut index = SearchIndex::open(":memory:")?;
|
||||
index.index_folder(root.path())?;
|
||||
fs::write(root.path().join("broken.pdf"), b"not a PDF")?;
|
||||
assert_eq!(index.index_folder(root.path())?.failures, 1);
|
||||
let result = index.search("broken.pdf", 10)?;
|
||||
assert!(result[0].extraction_error.as_ref().unwrap().contains("PDF"));
|
||||
assert_eq!(index.search("evidence", 10)?.len(), 1);
|
||||
fs::remove_file(root.path().join("broken.pdf"))?;
|
||||
docx(
|
||||
&root.path().join("broken.docx"),
|
||||
"<w:document><w:p><w:t>broken</w:p>",
|
||||
)?;
|
||||
assert_eq!(index.index_folder(root.path())?.failures, 1);
|
||||
assert!(index.search("broken.docx", 10)?[0]
|
||||
.extraction_error
|
||||
.is_some());
|
||||
pdf(&root.path().join("agreement.pdf"))?;
|
||||
index.index_folder(root.path())?;
|
||||
assert_eq!(index.search("cancellation", 10)?.len(), 1);
|
||||
fs::write(root.path().join("agreement.pdf"), "corrupted replacement")?;
|
||||
assert_eq!(index.index_folder(root.path())?.failures, 1);
|
||||
assert!(index.search("cancellation", 10)?.is_empty());
|
||||
assert!(index.search("agreement.pdf", 10)?[0]
|
||||
.extraction_error
|
||||
.is_some());
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
[package]
|
||||
name = "searchi-semantic"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
description = "Bounded offline CPU ONNX embedding adapter for Searchi"
|
||||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
ort = { version = "=2.0.0-rc.13", default-features = false, features = ["std", "load-dynamic", "api-27"] }
|
||||
sha2 = "0.10"
|
||||
tokenizers = { version = "=0.22.2", default-features = false, features = ["fancy-regex"] }
|
||||
|
||||
[dev-dependencies]
|
||||
serde_json.workspace = true
|
||||
@@ -0,0 +1,290 @@
|
||||
//! Offline text embeddings using a pinned quantized multilingual model.
|
||||
//!
|
||||
//! Model provisioning is explicit and separate. This crate never downloads a
|
||||
//! model or runtime and exposes no GPU execution-provider configuration.
|
||||
|
||||
use std::{
|
||||
fs::File,
|
||||
io::{BufReader, Read},
|
||||
path::{Path, PathBuf},
|
||||
sync::Mutex,
|
||||
};
|
||||
|
||||
use anyhow::{bail, ensure, Context, Result};
|
||||
use ort::{ep, session::Session, value::Tensor};
|
||||
use sha2::{Digest, Sha256};
|
||||
use tokenizers::{PaddingDirection, PaddingParams, Tokenizer, TruncationParams};
|
||||
|
||||
/// Embedding dimensionality; stored vectors must carry the model fingerprint.
|
||||
pub const DIMENSIONS: usize = 384;
|
||||
pub const MAX_TOKENS: usize = 128;
|
||||
pub const MAX_BATCH: usize = 8;
|
||||
pub const MAX_TEXT_BYTES: usize = 65536;
|
||||
/// Includes model revision, quantization, pooling, normalization and token cap.
|
||||
pub const MODEL_FINGERPRINT: &str =
|
||||
"minilm-multilingual:2c4055b12046f11709e9df2c122e59ffbdc2f900:int8:mean:l2:128";
|
||||
|
||||
const ARTIFACTS: [(&str, u64, &str); 2] = [
|
||||
(
|
||||
"onnx/model_quantized.onnx",
|
||||
118308126,
|
||||
"66fc00f5f29afcaff34092e1bdd20008ca3918265a82fb9695a551e510cc4ebc",
|
||||
),
|
||||
(
|
||||
"tokenizer.json",
|
||||
17082913,
|
||||
"b60b6b43406a48bf3638526314f3d232d97058bc93472ff2de930d43686fa441",
|
||||
),
|
||||
];
|
||||
// ONNX Runtime's library and environment are process-global. Prevent concurrent
|
||||
// initializers from selecting different libraries or silently overriding state.
|
||||
static RUNTIME_PATH: Mutex<Option<PathBuf>> = Mutex::new(None);
|
||||
|
||||
/// Local, mutable inference session. Call from a bounded background worker.
|
||||
pub struct CpuEmbedder {
|
||||
session: Session,
|
||||
tokenizer: Tokenizer,
|
||||
}
|
||||
|
||||
impl CpuEmbedder {
|
||||
/// Verify local artifacts, then load the explicitly supplied runtime.
|
||||
///
|
||||
/// The runtime must be initialized through this adapter before other `ort`
|
||||
/// code in this process. Threads are limited to 1–8; default caller policy
|
||||
/// should use one thread in a VM. Missing or damaged assets are errors.
|
||||
pub fn load(
|
||||
model_dir: impl AsRef<Path>,
|
||||
runtime_library: impl AsRef<Path>,
|
||||
threads: usize,
|
||||
) -> Result<Self> {
|
||||
ensure!(
|
||||
(1..=8).contains(&threads),
|
||||
"CPU threads must be between 1 and 8"
|
||||
);
|
||||
let model_dir = model_dir.as_ref();
|
||||
verify_artifacts(model_dir)?;
|
||||
let runtime_path = runtime_library
|
||||
.as_ref()
|
||||
.canonicalize()
|
||||
.context("Cannot resolve explicitly supplied ONNX Runtime library")?;
|
||||
ensure!(
|
||||
runtime_path.is_file(),
|
||||
"ONNX Runtime library must be a file"
|
||||
);
|
||||
{
|
||||
let mut initialized = RUNTIME_PATH
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("Runtime initialization lock poisoned"))?;
|
||||
if let Some(existing) = initialized.as_ref() {
|
||||
ensure!(
|
||||
*existing == runtime_path,
|
||||
"ONNX Runtime already initialized with a different library"
|
||||
);
|
||||
} else {
|
||||
let committed = ort::init_from(&runtime_path)?
|
||||
.with_telemetry(false)
|
||||
.with_execution_providers([ep::CPU::default().build()])
|
||||
.commit();
|
||||
ensure!(
|
||||
committed,
|
||||
"ONNX Runtime environment was already initialized outside the CPU adapter"
|
||||
);
|
||||
*initialized = Some(runtime_path);
|
||||
}
|
||||
}
|
||||
let session = Session::builder()?
|
||||
.with_execution_providers([ep::CPU::default().build()])
|
||||
.map_err(|e| anyhow::anyhow!("CPU session configuration: {e}"))?
|
||||
.with_intra_threads(threads)
|
||||
.map_err(|e| anyhow::anyhow!("CPU thread configuration: {e}"))?
|
||||
.with_inter_threads(1)
|
||||
.map_err(|e| anyhow::anyhow!("Inter-thread configuration: {e}"))?
|
||||
.with_parallel_execution(false)
|
||||
.map_err(|e| anyhow::anyhow!("Sequential session configuration: {e}"))?
|
||||
.commit_from_file(model_dir.join("onnx/model_quantized.onnx"))?;
|
||||
let mut tokenizer = Tokenizer::from_file(model_dir.join("tokenizer.json"))
|
||||
.map_err(|error| anyhow::anyhow!("Cannot load tokenizer: {error}"))?;
|
||||
tokenizer
|
||||
.with_truncation(Some(TruncationParams {
|
||||
max_length: MAX_TOKENS,
|
||||
..Default::default()
|
||||
}))
|
||||
.map_err(|error| anyhow::anyhow!("Cannot configure tokenizer: {error}"))?;
|
||||
tokenizer.with_padding(Some(PaddingParams {
|
||||
pad_id: 1,
|
||||
pad_token: "<pad>".into(),
|
||||
..Default::default()
|
||||
}));
|
||||
Ok(Self { session, tokenizer })
|
||||
}
|
||||
|
||||
/// Produce finite, L2-normalized vectors from genuine local model inference.
|
||||
///
|
||||
/// Inputs above 128 tokens are truncated. Production indexing should chunk
|
||||
/// documents first and keep chunk locations; never silently embed entire PDFs.
|
||||
pub fn encode(&mut self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
|
||||
validate_texts(texts)?;
|
||||
// Encode sequentially so tokenizers cannot create an unbounded global
|
||||
// Rayon worker pool independently of the caller's inference budget.
|
||||
let mut encoded = texts
|
||||
.iter()
|
||||
.map(|text| self.tokenizer.encode(*text, true))
|
||||
.collect::<std::result::Result<Vec<_>, _>>()
|
||||
.map_err(|error| anyhow::anyhow!("Tokenization failed: {error}"))?;
|
||||
let sequence_len = encoded.iter().map(|item| item.len()).max().unwrap_or(0);
|
||||
for item in &mut encoded {
|
||||
item.pad(sequence_len, 1, 0, "<pad>", PaddingDirection::Right);
|
||||
}
|
||||
ensure!(
|
||||
sequence_len > 0 && sequence_len <= MAX_TOKENS,
|
||||
"Invalid token sequence length"
|
||||
);
|
||||
let ids: Vec<i64> = encoded
|
||||
.iter()
|
||||
.flat_map(|item| item.get_ids().iter().map(|&v| i64::from(v)))
|
||||
.collect();
|
||||
let masks: Vec<i64> = encoded
|
||||
.iter()
|
||||
.flat_map(|item| item.get_attention_mask().iter().map(|&v| i64::from(v)))
|
||||
.collect();
|
||||
let types: Vec<i64> = encoded
|
||||
.iter()
|
||||
.flat_map(|item| item.get_type_ids().iter().map(|&v| i64::from(v)))
|
||||
.collect();
|
||||
let shape = [texts.len(), sequence_len];
|
||||
let outputs = self.session.run(ort::inputs![
|
||||
"input_ids" => Tensor::from_array((shape, ids))?,
|
||||
"attention_mask" => Tensor::from_array((shape, masks.clone()))?,
|
||||
"token_type_ids" => Tensor::from_array((shape, types))?
|
||||
])?;
|
||||
let (output_shape, hidden) = outputs[0].try_extract_tensor::<f32>()?;
|
||||
ensure!(
|
||||
**output_shape == [texts.len() as i64, sequence_len as i64, DIMENSIONS as i64],
|
||||
"Unexpected embedding output shape"
|
||||
);
|
||||
pool_and_normalize(hidden, &masks, texts.len(), sequence_len)
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_texts(texts: &[&str]) -> Result<()> {
|
||||
ensure!(
|
||||
!texts.is_empty() && texts.len() <= MAX_BATCH,
|
||||
"Batch size must be between 1 and {MAX_BATCH}"
|
||||
);
|
||||
ensure!(
|
||||
texts
|
||||
.iter()
|
||||
.all(|text| !text.trim().is_empty() && text.len() <= MAX_TEXT_BYTES),
|
||||
"Text must be nonempty and at most {MAX_TEXT_BYTES} UTF-8 bytes"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn verify_artifacts(model_dir: &Path) -> Result<()> {
|
||||
for (relative, bytes, expected) in ARTIFACTS {
|
||||
let path = model_dir.join(relative);
|
||||
let file = File::open(&path).with_context(|| {
|
||||
format!(
|
||||
"Missing local model artifact {}; run the explicit model downloader first",
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
ensure!(
|
||||
file.metadata()?.len() == bytes,
|
||||
"Model artifact size mismatch: {}",
|
||||
path.display()
|
||||
);
|
||||
let mut reader = BufReader::new(file);
|
||||
let mut hash = Sha256::new();
|
||||
let mut buffer = [0_u8; 65536];
|
||||
loop {
|
||||
let read = reader.read(&mut buffer)?;
|
||||
if read == 0 {
|
||||
break;
|
||||
}
|
||||
hash.update(&buffer[..read]);
|
||||
}
|
||||
let actual = format!("{:x}", hash.finalize());
|
||||
ensure!(
|
||||
actual == expected,
|
||||
"Model artifact SHA256 mismatch: {}",
|
||||
path.display()
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn pool_and_normalize(
|
||||
hidden: &[f32],
|
||||
masks: &[i64],
|
||||
batch: usize,
|
||||
tokens: usize,
|
||||
) -> Result<Vec<Vec<f32>>> {
|
||||
ensure!(
|
||||
hidden.len() == batch * tokens * DIMENSIONS && masks.len() == batch * tokens,
|
||||
"Invalid pooling input dimensions"
|
||||
);
|
||||
let mut vectors = Vec::with_capacity(batch);
|
||||
for item in 0..batch {
|
||||
let mut vector = vec![0_f32; DIMENSIONS];
|
||||
let mut count = 0;
|
||||
for token in 0..tokens {
|
||||
if masks[item * tokens + token] == 0 {
|
||||
continue;
|
||||
}
|
||||
count += 1;
|
||||
let offset = (item * tokens + token) * DIMENSIONS;
|
||||
for dimension in 0..DIMENSIONS {
|
||||
vector[dimension] += hidden[offset + dimension];
|
||||
}
|
||||
}
|
||||
ensure!(count > 0, "Cannot pool a fully masked sequence");
|
||||
for value in &mut vector {
|
||||
*value /= count as f32;
|
||||
}
|
||||
let norm = vector.iter().map(|value| value * value).sum::<f32>().sqrt();
|
||||
if !norm.is_finite() || norm <= f32::EPSILON {
|
||||
bail!("Model produced nonfinite or zero embeddings");
|
||||
}
|
||||
for value in &mut vector {
|
||||
*value /= norm;
|
||||
}
|
||||
ensure!(
|
||||
vector.iter().all(|value| value.is_finite()),
|
||||
"Model produced nonfinite embeddings"
|
||||
);
|
||||
vectors.push(vector);
|
||||
}
|
||||
Ok(vectors)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn padding_is_excluded_and_output_is_normalized() {
|
||||
let mut hidden = vec![0.0; 2 * DIMENSIONS];
|
||||
hidden[0] = 3.0;
|
||||
hidden[1] = 4.0;
|
||||
hidden[DIMENSIONS] = 999.0;
|
||||
let pooled = pool_and_normalize(&hidden, &[1, 0], 1, 2).unwrap();
|
||||
assert!((pooled[0][0] - 0.6).abs() < 1e-6);
|
||||
assert!((pooled[0][1] - 0.8).abs() < 1e-6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_unbounded_inputs_and_invalid_outputs() {
|
||||
assert!(validate_texts(&[]).is_err());
|
||||
assert!(validate_texts(&["x"; MAX_BATCH + 1]).is_err());
|
||||
assert!(validate_texts(&[" "]).is_err());
|
||||
assert!(validate_texts(&[&"x".repeat(MAX_TEXT_BYTES + 1)]).is_err());
|
||||
assert!(pool_and_normalize(&vec![0.0; DIMENSIONS], &[0], 1, 1).is_err());
|
||||
assert!(pool_and_normalize(&vec![f32::NAN; DIMENSIONS], &[1], 1, 1).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_model_does_not_load_or_download_runtime() {
|
||||
assert!(CpuEmbedder::load("this-model-does-not-exist", "missing-runtime", 1).is_err());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
//! Opt-in real-artifact test. Default tests never acquire model/runtime assets.
|
||||
use searchi_semantic::{CpuEmbedder, DIMENSIONS};
|
||||
use std::path::PathBuf;
|
||||
|
||||
#[test]
|
||||
#[ignore = "Requires explicit SEARCHI_MODEL_DIR and SEARCHI_ORT_LIBRARY local assets"]
|
||||
fn real_cpu_model_retrieves_english_and_german_fixture() {
|
||||
let model =
|
||||
PathBuf::from(std::env::var_os("SEARCHI_MODEL_DIR").expect("Set SEARCHI_MODEL_DIR"));
|
||||
let runtime =
|
||||
PathBuf::from(std::env::var_os("SEARCHI_ORT_LIBRARY").expect("Set SEARCHI_ORT_LIBRARY"));
|
||||
let fixture: serde_json::Value =
|
||||
serde_json::from_str(include_str!("../../../tools/semantic-proof/fixture.json")).unwrap();
|
||||
let documents = fixture["documents"].as_array().unwrap();
|
||||
let texts: Vec<_> = documents
|
||||
.iter()
|
||||
.map(|doc| doc["text"].as_str().unwrap())
|
||||
.collect();
|
||||
let mut embedder = CpuEmbedder::load(&model, &runtime, 1).unwrap();
|
||||
let vectors = embedder.encode(&texts).unwrap();
|
||||
assert!(vectors.iter().all(|vector| vector.len() == DIMENSIONS));
|
||||
for vector in &vectors {
|
||||
let norm = vector.iter().map(|v| v * v).sum::<f32>().sqrt();
|
||||
assert!((norm - 1.0).abs() < 1e-5);
|
||||
}
|
||||
let mut correct = 0;
|
||||
for query in fixture["queries"].as_array().unwrap() {
|
||||
let vector = embedder
|
||||
.encode(&[query["text"].as_str().unwrap()])
|
||||
.unwrap()
|
||||
.remove(0);
|
||||
let mut scores: Vec<_> = vectors
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, candidate)| {
|
||||
(
|
||||
i,
|
||||
candidate
|
||||
.iter()
|
||||
.zip(&vector)
|
||||
.map(|(a, b)| a * b)
|
||||
.sum::<f32>(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
scores.sort_by(|a, b| b.1.total_cmp(&a.1));
|
||||
let rank = scores
|
||||
.iter()
|
||||
.position(|(i, _)| documents[*i]["id"] == query["expected"])
|
||||
.unwrap()
|
||||
+ 1;
|
||||
println!(
|
||||
"{}: expected rank {rank}; top result {}",
|
||||
query["language"], documents[scores[0].0]["id"]
|
||||
);
|
||||
assert!(
|
||||
rank <= 3,
|
||||
"Real model lost expected passage for {}",
|
||||
query["text"]
|
||||
);
|
||||
correct += usize::from(rank == 1);
|
||||
}
|
||||
// This is the measured proof baseline, not the production quality gate.
|
||||
// A known product-release query currently returns the expected result second.
|
||||
assert!(
|
||||
correct >= 5,
|
||||
"Semantic proof regressed below measured 5/6 top-1 baseline"
|
||||
);
|
||||
assert!(embedder.encode(&[" "]).is_err());
|
||||
assert!(embedder.encode(&["x"; 9]).is_err());
|
||||
}
|
||||
@@ -33,6 +33,7 @@ enabled. Model downloads and release checks must be explicit operations.
|
||||
- Substring filename search scans the metadata table; the million-file target
|
||||
requires an additional substring index and measured query planning.
|
||||
- Initial indexing publishes after the root transaction finishes.
|
||||
- File extraction currently covers bounded UTF-8 text, not PDF/DOCX/media.
|
||||
- File extraction covers bounded UTF-8, PDF text and DOCX paragraphs. OCR and
|
||||
media extraction remain upcoming work; parser workers are not isolated yet.
|
||||
- No automatic watcher, persistent extraction queue or ANN index yet.
|
||||
- Native CI and signed package validation require actual platform runners.
|
||||
@@ -0,0 +1,46 @@
|
||||
# Build and release targets
|
||||
|
||||
Install stable Rust, Node 22+, and native Tauri development prerequisites.
|
||||
On Windows install Visual Studio C++ build tools and WebView2. Linux needs
|
||||
WebKitGTK 4.1 development libraries. macOS needs Xcode command-line tools.
|
||||
See https://v2.tauri.app/start/prerequisites/ for platform details.
|
||||
|
||||
```powershell
|
||||
cargo test --workspace --locked
|
||||
cd apps/desktop
|
||||
npm ci
|
||||
npm run build
|
||||
# Windows VM: all rendering and inference tests must use CPU.
|
||||
$env:WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS='--disable-gpu'
|
||||
npm run tauri -- dev
|
||||
```
|
||||
|
||||
The desktop is a standalone Cargo workspace; validate it separately:
|
||||
|
||||
```sh
|
||||
cargo test --manifest-path apps/desktop/src-tauri/Cargo.toml --locked
|
||||
```
|
||||
|
||||
## Cross-compilation
|
||||
|
||||
The Rust engine has no GUI dependency. Add the destination Rust target with
|
||||
`rustup target add TARGET`, then use `cargo build -p searchi-cli --target TARGET`.
|
||||
This requires the target linker, C toolchain and SDK because SQLite is compiled
|
||||
from bundled C. Target installation alone is insufficient. Document each
|
||||
verified host/target pair before treating it as supported.
|
||||
|
||||
Desktop builds also need the platform webview libraries and packaging tools.
|
||||
Use native runners for dependable release builds. macOS signing/notarization
|
||||
needs Apple credentials and tooling. Windows installer cross-builds have
|
||||
additional Tauri caveats: https://v2.tauri.app/distribute/windows-installer/.
|
||||
|
||||
## CI and packaging gates
|
||||
|
||||
Both Gitea and GitHub workflow files are provided. Gitea needs registered runners
|
||||
matching the configured OS labels; this repository does not provision runners.
|
||||
No Linux/macOS checks are considered passed until actual job evidence exists.
|
||||
|
||||
Current bundles are disabled during development. Enable installers only after
|
||||
packaging icons, native smoke tests, CPU model distribution and license review,
|
||||
upgrade/migration tests, signatures and rollback checks are ready. Do not place
|
||||
private signing keys, repository tokens or model credentials in source control.
|
||||
+9
-2
@@ -10,14 +10,14 @@ Status: implementation started; no production release yet. Windows VM tests use
|
||||
- [x] Persistence and filesystem lifecycle integration tests (Windows CPU).
|
||||
|
||||
## 2. Document and semantic retrieval
|
||||
- [ ] Text, Markdown, code, PDF and DOCX extraction with fixtures.
|
||||
- [x] Text, Markdown, code, PDF and DOCX extraction with real fixtures (Windows CPU).
|
||||
- [ ] Validate actual embedding model artifacts, license, CPU runtime and quantization.
|
||||
- [ ] Versioned embeddings, ANN index, exact-match priority and hybrid fusion.
|
||||
- [ ] English/German relevance corpus and measured top-10 recall.
|
||||
- [ ] Atomic model/index replacement and resumable persistent work queues.
|
||||
|
||||
## 3. Desktop
|
||||
- [ ] Tauri shell, accessible keyboard search, progressive results and previews.
|
||||
- [ ] Tauri shell, accessible keyboard search, progressive results and previews (shell built; UI interaction verification pending).
|
||||
- [ ] Folder selection, exclusion controls, resource budgets and indexing status.
|
||||
- [ ] Global shortcut, opening/revealing source files, PDF page navigation.
|
||||
- [ ] Watchers and reconciliation across Windows, macOS and Linux.
|
||||
@@ -45,6 +45,13 @@ Completion requires recorded evidence; targets are not measured results.
|
||||
- Actual CLI process integration tests and independent-reader concurrency test.
|
||||
- TypeScript desktop frontend build and dependency audit passed; native validation underway.
|
||||
- Initial README commit pushed to Gitea main via protected Infisical injection.
|
||||
- Native desktop build and guarded opening tests passed on Windows.
|
||||
- Genuine multilingual CPU model proof: 5/6 top-1, 6/6 recall@3; quality gate remains open.
|
||||
- Native Rust CPU embedding adapter verifies pinned model/tokenizer assets;
|
||||
integration with the search index and platform runtime packaging remain open.
|
||||
- PDF page and DOCX paragraph matches persist; corrupt/encrypted/over-limit
|
||||
documents remain filename-searchable and report extraction errors.
|
||||
- CLI exposes folder/type/size/date filters and additional directory exclusions.
|
||||
|
||||
The partial features above do not close broader production milestones such as
|
||||
crash-resumable jobs, fuzzy retrieval, watcher reliability or performance targets.
|
||||
@@ -0,0 +1,155 @@
|
||||
# CPU semantic inference proof
|
||||
|
||||
This milestone runs genuine neural embeddings locally. It does not yet connect
|
||||
semantic vectors to the SQLite search index or desktop interface. Production
|
||||
retrieval quality, large-corpus performance, and Linux/macOS runtime packaging
|
||||
remain unverified.
|
||||
|
||||
## Model and runtime decision
|
||||
|
||||
The first working baseline is the quantized ONNX export of
|
||||
`sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, published by
|
||||
`Xenova`. The upstream model card describes 384-dimensional text embeddings,
|
||||
attention-mask mean pooling, a 128-token maximum, and Apache-2.0 licensing.
|
||||
It covers multiple languages. This is a text-only baseline; it does not claim
|
||||
EmbeddingGemma 2's multimodal features or compatibility.
|
||||
|
||||
Sources reviewed on 2026-10-12:
|
||||
|
||||
- [Original model card and pooling architecture](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2)
|
||||
- [ONNX export publisher](https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2)
|
||||
- [ONNX Runtime Python session API](https://onnxruntime.ai/docs/api/python/api_summary.html)
|
||||
- [Rust ort dynamic-library initialization](https://docs.rs/ort/2.0.0-rc.13/ort/environment/fn.init_from.html)
|
||||
- [Rust ort session configuration](https://docs.rs/ort/2.0.0-rc.13/ort/session/builder/struct.SessionBuilder.html)
|
||||
|
||||
Inference explicitly selects `CPUExecutionProvider`. Python uses the CPU
|
||||
`onnxruntime` wheel, never `onnxruntime-gpu`. The native Rust crate disables ort
|
||||
default features: there are no binary downloads, GPU provider features, or model
|
||||
HTTP client features. It loads an explicitly supplied local runtime library and
|
||||
disables runtime telemetry. Tokenization is sequential; ONNX sessions use
|
||||
sequential graph execution with bounded intra-op threads and one inter-op thread.
|
||||
|
||||
## Reproduce on Windows
|
||||
|
||||
Run these commands from the repository root. Provisioning commands intentionally
|
||||
use the network; the inference command does not provision anything.
|
||||
|
||||
```powershell
|
||||
python -m venv tools/semantic-proof/.venv
|
||||
& tools/semantic-proof/.venv/Scripts/python.exe -m pip install --only-binary=:all: -r tools/semantic-proof/requirements.txt
|
||||
& tools/semantic-proof/.venv/Scripts/python.exe tools/semantic-proof/download.py
|
||||
& tools/semantic-proof/.venv/Scripts/python.exe tools/semantic-proof/proof.py --threads 1 --output tools/semantic-proof/windows-cpu-result.json
|
||||
```
|
||||
|
||||
On Linux/macOS, use `.venv/bin/python` instead of `.venv/Scripts/python.exe`.
|
||||
These instructions have only been executed on Windows. Wheel availability and
|
||||
native library distribution must be checked on other platforms.
|
||||
|
||||
`download.py` downloads exactly two revision-pinned artifacts, streams them to a
|
||||
temporary file, rejects oversized downloads, checks byte length and SHA256, then
|
||||
atomically replaces the local asset. Assets occupy approximately 136 MB in
|
||||
decimal units and are ignored by Git. Inference validates both files again and
|
||||
fails clearly when either is missing or damaged. It never contacts the model
|
||||
hub. The Python proof sets offline environment flags and rejects Python socket
|
||||
connect, name-resolution, and bind audit events throughout import and inference.
|
||||
This guard checks Python socket activity; it is not an operating-system firewall
|
||||
against arbitrary native libraries.
|
||||
|
||||
The proof intentionally exits nonzero if any fixture query misses top-1. Its
|
||||
JSON report is written before the quality gate, preserving failures for review.
|
||||
|
||||
## Pinned artifacts and licensing
|
||||
|
||||
Revision: `2c4055b12046f11709e9df2c122e59ffbdc2f900`.
|
||||
The manifest in `tools/semantic-proof/model-manifest.json` records the same
|
||||
model contract and hashes used by the native adapter.
|
||||
|
||||
| Artifact | Bytes | SHA256 |
|
||||
| --- | ---: | --- |
|
||||
| `onnx/model_quantized.onnx` | 118,308,126 | `66fc00f5f29afcaff34092e1bdd20008ca3918265a82fb9695a551e510cc4ebc` |
|
||||
| `tokenizer.json` | 17,082,913 | `b60b6b43406a48bf3638526314f3d232d97058bc93472ff2de930d43686fa441` |
|
||||
|
||||
The base model card specifies Apache-2.0. The export card identifies that base
|
||||
model but does not display its own license declaration. Before redistributing
|
||||
the ONNX export in installers, confirm export licensing and preserve the base
|
||||
license, attribution, and applicable notices. Current code explicitly downloads
|
||||
user-local assets and does not ship model weights or runtime DLLs in Git.
|
||||
ONNX Runtime uses MIT licensing; ort uses MIT or Apache-2.0 and tokenizers uses
|
||||
Apache-2.0. Release packaging must retain third-party notices and validate
|
||||
platform-specific runtime hashes independently; a model hash does not attest
|
||||
to the runtime library.
|
||||
|
||||
The tested Windows ONNX Runtime 1.31.0 DLL is 19,288,888 bytes with SHA256
|
||||
`90497e3d406c09f4d59d364c696a2f93dc02f6ef13741b3b89eec95bdb4a3806`.
|
||||
The adapter currently trusts the explicitly supplied runtime path; it validates
|
||||
model hashes, not runtime-library hashes. A release distribution must add a
|
||||
platform runtime manifest before shipping. `requirements-lock-windows.txt`
|
||||
captures the exact Python environment used for this proof.
|
||||
|
||||
## Measured Windows VM result
|
||||
|
||||
`tools/semantic-proof/windows-cpu-result.json` contains the full measurement and
|
||||
rankings. The run used Windows 11 AMD64, Python 3.14.8, ONNX Runtime 1.31.0,
|
||||
four visible logical CPUs, and **one inference thread with only the CPU provider**.
|
||||
|
||||
| Metric | Measurement |
|
||||
| --- | ---: |
|
||||
| Fixture documents / queries | 8 / 6 |
|
||||
| Timed warm query runs | 60 |
|
||||
| Artifact verification plus runtime/model load | approximately 3.4 seconds |
|
||||
| Corpus batch embedding | approximately 145 ms |
|
||||
| Warm query p50 / p95 | approximately 3.1 / 11.9 ms |
|
||||
| Top-1 accuracy | 5/6 (83.3%) |
|
||||
| Recall at 3 | 6/6 (100%) |
|
||||
|
||||
The corpus includes English and German documents, same-language semantic
|
||||
queries, and cross-language queries. A question about a postponed product
|
||||
release ranks the supplier contract above the expected meeting notes. That
|
||||
failure remains in the fixture and blocks the strict top-1 quality gate.
|
||||
This small demonstration supports the feasibility of CPU inference; it does
|
||||
not establish large-corpus latency, robust multilingual quality, or production
|
||||
readiness. Quantization and batch shape can affect similarity scores.
|
||||
|
||||
## Native integration contract
|
||||
|
||||
`searchi-semantic` exposes:
|
||||
|
||||
```rust
|
||||
CpuEmbedder::load(model_dir, runtime_library, threads) -> Result<CpuEmbedder>
|
||||
CpuEmbedder::encode(&mut self, texts: &[&str]) -> Result<Vec<Vec<f32>>>
|
||||
DIMENSIONS // 384
|
||||
MODEL_FINGERPRINT // revision + quantization + pooling + normalization + token cap
|
||||
```
|
||||
|
||||
Inputs are bounded to eight texts per batch, 64 KiB UTF-8 per text, and 128
|
||||
tokens. Outputs are finite, normalized `f32` vectors. The runtime library is
|
||||
process-global; adapter initialization rejects attempts to switch libraries or
|
||||
silently accept an environment initialized outside the adapter. Supply a known,
|
||||
trusted runtime before calling any other ort APIs in the process.
|
||||
|
||||
Default unit tests need no model or runtime. The real-artifact test is opt-in:
|
||||
|
||||
```powershell
|
||||
$env:SEARCHI_MODEL_DIR = (Resolve-Path tools/semantic-proof/model).Path
|
||||
$env:SEARCHI_ORT_LIBRARY = (Resolve-Path tools/semantic-proof/.venv/Lib/site-packages/onnxruntime/capi/onnxruntime.dll).Path
|
||||
cargo test -p searchi-semantic --test cpu_model -- --ignored --nocapture
|
||||
```
|
||||
|
||||
The smoke test checks actual local inference, vector normalization, English and
|
||||
German retrieval, recall at three, and the measured five-of-six top-1 baseline.
|
||||
It does not replace the strict production quality gate.
|
||||
|
||||
The native real-artifact smoke was executed successfully on the Windows VM,
|
||||
using that CPU-only DLL. It reproduced five-of-six top-1 and six-of-six recall
|
||||
at three. Three default unit tests passed; targeted Clippy passed with warnings
|
||||
denied. Linux and macOS native checks have not been executed.
|
||||
|
||||
The next integration should read bounded `SearchIndex::passage_batch` rows,
|
||||
split long passages into overlapping tokenizer windows, preserve source page or
|
||||
paragraph locations, and persist each vector with a content fingerprint and
|
||||
`MODEL_FINGERPRINT`. Passage IDs are not immutable content identities: changes
|
||||
must invalidate stale vectors. Queries should combine lexical and semantic
|
||||
retrieval while preserving exact filename priority. Model upgrades build a
|
||||
separate vector generation before an atomic swap. Cancellation, persistent
|
||||
embedding jobs, ANN retrieval, representative quality corpora, and signed
|
||||
cross-platform runtime packaging are still required.
|
||||
@@ -9,6 +9,23 @@ Initial foundation checks (2026-10-12):
|
||||
- `cargo clippy --workspace --all-targets -- -D warnings`: passed.
|
||||
- Rust formatting applied; final formatting verification follows integration.
|
||||
- Desktop TypeScript/Vite build passed; npm audit reported zero vulnerabilities.
|
||||
- Initial Windows desktop native check, embedded app build, strict Clippy and
|
||||
two desktop tests passed (index gate recovery and guarded source opening).
|
||||
- Desktop launched with WebView2 `--disable-gpu`. UI automation could not capture
|
||||
the window: `GetCursorPos` access denied, then monitor capture error
|
||||
`0x80070057`. Visual layout and end-to-end UI interactions remain unverified.
|
||||
- Real Python CPU ONNX embedding proof completed on 8 documents / 6 queries:
|
||||
5/6 top-1 accuracy, 6/6 recall@3. The strict top-1 quality gate is not met.
|
||||
See `tools/semantic-proof/windows-cpu-result.json` for raw timings and rankings.
|
||||
This tiny fixture is not evidence of large-corpus performance.
|
||||
- Native Rust CPU embedding adapter compiled, passed its three unit tests and
|
||||
the explicit local-model smoke test. The smoke reproduced the same 5/6 top-1
|
||||
limitation. Default CI omits this asset-dependent test; it must be executed
|
||||
separately with provisioned local artifacts.
|
||||
- Integrated workspace validation: 19 default tests passed; 1 local-model test
|
||||
is opt-in. Rust formatting and strict Clippy passed after lockfile refresh.
|
||||
- PDF/DOCX fixture tests include matching source locations, corrupt replacement,
|
||||
encrypted PDF behavior and compressed-input limits.
|
||||
|
||||
Not yet established: production latency targets, semantic inference, desktop
|
||||
runtime behavior, Linux/macOS execution, installer signing or upgrades.
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
.venv/
|
||||
model/
|
||||
__pycache__/
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Explicit provisioning only; inference never imports this downloader."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import urllib.request
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def digest(path):
|
||||
with path.open("rb") as source:
|
||||
return hashlib.file_digest(source, "sha256").hexdigest()
|
||||
|
||||
|
||||
def verified(path, spec):
|
||||
return path.is_file() and path.stat().st_size == spec["bytes"] and digest(path) == spec["sha256"]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Explicitly download ~136 MB of pinned, verified model artifacts")
|
||||
parser.add_argument("--model-dir", type=Path, default=HERE / "model")
|
||||
args = parser.parse_args()
|
||||
manifest = json.loads((HERE / "model-manifest.json").read_text())
|
||||
for spec in manifest["files"]:
|
||||
target = args.model_dir / spec["path"]
|
||||
if verified(target, spec):
|
||||
print(f"Verified existing {spec['path']}")
|
||||
continue
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = target.with_suffix(target.suffix + ".part")
|
||||
url = f"https://huggingface.co/{manifest['model_id']}/resolve/{manifest['revision']}/{spec['path']}"
|
||||
print(f"Downloading {spec['path']} ({spec['bytes']} bytes)", flush=True)
|
||||
try:
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "Searchi-semantic-proof/0.1"})
|
||||
with urllib.request.urlopen(request, timeout=60) as response, temporary.open("wb") as output:
|
||||
total = 0
|
||||
while chunk := response.read(1024 * 1024):
|
||||
total += len(chunk)
|
||||
if total > spec["bytes"]:
|
||||
raise ValueError("Download exceeds manifest size")
|
||||
output.write(chunk)
|
||||
if not verified(temporary, spec):
|
||||
raise ValueError(f"Size or SHA256 mismatch for {spec['path']}")
|
||||
os.replace(temporary, target)
|
||||
print(f"Verified SHA256 {spec['sha256']}")
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"documents": [
|
||||
{"id": "supplier-contract", "text": "The supplier agreement may be ended by giving sixty days written notice. Early cancellation incurs no additional fee."},
|
||||
{"id": "backup-policy", "text": "Backups are encrypted and stored each night. Recovery copies are retained for thirty days."},
|
||||
{"id": "travel-expenses", "text": "Business train tickets and hotel stays can be reimbursed. Submit receipts with the monthly expense report."},
|
||||
{"id": "meeting-notes", "text": "The design team decided to move the launch to November because accessibility testing needs another week."},
|
||||
{"id": "apartment-lease", "text": "Die Wohnung kostet monatlich 950 Euro. Die Kaution beträgt drei Monatsmieten und wird nach dem Auszug zurückgezahlt."},
|
||||
{"id": "garden-guide", "text": "Tomaten brauchen viel Sonnenlicht und regelmäßiges Gießen. Die Pflanzen sollten vor Frost geschützt werden."},
|
||||
{"id": "password-reset", "text": "To recover your account, request a password reset link. The link expires after fifteen minutes."},
|
||||
{"id": "invoice", "text": "Invoice 1042 lists five software licenses at forty euros each. Payment is due on the first of December."}
|
||||
],
|
||||
"queries": [
|
||||
{"text": "How do we stop working with the vendor?", "expected": "supplier-contract", "language": "en"},
|
||||
{"text": "Wie können wir die Zusammenarbeit mit dem Lieferanten beenden?", "expected": "supplier-contract", "language": "de-crosslingual"},
|
||||
{"text": "Wo finde ich die Regeln zur Erstattung meiner Dienstreise?", "expected": "travel-expenses", "language": "de-crosslingual"},
|
||||
{"text": "Why was the product release postponed?", "expected": "meeting-notes", "language": "en"},
|
||||
{"text": "How much money must I leave as a security deposit for the flat?", "expected": "apartment-lease", "language": "en-crosslingual"},
|
||||
{"text": "Wie pflege ich die Gemüsepflanzen im Garten?", "expected": "garden-guide", "language": "de"}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"model_id": "Xenova/paraphrase-multilingual-MiniLM-L12-v2",
|
||||
"revision": "2c4055b12046f11709e9df2c122e59ffbdc2f900",
|
||||
"base_model": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
|
||||
"base_license": "Apache-2.0",
|
||||
"dimensions": 384,
|
||||
"max_tokens": 128,
|
||||
"pooling": "attention-mask-mean",
|
||||
"normalization": "l2",
|
||||
"files": [
|
||||
{"path": "onnx/model_quantized.onnx", "bytes": 118308126, "sha256": "66fc00f5f29afcaff34092e1bdd20008ca3918265a82fb9695a551e510cc4ebc"},
|
||||
{"path": "tokenizer.json", "bytes": 17082913, "sha256": "b60b6b43406a48bf3638526314f3d232d97058bc93472ff2de930d43686fa441"}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
"""Real, local ONNX text embeddings with an enforced CPU-only session."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
|
||||
# Fail rather than silently contacting a model service or downloading an asset.
|
||||
os.environ["HF_HUB_OFFLINE"] = "1"
|
||||
os.environ["TRANSFORMERS_OFFLINE"] = "1"
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
|
||||
|
||||
def reject_network(event, args):
|
||||
if event in {"socket.connect", "socket.getaddrinfo", "socket.bind"}:
|
||||
raise RuntimeError("Semantic proof inference is offline; network access is forbidden")
|
||||
|
||||
|
||||
sys.addaudithook(reject_network)
|
||||
|
||||
import numpy as np
|
||||
import onnxruntime as ort
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
class CpuEmbedder:
|
||||
"""Bounded reference contract for a future Rust ONNX adapter."""
|
||||
|
||||
def __init__(self, model_dir, threads=1):
|
||||
if not 1 <= threads <= 8:
|
||||
raise ValueError("threads must be between 1 and 8")
|
||||
self.manifest = json.loads((HERE / "model-manifest.json").read_text())
|
||||
for spec in self.manifest["files"]:
|
||||
path = model_dir / spec["path"]
|
||||
if not path.is_file():
|
||||
raise ValueError(f"Missing local artifact {path}; explicitly run download.py first")
|
||||
with path.open("rb") as source:
|
||||
checksum = hashlib.file_digest(source, "sha256").hexdigest()
|
||||
if path.stat().st_size != spec["bytes"] or checksum != spec["sha256"]:
|
||||
raise ValueError(f"Artifact size/SHA256 mismatch: {path}")
|
||||
options = ort.SessionOptions()
|
||||
options.intra_op_num_threads = threads
|
||||
options.inter_op_num_threads = 1
|
||||
options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
|
||||
self.session = ort.InferenceSession(
|
||||
str(model_dir / "onnx/model_quantized.onnx"),
|
||||
sess_options=options,
|
||||
providers=["CPUExecutionProvider"],
|
||||
)
|
||||
if self.session.get_providers() != ["CPUExecutionProvider"]:
|
||||
raise RuntimeError("Only CPUExecutionProvider is allowed")
|
||||
self.tokenizer = Tokenizer.from_file(str(model_dir / "tokenizer.json"))
|
||||
self.tokenizer.enable_truncation(max_length=self.manifest["max_tokens"])
|
||||
self.tokenizer.enable_padding(pad_id=1, pad_token="<pad>")
|
||||
|
||||
def encode(self, texts):
|
||||
if not texts or len(texts) > 8:
|
||||
raise ValueError("Batch size must be between 1 and 8")
|
||||
if any(not text.strip() or len(text.encode("utf-8")) > 65536 for text in texts):
|
||||
raise ValueError("Text must be nonempty and at most 64 KiB UTF-8")
|
||||
encoded = self.tokenizer.encode_batch(texts)
|
||||
available = {
|
||||
"input_ids": np.asarray([item.ids for item in encoded], dtype=np.int64),
|
||||
"attention_mask": np.asarray([item.attention_mask for item in encoded], dtype=np.int64),
|
||||
"token_type_ids": np.asarray([item.type_ids for item in encoded], dtype=np.int64),
|
||||
}
|
||||
inputs = {item.name: available[item.name] for item in self.session.get_inputs()}
|
||||
hidden = self.session.run(None, inputs)[0]
|
||||
mask = available["attention_mask"][..., None].astype(np.float32)
|
||||
pooled = (hidden * mask).sum(axis=1) / np.maximum(mask.sum(axis=1), 1e-9)
|
||||
vectors = pooled / np.maximum(np.linalg.norm(pooled, axis=1, keepdims=True), 1e-9)
|
||||
if vectors.shape != (len(texts), self.manifest["dimensions"]) or not np.isfinite(vectors).all():
|
||||
raise RuntimeError("Invalid model embedding shape or nonfinite values")
|
||||
return vectors.astype(np.float32)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--model-dir", type=Path, default=HERE / "model")
|
||||
parser.add_argument("--fixture", type=Path, default=HERE / "fixture.json")
|
||||
parser.add_argument("--output", type=Path)
|
||||
parser.add_argument("--threads", type=int, default=1, choices=range(1, 9))
|
||||
parser.add_argument("--repeats", type=int, default=10, choices=range(1, 101))
|
||||
args = parser.parse_args()
|
||||
fixture = json.loads(args.fixture.read_text(encoding="utf-8"))
|
||||
started = time.perf_counter()
|
||||
embedder = CpuEmbedder(args.model_dir, args.threads)
|
||||
load_ms = (time.perf_counter() - started) * 1000
|
||||
started = time.perf_counter()
|
||||
vectors = embedder.encode([item["text"] for item in fixture["documents"]])
|
||||
corpus_ms = (time.perf_counter() - started) * 1000
|
||||
rankings = []
|
||||
timings = []
|
||||
for query in fixture["queries"]:
|
||||
vector = embedder.encode([query["text"]])[0] # Warm this input shape.
|
||||
for _ in range(args.repeats):
|
||||
started = time.perf_counter()
|
||||
vector = embedder.encode([query["text"]])[0]
|
||||
scores = vectors @ vector
|
||||
order = np.argsort(-scores)
|
||||
timings.append((time.perf_counter() - started) * 1000)
|
||||
ranked = [{"id": fixture["documents"][int(i)]["id"], "cosine": float(scores[i])} for i in order]
|
||||
expected_rank = next(i + 1 for i, result in enumerate(ranked) if result["id"] == query["expected"])
|
||||
rankings.append({**query, "expected_rank": expected_rank, "top3": ranked[:3]})
|
||||
report = {
|
||||
"platform": platform.platform(), "python": platform.python_version(),
|
||||
"machine": platform.machine(), "logical_cpus": os.cpu_count(),
|
||||
"onnxruntime": ort.__version__, "numpy": np.__version__,
|
||||
"providers": embedder.session.get_providers(), "threads": args.threads,
|
||||
"model": embedder.manifest, "documents": len(fixture["documents"]),
|
||||
"queries": len(rankings), "timed_query_runs": len(timings),
|
||||
"artifact_verify_and_load_ms": load_ms, "corpus_embedding_ms": corpus_ms,
|
||||
"query_p50_ms": statistics.median(timings),
|
||||
"query_p95_ms": float(np.percentile(timings, 95)),
|
||||
"top1_accuracy": sum(item["expected_rank"] == 1 for item in rankings) / len(rankings),
|
||||
"recall_at_3": sum(item["expected_rank"] <= 3 for item in rankings) / len(rankings),
|
||||
"network": "Python socket audit guard active during import, model load, and inference",
|
||||
"rankings": rankings,
|
||||
}
|
||||
serialized = json.dumps(report, ensure_ascii=False, indent=2)
|
||||
if args.output:
|
||||
args.output.write_text(serialized + "\n", encoding="utf-8")
|
||||
print(serialized)
|
||||
if report["top1_accuracy"] < 1:
|
||||
raise SystemExit("Fixture top-1 quality gate failed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,21 @@
|
||||
anyio==4.15.1
|
||||
click==8.5.0
|
||||
colorama==0.4.6
|
||||
filelock==4.1.1
|
||||
flatbuffers==25.12.19
|
||||
fsspec==2026.9.0
|
||||
h11==0.16.0
|
||||
hf-xet==1.7.0
|
||||
httpcore2==2.13.1
|
||||
httpx2==2.13.1
|
||||
huggingface_hub==2.2.0
|
||||
idna==3.20
|
||||
numpy==2.5.4
|
||||
onnxruntime==1.31.0
|
||||
packaging==26.3
|
||||
protobuf==7.36.2
|
||||
PyYAML==6.0.3
|
||||
tokenizers==0.23.3
|
||||
tqdm==4.70.1
|
||||
truststore==0.10.4
|
||||
typing_extensions==4.16.0
|
||||
@@ -0,0 +1,4 @@
|
||||
# CPU wheel only. Never replace with onnxruntime-gpu.
|
||||
onnxruntime==1.31.0
|
||||
tokenizers==0.23.3
|
||||
numpy==2.5.4
|
||||
@@ -0,0 +1,166 @@
|
||||
{
|
||||
"platform": "Windows-11-10.0.26300-SP0",
|
||||
"python": "3.14.8",
|
||||
"machine": "AMD64",
|
||||
"logical_cpus": 4,
|
||||
"onnxruntime": "1.31.0",
|
||||
"numpy": "2.5.4",
|
||||
"providers": [
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"threads": 1,
|
||||
"model": {
|
||||
"model_id": "Xenova/paraphrase-multilingual-MiniLM-L12-v2",
|
||||
"revision": "2c4055b12046f11709e9df2c122e59ffbdc2f900",
|
||||
"base_model": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
|
||||
"base_license": "Apache-2.0",
|
||||
"dimensions": 384,
|
||||
"max_tokens": 128,
|
||||
"pooling": "attention-mask-mean",
|
||||
"normalization": "l2",
|
||||
"files": [
|
||||
{
|
||||
"path": "onnx/model_quantized.onnx",
|
||||
"bytes": 118308126,
|
||||
"sha256": "66fc00f5f29afcaff34092e1bdd20008ca3918265a82fb9695a551e510cc4ebc"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer.json",
|
||||
"bytes": 17082913,
|
||||
"sha256": "b60b6b43406a48bf3638526314f3d232d97058bc93472ff2de930d43686fa441"
|
||||
}
|
||||
]
|
||||
},
|
||||
"documents": 8,
|
||||
"queries": 6,
|
||||
"timed_query_runs": 60,
|
||||
"artifact_verify_and_load_ms": 3399.298300006194,
|
||||
"corpus_embedding_ms": 145.497699995758,
|
||||
"query_p50_ms": 3.0544000037480146,
|
||||
"query_p95_ms": 11.88376000645803,
|
||||
"top1_accuracy": 0.8333333333333334,
|
||||
"recall_at_3": 1.0,
|
||||
"network": "Python socket audit guard active during import, model load, and inference",
|
||||
"rankings": [
|
||||
{
|
||||
"text": "How do we stop working with the vendor?",
|
||||
"expected": "supplier-contract",
|
||||
"language": "en",
|
||||
"expected_rank": 1,
|
||||
"top3": [
|
||||
{
|
||||
"id": "supplier-contract",
|
||||
"cosine": 0.389570415019989
|
||||
},
|
||||
{
|
||||
"id": "invoice",
|
||||
"cosine": 0.16107530891895294
|
||||
},
|
||||
{
|
||||
"id": "travel-expenses",
|
||||
"cosine": 0.06973522156476974
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"text": "Wie können wir die Zusammenarbeit mit dem Lieferanten beenden?",
|
||||
"expected": "supplier-contract",
|
||||
"language": "de-crosslingual",
|
||||
"expected_rank": 1,
|
||||
"top3": [
|
||||
{
|
||||
"id": "supplier-contract",
|
||||
"cosine": 0.5199419260025024
|
||||
},
|
||||
{
|
||||
"id": "invoice",
|
||||
"cosine": 0.21696728467941284
|
||||
},
|
||||
{
|
||||
"id": "meeting-notes",
|
||||
"cosine": 0.14755962789058685
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"text": "Wo finde ich die Regeln zur Erstattung meiner Dienstreise?",
|
||||
"expected": "travel-expenses",
|
||||
"language": "de-crosslingual",
|
||||
"expected_rank": 1,
|
||||
"top3": [
|
||||
{
|
||||
"id": "travel-expenses",
|
||||
"cosine": 0.5095880627632141
|
||||
},
|
||||
{
|
||||
"id": "apartment-lease",
|
||||
"cosine": 0.3584939241409302
|
||||
},
|
||||
{
|
||||
"id": "invoice",
|
||||
"cosine": 0.2985326647758484
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"text": "Why was the product release postponed?",
|
||||
"expected": "meeting-notes",
|
||||
"language": "en",
|
||||
"expected_rank": 2,
|
||||
"top3": [
|
||||
{
|
||||
"id": "supplier-contract",
|
||||
"cosine": 0.3999302387237549
|
||||
},
|
||||
{
|
||||
"id": "meeting-notes",
|
||||
"cosine": 0.3511156439781189
|
||||
},
|
||||
{
|
||||
"id": "backup-policy",
|
||||
"cosine": 0.19898240268230438
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"text": "How much money must I leave as a security deposit for the flat?",
|
||||
"expected": "apartment-lease",
|
||||
"language": "en-crosslingual",
|
||||
"expected_rank": 1,
|
||||
"top3": [
|
||||
{
|
||||
"id": "apartment-lease",
|
||||
"cosine": 0.5188605785369873
|
||||
},
|
||||
{
|
||||
"id": "travel-expenses",
|
||||
"cosine": 0.2738097310066223
|
||||
},
|
||||
{
|
||||
"id": "backup-policy",
|
||||
"cosine": 0.2614522874355316
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"text": "Wie pflege ich die Gemüsepflanzen im Garten?",
|
||||
"expected": "garden-guide",
|
||||
"language": "de",
|
||||
"expected_rank": 1,
|
||||
"top3": [
|
||||
{
|
||||
"id": "garden-guide",
|
||||
"cosine": 0.5030614733695984
|
||||
},
|
||||
{
|
||||
"id": "apartment-lease",
|
||||
"cosine": 0.03017299994826317
|
||||
},
|
||||
{
|
||||
"id": "meeting-notes",
|
||||
"cosine": 0.030125245451927185
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in new issue
Block a user