karl
caf6a2f82e
Core CI / core (ubuntu-latest) (push) Failing after 3m27s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m45s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
53 lines
2.8 KiB
Markdown
53 lines
2.8 KiB
Markdown
# Searchi assets
|
|
|
|
GUI-free model provisioning shared by desktop setup and headless CLI setup.
|
|
Search never invokes this downloader automatically. Inference and GPU providers
|
|
are outside this crate; provisioning tests run on the Windows VM using CPU only.
|
|
|
|
The compiled [model manifest](../../tools/embeddinggemma2/model-manifest.json)
|
|
pins Google EmbeddingGemma 2's text GGUF and multimedia projection to one
|
|
Hugging Face export commit, exact byte lengths, and SHA-256 digests. The two
|
|
artifacts total 864,676,480 bytes. Source URLs cannot be overridden through the
|
|
public API. Executable embedding workers remain bundled release resources and
|
|
are never downloaded by this crate.
|
|
|
|
```rust,no_run
|
|
use std::sync::atomic::AtomicU64;
|
|
let directory = searchi_assets::default_model_directory()?;
|
|
let progress = AtomicU64::new(0);
|
|
if !searchi_assets::verify_cached(&directory)? {
|
|
// Only after the user explicitly chooses setup/download.
|
|
searchi_assets::provision(&directory, &progress)?;
|
|
}
|
|
# Ok::<(), anyhow::Error>(())
|
|
```
|
|
|
|
- `default_model_directory()` computes the platform-local application data path
|
|
via `directories::ProjectDirs`, then appends `models/embeddinggemma2`.
|
|
- `verify_cached(directory)` performs offline size and full-digest validation.
|
|
It does not trust a same-size file or create directories.
|
|
- `provision(directory, progress)` verifies/reuses cached artifacts and downloads
|
|
only missing or corrupt files. Call it on a blocking setup worker.
|
|
- `TOTAL_BYTES` supplies the progress total to GUI and CLI consumers.
|
|
|
|
New files are streamed through a 64 KiB buffer and rejected immediately if the
|
|
body exceeds the pinned size. Content-Length, when supplied, must also match.
|
|
The final byte count and SHA-256 must match before the temporary file is synced
|
|
and atomically replaces the destination in the same directory. A failed attempt
|
|
leaves any existing destination intact and deletes its temporary file through
|
|
RAII. Unix directory metadata is also synced after replacement.
|
|
|
|
There are three attempts per artifact, with 250/500 ms retry delays. Connections
|
|
time out after 20 seconds, and an entire request including streamed body reads
|
|
after 900 seconds. Progress includes completed verified artifacts and the current
|
|
attempt's received bytes; it resets to completed verified bytes after failure.
|
|
Hard process termination can leave orphan `.part` files, which are never treated
|
|
as valid models. Downloads resume by reusing completed verified artifacts rather
|
|
than trusting partial files.
|
|
|
|
Tests use a private local HTTP fixture and small authentic byte payloads; they
|
|
do not download the production model or rely on a public service. They verify
|
|
same-size corruption, cache reuse without URL requests, interrupted-response
|
|
retry, streaming size bounds, incorrect digests, atomic replacement, and partial
|
|
file cleanup. Linux/macOS runtime checks remain pending.
|