karl
b8400c4fbf
Core CI / core (ubuntu-latest) (push) Failing after 3m31s
Core CI / core (macos-latest) (push) Canceled after 0s
Core CI / core (windows-latest) (push) Canceled after 0s
Core CI / desktop (ubuntu-latest) (push) Successful in 5m39s
Core CI / desktop (macos-latest) (push) Canceled after 0s
Core CI / desktop (windows-latest) (push) Canceled after 0s
71 lines
4.3 KiB
Markdown
71 lines
4.3 KiB
Markdown
# Local media preparation
|
|
|
|
`searchi-media` prepares actual files for the Google multimodal embedder. It
|
|
does not download programs or models and has no UI dependency. Callers supply
|
|
verified packaged FFmpeg/ffprobe paths through `FfmpegTools::new`, or explicitly
|
|
set both `SEARCHI_FFMPEG_PATH` and `SEARCHI_FFPROBE_PATH`.
|
|
|
|
FFmpeg handles maintained codecs and container timestamps; serde parses its
|
|
bounded JSON, tempfile owns frame cleanup, and sha2 validates file identity.
|
|
The embedding crate uses the Rust image library for bounded raster decoding.
|
|
USearch supplies the service's persisted HNSW index; no codec or vector index
|
|
is reimplemented here.
|
|
|
|
Every FFmpeg/ffprobe media input restricts protocols to `file,pipe`. A selected
|
|
local playlist cannot fetch HTTP/HTTPS/RTSP references. A real loopback-listener
|
|
regression verifies probing and both decoding paths never connect. This is a
|
|
protocol restriction, not a filesystem sandbox: explicitly selected files on
|
|
network-mounted drives still use their operating system's filesystem access.
|
|
|
|
Windows VM tests use CPU only. Decoder commands specify one thread and
|
|
`-hwaccel none`; subprocess windows are hidden. Portable argument arrays avoid
|
|
shell interpretation. Production hardware decode and reliable CPU fallback
|
|
remain unimplemented; this crate does not silently enable accelerators.
|
|
|
|
Supported images are PNG, JPEG, WebP and BMP (actual bounded embedding-library
|
|
decode, including EXIF orientation). AVIF, HEIC, TIFF, animated GIF and embedded
|
|
album artwork are not supported yet. Movie containers include MP4, MKV, MOV,
|
|
AVI, WebM, M4V, MPEG/MPG; audio includes MP3, FLAC, WAV, M4A, Ogg, Opus, AAC,
|
|
AIFF. Actual codec support comes from the packaged FFmpeg build. Unknown or
|
|
corrupt files remain filename-searchable with explicit indexing errors.
|
|
|
|
Default limits: 16 GiB media input; four-hour known duration; 16 million source
|
|
video pixels; 45 seconds per decoder command; 64 KiB ffprobe JSON; 256 KiB
|
|
frame diagnostic output; 8 MiB encoded PNG frame. Images are separately limited
|
|
to 64 MiB input. Captured pipe output is bounded and timed processes are killed.
|
|
Process address-space sandboxing remains a release gate; these limits do not
|
|
guarantee a codec's internal allocations fit a fixed RAM ceiling.
|
|
|
|
Video samples span the whole timeline at two-second intervals, becoming coarser
|
|
when the default 240 distributed-frame budget is reached. Up to 120 additional
|
|
scene-cut frames are captured within 1.5-second windows around sample anchors.
|
|
This is **partial visual coverage**, not exhaustive scene detection. Actual
|
|
decoded presentation timestamps are retained relative to container start. The
|
|
primary non-cover-art video stream is selected. Full images use model budget
|
|
280; video frames use 70. No fabricated captions are stored.
|
|
|
|
The primary audio stream is decoded to mono 16 kHz PCM, embedded in 30-second
|
|
windows with two seconds of overlap. The complete stream is covered or the
|
|
600-window budget returns an explicit failure. Standalone music uses the same
|
|
real audio embedding; artist/title/album/genre tags are merged from genuine
|
|
stream and format metadata and stored separately for lexical search. Audio
|
|
embeddings are not transcripts; exact spoken-word retrieval is not provided.
|
|
Alternative language tracks and isolated channels remain a release gate.
|
|
|
|
The service publishes text, image, video and audio vectors as one generation.
|
|
It persists origin and page/timestamp locations, policy/model fingerprints,
|
|
full file SHA at indexing, and size/mtime guards. Corrupt-file failures discard
|
|
that file's partial vectors and persist reasons. Model inference failure aborts
|
|
the generation. At query time only actual media candidate hits are stat-checked;
|
|
no corpus-wide hash/stat scan occurs. Changes preserving both size and mtime
|
|
cannot be detected by this fast guard; periodic content-hash reconciliation is
|
|
a release gate. Builds currently rebuild a complete generation, with a
|
|
100,000-vector / 10,000-media-file bound; unchanged-vector reuse is future work.
|
|
|
|
Offline tests cover complete audio windows, frame budgets and identity changes.
|
|
The ignored `real_decoder_preserves_frame_timestamps_and_audio` test has been
|
|
executed successfully in the CPU Windows VM with provisioned FFmpeg 9.0.2.
|
|
It creates genuine FFV1/PCM media, validates decoded two-second PTS and actual
|
|
mono sine samples. Linux/macOS execution and installer codec/license delivery
|
|
have not been validated.
|