searchi/crates/searchi-media/README.md

Local media preparation

searchi-media prepares actual files for the Google multimodal embedder. It does not download programs or models and has no UI dependency. Callers supply verified packaged FFmpeg/ffprobe paths through FfmpegTools::new, or explicitly set both SEARCHI_FFMPEG_PATH and SEARCHI_FFPROBE_PATH.

FFmpeg handles maintained codecs and container timestamps; serde parses its bounded JSON, tempfile owns frame cleanup, and sha2 validates file identity. The embedding crate uses the Rust image library for bounded raster decoding. USearch supplies the service's persisted HNSW index; no codec or vector index is reimplemented here.

Every FFmpeg/ffprobe media input restricts protocols to file,pipe. A selected local playlist cannot fetch HTTP/HTTPS/RTSP references. A real loopback-listener regression verifies probing and both decoding paths never connect. This is a protocol restriction, not a filesystem sandbox: explicitly selected files on network-mounted drives still use their operating system's filesystem access.

Windows VM tests use CPU only. Decoder commands specify one thread and -hwaccel none; subprocess windows are hidden. Portable argument arrays avoid shell interpretation. Production hardware decode and reliable CPU fallback remain unimplemented; this crate does not silently enable accelerators.

Supported images are PNG, JPEG, WebP and BMP (actual bounded embedding-library decode, including EXIF orientation). AVIF, HEIC, TIFF, animated GIF and embedded album artwork are not supported yet. Movie containers include MP4, MKV, MOV, AVI, WebM, M4V, MPEG/MPG; audio includes MP3, FLAC, WAV, M4A, Ogg, Opus, AAC, AIFF. Actual codec support comes from the packaged FFmpeg build. Unknown or corrupt files remain filename-searchable with explicit indexing errors.

Default limits: 16 GiB media input; four-hour known duration; 16 million source video pixels; 45 seconds per decoder command; 64 KiB ffprobe JSON; 256 KiB frame diagnostic output; 8 MiB encoded PNG frame. Images are separately limited to 64 MiB input. Captured pipe output is bounded and timed processes are killed. Process address-space sandboxing remains a release gate; these limits do not guarantee a codec's internal allocations fit a fixed RAM ceiling.

Video samples span the whole timeline at two-second intervals, becoming coarser when the default 240 distributed-frame budget is reached. Up to 120 additional scene-cut frames are captured within 1.5-second windows around sample anchors. This is partial visual coverage, not exhaustive scene detection. Actual decoded presentation timestamps are retained relative to container start. The primary non-cover-art video stream is selected. Full images use model budget 280; video frames use 70. No fabricated captions are stored.

The primary audio stream is decoded to mono 16 kHz PCM, embedded in 30-second windows with two seconds of overlap. The complete stream is covered or the 600-window budget returns an explicit failure. Standalone music uses the same real audio embedding; artist/title/album/genre tags are merged from genuine stream and format metadata and stored separately for lexical search. Audio embeddings are not transcripts; exact spoken-word retrieval is not provided. Alternative language tracks and isolated channels remain a release gate.

The service publishes text, image, video and audio vectors as one generation. It persists origin and page/timestamp locations, policy/model fingerprints, full file SHA at indexing, and size/mtime guards. Corrupt-file failures discard that file's partial vectors and persist reasons. Model inference failure aborts the generation. At query time only actual media candidate hits are stat-checked; no corpus-wide hash/stat scan occurs. Changes preserving both size and mtime cannot be detected by this fast guard; periodic content-hash reconciliation is a release gate. Builds currently rebuild a complete generation, with a 100,000-vector / 10,000-media-file bound; unchanged-vector reuse is future work.

Offline tests cover complete audio windows, frame budgets and identity changes. The ignored real_decoder_preserves_frame_timestamps_and_audio test has been executed successfully in the CPU Windows VM with provisioned FFmpeg 9.0.2. It creates genuine FFV1/PCM media, validates decoded two-second PTS and actual mono sine samples. Linux/macOS execution and installer codec/license delivery have not been validated.