3.5 KiB
Windows Unicode model paths
The native worker must preserve filenames from Windows command-line arguments.
Windows narrow CRT char** argv uses the active ANSI code page; the pinned
llama/ggml model loader expects UTF-8 filenames. UTF-8 JSON requests alone do
not correct the model path supplied at startup.
Observed failure, 2026-10-13
An actual CPU query-role startup used a hardlink to the verified Google backbone
under Gemäß_日本_😀. The model bytes were unchanged (309,855,456 bytes,
SHA256 2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135).
On this Windows VM the active ANSI code page was 1252. The normal worker with
SHA256 1272797af94c21db1ccabaa3ad996ab57d64f1bf9d33280c61d65c51360deab7
exited with code 1 before a ready handshake. gguf_init_from_file reported an
invalid argument with the Japanese and emoji components replaced by question
marks, and the German component no longer valid UTF-8. The captured raw stderr
is preserved locally in .searchi/qa/windows-unicode-model-before-stderr.bin.
Implementation
Windows enters through wmain, converts the original UTF-16 arguments to UTF-8
with WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS), then calls the shared
worker entry. Invalid Unicode fails explicitly. Unix continues to pass its
existing byte arguments through main. Normal CPU/GPU selection, protocol,
model validation and role behavior remain unchanged. MinGW receives its required
-municode link option, but a MinGW build has not been executed or certified.
After the fix, normal MSVC worker SHA256
c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73
passed the real Unicode query/indexing regression (21.03 s). A strengthened
regression also relocated the worker and every adjacent DLL as hardlinks under
the Unicode folder: actual query and full document indexing passed (22.45 s),
with ASCII/Unicode query-vector cosine 1.00000001. The temporary fixture names
were verified absent afterward and the original backbone remained intact.
The unchanged query-context/parity/boundary regression passed again (25.83 s).
A separate native after-probe reported query role, 512-token context, CPU,
ggml-cpu-cascadelake.dll, no Vulkan module, and a normalized 768-dimensional
vector. Metadata is retained in .searchi/qa/windows-unicode-model-after.json.
The ad-hoc model hardlink was removed only after verifying its absolute path
under the owned QA root and that the directory contained exactly that link.
The original verified model remains present. No GPU was executed.
The actual regression creates a uniquely owned temporary directory on the model's volume, hardlinks the backbone under a Unicode child, relocates adjacent runtime files as hardlinks, and compares ASCII/Unicode CPU query vectors. It also opens the full indexing role through that Unicode path and encodes a real document. Temporary-name cleanup leaves the original verified model intact. No duplicate model download or copy is needed; hardware acceleration is not executed in this VM.
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_GEMMA_MODEL_DIR = 'PATH/TO/VERIFIED/MODELS'
$env:SEARCHI_GEMMA_WORKER = 'PATH/TO/CURRENT/searchi-gemma-worker.exe'
cargo test -p searchi-semantic --test unicode_windows_cpu --locked -- --ignored --nocapture
Native package restaging and installed Unicode-profile paths require their own checks. This focused model-path regression does not certify every filesystem, codec, launcher or installer path, or any GPU hardware.