4.3 KiB
Windows Unicode model paths
The native worker must preserve filenames from Windows command-line arguments.
Windows narrow CRT char** argv uses the active ANSI code page; the pinned
llama/ggml model loader expects UTF-8 filenames. UTF-8 JSON requests alone do
not correct the model path supplied at startup.
Observed failure, 2026-10-13
An actual CPU query-role startup used a hardlink to the verified Google backbone
under Gemäß_日本_😀. The model bytes were unchanged (309,855,456 bytes,
SHA256 2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135).
On this Windows VM the active ANSI code page was 1252. The normal worker with
SHA256 1272797af94c21db1ccabaa3ad996ab57d64f1bf9d33280c61d65c51360deab7
exited with code 1 before a ready handshake. gguf_init_from_file reported an
invalid argument with the Japanese and emoji components replaced by question
marks, and the German component no longer valid UTF-8. The captured raw stderr
is preserved locally in .searchi/qa/windows-unicode-model-before-stderr.bin.
Implementation
Windows enters through wmain, converts the original UTF-16 arguments to UTF-8
with WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS), then calls the shared
worker entry. Invalid Unicode fails explicitly. Unix continues to pass its
existing byte arguments through main. Normal CPU/GPU selection, protocol,
model validation and role behavior remain unchanged. MinGW receives its required
-municode link option, but a MinGW build has not been executed or certified.
After the fix, normal MSVC worker SHA256
c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73
passed the real Unicode query/indexing regression (21.03 s). A strengthened
regression also relocated the worker and every adjacent DLL as hardlinks under
the Unicode folder: actual query and full document indexing passed (22.45 s),
with ASCII/Unicode query-vector cosine 1.00000001. The temporary fixture names
were verified absent afterward and the original backbone remained intact.
The unchanged query-context/parity/boundary regression passed again (25.83 s).
A separate native after-probe reported query role, 512-token context, CPU,
ggml-cpu-cascadelake.dll, no Vulkan module, and a normalized 768-dimensional
vector. Metadata is retained in .searchi/qa/windows-unicode-model-after.json.
The ad-hoc model hardlink was removed only after verifying its absolute path
under the owned QA root and that the directory contained exactly that link.
The original verified model remains present. No GPU was executed.
The actual regression creates a uniquely owned temporary directory on the model's volume, hardlinks the backbone under a Unicode child, relocates adjacent runtime files as hardlinks, and compares ASCII/Unicode CPU query vectors. It also opens the full indexing role through that Unicode path and encodes a real document. Temporary-name cleanup leaves the original verified model intact. No duplicate model download or copy is needed; hardware acceleration is not executed in this VM.
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_GEMMA_MODEL_DIR = 'PATH/TO/VERIFIED/MODELS'
$env:SEARCHI_GEMMA_WORKER = 'PATH/TO/CURRENT/searchi-gemma-worker.exe'
cargo test -p searchi-semantic --test unicode_windows_cpu --locked -- --ignored --nocapture
Updated development archive
The normal worker was restaged and the embedded desktop Release rebuilt in
2m 02s. Executable: 23,924,224 bytes, SHA-256
bce713e7d081095e650756d1b0ed497cb21e451ea2e47bead6676a22327729b9.
The unsigned Windows archive is 145,409,579 bytes, SHA-256
512be4960da197a45710d49bdbc27b84c18e0270364e41bbcc546b8791826842.
Both actual package checks extracted under a Unicode directory and verified all
39 recorded files. Default auto mode used CPU with Vulkan registration disabled
only in the child: optimized indexing context 2048 and baseline-only query
context 512 both produced normalized 768-dimensional real Google vectors.
The packaged FFprobe started successfully in that directory. Local reports are
.searchi/packages/smoke-unicode-{indexing,query-baseline}.json.
These are worker/dependency checks outside the build tree, not complete installed desktop interaction, Unicode user-profile, clean-machine, signing or older physical-CPU certification. This regression does not certify every filesystem, codec, launcher or installer path, or any GPU hardware.