62 lines
3.5 KiB
Markdown
62 lines
3.5 KiB
Markdown
|
# Windows Unicode model paths
|
||
|
|
|
||
|
|
The native worker must preserve filenames from Windows command-line arguments.
|
||
|
|
Windows narrow CRT `char** argv` uses the active ANSI code page; the pinned
|
||
|
|
llama/ggml model loader expects UTF-8 filenames. UTF-8 JSON requests alone do
|
||
|
|
not correct the model path supplied at startup.
|
||
|
|
|
||
|
|
## Observed failure, 2026-10-13
|
||
|
|
|
||
|
|
An actual CPU query-role startup used a hardlink to the verified Google backbone
|
||
|
|
under `Gemäß_日本_😀`. The model bytes were unchanged (309,855,456 bytes,
|
||
|
|
SHA256 `2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135`).
|
||
|
|
On this Windows VM the active ANSI code page was 1252. The normal worker with
|
||
|
|
SHA256 `1272797af94c21db1ccabaa3ad996ab57d64f1bf9d33280c61d65c51360deab7`
|
||
|
|
exited with code 1 before a ready handshake. `gguf_init_from_file` reported an
|
||
|
|
invalid argument with the Japanese and emoji components replaced by question
|
||
|
|
marks, and the German component no longer valid UTF-8. The captured raw stderr
|
||
|
|
is preserved locally in `.searchi/qa/windows-unicode-model-before-stderr.bin`.
|
||
|
|
|
||
|
|
## Implementation
|
||
|
|
|
||
|
|
Windows enters through `wmain`, converts the original UTF-16 arguments to UTF-8
|
||
|
|
with `WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS)`, then calls the shared
|
||
|
|
worker entry. Invalid Unicode fails explicitly. Unix continues to pass its
|
||
|
|
existing byte arguments through `main`. Normal CPU/GPU selection, protocol,
|
||
|
|
model validation and role behavior remain unchanged. MinGW receives its required
|
||
|
|
`-municode` link option, but a MinGW build has not been executed or certified.
|
||
|
|
|
||
|
|
After the fix, normal MSVC worker SHA256
|
||
|
|
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`
|
||
|
|
passed the real Unicode query/indexing regression (21.03 s). A strengthened
|
||
|
|
regression also relocated the worker and every adjacent DLL as hardlinks under
|
||
|
|
the Unicode folder: actual query and full document indexing passed (22.45 s),
|
||
|
|
with ASCII/Unicode query-vector cosine 1.00000001. The temporary fixture names
|
||
|
|
were verified absent afterward and the original backbone remained intact.
|
||
|
|
The unchanged query-context/parity/boundary regression passed again (25.83 s).
|
||
|
|
|
||
|
|
A separate native after-probe reported query role, 512-token context, CPU,
|
||
|
|
`ggml-cpu-cascadelake.dll`, no Vulkan module, and a normalized 768-dimensional
|
||
|
|
vector. Metadata is retained in `.searchi/qa/windows-unicode-model-after.json`.
|
||
|
|
The ad-hoc model hardlink was removed only after verifying its absolute path
|
||
|
|
under the owned QA root and that the directory contained exactly that link.
|
||
|
|
The original verified model remains present. No GPU was executed.
|
||
|
|
|
||
|
|
The actual regression creates a uniquely owned temporary directory on the
|
||
|
|
model's volume, hardlinks the backbone under a Unicode child, relocates adjacent
|
||
|
|
runtime files as hardlinks, and compares
|
||
|
|
ASCII/Unicode CPU query vectors. It also opens the full indexing role through
|
||
|
|
that Unicode path and encodes a real document. Temporary-name cleanup leaves
|
||
|
|
the original verified model intact. No duplicate model download or copy is
|
||
|
|
needed; hardware acceleration is not executed in this VM.
|
||
|
|
|
||
|
|
```powershell
|
||
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
||
|
|
$env:SEARCHI_GEMMA_MODEL_DIR = 'PATH/TO/VERIFIED/MODELS'
|
||
|
|
$env:SEARCHI_GEMMA_WORKER = 'PATH/TO/CURRENT/searchi-gemma-worker.exe'
|
||
|
|
cargo test -p searchi-semantic --test unicode_windows_cpu --locked -- --ignored --nocapture
|
||
|
|
```
|
||
|
|
|
||
|
|
Native package restaging and installed Unicode-profile paths require their own
|
||
|
|
checks. This focused model-path regression does not certify every filesystem,
|
||
|
|
codec, launcher or installer path, or any GPU hardware.
|