79 lines
4.3 KiB
Markdown
79 lines
4.3 KiB
Markdown
# Windows Unicode model paths
|
|
|
|
The native worker must preserve filenames from Windows command-line arguments.
|
|
Windows narrow CRT `char** argv` uses the active ANSI code page; the pinned
|
|
llama/ggml model loader expects UTF-8 filenames. UTF-8 JSON requests alone do
|
|
not correct the model path supplied at startup.
|
|
|
|
## Observed failure, 2026-10-13
|
|
|
|
An actual CPU query-role startup used a hardlink to the verified Google backbone
|
|
under `Gemäß_日本_😀`. The model bytes were unchanged (309,855,456 bytes,
|
|
SHA256 `2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135`).
|
|
On this Windows VM the active ANSI code page was 1252. The normal worker with
|
|
SHA256 `1272797af94c21db1ccabaa3ad996ab57d64f1bf9d33280c61d65c51360deab7`
|
|
exited with code 1 before a ready handshake. `gguf_init_from_file` reported an
|
|
invalid argument with the Japanese and emoji components replaced by question
|
|
marks, and the German component no longer valid UTF-8. The captured raw stderr
|
|
is preserved locally in `.searchi/qa/windows-unicode-model-before-stderr.bin`.
|
|
|
|
## Implementation
|
|
|
|
Windows enters through `wmain`, converts the original UTF-16 arguments to UTF-8
|
|
with `WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS)`, then calls the shared
|
|
worker entry. Invalid Unicode fails explicitly. Unix continues to pass its
|
|
existing byte arguments through `main`. Normal CPU/GPU selection, protocol,
|
|
model validation and role behavior remain unchanged. MinGW receives its required
|
|
`-municode` link option, but a MinGW build has not been executed or certified.
|
|
|
|
After the fix, normal MSVC worker SHA256
|
|
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`
|
|
passed the real Unicode query/indexing regression (21.03 s). A strengthened
|
|
regression also relocated the worker and every adjacent DLL as hardlinks under
|
|
the Unicode folder: actual query and full document indexing passed (22.45 s),
|
|
with ASCII/Unicode query-vector cosine 1.00000001. The temporary fixture names
|
|
were verified absent afterward and the original backbone remained intact.
|
|
The unchanged query-context/parity/boundary regression passed again (25.83 s).
|
|
|
|
A separate native after-probe reported query role, 512-token context, CPU,
|
|
`ggml-cpu-cascadelake.dll`, no Vulkan module, and a normalized 768-dimensional
|
|
vector. Metadata is retained in `.searchi/qa/windows-unicode-model-after.json`.
|
|
The ad-hoc model hardlink was removed only after verifying its absolute path
|
|
under the owned QA root and that the directory contained exactly that link.
|
|
The original verified model remains present. No GPU was executed.
|
|
|
|
The actual regression creates a uniquely owned temporary directory on the
|
|
model's volume, hardlinks the backbone under a Unicode child, relocates adjacent
|
|
runtime files as hardlinks, and compares
|
|
ASCII/Unicode CPU query vectors. It also opens the full indexing role through
|
|
that Unicode path and encodes a real document. Temporary-name cleanup leaves
|
|
the original verified model intact. No duplicate model download or copy is
|
|
needed; hardware acceleration is not executed in this VM.
|
|
|
|
```powershell
|
|
$env:SEARCHI_FORCE_CPU = '1'
|
|
$env:SEARCHI_GEMMA_MODEL_DIR = 'PATH/TO/VERIFIED/MODELS'
|
|
$env:SEARCHI_GEMMA_WORKER = 'PATH/TO/CURRENT/searchi-gemma-worker.exe'
|
|
cargo test -p searchi-semantic --test unicode_windows_cpu --locked -- --ignored --nocapture
|
|
```
|
|
|
|
## Updated development archive
|
|
|
|
The normal worker was restaged and the embedded desktop Release rebuilt in
|
|
2m 02s. Executable: 23,924,224 bytes, SHA-256
|
|
`bce713e7d081095e650756d1b0ed497cb21e451ea2e47bead6676a22327729b9`.
|
|
The unsigned Windows archive is 145,409,579 bytes, SHA-256
|
|
`512be4960da197a45710d49bdbc27b84c18e0270364e41bbcc546b8791826842`.
|
|
|
|
Both actual package checks extracted under a Unicode directory and verified all
|
|
39 recorded files. Default `auto` mode used CPU with Vulkan registration disabled
|
|
only in the child: optimized indexing context 2048 and baseline-only query
|
|
context 512 both produced normalized 768-dimensional real Google vectors.
|
|
The packaged FFprobe started successfully in that directory. Local reports are
|
|
`.searchi/packages/smoke-unicode-{indexing,query-baseline}.json`.
|
|
|
|
These are worker/dependency checks outside the build tree, not complete installed
|
|
desktop interaction, Unicode user-profile, clean-machine, signing or older
|
|
physical-CPU certification. This regression does not certify every filesystem,
|
|
codec, launcher or installer path, or any GPU hardware.
|