searchi/docs/WINDOWS-UNICODE.md

# Windows Unicode model paths

The native worker must preserve filenames from Windows command-line arguments.
Windows narrow CRT `char** argv` uses the active ANSI code page; the pinned
llama/ggml model loader expects UTF-8 filenames. UTF-8 JSON requests alone do
not correct the model path supplied at startup.

## Observed failure, 2026-10-13

An actual CPU query-role startup used a hardlink to the verified Google backbone
under `Gemäß_日本_😀`. The model bytes were unchanged (309,855,456 bytes,
SHA256 `2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135`).
On this Windows VM the active ANSI code page was 1252. The normal worker with
SHA256 `1272797af94c21db1ccabaa3ad996ab57d64f1bf9d33280c61d65c51360deab7`
exited with code 1 before a ready handshake. `gguf_init_from_file` reported an
invalid argument with the Japanese and emoji components replaced by question
marks, and the German component no longer valid UTF-8. The captured raw stderr
is preserved locally in `.searchi/qa/windows-unicode-model-before-stderr.bin`.

## Implementation

Windows enters through `wmain`, converts the original UTF-16 arguments to UTF-8
with `WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS)`, then calls the shared
worker entry. Invalid Unicode fails explicitly. Unix continues to pass its
existing byte arguments through `main`. Normal CPU/GPU selection, protocol,
model validation and role behavior remain unchanged. MinGW receives its required
`-municode` link option, but a MinGW build has not been executed or certified.

After the fix, normal MSVC worker SHA256
`c01cb3ec13cfca722ee768e8350e1399a8b9a4d14a24a35adc3473a5cfc82a73`
passed the real Unicode query/indexing regression (21.03 s). A strengthened
regression also relocated the worker and every adjacent DLL as hardlinks under
the Unicode folder: actual query and full document indexing passed (22.45 s),
with ASCII/Unicode query-vector cosine 1.00000001. The temporary fixture names
were verified absent afterward and the original backbone remained intact.
The unchanged query-context/parity/boundary regression passed again (25.83 s).

A separate native after-probe reported query role, 512-token context, CPU,
`ggml-cpu-cascadelake.dll`, no Vulkan module, and a normalized 768-dimensional
vector. Metadata is retained in `.searchi/qa/windows-unicode-model-after.json`.
The ad-hoc model hardlink was removed only after verifying its absolute path
under the owned QA root and that the directory contained exactly that link.
The original verified model remains present. No GPU was executed.

The actual regression creates a uniquely owned temporary directory on the
model's volume, hardlinks the backbone under a Unicode child, relocates adjacent
runtime files as hardlinks, and compares
ASCII/Unicode CPU query vectors. It also opens the full indexing role through
that Unicode path and encodes a real document. Temporary-name cleanup leaves
the original verified model intact. No duplicate model download or copy is
needed; hardware acceleration is not executed in this VM.

```powershell
$env:SEARCHI_FORCE_CPU = '1'
$env:SEARCHI_GEMMA_MODEL_DIR = 'PATH/TO/VERIFIED/MODELS'
$env:SEARCHI_GEMMA_WORKER = 'PATH/TO/CURRENT/searchi-gemma-worker.exe'
cargo test -p searchi-semantic --test unicode_windows_cpu --locked -- --ignored --nocapture
```

## Updated development archive

The normal worker was restaged and the embedded desktop Release rebuilt in
2m 02s. Executable: 23,924,224 bytes, SHA-256
`bce713e7d081095e650756d1b0ed497cb21e451ea2e47bead6676a22327729b9`.
The unsigned Windows archive is 145,409,579 bytes, SHA-256
`512be4960da197a45710d49bdbc27b84c18e0270364e41bbcc546b8791826842`.

Both actual package checks extracted under a Unicode directory and verified all
39 recorded files. Default `auto` mode used CPU with Vulkan registration disabled
only in the child: optimized indexing context 2048 and baseline-only query
context 512 both produced normalized 768-dimensional real Google vectors.
The packaged FFprobe started successfully in that directory. Local reports are
`.searchi/packages/smoke-unicode-{indexing,query-baseline}.json`.

These are worker/dependency checks outside the build tree, not complete installed
desktop interaction, Unicode user-profile, clean-machine, signing or older
physical-CPU certification. This regression does not certify every filesystem,
codec, launcher or installer path, or any GPU hardware.