Decode and preserve BOM-marked UTF-16 text files

[bholmesdev]

What happened?

Hubble renders BOM-marked UTF-16 text files as mojibake instead of decoding them. This was reported during WinGet verification with ChangedFiles.txt and is reproducible on current main (0.1.25).

The editor currently reads every editable text file with fs.readFile(resolved, "utf8"). A UTF-16 file therefore appears as text such as ��\0C\0h\0a\0n\0g\0e\0d....

How can we reproduce it?

1. Create ChangedFiles.txt encoded as UTF-16LE or UTF-16BE with a BOM. Windows PowerShell 5.1 redirection commonly produces UTF-16LE files.
2. Open its parent folder in Hubble Desktop.
3. Open ChangedFiles.txt.
4. Observe replacement characters and NUL-separated letters instead of the file contents.

What did you expect to happen?

Hubble should detect a UTF-16 BOM, decode the file correctly, and preserve its encoding when saving edits.

What actually happened?

The file is decoded as UTF-8 and displayed as mojibake. Editing and saving is higher risk: the preload encodes the corrupted editor string as UTF-8, which can irreversibly rewrite the original file.

Screenshots, videos, logs, or notes

Original WinGet verification report and screenshot: microsoft/winget-pkgs#408140 (comment)

Current implementation areas:

• apps/desktop/electron/main.ts: desktop:read-file-text assumes UTF-8
• apps/desktop/electron/main.ts: content search also assumes UTF-8
• apps/desktop/electron/preload.ts: saves encode editor strings as UTF-8

No duplicate issue found.

App version

Reproduced on current main (0.1.25); originally reported against 0.1.21 on Windows.

Does this block your work?

Partially — I have a workaround

Acceptance criteria

• BOM-marked UTF-16LE and UTF-16BE .txt and Markdown files display correctly.
• Saving an existing UTF-16 file preserves its original encoding and BOM.
• UTF-8 reads and saves remain unchanged.
• Content search decodes supported text encodings consistently with the editor.
• Regression tests cover UTF-16LE/BE decode and edit/save round trips.

Blocked by

None - can start immediately

[github-actions]

Routing this to needs-discussion. The bug report is accurate and the decode half is straightforward, but the acceptance criterion "saving preserves the original encoding and BOM" is an open design/product decision that spans the read → editor → save → sync contracts.

Confirmed on main:

• apps/desktop/electron/main.ts — desktop:read-file-text does fs.readFile(resolved, "utf8"), so UTF-16 shows as mojibake.
• apps/desktop/electron/main.ts — desktop:search-file-contents reads candidates with "utf8" too.
• apps/desktop/electron/preload.ts — writeFileText encodes with new TextEncoder() (UTF-8 only) and main writes those bytes verbatim, deliberately, per Electron main UTF-8 truncation when saving rich transcript notes #126. So today an edit+save on a UTF-16 file rewrites it as UTF-8 mojibake — the destructive part of the report.

Why preservation isn't a one-pass change:

• readFileText returns a bare string and the whole pipeline compares strings (self-save tracking, classifyFileChange, external-change watcher in apps/desktop/src/store/actions.ts). Preserving encoding means either changing the DesktopApi.readFileText/writeFileText contract to carry encoding metadata (touches apps/desktop/src/desktopApi/types.ts and ~40 call sites/tests in apps/desktop/src/store/actions.test.ts), or keeping a per-path encoding map in main — each with different failure modes (stale entries, renames, files written by other tools).
• TextEncoder cannot emit UTF-16; the encode step would move to Buffer in preload or back to main, which re-opens the crossing that Electron main UTF-8 truncation when saving rich transcript notes #126 fixed.
• Scope of "supported encodings" is undefined. isEditableFile covers Markdown, .txt, and ~60 code extensions, all through the same read path. BOM-less UTF-16 and UTF-8-BOM (a leading BOM currently leaks into editor content and front matter parsing) are not addressed by the criteria.
• Cloud Sync assumes UTF-8 strings (FileSystem.readFile(): Promise<string>, contentHash via TextEncoder in packages/sync/src/fs.ts). A synced UTF-16 Markdown file would still round-trip as UTF-8 through Convex, so "preserve encoding" is only true for local-only files unless sync is in scope.

Open questions:

• Preserve or normalize? Reading UTF-16 correctly and saving as UTF-8 (with a one-time notice) is a much smaller change than round-trip preservation. Which is the intended product behavior?
• Which encodings are in scope: BOM UTF-16LE/BE only, or also UTF-8 BOM and BOM-less UTF-16?
• Which file kinds: .txt + Markdown only, or every isEditableFile type?
• If preserving: encoding metadata on the IPC contract, or a main-process per-path map? What happens when the on-disk encoding changes underneath an open editor?
• Is Cloud Sync in scope, or explicitly out (local-only guarantee)?
• Interim safety: should Hubble block or warn on save for non-UTF-8 files before full support lands, so the destructive rewrite stops immediately?

A scoped split would likely unblock this fast: (1) decode BOM-marked UTF-16 on read + search and make saves non-destructive, then (2) a follow-up for encoding preservation and sync semantics.

---

Triaged by a Warp Factory agent.