Skip to content

fix: decode Windows shell output in the console code page - #251

Merged
its-janghoon merged 1 commit into
developfrom
fix/winencoding
Nov 13, 2026
Merged

its-janghoon merged 1 commit into
developfrom
fix/winencoding

Conversation

@its-janghoon

Copy link
Copy Markdown
Member

What's done

Shell tool output on Korean Windows (code page 949) showed Korean as ����. The shell tool and the ! user shell decoded child output as UTF-8, but PowerShell 5.1, cmd builtins and Python write to a pipe in the console/ANSI code page.

  • Shell.utf8(shell, command): on Windows, prefixes PowerShell commands with [Console]::OutputEncoding=$OutputEncoding=[Text.UTF8Encoding]::new($false); (no BOM, same idea as Codex) and cmd commands with chcp 65001>nul&.
  • Shell.env(): on Windows, PYTHONUTF8=1 and PYTHONIOENCODING=utf-8 as child env defaults. The user's own env and the shell.env plugin hook still win.
  • Shell.decoder(): streaming UTF-8 decode. On Windows only, once a chunk is not valid UTF-8, the rest of the stream is decoded with the system ANSI code page (read once from HKLM\...\Nls\CodePage\ACP), including a lead byte held back from the previous chunk.
  • Used in packages/redrob/src/tool/shell.ts and the user shell in packages/redrob/src/session/prompt.ts.

Linux and macOS: no prefix, no env change, and the decoder is the same non-fatal UTF-8 decode as Stream.decodeText.

What's not done

  • The core v2 bash tool (packages/core/src/tool/bash.ts) is left as is; it is not the tool the engine uses today.
  • PTY terminals are not touched.

What to review

  • packages/core/src/shell.ts: the new functions at the end.
  • On a console shared with the parent (TUI started from a terminal), the PowerShell/cmd prefix sets that console's output code page to 65001.

Intentions

Fix at the source (ask the programs for UTF-8) and keep a code-page fallback for programs that ignore it.

Verified on a Korean Windows PC (code page 949), redrob run --format json with the Redrob login, running python -c "print('한글 테스트')", Write-Output '한글 테스트', cmd /c echo 한글:

shell origin/develop this branch
PowerShell 5.1 (default) �ѱ� �׽�Ʈ for all three 한글 테스트 / 한글 테스트 / 한글
cmd.exe �ѱ� �׽�Ʈ for all three 한글 테스트 / 한글

Also on that PC, Shell.decoder() decodes raw CP949 bytes to 한글 using the registry code page.

Tests: packages/core/test/shell.test.ts 13 pass (7 new). Core suite: the same 17 failures as origin/develop (ModelsDev/RedrobPlugin network fixtures), none new. Redrob suite: 3564 pass, 3 fail; 2 fail on origin/develop too, and httpapi-v2-location timed out under load and passes alone. bun typecheck passes.

@its-janghoon

Copy link
Copy Markdown
Member Author

Self-review: the change is opt-in by platform. Every new path checks platform === 'win32', so Linux/macOS keep the exact UTF-8 decode they had. On Windows the risk is a wrong fallback code page; it only kicks in after output has already proven not to be UTF-8, so the worst case is the same garbage as before, never worse. The PowerShell prefix uses UTF8Encoding without a BOM so piped output does not start with U+FEFF. The cmd prefix adds one chcp.com spawn per command (~20 ms). The decoder holds back an incomplete UTF-8 lead byte so a CP949 character split across chunks still decodes. Checked before/after on a Korean Windows PC with redrob run.

Ask PowerShell, cmd and Python for UTF-8 output on Windows, and decode output that is still not UTF-8 with the system ANSI code page instead of U+FFFD.
@its-janghoon
its-janghoon merged commit a5b0311 into develop Nov 13, 2026
11 checks passed
@its-janghoon
its-janghoon deleted the fix/winencoding branch November 13, 2026 06:10
@its-janghoon its-janghoon mentioned this pull request Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant