Skip to content

A string literal past the BMP keeps its character on Linux - #782

Merged
ASDAlexander77 merged 2 commits into
mainfrom
non-bmp-literal
Nov 8, 2026
Merged

ASDAlexander77 merged 2 commits into
mainfrom
non-bmp-literal

Conversation

@ASDAlexander77

Copy link
Copy Markdown
Owner

What

The alpha118 release run failed on every Linux runner: DefaultLib's new string_fromCharCode.ts (DefaultLib #68) asserts String.fromCharCode(0xD83D, 0xDE00) == "😀", and on Linux the literal "😀" was "". fromCharCode was right; the compiler dropped the literal.

  • Linux: a literal past the BMP came out empty. The parser holds a source as UTF-16 units (stows reads it with codecvt_utf8_utf16), so 😀 is a surrogate pair. Where wchar_t has 32 bits, llvm::convertWideToUTF8 reads the wide string as UTF-32, a surrogate is invalid there, and the conversion failed with an empty result. convertWideToUTF8 / convertUTF8toWide (MLIRHelper.h) now convert UTF-16 on Linux as well; Windows keeps the same calls as before.
  • Everywhere: "\u{1F600}" was U+F600. The scanner's UTF16Encoding put a code point past U+FFFF in a single wchar_t, which kept its low 16 bits on Windows. It is a surrogate pair now, as the ES spec says. Found by the new test.

Testing

  • New 00string_non_bmp.ts (compiled and JIT): literals, a global, templates, \u{...} and 😀 escapes, mixed 2/3/4-byte text. It fails on Linux without the fix.
  • ctest: Windows 6689/6689, Linux (WSL, prebuilt LLVM 22.1.8) 6661/6661.
  • DefaultLib main (f5d8edb, with string_fromCharCode.ts and console_readLine.ts) built and tested on Linux: rc, none and own all pass, release and debug, compiled and JIT. (gc fails weakref_basic.ts only in the release passes, and only with my locally built libgc.a; with alpha117's it passes. alpha117's release run passed it on all three Ubuntu runners.)

After merge, the v0.0-pre-alpha118 tag needs to move to the merge commit for the release to rebuild.

🤖 Generated with Claude Code

ASDAlexander77 and others added 2 commits November 8, 2026 12:42
The Performance table still gave the 2026-10-21 numbers (gc 1298 ms,
26x slower than C++). Since then @value, parameters captured by copy,
direct calls and tagless null took the tslang builds to 70-110 ms,
1.3-2.0x of idiomatic Rust; the table now gives those, as
tslang/benchmarks/raytrace/README.md does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parser holds a source as UTF-16 units (stows reads it with
codecvt_utf8_utf16), so a character outside the BMP is a surrogate
pair. On Linux wchar_t has 32 bits, and llvm::convertWideToUTF8 read
the wide string as UTF-32, where a surrogate is invalid: the conversion
failed and a literal such as "😀" came out as "". It broke DefaultLib's
string_fromCharCode.ts on every Linux runner of the alpha118 release.
convertWideToUTF8 / convertUTF8toWide now convert UTF-16 there too;
Windows keeps its code.

The scanner's UTF16Encoding put a code point past U+FFFF in a single
wchar_t, so "\u{1F600}" was U+F600 on Windows. It is a surrogate pair
now, as the ES spec says.

00string_non_bmp.ts covers literals, templates and both escapes.
ctest: Windows 6689/6689, Linux (WSL) 6661/6661; DefaultLib main
(f5d8edb) on Linux passes rc/none/own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ASDAlexander77
ASDAlexander77 merged commit cb1fe12 into main Nov 8, 2026
2 checks passed
@ASDAlexander77
ASDAlexander77 deleted the non-bmp-literal branch November 8, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant