Skip to content

A leading U+FEFF is stripped from every string literal, so "\uFEFF" compiles to the empty string #243

Description

@2youg1

A string literal loses exactly one U+FEFF when that character is at position
0. The build is clean and the type checker is correct; only the emitted
program disagrees with Node. Because "\uFEFF" is how JavaScript spells a
byte-order mark, every BOM-detection idiom inverts in a compiled binary, and it
inverts silently — on input that contains no BOM at all.

Repro

const plain = "hello";
console.log(plain.startsWith("\uFEFF")); // node: false    scriptc: true
console.log(plain.indexOf("\uFEFF"));    // node: -1       scriptc: 0
console.log(plain.split("\uFEFF").length); // node: 1      scriptc: 5
console.log("\uFEFF".length);            // node: 1        scriptc: 0
$ node --experimental-strip-types bom.ts
false
-1
1
1

$ scriptc build bom.ts -o bom && ./bom
true
0
5
0

Exactly one mark, only at position 0

function show(label: string, value: string): void {
  const codes: number[] = [];
  for (let i = 0; i < value.length; i++) codes.push(value.charCodeAt(i));
  console.log(`${label} len=${value.length} codes=[${codes.join(",")}]`);
}
show("leading   ", "\uFEFFabc");
show("trailing  ", "abc\uFEFF");
show("middle    ", "a\uFEFFb");
show("alone     ", "\uFEFF");
show("doubled   ", "\uFEFF\uFEFF");
show("template  ", `\uFEFFabc`);
show("fromCharCode", String.fromCharCode(0xfeff));
show("zwsp U+200B", "\u200B");
show("e-acute U+00E9", "\u00E9");
expression node scriptc 0.0.35
"\uFEFFabc" len=4 [65279,97,98,99] len=3 [97,98,99]
"abc\uFEFF" len=4 [97,98,99,65279] len=4 [97,98,99,65279]
"a\uFEFFb" len=3 [97,65279,98] len=3 [97,65279,98]
"\uFEFF" len=1 [65279] len=0 []
"\uFEFF\uFEFF" len=2 [65279,65279] len=1 [65279]
`\uFEFFabc` len=4 [65279,97,98,99] len=3 [97,98,99]
String.fromCharCode(0xfeff) len=1 [65279] len=1 [65279]
"\u200B" len=1 [8203] len=1 [8203]
"\u00E9" len=1 [233] len=1 [233]

Trailing and interior marks survive; other escapes survive; String.fromCharCode
survives. A doubled mark loses one and keeps one. The rule is one leading
U+FEFF removed per literal, which reads like the source-file BOM strip being
applied to each literal's value instead of once to the file text.

The frontend is correct; the value is lost later

console.log("\uFEFF" === "");
$ scriptc build bom.ts -o bom
bom.ts:1:13 - error SC0001: This comparison appears to be unintentional
because the types '""' and '""' have no overlap.

The checker types the literal as "" — one character, not empty — and rejects
the comparison on that basis. So the character is present through type
checking and is dropped somewhere after it.

Why it is easy to hit

Stripping a BOM after reading a file is the ordinary use of this escape, and
every shape below breaks. What makes it hard to see: the mark read from disk
is a real code unit (charCodeAt(0) returns 65279), so the data is intact
while every literal written to test it is empty.

if (text.startsWith("\uFEFF")) text = text.slice(1); // always true
if (text.indexOf("\uFEFF") === 0) { /* ... */ }      // always taken
text = text.slice("\uFEFF".length);                  // slice(0), strips nothing
const parts = text.split("\uFEFF");                  // splits into characters

(text.replace("\uFEFF", "") is not in the list because a string-argument
replace is rejected by a static build, so I could not measure it.)

Found by a repository gate that reads every tracked file and compares the head
of each one. It passed under Bun and failed as a compiled executable on the two
UTF-8-with-BOM files in the tree, which is exactly the divergence a compiled
build is supposed to rule out.

Workaround

String.fromCharCode(0xfeff) and text.charCodeAt(0) === 0xfeff both behave,
so a comparison by code point is safe today.

Environment

  • scriptc 0.0.35, @scriptc/compiler 0.0.35 (current latest on npm)
  • Windows 11 x64 (10.0.26220), static build, default backend, no --dynamic
  • Reference runtime: Node v24.15.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions