diff --git a/TSLANG_ISSUES.md b/TSLANG_ISSUES.md index 6bdae4328..0304ed489 100644 --- a/TSLANG_ISSUES.md +++ b/TSLANG_ISSUES.md @@ -35,3 +35,127 @@ a local `values` broke the default library's `Array`): a generic declared out now sees no local of another function; one declared in a function still reads that function's locals, and lambdas in a generic still capture the generic's own. Regression test `tslang/test/tester/tests/00tslang_issue_30.ts`. +Items 34-36, found in the default library on 2026-11-08, are fixed or answered too: 34 (a string +literal starting with U+FEFF lost it, so its `.length` was 0): LLVM's UTF-16 to UTF-8 +conversion took a leading U+FEFF for a byte order mark and skipped it, and would have byte-swapped a +literal starting with U+FFFE. Literals are now converted as they are. 35 (`"\\n"` read as a line feed) +did not reproduce: on the same build the repro prints `2 6 4 2 2 3` and `92 2`, as expected. The repro +file was most likely written through a shell that turns `\\` into `\`, which gives exactly the output +reported. Both are covered by `tslang/test/tester/tests/00string_literal_escapes.ts`. 36 (an older +runtime on `TSLANG_LIB_PATH` failed every `--emit=dll` with a bare LNK2001 for `__tslang_heap_shared`): +tslang now checks the runtime's `TypeScriptAsyncRuntime.lib` before linking a library and says it is +older than the compiler. `-v` is an alias for `--verbose`, which prints the linker's command line, so +the linker's "use -v" hint works. + +Items 31-33 are `-mm=own` limits found in the default library (TypeScriptCompilerDefaultLib), not in +native_ts_graphics. There the tests `thread_spawn.ts` and `atomics_wait.ts` are marked `// not own` +because of them. Each repro was compiled with `--emit=exe` (the JIT's default-library DLL didn't +link against this build) and runs under `-mm=gc` with the output shown. + +### 31. `-mm=own`: two closures can't capture the same array + +tslang (commit 94b8bdb2, built with uncommitted local changes). Found 2026-11-08. + +A closure passed to `Thread.spawn` moves what it captures, so a second closure capturing the same +`Int32Array` is a use after move. Threads can't share memory under `-mm=own`, which is what +`Atomics` is for. A single capture is fine: in the third repro the main thread still reads the +array after the thread captured it. The error names `'this value'`, not `counter`, and points at the +second arrow function rather than at `counter` inside it. + +Repro (`share_two.ts`): +```ts +// -mm=own: two threads that capture the same Int32Array +function main() { + const counter = new Int32Array([0]); + const a = Thread.spawn((): int => { Atomics.add(counter, 0, 1); return 0; }); + const b = Thread.spawn((): int => { Atomics.add(counter, 0, 1); return 0; }); + a.join(); + b.join(); + print(Atomics.load(counter, 0)); +} +``` + +`-mm=gc` prints `2`. `-mm=own`: +``` +share_two.ts:5:28: error: 'this value' is used here after its value was moved + const b = Thread.spawn((): int => { Atomics.add(counter, 0, 1); return 0; }); + ^ +share_two.ts:4:28: note: value moved here + const a = Thread.spawn((): int => { Atomics.add(counter, 0, 1); return 0; }); + ^ +``` + +The same in a loop (`share_loop.ts`), the usual way to start workers, gives a different error: +```ts +// -mm=own: threads spawned in a loop, each capturing the same Int32Array +function main() { + const counter = new Int32Array([0]); + const workers: ThreadHandle[] = []; + for (let t = 0; t < 4; t++) { + workers.push(Thread.spawn((): int => { Atomics.add(counter, 0, 1); return 0; })); + } + + for (const w of workers) w.join(); + print(Atomics.load(counter, 0)); +} +``` + +`-mm=gc` prints `4`. `-mm=own`: +``` +share_loop.ts:6:35: error: 'this value' is moved inside a loop but was made outside it; only a borrow could do that, and -mm=own cannot prove one yet + workers.push(Thread.spawn((): int => { Atomics.add(counter, 0, 1); return 0; })); + ^ +``` + +One capture compiles and runs under both models, printing `1` (`share_after.ts`): +```ts +// -mm=own: the main thread uses the Int32Array after a thread captured it +function main() { + const counter = new Int32Array([0]); + const a = Thread.spawn((): int => { Atomics.add(counter, 0, 1); return 0; }); + a.join(); + print(Atomics.load(counter, 0)); +} +``` + +Expected: a way to share one array between threads under `-mm=own`, such as `Shared` captured by +each closure, or a capture that borrows when the closure can't outlive the array. + +### 32. `-mm=own`: the error for a move in a loop doesn't say what was moved + +tslang (commit 94b8bdb2, built with uncommitted local changes). Found 2026-11-08. + +Both errors in 31 call the captured variable `'this value'`. In a program with several captures, +nothing says which one is moved. The error should name `counter` and point at its use inside the +arrow function. + +### 33. `-mm=own`: a thread can't return an object, and the error points into the default library + +tslang (commit 94b8bdb2, built with uncommitted local changes). Found 2026-11-08. + +`Thread.spawn` keeps the function's result in a field of `ThreadStart` for `join()` to return. +When `R` is a class, storing it is rejected. The error is reported in the default library's +`lib.generics.ts` (`this.result = result;` in `ThreadStart.run`) with no line in the user's program, +so nothing points at the `Thread.spawn` call that caused it. A string or a number works. + +Repro (`return_object.ts`): +```ts +// -mm=own: a thread whose function returns an object +class Box { + constructor(public value: int) {} +} + +function main() { + const t = Thread.spawn((): Box => new Box(42)); + print(t.join().value); +} +``` + +`-mm=gc` prints `42`. `-mm=own`: +``` +.\__build\defaultlib\generics/lib.generics.ts:857:13: error: 'this value' takes a second reference; -mm=own cannot prove a move or a borrow here yet +``` + +`const result = this.fn(); this.result = result;` moves a value the function just returned, which +the field should be able to take. Expected: the move is accepted. Failing that, the error should +also name the instantiating call in the user's file. diff --git a/tslang/include/TypeScript/MLIRLogic/MLIRHelper.h b/tslang/include/TypeScript/MLIRLogic/MLIRHelper.h index 805e8eb9a..8e943e56d 100644 --- a/tslang/include/TypeScript/MLIRLogic/MLIRHelper.h +++ b/tslang/include/TypeScript/MLIRLogic/MLIRHelper.h @@ -24,36 +24,50 @@ namespace typescript // codecvt_utf8_utf16, as TypeScript counts a string in UTF-16 units - so a character outside the BMP // is a surrogate pair even where wchar_t has 32 bits (Linux). llvm::convertWideToUTF8 takes a 32-bit // wchar_t string for UTF-32, where a surrogate is invalid, and gave "" for a literal such as "😀". +// +// The units are converted as they are: llvm::convertUTF16ToUTF8String reads a leading U+FEFF as a byte +// order mark and skips it (and a leading U+FFFE byte-swaps the whole string), so a string literal +// starting with \uFEFF lost it (TSLANG_ISSUES 34). static std::string convertWideToUTF8(const std::wstring &ws) { - std::string s; - if constexpr (sizeof(wchar_t) == sizeof(llvm::UTF16)) - { - llvm::convertWideToUTF8(ws, s); - } - else + llvm::SmallVector units; + units.reserve(ws.size()); + for (auto c : ws) { - llvm::SmallVector units; - units.reserve(ws.size()); - for (auto c : ws) + auto code = static_cast(c); + if (code > 0xFFFF && code <= 0x10FFFF) { - auto code = static_cast(c); - if (code > 0xFFFF && code <= 0x10FFFF) - { - // a code point made elsewhere (not by stows) - its surrogate pair - code -= 0x10000; - units.push_back(static_cast(0xD800 + (code >> 10))); - units.push_back(static_cast(0xDC00 + (code & 0x3FF))); - } - else - { - units.push_back(static_cast(code)); - } + // a code point made elsewhere (not by stows) - its surrogate pair + code -= 0x10000; + units.push_back(static_cast(0xD800 + (code >> 10))); + units.push_back(static_cast(0xDC00 + (code & 0x3FF))); } + else + { + units.push_back(static_cast(code)); + } + } - llvm::convertUTF16ToUTF8String(units, s); + std::string s; + if (units.empty()) + { + return s; + } + + // room for every unit, as convertUTF16ToUTF8String allocates + s.resize(units.size() * UNI_MAX_UTF8_BYTES_PER_CODE_POINT + 1); + const llvm::UTF16 *src = units.begin(); + auto *dstStart = reinterpret_cast(&s[0]); + auto *dst = dstStart; + auto result = llvm::ConvertUTF16toUTF8(&src, units.end(), &dst, dstStart + s.size(), llvm::strictConversion); + if (result != llvm::conversionOK) + { + // as convertUTF16ToUTF8String: an invalid sequence (a lone surrogate) gives "" + s.clear(); + return s; } + s.resize(dst - dstStart); return s; } diff --git a/tslang/test/tester/CMakeLists.txt b/tslang/test/tester/CMakeLists.txt index 6f17f8849..ba5179f46 100644 --- a/tslang/test/tester/CMakeLists.txt +++ b/tslang/test/tester/CMakeLists.txt @@ -1814,6 +1814,12 @@ foreach(issues_mm gc rc none own) tslang_add_test(NAME test-jit-00-tslang-issue-29-thread-${issues_mm} COMMAND test-runner -jit -mm=${issues_mm} "${PROJECT_SOURCE_DIR}/test/tester/tests/${issue_29_thread_test}") tslang_add_test(NAME test-compile-00-tslang-issue-29-thread-${issues_mm} COMMAND test-runner -mm=${issues_mm} "${PROJECT_SOURCE_DIR}/test/tester/tests/${issue_29_thread_test}") endforeach() +# TSLANG 34/35: a string literal holds what its escapes say: a leading U+FEFF is kept, and "\\n" is a +# backslash and n +foreach(issues_mm gc rc none own) + tslang_add_test(NAME test-jit-00-string-literal-escapes-${issues_mm} COMMAND test-runner -jit -mm=${issues_mm} "${PROJECT_SOURCE_DIR}/test/tester/tests/00string_literal_escapes.ts") + tslang_add_test(NAME test-compile-00-string-literal-escapes-${issues_mm} COMMAND test-runner -mm=${issues_mm} "${PROJECT_SOURCE_DIR}/test/tester/tests/00string_literal_escapes.ts") +endforeach() # an Opaque out of an `any` is as wide as the value in the box: `0` is null (the # padding after a 4-byte value was read as the pointer's high bytes) foreach(issues_mm gc rc none own) diff --git a/tslang/test/tester/tests/00string_literal_escapes.ts b/tslang/test/tester/tests/00string_literal_escapes.ts new file mode 100644 index 000000000..9edf1a00d --- /dev/null +++ b/tslang/test/tester/tests/00string_literal_escapes.ts @@ -0,0 +1,28 @@ +// A string literal holds what its escapes say, and .length counts its UTF-8 bytes. +// TSLANG_ISSUES 34: a literal starting with U+FEFF lost it - the conversion to UTF-8 took it for a +// byte order mark. TSLANG_ISSUES 35: "\\" is one backslash, so "\\n" is the two characters \ and n. + +function main() { + // 34: U+FEFF is three bytes wherever it is, written as an escape or as itself + assert("\uFEFF".length == 3, "a literal that is U+FEFF"); + assert("\uFEFFx".length == 4, "a literal starting with U+FEFF"); + assert("a\uFEFFb".length == 5, "U+FEFF in the middle"); + assert("x".length == 4, "a literal starting with U+FEFF typed as is"); + assert("\uFEFFx" == "\u{FEFF}x", "the same as the code point escape"); + // U+FFFE is no byte-swapped order mark either: the rest is not swapped + assert("\uFFFEab".length == 5, "a literal starting with U+FFFE"); + assert("\uFFFEab" != "\uFFFE\u6100\u6200", "keeps its other characters as they are"); + assert("\u00E9".length == 2 && "\u200B".length == 3 && "\uFFFD".length == 3, "other escapes"); + + // 35: an escaped backslash is one backslash, and what follows it is itself + assert("\\n".length == 2, "\\n"); + assert("\\u0041".length == 6, "\\u0041"); + assert("\\x41".length == 4, "\\x41"); + assert("\\d".length == 2, "\\d"); + assert("\\\\".length == 2, "\\\\"); + assert("a\\b".length == 3, "a\\b"); + const s = "\\n"; + assert(s == "\x5Cn" && s == "\u005Cn" && s != "\n", "a backslash, then n"); + + print("done."); +} diff --git a/tslang/tslang/exe.cpp b/tslang/tslang/exe.cpp index 3cbc28f50..a176a0192 100644 --- a/tslang/tslang/exe.cpp +++ b/tslang/tslang/exe.cpp @@ -488,6 +488,40 @@ static bool resolveWindowsLibPath(const llvm::Triple &triple, std::string &path, return checkWindowsBinaryMachine(triple, lib); } +// A library tslang builds on Windows links with `-include:__tslang_heap_shared`, which the runtime +// (TypeScriptAsyncRuntime.lib) defines since #784. A runtime older than that failed every +// --emit=dll with a bare "unresolved external symbol __tslang_heap_shared" (TSLANG_ISSUES 36): say +// which runtime it is and that it is older than the compiler. `path` is the directory the link +// takes the runtime from (resolveWindowsLibPath has made it the x86 one for x86); a runtime not +// found there is the linker's to report. +static bool checkRuntimeSharesHeap(llvm::StringRef path) +{ + if (path.empty()) + { + return true; + } + + llvm::SmallString<256> lib(path); + llvm::sys::path::append(lib, "TypeScriptAsyncRuntime.lib"); + auto buffer = llvm::MemoryBuffer::getFile(lib, /*IsText=*/false, /*RequiresNullTerminator=*/false); + if (!buffer) + { + return true; + } + + // the archive's symbol table names it (x86 decorates it with one more underscore) + if ((*buffer)->getBuffer().contains("__tslang_heap_shared")) + { + return true; + } + + llvm::WithColor::error(llvm::errs(), "tslang") + << "the TypeScript runtime " << lib << " is older than this tslang: it has no __tslang_heap_shared, " + << "which a library (--emit=dll) links with. Point --tslang-lib-path (or TSLANG_LIB_PATH) at the " + << "runtime that came with this tslang.\n"; + return false; +} + void addCommandArgs(clang::driver::Compilation *c, llvm::ArrayRef cmdParts) { for (auto &job : c->getJobs()) @@ -975,6 +1009,11 @@ int linkBinary(const char *argv0, const LinkRequest &request, CompileOptions &co return 1; } + if (shared && winX86OrX64 && !checkRuntimeSharesHeap(tslangLibPath)) + { + return 1; + } + tslangLibPathOpt = getLibsPathOpt(tslangLibPath); if (!tslangLibPathOpt.empty()) { diff --git a/tslang/tslang/tslang.cpp b/tslang/tslang/tslang.cpp index 5fbeb5da4..7293a5d3a 100644 --- a/tslang/tslang/tslang.cpp +++ b/tslang/tslang/tslang.cpp @@ -133,7 +133,9 @@ cl::opt jitCache{"jit-cache", cl::desc("--emit=jit: compile the program an cl::opt jitNative{"jit-native", cl::desc("--emit=jit on Windows: link the JIT cache's objects into a DLL with a PDB and load it with the system loader, so that WinDbg and Visual Studio can debug the program with its symbols and source (implies --di unless --opt is given; needs a linker and the JIT cache)"), cl::cat(TypeScriptCompilerCategory)}; cl::opt jitCacheDir{"jit-cache-dir", cl::desc("--emit=jit: the folder the JIT cache keeps its objects in (default: '__jit' beside each source file)"), cl::value_desc("folder"), cl::cat(TypeScriptCompilerCategory)}; -cl::opt verbose{"verbose", cl::Hidden, cl::desc("Verbose output"), cl::init(false), cl::cat(TypeScriptCompilerDebugCategory)}; +cl::opt verbose{"verbose", cl::desc("Verbose output: print the linker's command line and what else is run"), cl::init(false), cl::cat(TypeScriptCompilerDebugCategory)}; +// the linker's own failure message says "use -v to see invocation" (TSLANG_ISSUES 36) +cl::alias verboseShort{"v", cl::desc("Alias for --verbose"), cl::aliasopt(verbose)}; cl::opt printOp{"print-op", cl::Hidden, cl::desc("Print Op on Diagnostic"), cl::init(false), cl::cat(TypeScriptCompilerDebugCategory)}; cl::opt printStackTrace{"print-stack-trace", cl::Hidden, cl::desc("Print stack trace on Diagnostic"), cl::init(false), cl::cat(TypeScriptCompilerDebugCategory)};