Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/create-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,7 @@ jobs:
cp ./tslang/ninja/release/lib/libTypeScriptRuntime.so ./__stage/
cp ./gc/ninja/release/libgc.a ./__stage/
cp ./tslang/ninja/release/lib/libTypeScriptAsyncRuntime.a ./__stage/
cp ./tslang/ninja/release/lib/libTypeScriptMimalloc.a ./__stage/
cp ../3rdParty/llvm/release/bin/wasm-ld ./__stage/
# Add the default library as a whole tree so its per-target, per-build, per-memory-model
# subfolders are preserved:
Expand Down
80 changes: 56 additions & 24 deletions docs/memory-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,40 +188,72 @@ two throwaway heap allocations of the same size back at the same point and the p
at the unpromoted speed (157 ms against 158). Expect the same kind of swing, in either direction,
from any change to what a program allocates early.

## The heap on Windows
## The heap

Under `-mm=rc`, `-mm=own` and `-mm=none`, a Windows program allocates from
[mimalloc](https://github.com/microsoft/mimalloc) rather than the Windows process heap
(`HeapAlloc`):
Under `-mm=rc`, `-mm=own` and `-mm=none`, a program on Windows or Linux allocates from
[mimalloc](https://github.com/microsoft/mimalloc) rather than the system's heap. Linux works
differently from Windows; see [On Linux](#on-linux) below.

### On Windows

A Windows program allocates from mimalloc rather than the Windows process heap (`HeapAlloc`):

- an executable `tslang --emit=exe` builds links `TypeScriptMimalloc.lib` (it ships beside
`TypeScriptAsyncRuntime.lib`);
- a program run with `--emit=jit` gets the same.

There is one exception, because a block allocated by one module is often freed by another. A
program that loads a library tslang built keeps the process heap, and so does that library. That
covers importing a `--emit=dll` library, the default library's DLL under the JIT, and a library
given with `--shared-libs`. A program you link yourself, against `TypeScriptAsyncRuntime.lib`, also
keeps the process heap unless you link `TypeScriptMimalloc.lib` ahead of it. Do that only for an
executable that loads no library tslang built.
A block allocated by one module is often freed by another, so a process has one heap, and it is
the program's. A library tslang built - a `--emit=dll` library the program imports, the default
library's DLL under the JIT, a library given with `--shared-libs` - allocates from the program's
heap, which it finds through the entry points the program exports. Under a program that exports
none (one tslang did not build), a library allocates from the process heap.

A library built by an older tslang keeps a heap of its own. A program on mimalloc refuses to load it
and says to rebuild it; under the JIT, such a library given with `--shared-libs` keeps the program
on the process heap instead.

A program you link yourself, against `TypeScriptAsyncRuntime.lib`, keeps the process heap unless you
link `TypeScriptMimalloc.lib` ahead of it.

`--process-heap` keeps the process heap anyway, for heap debugging tools such as page heap or
Application Verifier. `-mm=gc` allocates from the collector, and other platforms from the C
library's `malloc`; neither changes.
Application Verifier.

Measured, median of eleven runs, ahead-of-time build with `--opt --opt_level=3`, in milliseconds.
`--process-heap` → default:
### On Linux

| program | model | `--process-heap` → default |
| --- | --- | --- |
| ray tracer, every vector and colour an object on the heap | `-mm=own` | 529 → 180 (2.9×) |
| ray tracer, vectors and colours `@value` types | `-mm=own` | 78 → 74 |
| ray tracer, vectors and colours `@value` types | `-mm=rc` | 111 → 106 |
| 20,000 arrays of 200 new objects | `-mm=rc` / `-mm=own` | 131 → 48 (2.7×) |
| 5 million strings built by `+` | `-mm=own` | 910 → 696 (1.3×) |

The gain is in programs that make and drop many small objects, arrays and strings. Where `@value`
types or stack promotion already keep a program's data off the heap, the heap barely matters.
An executable `tslang --emit=exe` builds links `libTypeScriptMimalloc.a`, which defines `malloc`,
`free` and the rest. On Linux, the `malloc` an executable defines is the one every shared object in
the process uses: the default library's `.so`, a library tslang built, and the C library itself. So
the whole process has one heap without any further step, and a library built by an older tslang
needs no rebuild.

tslang itself is linked the same way, so a program run with `--emit=jit`, and every library it
loads, allocates from tslang's mimalloc. Under the JIT, `--process-heap` has no effect on Linux.
A tslang built with a sanitizer keeps the C library's `malloc`, because the sanitizer needs to own
it.

A program you link yourself keeps the C library's `malloc` unless you link
`-Wl,--whole-archive -lTypeScriptMimalloc -Wl,--no-whole-archive`. `--process-heap` keeps the C
library's `malloc` in an executable tslang links.

### Everywhere

`-mm=gc` allocates from the collector, and other platforms (wasm, Android) from the C library's
`malloc`; neither changes.

Measured, median of eleven runs, ahead-of-time build with `--opt --opt_level=3`, in milliseconds,
`--process-heap` → default. Windows 11 and Linux (Ubuntu under WSL 2, glibc) on the same machine:

| program | model | Windows | Linux |
| --- | --- | --- | --- |
| ray tracer, every vector and colour an object on the heap | `-mm=own` | 529 → 180 (2.9×) | 131 → 118 (1.1×) |
| ray tracer, vectors and colours `@value` types | `-mm=own` | 78 → 74 | 60 → 60 |
| ray tracer, vectors and colours `@value` types | `-mm=rc` | 111 → 106 | 101 → 102 |
| 20,000 arrays of 200 new objects | `-mm=rc` / `-mm=own` | 131 → 48 (2.7×) | 68 → 35 (1.9×) |
| 5 million strings built by `+` | `-mm=own` | 910 → 696 (1.3×) | 431 → 440 |

The gain is in programs that make and drop many small objects, arrays and strings, and it is larger
on Windows, whose process heap is slower than glibc's `malloc`. Where `@value` types or stack
promotion already keep a program's data off the heap, the heap barely matters.

## Other limits of `-mm=rc`

Expand Down
33 changes: 30 additions & 3 deletions tslang/include/TypeScript/ProcessHeap.h
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,11 @@
// list of its blocks per copy, while tslang.exe's own `malloc` is rpmalloc when LLVM is the
// prebuilt package. The process heap is the same heap in every module and every build.
//
// An executable that loads no library tslang built is one module, so it allocates from mimalloc
// instead: tslang links TypeScriptMimalloc.lib, whose definitions of the same entry points win over
// these (lib/TypeScriptMimalloc), and the JIT does the same for the program it runs (jit.cpp).
// The process's one heap is the program's, and an executable tslang links has mimalloc for it
// (TypeScriptMimalloc.lib, whose definitions of the entry points win over these), as does a program
// the JIT runs (jit.cpp). A library tslang built allocates from the program's heap through the
// entry points the program exports, and from the process heap only when the program exports none
// (lib/ProcessHeapExports.inc).
//
// Header-only, so each place that has to provide them - the async runtime library a program links,
// TypeScriptRuntime.dll and the JIT - compiles the same code.
Expand Down Expand Up @@ -91,6 +93,31 @@ inline void *allocateAligned(size_t align, size_t size)
return allocate(size);
}

// Whether the program allocates from mimalloc (an executable that links TypeScriptMimalloc exports
// this marker; the JIT knows for itself).
inline bool programUsesMimalloc()
{
return GetProcAddress(GetModuleHandleW(nullptr), "__tslang_heap_mimalloc") != nullptr;
}

// Whether a library allocates from the program's heap (lib/ProcessHeapExports.inc exports the
// marker) rather than keeping one of its own, as a library a tslang before it built does. Read from
// the library mapped without running any of its code or loading what it imports, found where
// LoadLibrary would find it - so it can be asked before the library is loaded. A library that is not
// there says yes: loading it fails on its own.
inline bool librarySharesHeap(const wchar_t *fileName)
{
auto library = LoadLibraryExW(fileName, nullptr, DONT_RESOLVE_DLL_REFERENCES);
if (library == nullptr)
{
return true;
}

auto shares = GetProcAddress(library, "__tslang_heap_shared") != nullptr;
FreeLibrary(library);
return shares;
}

} // namespace process_heap
} // namespace typescript

Expand Down
129 changes: 122 additions & 7 deletions tslang/lib/ProcessHeapExports.inc
Original file line number Diff line number Diff line change
@@ -1,34 +1,149 @@
// The process-heap allocator under the names ProcessHeapPass calls (PROCESS_HEAP_* in
// The heap entry points under the names ProcessHeapPass calls (PROCESS_HEAP_* in
// TypeScript/Defines.h). Included by the async runtime library, which a program and a library link,
// and by TypeScriptRuntime.dll, which the JIT loads; see TypeScript/ProcessHeap.h for why.
// and by TypeScriptRuntime.dll, which the JIT loads; see TypeScript/ProcessHeap.h for why there is
// one heap per process.
//
// The program owns the heap. In the program's own module these are the process heap; an executable
// that links TypeScriptMimalloc gets that library's definitions instead (mimalloc), and tslang.exe
// gives a program it runs its own (jit.cpp). Every one of them is exported, so a library - a
// module other than the process's executable - looks them up there at its first allocation and
// calls the program's: the library's blocks and the program's come from one heap, whichever it is.
// A program that exports none (a host tslang did not build, or one built before this) leaves a
// library on the process heap, which is what such a program allocates from too.
//
// `__tslang_heap_shared` says a module does this, so that a program on mimalloc can refuse a library
// that would keep a heap of its own (ProcessHeap.h, librarySharesHeap).

#ifdef _WIN32

#include <atomic>

#include "TypeScript/Defines.h"
#include "TypeScript/ProcessHeap.h"

extern "C" void *__tslang_heap_malloc(size_t size)
extern "C" __declspec(dllexport) const int __tslang_heap_shared = 1;

namespace
{

using HeapMallocFn = void *(*)(size_t);
using HeapCallocFn = void *(*)(size_t, size_t);
using HeapReallocFn = void *(*)(void *, size_t);
using HeapFreeFn = void (*)(void *);
using HeapAlignedAllocFn = void *(*)(size_t, size_t);

void *processHeapMalloc(size_t size)
{
return typescript::process_heap::allocate(size);
}

extern "C" void *__tslang_heap_calloc(size_t count, size_t size)
void *processHeapCalloc(size_t count, size_t size)
{
return typescript::process_heap::allocateZeroed(count, size);
}

extern "C" void *__tslang_heap_realloc(void *ptr, size_t size)
void *processHeapRealloc(void *ptr, size_t size)
{
return typescript::process_heap::reallocate(ptr, size);
}

extern "C" void __tslang_heap_free(void *ptr)
void processHeapFree(void *ptr)
{
typescript::process_heap::release(ptr);
}

extern "C" void *__tslang_heap_aligned_alloc(size_t alignment, size_t size)
void *processHeapAlignedAlloc(size_t alignment, size_t size)
{
return typescript::process_heap::allocateAligned(alignment, size);
}

// Filled once, by whichever thread allocates first; every thread that races to it computes the
// same five functions. The library's own global constructors may allocate before anything else
// in it runs, so this is not left to a constructor of this file.
std::atomic<HeapMallocFn> heapMalloc{nullptr};
std::atomic<HeapCallocFn> heapCalloc{nullptr};
std::atomic<HeapReallocFn> heapRealloc{nullptr};
std::atomic<HeapFreeFn> heapFree{nullptr};
std::atomic<HeapAlignedAllocFn> heapAlignedAlloc{nullptr};

void resolveHeap()
{
HeapMallocFn mallocFn = processHeapMalloc;
HeapCallocFn callocFn = processHeapCalloc;
HeapReallocFn reallocFn = processHeapRealloc;
HeapFreeFn freeFn = processHeapFree;
HeapAlignedAllocFn alignedAllocFn = processHeapAlignedAlloc;

// the program's heap, unless this is the program
HMODULE self = nullptr;
auto program = GetModuleHandleW(nullptr);
if (GetModuleHandleExW(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
reinterpret_cast<LPCWSTR>(&resolveHeap), &self)
&& program != nullptr && program != self)
{
auto programMalloc = reinterpret_cast<HeapMallocFn>(GetProcAddress(program, PROCESS_HEAP_MALLOC));
auto programCalloc = reinterpret_cast<HeapCallocFn>(GetProcAddress(program, PROCESS_HEAP_CALLOC));
auto programRealloc = reinterpret_cast<HeapReallocFn>(GetProcAddress(program, PROCESS_HEAP_REALLOC));
auto programFree = reinterpret_cast<HeapFreeFn>(GetProcAddress(program, PROCESS_HEAP_FREE));
auto programAlignedAlloc =
reinterpret_cast<HeapAlignedAllocFn>(GetProcAddress(program, PROCESS_HEAP_ALIGNED_ALLOC));
// all five or none: half of one heap and half of another frees blocks on the wrong one
if (programMalloc && programCalloc && programRealloc && programFree && programAlignedAlloc)
{
mallocFn = programMalloc;
callocFn = programCalloc;
reallocFn = programRealloc;
freeFn = programFree;
alignedAllocFn = programAlignedAlloc;
}
}

heapCalloc.store(callocFn, std::memory_order_relaxed);
heapRealloc.store(reallocFn, std::memory_order_relaxed);
heapFree.store(freeFn, std::memory_order_relaxed);
heapAlignedAlloc.store(alignedAllocFn, std::memory_order_relaxed);
// last, and released: a thread that sees malloc sees the other four
heapMalloc.store(mallocFn, std::memory_order_release);
}

void ensureHeap()
{
if (!heapMalloc.load(std::memory_order_acquire))
{
resolveHeap();
}
}

} // namespace

extern "C" __declspec(dllexport) void *__tslang_heap_malloc(size_t size)
{
ensureHeap();
return heapMalloc.load(std::memory_order_relaxed)(size);
}

extern "C" __declspec(dllexport) void *__tslang_heap_calloc(size_t count, size_t size)
{
ensureHeap();
return heapCalloc.load(std::memory_order_relaxed)(count, size);
}

extern "C" __declspec(dllexport) void *__tslang_heap_realloc(void *ptr, size_t size)
{
ensureHeap();
return heapRealloc.load(std::memory_order_relaxed)(ptr, size);
}

extern "C" __declspec(dllexport) void __tslang_heap_free(void *ptr)
{
ensureHeap();
heapFree.load(std::memory_order_relaxed)(ptr);
}

extern "C" __declspec(dllexport) void *__tslang_heap_aligned_alloc(size_t alignment, size_t size)
{
ensureHeap();
return heapAlignedAlloc.load(std::memory_order_relaxed)(alignment, size);
}

#endif // _WIN32
17 changes: 17 additions & 0 deletions tslang/lib/TypeScriptAsyncRuntime/DynamicRuntime.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@
#define WIN32_LEAN_AND_MEAN
#define NOMINMAX
#include <windows.h>

#include <cstdio>

#include "TypeScript/ProcessHeap.h"
#else
#include <dlfcn.h>
#endif
Expand Down Expand Up @@ -56,6 +60,19 @@ void *openLibrary(const char *fileName)
return nullptr;
}

// A program on mimalloc lends it to every library it loads (ProcessHeapExports.inc). A library
// that keeps a heap of its own - one a tslang before that built - would free the program's
// blocks on it, and the program its: refused here, before any of its code runs.
if (typescript::process_heap::programUsesMimalloc()
&& !typescript::process_heap::librarySharesHeap(wideFileName.c_str()))
{
fprintf(stderr,
"tslang runtime: %s does not allocate from the program's heap (a tslang before this one built it); "
"rebuild it, or build the program with --process-heap\n",
fileName);
return nullptr;
}

return reinterpret_cast<void *>(LoadLibraryW(wideFileName.c_str()));
#else
return dlopen(fileName, RTLD_LAZY | RTLD_GLOBAL);
Expand Down
Loading
Loading