Dynamic linking, threading, and cross-library TLS

[alexcrichton]

Over in wasi-sdk land we've recently been trying to get the whole 9 yards of major ABI-related features all working at the same time for componentize-py which uses:

• Dynamic linking - used so the python interpreter can "dlopen" native extensions, although for our purposes these are pre-bundled into a single component so it's mostly dynamically discovering where things were placed by the linker.
• Threads - with wasip3 we're in the process of stressing/testing coop threads and making sure it works in all the places
• Exceptions - some popular Python extensions use C++ exceptions as well

I think we're almost to the point where we can get everything working, including some refactors to how TLS works in wasip3, but there's one final piece that I'm not sure how to get working. I think that this is a limitation of tooling today which can't be supported, but I'm not sure of this -- hence the issue!

The specific case we're running into is that I'm not sure how dynamic library A can refer to a thread-local variable in dynamic library B. One example of this is the __wasm_lpad_context symbol used in the codegen of C++ exceptions. Right now libraries import a single "TLS base" value, either a global or a __wasm_tls_base function, which is the base of that dynamic library's TLS block. References to TLS values are then static offsets from that base, but this doesn't work when library A tries to access library B's TLS since A only has access to its own base pointer. The example in this situation is that the main module is trying to modify __wasm_lpad_context in the libunwind.so module, but it has no means of getting the base pointer to that location.

This issue boils down to: is it possible for cross-library thread-local symbols to work right now in the face of threading being enabled? We worked around the one instance of this in wasi-libc, for example, by defining errno to be *__errno_location() like it is on some native platforms as well. In the case of __wasm_lpad_context the accesses here are internal within LLVM's codegen for exception handling so it's not as easy to modify.

[alexcrichton]

cc @aheejin on this -- if it ends up that this can't work today, what would you think of updating WasmEHPrepare.cpp to call a function to get a pointer to __wasm_lpad_context as opposed to referencing it via a symbol? That's effectively the workaround we had for wasi-libc's errno definition. I'm not sure what __wasm_lpad_context is used for myself, though, so I'm not sure how applicable this would be.

[aheejin]

@sbc100 Is there a general way to handle this kind of cross-library references of thread local symbols in dynamic linking?

@alexcrichton In case there is not a good solution, adding a function call sounds fine to me. It would be nicer if we can do it only when dynamic linking is enabled, given that it adds a call instruction per catch clause. (This excludes both catch (...) and destructors, so the number of the those sites wouldn't be very many though)

[sbc100]

We did make this work (importing TLS address from one thread to DLL to another) in emscripten. IIRC this is how it works:

1. Addresses for external symbol are imported as GOT.data.<foo> mutable globals when DLLs are first loaded.
2. The dynamic linker (in our case written in JS) creates new Wasm globals of the form GOT.data.<foo> on demand using a JS proxy object: https://github.com/emscripten-core/emscripten/blob/7cc2974d1e7dd0c49aa30ccd9f155fd74dc37ddb/src/lib/libdylink.js#L786-L793
3. During DLL loading (in order to support dependency cycles between module, of which is have a lot in emscripten), symbols are not required to be defined, and in that case these globals will contains a zero address.
4. TLS symbols are exported from DLLs as TLS-relative Wasm globals. The dylink0 metadata section tells the dynamic linker which symbols are TLS-relative: https://github.com/emscripten-core/emscripten/blob/7cc2974d1e7dd0c49aa30ccd9f155fd74dc37ddb/src/lib/libdylink.js#L520-L529
5. Once a DLL is fully loaded on a given thread and its TLS init function has been called we then update any GOT entries for its TLS symbols: https://github.com/emscripten-core/emscripten/blob/7cc2974d1e7dd0c49aa30ccd9f155fd74dc37ddb/src/lib/libpthread.js#L675-L693

I guess the key here is that there has to a central / global symbol table which includes all the GOT symbols and it needs to be possible to update the GOT symbols after running TLS init. In our case that symbol table is all managed in JS, but I imagine its possible to write that on the wasm side too.

[sbc100]

BTW, the reason we have a lot of dependency cycles in emscripten is that main module always embeds all systems libraries (libc, libc++, compiler-rt, etc). This means that all system library references in DLLs end up being back-references to main program. We rely on JS stubs and also indirection through table slots that we can update to handle these back references.

[sbc100]

Also, note that just because a module defines a TLS data address, it might not win the race at runtime to define it. For example, imagine 3 different DLLs all exporting the same TLS data address. Its likely they they will both import and export the address, to handle the case that their definition is not the one that use chosen at runtime by the dynamic linker.

Whichever DLL is loaded first ends up winning the race to define a given symbol and then other DLLs will import that address, effectively ignoring their local definition completely. This mirrors the behavior of ELF, and can be controlled using the -Bsymbolic linker flag (this flag tells the linker to bind references the symbols defined in a DLL locally rather than allowing another DLL to override them).

[alexcrichton]

@sbc100 is this perhaps something or some flag we need to configure on the wasip3 target? I've got foo.c as:

__attribute__((visibility("default")))
_Thread_local int foo;

and bar.c as:

extern _Thread_local int foo;

__attribute__((visibility("default")))
int get() {
  return foo;
}

If I try to compile for wasm32-wasip1-threads, which is sort of the closest to what we want, I get:

$ wasm32-wasip1-threads-clang -fPIC foo.c -shared -o libfoo.so -nostdlib -Wl,--no-entry
wasm-ld: warning: creating shared libraries, with -shared, is not yet stable
$ wasm32-wasip1-threads-clang -fPIC bar.c -shared -o libbar.so -nostdlib -Wl,--no-entry ./libfoo.so
wasm-ld: warning: creating shared libraries, with -shared, is not yet stable
wasm-ld: error: /tmp/bar-76f1bf.o: relocation R_WASM_MEMORY_ADDR_TLS_SLEB cannot be used against non-TLS symbol `foo`
wasm32-wasip1-threads-clang: error: linker command failed with exit code 1 (use -v to see invocation)

I'm not sure where this error comes from, but when compiling for wasm32-wasip3 instead I get:

$ wasm32-wasip3-clang -fPIC foo.c -shared -o libfoo.so -nostdlib -Wl,--no-entry
$ wasm32-wasip3-clang -fPIC bar.c -shared -o libbar.so -nostdlib -Wl,--no-entry ./libfoo.so

This succeeds and the output is:

(module $libbar.so
  (@dylink.0
    (mem-info)
    (needed "libfoo.so")
  )
  (type (;0;) (func (param i32)))
  (type (;1;) (func (result i32)))
  (type (;2;) (func))
  (import "env" "memory" (memory (;0;) 0))
  (import "env" "__memory_base" (global $__memory_base (;0;) i32))
  (import "env" "__table_base" (global $__table_base (;1;) i32))
  (import "env" "__wasm_set_tls_base" (func $__wasm_set_tls_base (;0;) (type 0)))
  (import "env" "__wasm_get_tls_base" (func $__wasm_get_tls_base (;1;) (type 1)))
  (global $__init_tls_base (;2;) (mut i32) i32.const 0)
  (global $__tls_size (;3;) i32 i32.const 0)
  (global $__tls_align (;4;) i32 i32.const 0)
  (export "get" (func $get))
  (func $__wasm_call_ctors (;2;) (type 2))
  (func $get (;3;) (type 1) (result i32)
    i32.const 0
    call $__wasm_get_tls_base
    i32.add
    i32.load
    return
  )
  (@producers
    (processed-by "clang" "23.1.0-wasi-sdk (https://github.com/llvm/llvm-project 278c31bfb8ceb7ea17dbfd11a4fb21e6634af957)")
  )
  (@custom "target_features" (after code) "\08+\0bbulk-memory+\0fbulk-memory-opt+\16call-indirect-overlong+\0amultivalue+\0fmutable-globals+\13nontrapping-fptoint+\0freference-types+\08sign-ext")
)

Here there's no global imported for the foo symbol. On a non-wasip3 build I'd expect call $__wasm_get_tls_base to get replaced with global.get $__tls_base, but that wouldn't fundamentally change things here. How is Emscripten generating globals for data imports in this case?

I'll note that even if we could get global imports to work they wouldn't work for the wasip3 target, though, because that's not taking the approach of an instance-per-thread model which it sounds like is basically how this scheme works in Emscripten. I realize Emscripten and the web in general have instance-per-thread, but I mean specifically for this scheme of cross-library references to TLS it relies on multiple unique globals per thread (per instance on the web) where for wasip3 we've got just one context slot which is unique-per-thread.

[sbc100]

Can you confirm that if you just compile bar.c above you see global.get 0 <GOT.mem.foo> in the generated code:

$ emcc -fPIC -c bar.c
$ wasm-objdump -d bar.o 

bar.o:	file format wasm 0x1

Code Disassembly:

000050 func[0] <get>:
 000051: 23 80 80 80 80 00          | global.get 0 <GOT.mem.foo>
 000057: 28 02 00                   | i32.load 2 0
 00005a: 0f                         | return
 00005b: 0b                         | end

I see the same thing once I link to a shared object:

$ emcc -shared bar.o -o libbar.so
$ wasm-objdump -d libbar.so 

libbar.so:	file format wasm 0x1

Code Disassembly:

000092 func[0] <__wasm_call_ctors>:
 000093: 0b                         | end
000095 func[1] <get>:
 000096: 23 82 80 80 80 00          | global.get 2 <GOT.mem.foo>
 00009c: 28 02 00                   | i32.load 2 0
 00009f: 0f                         | return
 0000a0: 0b                         | end

[sbc100]

More details of the final .so file (edited, I forgot to add -pthread):

$ wasm-objdump -d -r -x libbar.so 

libbar.so:	file format wasm 0x1

Section Details:

Custom:
 - name: "dylink.0"
 - mem_size     : 0
 - mem_p2align  : 0
 - table_size   : 0
 - table_p2align: 0
Type[4]:
 - type[0] () -> i32
 - type[1] (i32, i32) -> i32
 - type[2] () -> nil
 - type[3] (i32) -> nil
Import[6]:
 - memory[0] pages: initial=0 max=65536 shared <- env.memory
 - global[0] i32 mutable=0 <- env.__memory_base
 - global[1] i32 mutable=0 <- env.__table_base
 - func[0] sig=0 <env.pthread_self> <- env.pthread_self
 - func[1] sig=1 <env.emscripten_builtin_memalign> <- env.emscripten_builtin_memalign
 - global[2] i32 mutable=1 <- GOT.mem.foo
Function[4]:
 - func[2] sig=2 <__wasm_call_ctors>
 - func[3] sig=3
 - func[4] sig=0 <get>
 - func[5] sig=0 <_emscripten_tls_init>
Global[4]:
 - global[3] i32 mutable=1 - init i32=0
 - global[4] i32 mutable=0 - init i32=0
 - global[5] i32 mutable=0 - init i32=0
 - global[6] i32 mutable=1 - init i32=0
Export[3]:
 - func[2] <__wasm_call_ctors> -> "__wasm_call_ctors"
 - func[4] <get> -> "get"
 - func[5] <_emscripten_tls_init> -> "_emscripten_tls_init"
Code[4]:
 - func[2] size=2 <__wasm_call_ctors>
 - func[3] size=2
 - func[4] size=12 <get>
 - func[5] size=122 <_emscripten_tls_init>
Custom:
 - name: "target_features"
  - [+] atomics
  - [+] bulk-memory
  - [+] bulk-memory-opt
  - [+] call-indirect-overlong
  - [+] multivalue
  - [+] mutable-globals
  - [+] nontrapping-fptoint
  - [+] reference-types
  - [+] sign-ext

Code Disassembly:

000105 func[2] <__wasm_call_ctors>:
 000106: 0b                         | end
000108 func[3]:
 000109: 0b                         | end
00010b func[4] <get>:
 00010c: 23 82 80 80 80 00          | global.get 2 <GOT.mem.foo>
 000112: 28 02 00                   | i32.load 2 0
 000115: 0f                         | return
 000116: 0b                         | end
000118 func[5] <_emscripten_tls_init>:
 000119: 04 7f                      | local[0..3] type=i32
 00011b: 23 83 80 80 80 00          | global.get 3
 000121: 21 00                      | local.set 0
 000123: 10 80 80 80 80 00          | call 0 <env.pthread_self>
 000129: 22 01                      | local.tee 1
 00012b: 28 02 64                   | i32.load 2 100
 00012e: 21 02                      | local.set 2
 000130: 23 84 80 80 80 00          | global.get 4
 000136: 21 03                      | local.set 3
 000138: 02 40                      | block
 00013a: 20 02                      |   local.get 2
 00013c: 45                         |   i32.eqz
 00013d: 0d 00                      |   br_if 0
 00013f: 20 01                      |   local.get 1
 000141: 41 00                      |   i32.const 0
 000143: 36 02 64                   |   i32.store 2 100
 000146: 20 02                      |   local.get 2
 000148: 22 02                      |   local.tee 2
 00014a: 10 83 80 80 80 00          |   call 3
 000150: 20 02                      |   local.get 2
 000152: 0f                         |   return
 000153: 0b                         | end
 000154: 23 86 80 80 80 00          | global.get 6
 00015a: 21 02                      | local.set 2
 00015c: 02 40                      | block
 00015e: 02 40                      |   block
 000160: 20 02                      |     local.get 2
 000162: 0d 00                      |     br_if 0
 000164: 20 00                      |     local.get 0
 000166: 0d 01                      |     br_if 1
 000168: 20 03                      |     local.get 3
 00016a: 45                         |     i32.eqz
 00016b: 0d 01                      |     br_if 1
 00016d: 0b                         |   end
 00016e: 41 01                      |   i32.const 1
 000170: 24 86 80 80 80 00          |   global.set 6
 000176: 23 85 80 80 80 00          |   global.get 5
 00017c: 20 03                      |   local.get 3
 00017e: 10 81 80 80 80 00          |   call 1 <env.emscripten_builtin_memalign>
 000184: 21 00                      |   local.set 0
 000186: 0b                         | end
 000187: 20 00                      | local.get 0
 000189: 10 83 80 80 80 00          | call 3
 00018f: 20 00                      | local.get 0
 000191: 0b                         | end

[sbc100]

And libfoo.so, which contains that TLS info for the exports:

libfoo.so:	file format wasm 0x1

Section Details:

Custom:
 - name: "dylink.0"
 - mem_size     : 8
 - mem_p2align  : 2
 - table_size   : 0
 - table_p2align: 0
 - exports[1]:
  - foo [ tls binding=global vis=default ]
Type[4]:
 - type[0] () -> i32
 - type[1] (i32, i32) -> i32
 - type[2] () -> nil
 - type[3] (i32) -> nil
Import[5]:
 - memory[0] pages: initial=1 max=65536 shared <- env.memory
 - global[0] i32 mutable=0 <- env.__memory_base
 - global[1] i32 mutable=0 <- env.__table_base
 - func[0] sig=0 <env.pthread_self> <- env.pthread_self
 - func[1] sig=1 <env.emscripten_builtin_memalign> <- env.emscripten_builtin_memalign
Function[4]:
 - func[2] sig=2 <__wasm_call_ctors>
 - func[3] sig=3
 - func[4] sig=2
 - func[5] sig=0 <_emscripten_tls_init>
Global[5]:
 - global[2] i32 mutable=1 - init i32=0
 - global[3] i32 mutable=0 - init i32=4
 - global[4] i32 mutable=0 - init i32=4
 - global[5] i32 mutable=1 - init i32=0
 - global[6] i32 mutable=0 <foo> - init i32=0
Export[3]:
 - func[2] <__wasm_call_ctors> -> "__wasm_call_ctors"
 - global[6] -> "foo"
 - func[5] <_emscripten_tls_init> -> "_emscripten_tls_init"
Start:
 - start function: 4
DataCount:
 - data count: 1
Code[4]:
 - func[2] size=2 <__wasm_call_ctors>
 - func[3] size=16
 - func[4] size=84
 - func[5] size=122 <_emscripten_tls_init>
Data[1]:
 - segment[0] passive size=4
  - 0000000: 0000 0000                                ....
Custom:
 - name: "target_features"
  - [+] atomics
  - [+] bulk-memory
  - [+] bulk-memory-opt
  - [+] call-indirect-overlong
  - [+] multivalue
  - [+] mutable-globals
  - [+] nontrapping-fptoint
  - [+] reference-types
  - [+] sign-ext

Code Disassembly:

00010a func[2] <__wasm_call_ctors>:
 00010b: 0b                         | end
00010d func[3]:
 00010e: 20 00                      | local.get 0
 000110: 24 02                      | global.set 2
 000112: 20 00                      | local.get 0
 000114: 41 00                      | i32.const 0
 000116: 41 04                      | i32.const 4
 000118: fc 08 00 00                | memory.init 0 0
 00011c: 0b                         | end
00011e func[4]:
 00011f: 02 7f                      | local[0..1] type=i32
 000121: 23 00                      | global.get 0 <env.__memory_base>
 000123: 41 04                      | i32.const 4
 000125: 6a                         | i32.add
 000126: 21 00                      | local.set 0
 000128: 02 40                      | block
 00012a: 02 40                      |   block
 00012c: 02 40                      |     block
 00012e: 20 00                      |       local.get 0
 000130: 41 00                      |       i32.const 0
 000132: 41 01                      |       i32.const 1
 000134: fe 48 02 00                |       i32.atomic.rmw.cmpxchg 2 0
 000138: 0e 02 00 01 02             |       br_table 0 1 2
 00013d: 0b                         |     end
 00013e: 41 00                      |     i32.const 0
 000140: 23 00                      |     global.get 0 <env.__memory_base>
 000142: 6a                         |     i32.add
 000143: 22 01                      |     local.tee 1
 000145: 24 02                      |     global.set 2
 000147: 20 01                      |     local.get 1
 000149: 41 00                      |     i32.const 0
 00014b: 41 04                      |     i32.const 4
 00014d: fc 08 00 00                |     memory.init 0 0
 000151: 20 00                      |     local.get 0
 000153: 41 02                      |     i32.const 2
 000155: fe 17 02 00                |     i32.atomic.store 2 0
 000159: 20 00                      |     local.get 0
 00015b: 41 7f                      |     i32.const 4294967295
 00015d: fe 00 02 00                |     memory.atomic.notify 2 0
 000161: 1a                         |     drop
 000162: 0c 01                      |     br 1
 000164: 0b                         |   end
 000165: 20 00                      |   local.get 0
 000167: 41 01                      |   i32.const 1
 000169: 42 7f                      |   i64.const 18446744073709551615
 00016b: fe 01 02 00                |   memory.atomic.wait32 2 0
 00016f: 1a                         |   drop
 000170: 0b                         | end
 000171: 0b                         | end
000173 func[5] <_emscripten_tls_init>:
 000174: 04 7f                      | local[0..3] type=i32
 000176: 23 82 80 80 80 00          | global.get 2
 00017c: 21 00                      | local.set 0
 00017e: 10 80 80 80 80 00          | call 0 <env.pthread_self>
 000184: 22 01                      | local.tee 1
 000186: 28 02 64                   | i32.load 2 100
 000189: 21 02                      | local.set 2
 00018b: 23 83 80 80 80 00          | global.get 3
 000191: 21 03                      | local.set 3
 000193: 02 40                      | block
 000195: 20 02                      |   local.get 2
 000197: 45                         |   i32.eqz
 000198: 0d 00                      |   br_if 0
 00019a: 20 01                      |   local.get 1
 00019c: 41 00                      |   i32.const 0
 00019e: 36 02 64                   |   i32.store 2 100
 0001a1: 20 02                      |   local.get 2
 0001a3: 22 02                      |   local.tee 2
 0001a5: 10 83 80 80 80 00          |   call 3
 0001ab: 20 02                      |   local.get 2
 0001ad: 0f                         |   return
 0001ae: 0b                         | end
 0001af: 23 85 80 80 80 00          | global.get 5
 0001b5: 21 02                      | local.set 2
 0001b7: 02 40                      | block
 0001b9: 02 40                      |   block
 0001bb: 20 02                      |     local.get 2
 0001bd: 0d 00                      |     br_if 0
 0001bf: 20 00                      |     local.get 0
 0001c1: 0d 01                      |     br_if 1
 0001c3: 20 03                      |     local.get 3
 0001c5: 45                         |     i32.eqz
 0001c6: 0d 01                      |     br_if 1
 0001c8: 0b                         |   end
 0001c9: 41 01                      |   i32.const 1
 0001cb: 24 85 80 80 80 00          |   global.set 5
 0001d1: 23 84 80 80 80 00          |   global.get 4
 0001d7: 20 03                      |   local.get 3
 0001d9: 10 81 80 80 80 00          |   call 1 <env.emscripten_builtin_memalign>
 0001df: 21 00                      |   local.set 0
 0001e1: 0b                         | end
 0001e2: 20 00                      | local.get 0
 0001e4: 10 83 80 80 80 00          | call 3
 0001ea: 20 00                      | local.get 0
 0001ec: 0b                         | end

[sbc100]

For some reason it looks like your libbar.so thinks that foo is a DLL-local symbol.. and its not importing it.

[alexcrichton]

For wasm32-wasip1-threads I see a global.get, yeah, but for wasm32-wasip3 I only see a tls_base+constant. This is probably a wasip3-specific path somewhere in LLVM it seems.

It's ok though because even if that were actually changed it still wouldn't work, we don't want thread switching to require updating an arbitrary number of globals w.r.t. TLS references. The closest I can think of to what we want for wasip3 is a new kind of relocation where each library, at runtime, gets a unique number and then the __wasm_get_tls_base intrinsic has a second form which takes the ilbrary index to get the base of. That way one library would be able to get the base of another library by importing a global or similar corresponding to the library's dynamically-assigned-at-runtime index. That's pretty involved, though.

[sbc100]

I see, you the imported symbols from a TLS external address foo would be GOT.mem.foo but it would be relative to its defining DLL, instead of absolute (so it would be the same on each thread).

Then you have to do __wasm_get_tls_base(DSO_handle_foo) + GOT.mem.foo to get the actual address at runtime. i.e. you would make accessing TLS data more expensive but you keep thread switching cheap (since GOT.mem.foo doesn't need updating on thread switch).

[alexcrichton]

Right yeah, and the expense is only paid for cross-library TLS really. We've already got each library's TLS laid out in a block-of-pointers so each library already has an integer assigned to it, it's just not exposed anywhere. So from a runtime perspective the pieces are all there -- it's just from a compiler-perspective that we'd need the generated code to change (maybe more relocations, I'm not entirely sure). I realize though that this would all be specific for wasip3 at this time, though, so I suspect the most expedient route to solving our initial problem, exceptions not working, would probably be to have a mode where that calls a function rather than using a TLS symbol.