Is `longjmp` through rust code UB

[chorman0773]

What is the result of using setjmp in C, entering rust code, then calling longjmp, either from Rust, or after first returning into C. Is this considered undefined behaviour or defined behaviour.

Because of Pin (and likely others, things like replace_with), this cannot be sound in general, but there is a question of whether it is considered immediately undefined behaviour or whether it is defined in Rust.

This intersects with @rust-lang/wg-ffi-unwind. From https://rust-lang.zulipchat.com/#narrow/stream/210922-project-ffi-unwind/topic/cost.20of.20supporting.20longjmp.20without.20annotations and https://rust-lang.zulipchat.com/#narrow/stream/210922-project-ffi-unwind/topic/longjmp.20rules the likely answer is "Undefined Behaviour" even through frames without destructors/catch_unwind.

In addition to determining the desired behaviour, we should consider what guarantees we've made that would make longjmp unsound (beyond Pin).

(Cc prior broader discussion: #211)

[digama0]

I think it should be possible for at least some trivial versions of setjmp/longjmp to be sound. For example if both setjmp and longjmp calls are in C and they don't jump over any rust stack frames, then it should be possible to argue that this is just FFI having some lang-specific behavior which isn't relevant, as long as it does not explicitly manifest as "impossible behavior" from the rust code's perspective.

The next level of difficulty is jumping from C to C over a Rust stack frame which has no destructors, and here I think it's still fine, since this is as-if you just unwind those frames. (We may want to make this UB if there is a "nounwind" annotation of some kind. Possibly this is already true of all C calls, in which case this and everything after is UB at the unwind.)

If you jump from C to C over a Rust stack frame which has destructors, I still think it is fine as regards immediate UB, but this is unsound behavior and may trigger UB in subsequent rust code since a destructor was skipped.

For C to Rust or Rust to C, the question does not arise because (for a variety of reasons) the libc crate does not support setjmp or longjmp, last I checked. So you would have to go design that interface first.

[thomcc]

The next level of difficulty is jumping from C to C over a Rust stack frame which has no destructors, and here I think it's still fine, since this is as-if you just unwind those frames.
...
For C to Rust or Rust to C, the question does not arise because (for a variety of reasons) the libc crate does not support setjmp or longjmp, last I checked. So you would have to go design that interface first.

So, at work we call setjmp (sigsetjmp, really), and then immediately call into C, which may longjmp over Rust stack frames (ones which have no destructors) back to our setjmp. At the moment, it's almost certainly a form of UB (one which works in practice very reliably at the moment), since we don't use the (still-unstable) "ffi_returns_twice" attribute¹

I would strongly prefer this not be immediately UB though, since this kind of thing would be desirable to support and isn't that uncommon -- any Rust library that wants to call a C library that uses sjlj-based error-handling will want to do something like this.

Footnotes

1. We'd probably have to redesign this interface to effectively use without a significant performance cost, it inhibits inlining, so we'd have to use macros here instead of functions. ↩

[Amanieu]

For C to Rust or Rust to C, the question does not arise because (for a variety of reasons) the libc crate does not support setjmp or longjmp, last I checked. So you would have to go design that interface first.

We don't support setjmp because it has a weird calling convention that returns twice, however just calling longjmp from Rust is fine. It pretty much works like C to C over a Rust stack frame.

Calling setjmp is never going to be supported by Rust. The only way to safely use it is to call it using inline asm wrapper which would look something like this (credits to @nbdd0121, from Zulip):

unsafe fn setjmp<F: FnOnce() -> c_int>(env: *mut jmp_buf, f: F) -> c_int {
    let mut f = core::mem::MaybeUninit::new(f);
    extern "C" fn call_f<F: FnOnce() -> c_int>(f: *mut F) -> c_int {
        (unsafe { f.read() })()
    }
    let ret: c_int;
    std::arch::asm!(
        "call setjmp",
        "test eax, eax",
        "jne 1f",
        "mov rdi, r12",
        "call {}",
        "1:",
        sym call_f::<F>,
        in("rdi") env,
        in("r12") f.as_mut_ptr(),
        lateout("rax") ret,
        clobber_abi("C"),
    );
    ret
}

[chorman0773]

I would strongly prefer this not be immediately UB though, since this kind of thing would be desirable to support and isn't that uncommon -- any Rust library that wants to call a C library that uses sjlj-based error-handling will want to do something like this.

Note that any library that has that would have to be careful arround Pin anyways, and probably also destructors (and catch_unwind) in general, since windows and unix are quite inconsistent wrt. longjmp evaluating destructors.

[chorman0773]

Things that definitely make longjmp through rust unsound:

• Pin
• catch_unwind and Destructors, which may be guarding unsafe code

Things that may make longjmp through certain rust frames UB (or otherwise must be unspecified):

• catch_unwind and Destructors, which are run on Windows IIRC, and nowhere else.
• I'd assume abort-on-unwind handlers for extern "C" fns (and into extern "C-unwind" in an panic=abort crate), for same reason (not 100% sure on this).

[digama0]

By the way, I don't think that soundness of longjmp is our (t-opsem's) concern. We are only concerned about what causes immediate UB, and soundness is a derived property of libraries that don't cause UB. So Pin basically doesn't matter to us at all. (Obviously we do need to care about it for documentation purposes, but that doesn't seem to be the title question.)

[comex]

See also: https://github.com/rust-lang/project-ffi-unwind/blob/master/rfcs/0000-longjmp-pof-annotation.md

[RalfJung]

By the way, I don't think that soundness of longjmp is our (t-opsem's) concern. We are only concerned about what causes immediate UB, and soundness is a derived property of libraries that don't cause UB. So Pin basically doesn't matter to us at all. (Obviously we do need to care about it for documentation purposes, but that doesn't seem to be the title question.)

Well we do care about defining the semantics in a way that Pin is sound. ;)

However, setjmp/longjmp cannot be UB in the sense of "the AM stops with an error when you do this", since the AM doesn't even have such an operation. So this is really a subquestion of the general question of what FFI/asm can do to the Rust AM implementation state, which seems to live at the intersection of t-opsem and t-compiler.

[nbdd0121]

We define frame deallocation of non-POFs to be undefined.

[nbdd0121]

If frames to be deallocated consist of only POFs, then from Rust AM's perspective that's no different from just unwinding, given that no code is to be executed anyway?

[Amanieu]

If frames to be deallocated consist of only POFs, then from Rust AM's perspective that's no different from just unwinding, given that no code is to be executed anyway?

The question basically boils down to: is the compiler allowed to add unwind landing pads when the original code doesn't have any? If so then it is UB to skip these with longjmp, even if the original program has no droppable types in a frame.

[digama0]

However, setjmp/longjmp cannot be UB in the sense of "the AM stops with an error when you do this", since the AM doesn't even have such an operation.

It could, though. It would be something like an unwind from a function with a flag in the exception state to skip all destructors (call it "forced unwind"). This would then otherwise act just like an unwind, except that destructors would not be run on the way out. If you hit a nounwind function, you get UB. (There might also be functions which are no-normal-unwind which are okay with an unwind that skips destructors, in which case only "normal" unwinding is UB, not forced unwind. This is I think the common case for function calls that lack a cleanup block continuation.)

Obviously, forced unwind is unsafe since it breaks safety invariants, but it's not undefinable in the AM, and having a sane definition of what it does on the AM will avoid some of the weirder states this operation would otherwise end up in. Of course adding an operation to the AM means that it is now a thing you have to worry about happening from any opaque or external function, but you can control the fallout from that by liberally sprinkling "no-forced-unwind" on such functions until you can be sure they can be supported.

[chorman0773]

On Wed, May 24, 2023 at 16:57 Mario Carneiro ***@***.***> wrote: However, setjmp/longjmp cannot be UB in the sense of "the AM stops with an error when you do this", since the AM doesn't even have such an operation. It could, though. It would be something like an unwind from a function with a flag in the exception state to skip all destructors (call it "forced unwind"). This would then otherwise act just like an unwind, except that destructors would not be run on the way out. If you hit a nounwind function, you get UB. (There might also be functions which are no-normal-unwind which are okay with an unwind that skips destructors, in which case only "normal" unwinding is UB, not forced unwind. This is I think the common case for function calls that lack a cleanup block continuation.)

Note that skipping destructors is not even guaranteed - on SEH destructors are run and catches will "catch" a longjmp. I do not know enough about how panics are implemented on SEH to say whether this can even be fiesibly disabled.

…

Obviously, forced unwind is unsafe since it breaks safety invariants, but it's not *undefinable* in the AM, and having a sane definition of what it does on the AM will avoid some of the weirder states this operation would otherwise end up in. Of course adding an operation to the AM means that it is now a thing you have to worry about happening from any opaque or external function, but you can control the fallout from that by liberally sprinkling "no-forced-unwind" on such functions until you can be sure they can be supported. — Reply to this email directly, view it on GitHub <#404 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGLD25XNOYXHNN7C6RJRE3XHZY3NANCNFSM6AAAAAAYMO67D4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[digama0]

Note that skipping destructors is not even guaranteed - on SEH destructors
are run and catches will "catch" a longjmp. I do not know enough about how
panics are implemented on SEH to say whether this can even be fiesibly
disabled.

I think that would be handled by having longjmp do a normal unwind on windows, assuming it calls all destructors and rust drop impls count as destructors for this purpose. It would still be possible to implement forced unwind on windows using inline assembly, but I don't know whether we need to declare that UB for other reasons (i.e. windows gets messed up because it needs all unwinds to go through SEH for some reason).

[bjorn3]

LLVM will optimize out destructors only reachable when unwinding from calls to nounwind functions, yet it allows to longjmp out of a nounwind function just fine. This means that even on windows is undefined if destructors run or not and which ones do run.