error: could not compile `gkrust` since Rust 1.43 on SPARC Solaris

[psumbera]

View all comments

When I build Firefox on Solaris SPARC it fails with:

   Compiling gkrust v0.1.0 (/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/firefox-68.10.0/toolkit/library/rust)
     Running `CARGO=/builds/psumbera/rustc-1.44.1/bin/cargo CARGO_MANIFEST_DIR=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/firefox-68.10.0/toolkit/library/rust CARGO_PKG_AUTHORS='nobody@mozilla.org' CARGO_PKG_DESCRIPTION='Rust code for libxul' CARGO_PKG_HOMEPAGE= CARGO_PKG_NAME=gkrust CARGO_PKG_REPOSITORY= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= LD_LIBRARY_PATH='/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/release/deps:/builds/psumbera/rustc-1.44.1/lib' /builds/psumbera/rustc-1.44.1/bin/rustc --crate-name gkrust toolkit/library/rust/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C codegen-units=1 -C lto --cfg 'feature="bindgen"' --cfg 'feature="cubeb_pulse_rust"' --cfg 'feature="moz_memory"' --cfg 'feature="moz_places"' --cfg 'feature="quantum_render"' --cfg 'feature="servo"' -C metadata=02c56202166b3eec -C extra-filename=-02c56202166b3eec --out-dir /builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/deps --target sparcv9-sun-solaris -C linker=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/firefox-68.10.0/build/cargo-linker -L dependency=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/deps -L dependency=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/release/deps --extern gkrust_shared=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/deps/libgkrust_shared-9712e354bc55934b.rlib --extern mozilla_central_workspace_hack=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/deps/libmozilla_central_workspace_hack-4044d7b7b8cbdafc.rlib -C opt-level=2 --cap-lints warn -L native=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/build/lmdb-rkv-sys-c379f0737c738302/out`
error: could not compile `gkrust`.

Caused by:
  process didn't exit successfully: `CARGO=/builds/psumbera/rustc-1.44.1/bin/cargo CARGO_MANIFEST_DIR=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/firefox-68.10.0/toolkit/library/rust CARGO_PKG_AUTHORS='nobody@mozilla.org' CARGO_PKG_DESCRIPTION='Rust code for libxul' CARGO_PKG_HOMEPAGE= CARGO_PKG_NAME=gkrust CARGO_PKG_REPOSITORY= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= LD_LIBRARY_PATH='/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/release/deps:/builds/psumbera/rustc-1.44.1/lib' /builds/psumbera/rustc-1.44.1/bin/rustc --crate-name gkrust toolkit/library/rust/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C codegen-units=1 -C lto --cfg 'feature="bindgen"' --cfg 'feature="cubeb_pulse_rust"' --cfg 'feature="moz_memory"' --cfg 'feature="moz_places"' --cfg 'feature="quantum_render"' --cfg 'feature="servo"' -C metadata=02c56202166b3eec -C extra-filename=-02c56202166b3eec --out-dir /builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/deps --target sparcv9-sun-solaris -C linker=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/firefox-68.10.0/build/cargo-linker -L dependency=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/deps -L dependency=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/release/deps --extern gkrust_shared=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/deps/libgkrust_shared-9712e354bc55934b.rlib --extern mozilla_central_workspace_hack=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/deps/libmozilla_central_workspace_hack-4044d7b7b8cbdafc.rlib -C opt-level=2 --cap-lints warn -L native=/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/sparcv9-sun-solaris/release/build/lmdb-rkv-sys-c379f0737c738302/out` (signal: 11, SIGSEGV: invalid memory reference)
make[4]: *** [/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/firefox-68.10.0/config/makefiles/rust.mk:240: force-cargo-library-build] Error 101
make[4]: Leaving directory '/builds/psumbera/userland-ff-68.10.0/components/desktop/firefox/build/sparcv9/toolkit/library/rust'

This worked well till Rust 1.42. Since Rust 1.43 it fails. There is no problem on X86 Solaris.

I was able to bisect this to following change:

e82ec2315e5adb1c291c3702cd2ac1f46ecd0fcf is the first bad commit
commit e82ec2315e5adb1c291c3702cd2ac1f46ecd0fcf
Author: Dylan MacKenzie <ecstaticmorse@gmail.com>
Date:   Tue Mar 3 11:26:51 2020 -0800

    Use correct place for `enum_place`

    PR #69562, which fixed a bug that was causing clippy to ICE, passed the
    place for the *result* of `Rvalue::Discriminant` instead of the
    *operand* to `apply_discriminant_switch_effect`. As a result, no effect
    was applied at all, and we lost the perf benefits from marking
    inactive enum variants as uninitialized.

e82ec23

[psumbera]

@ecstatic-morse any idea what can be wrong with it on SPARC (64 bits big endian system)?

There is also core file generated:

0007fdd75ddddcf1 libc.so.1`_memcpy%sun4v-hwcap3+0x108(9fd94b2180, 10, 1, 10, 7fdd774a58000, 0)
0007fdd75dddde51 libstd-18d6dd3f485e207d.so`_$LT$$RF$str$u20$as$u20$std..ffi..c_str..CString..new..SpecIntoVec$GT$::into_vec::h866051ec29816e57+0xb4(7fdd75dddec00, 1, 10, 0, 0, 9fd94b2210)
0007fdd75ddddf31 librustc_driver-64d0fcc570abb6ec.so`rustc_codegen_llvm::back::lto::run_fat::hcb53787b9ab702b0+0xfe4(9fd945aac0, 2, 7fdd75dddec00, 9fd945aaf8, 7fdd75dddeb68, 7fdd75dddeb61)
0007fdd75ddde431 librustc_driver-64d0fcc570abb6ec.so`rustc_codegen_ssa::back::write::generate_lto_work::h29d8f36a78087208+0xbc(7fdd75dddf370, 7fdd75dddefc8, 7fdd75dddf4f0, 7fdd75dddf3d0,
7fdd75dddf5c0, 9fda425120)
0007fdd75ddde5d1 librustc_driver-64d0fcc570abb6ec.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h122f5d501f74f5dc+0x400(7fdd75dddf370, 7fdd75dddf5c8, 7fdd75dddf3d8, 0, 0,
7fdd75dddf5f8)
0007fdd75dddeee1 librustc_driver-64d0fcc570abb6ec.so`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h784dca9a9559144b+0x98(7fdd75dddfc28, 7fdd75dddfb40, 9fda3fce90,
7fdd75dddf870, 168, 0)
0007fdd75dddf5a1 libstd-18d6dd3f485e207d.so`std::sys::unix::thread::Thread::new::thread_start::h4f6832cd8c1f06fc+0x20(7fdd774004000, 7fdd77871be94, 9fda3e2c90, 9fda3fce90, 7fdd77e5558b0, 0)
0007fdd75dddf671 libc.so.1`_lwp_start(0, 0, 0, 0, 0, 0)

[ecstatic-morse]

This is concerning. To confirm, this occurs with recent nightlies as well?

The commit you bisected to (part of #69676) re-enabled the optimization around drop elaboration in #68528, which could have plausibly caused this kind of issue. There's nothing target-dependent about that PR, so while it's possible that #68528 triggered a SPARC-only LLVM bug, it's very possible that there's a latent miscompilation on other platforms as well that manifests as a segmentation fault due to some detail of SPARC. However, it's surprising that we didn't see any issues from users on Intel machines.

The innermost frame points to this implementation detail of Cstring::new. There's no match on a type that has a drop implementation along that code path, so presumably the root cause is elsewhere. My next step for debugging this would probably be ASan, although I haven't had to use it in anger on Rust code yet. WDYT @psumbera?

Nominating for @rust-lang/compiler discussion to put this on their radar, since it could affect tier 1 platforms as well.

[psumbera]

This is concerning. To confirm, this occurs with recent nightlies as well?

Latest version I have tested is Rust 1.44.1. I can revert the commit and I can use it to build Firefox.

I will try latest nightly build as well. But I will need to build Rust 1.45 first (there are no bootstrap archives for Solaris).

The innermost frame points to this implementation detail of Cstring::new. There's no match on a type that has a drop implementation along that code path, so presumably the root cause is elsewhere. My next step for debugging this would probably be ASan, although I haven't had to use it in anger on Rust code yet. WDYT @psumbera?

Unfortunately my knowledge of Rust and its internals are very limited. Plus I believe ASan is not available for SPARC.

[psumbera]

I will try latest nightly build as well. But I will need to build Rust 1.45 first (there are no bootstrap archives for Solaris).

I'm not able to build it now because of #74628 .

[tmiasko]

This kind of issues are often caught by LLVM assertions, so as a starting step
I would recommend enabling them if you haven't tried that already. Possibly
together with debuginfo=1 for more complete backtrace.