Support `RUSTC_BOOTSTRAP`

[ojeda]

It would be nice to be able to use nightly features in stable releases. The particular mechanism or name for the escape hatch is not important (although I would say using the same one as rustc and the other tools would be nice).

This is a follow-up of #3900 since that did not get merged.

[calebcartwright]

Thank you for reaching out but I'm going to close this.

There's really two core parts here: whether we can enable the ability for folks to opt in to unstable config options on stable, and if so, then how.

As far as the "how", we definitely would not want to do that via RUSTC_BOOTSTRAP as that would lead to pervasive use throughout the ecosystem and inevitably some unintended, and nasty, consequences.

This is a capability we'd like to be able to provide and the most likely mechanism would be via a separate config option which folks could function as a buyer-beware of sorts if they're willing to accept potentially broken/invalid code, dropped comments, etc.

Unfortunately though the first part really is the bigger blocker as we received a ton of push back, including from folks on the core team, about letting permitting the usage of unstable opts on stable. I don't anticipate that changing any time soon as "unstable on stable" tends to be anathema in the Rust ecosystem.

[ojeda]

As far as the "how", we definitely would not want to do that via RUSTC_BOOTSTRAP as that would lead to pervasive use throughout the ecosystem and inevitably some unintended, and nasty, consequences.

I do not follow why RUSTC_BOOTSTRAP would be any different than any other mechanism.

if they're willing to accept potentially broken/invalid code, dropped comments, etc.

Please note that if a feature is buggy or untested (to the point of losing data), then it should be definitely labeled as "alpha" or "experimental" or similar, not just as unstable.

For me, as a user, "unstable" (in the context of Rust) means a feature that may change or go away entirely. It does not automatically mean it is dangerous to use. In fact, rustc warns about unstable features that are quite incomplete, buggy, potentially unsound etc. (e.g. specialization).

Unfortunately though the first part really is the bigger blocker as we received a ton of push back, including from folks on the core team, about letting permitting the usage of unstable opts on stable. I don't anticipate that changing any time soon as "unstable on stable" tends to be anathema in the Rust ecosystem.

Preventing unstable features from being used in stable compiler by mistake (or by newcomers etc.) is a very good thing.

However, not having any way of overriding it is a mistake, because some projects/companies require official releases or stable tags to be used. And that is not a bad thing: those projects get early access to some features and, for you as a maintainer, is another feedback channel.

On top of that, rustc, rustdoc, cargo, etc. already have the same mechanism (even if it was originally for different reasons), so it is very easy to argue for it here ;)

Now, your concern about buggy features is understandable (and commendable!), but those should be labeled as such (with a warning like rustc or gated under another mechanism/flag (e.g. -Zallow-alpha-features-that-may-cause-data-loss).

In any case, please note that, at least for rustfmt, it is usually not a big deal to have all the features or not, so it is OK to keep them hidden, but please note that you may leaving out some projects from being able to give back feedback on the unstable features.

[est31]

If you change the policy that developers should use the nightly rustc toolchain, and the stable toolchain should be used by downstream users, in CI, etc, there is not really a need for RUSTC_BOOTSTRAP.

You can unlock all the advantages you mention of early access and testing, and in fact being on nightly gives even more direct feedback to developers: you'll be filing bugs a few days after they get merged to master, not 40-80 days.

[Ericson2314]

I must say, I resent the frequent suggestion that projects wanting to use unstable feature should also be using nightly compilers. Wanting unstable features (interfaces) != wanting an unstable implementation.

Because the Rust compiler and the standard library use numerous unstable features, it is perfectly valid to say while released versions unstable features are no less subject to future change, they are at least better tested. It is perfectly reasonable for some to want an unstable feature, knowingly consent to that feature changing in the future, and still want the most battle-tested implementation of that feature.

Let me just remind all of you who @ojeda is too -- the point person on Rust for Linux. If getting Rust support in Linux requires unstable features, that is not ideal but perfectly acceptable --- after all it uses tons of C extensions, obscure assembly directives etc. etc., and e.g. the "clang-built Linux" folks already are pinning specific version as they try to match the long tail of underspecified GNU tool functionality. Rust unstable features are indeed analogous to this.

However, if Linux devs, to format their Rust code, are being pushed to grab arbitrary nightly Rust toolchains, we run the risk of kernels crashing not because these features are unstable, but because the compiler implementation is --- indeed, it is not unreasonable for nightly compiler to have an issue with stable-only code too; no testsuite is perfect.

IMO this would extremely unfortunate collaborative damage, which does nothing to help the case for Rust in Linux, and also does nothing to avoid yes, pernicious "de facto stability" , since the Linux devs are already with full consent opting into using into unstable features.

I thus see this prudishness around the RUSTC_BOOTSTRAP escape hatch as being completely counter-productive: not necessary for its stated goal, and also opening the door to other serious problems.

[est31]

Because the Rust compiler and the standard library use numerous unstable features, it is perfectly valid to say while released versions unstable features are no less subject to future change, they are at least better tested.

There are no ICE/etc backports going on for nightly features. Partially that's because usually if there is a breakage it also affects stable features, but partially because nightly features are little used anyways so issues specific to nightly features will not be discovered anyways. In general, if you want stable implementations, stay away from nightly features, nightly compiler or not.

If getting Rust support in Linux requires unstable features, that is not ideal but perfectly acceptable --- after all it uses tons of C extensions, obscure assembly directives etc. etc.

I fully agree about those obscure extensions that only a kernel would need... say some option that disables float formatting in libcore because the kernel doesn't touch the FPU. But these are not rustfmt options. If you scan through the tracking issue for rustfmt the only unstable things the kernel wants from rustfmt are options to format comments and impl items... none of that should be a blocker for a kernel, at least not with that argument.

IMO the best argument for RUSTC_BOOTSTRAP that I know of is that rustc is also using rustfmt with nightly options and might require RUSTC_BOOSTRAP for rustfmt. If you see the date of my comment and the one I linked, that's something I became aware of after I made the comment above. I haven't changed my opinion on RUSTC_BOOTSTRAP for rustfmt however, the existence of RUSTC_BOOTSTRAP is bad, even if it's used by the compiler.

However, if Linux devs, to format their Rust code, are being pushed to grab arbitrary nightly Rust toolchains, we run the risk of kernels crashing not because these features are unstable, but because the compiler implementation is

Linux devs are not distributing the kernel in binary form, but in source form. Distributors are not bound to the kernel's rustfmt policies so they can just not use it for their local patches. Even if they do, most distros still have the individual packager not upload binaries any more but instead just the patch files, and it's then built by CI.

I think it's been a common suggestion for a long time to use nightly clippy together with stable rustc, as bugfixes arrive more quickly (recently there has been some distancing from that though). So using nightly rustfmt together with stable rustc shouldn't be that uncommon. In general for kernel development, given that the newest rustc doesn't work with the mainline kernel, it might make sense to use rustup anyways.

[ojeda]

Thanks @Ericson2314 for reviving this issue!

If you change the policy that developers should use the nightly rustc toolchain, and the stable toolchain should be used by downstream users, in CI, etc, there is not really a need for RUSTC_BOOTSTRAP.

Asking every kernel developer to use nightly releases will not work. We did use nightly a long time ago and moved to stable releases for a reason.

You can unlock all the advantages you mention of early access and testing, and in fact being on nightly gives even more direct feedback to developers: you'll be filing bugs a few days after they get merged to master, not 40-80 days.

That seems orthogonal to me -- we could still test beta/nightlies or stable pre-releases to catch bugs sooner. In fact, I did that sometimes pre-merge. We also discussed with upstream Rust about eventually including the kernel in crater runs.

the existence of RUSTC_BOOTSTRAP is bad, even if it's used by the compiler.

How so? There are use cases for it.

In general for kernel development, given that the newest rustc doesn't work with the mainline kernel, it might make sense to use rustup anyways.

We don't want to force everyone to use rustup -- there is a reason we also mention the standalone installers in our docs. But, anyway, it is orthogonal: people could also install nightly toolchains manually, i.e. the issue is not rustup or not.

In any case, please note that the goal is to declare a minimum Rust version as soon as possible.

Also, I agree this issue is not critical or a blocker for the kernel. But it is painful for no good reason.

[Ericson2314]

I suppose the fear is always https://xkcd.com/1172/ type stuff with "but I didn't know RUSTC_BOOSTRAP was set", but is there evidence that this has actually occurred?

[deifactor]

So using nightly rustfmt together with stable rustc shouldn't be that uncommon.

I would be fine with this being the answer if there was an easy way to specify that you're doing that in your rust-toolchain.toml (or similar); as far as I can tell there isn't a nice way to say "use this component from nightly, all others from X stable version".

[RalfJung]

@calebcartwright is there any way this could be reconsidered? rustfmt is the "odd one out" among Rust tools by not supporting some way to opt-in to unstable features in stable builds. As a consequence of this, ./x in the rustc repo has to download two entire rust toolchains: a beta toolchain for the build, and a nightly toolchain for formatting. If you're trying to develop somewhere with poor connectivity, doubling the download size makes quite a difference... and even with fast internet, this still consumes unnecessary resources and causes confusion.