Make the dependency on zerocopy optional

[hpenne]

Background

Projects that need to vet their dependencies will suffer greatly now that you have introduced a dependency on zerocopy, which is a big crate with a huge amount of unsafe code. Since just about any major project uses rand, this now makes "every" major Rust project depend on zerocopy as well.
This is really problematic. Please make this optional and off by default.

I understand that some users want the absolute best performance and that you want to support this, but most applications do not need this and introducing a dependency like zerocopy will have a significant negative impact on many users, one that it is currently almost impossible to get out of.

Feature request

Please make the performance optimizations that require zerocopy optional and off by default.

[hpenne]

Looking quickly at the commit that introduced this, it may seem like zerocopy (which contains lots of unsafe code) was introduced to remove a very small amount of unsafe code from rand. Was that the only motivation? If so, then this change is a huge negative for many users, as the amount of unsafe code in total in our projects has now suddenly increased by a very significant number. If this was the only motivation then please consider reverting this change.

[benjamin-lieser]

Using zerocopy should make things safer and not more unsafe. It replaces potentially unsound unsafe code with safe semantics which are carefully checked and relay more on the compiler than on humans to verify.

I think it is a bit misguided to only focus on the amount of unsafe code. zerocopy reduces the possibility to write unsound code and that is a good thing in my opinion.

You could argue that we can leave some performance and remove unsafe code and zerocopy, but this is a different story.

[dhardy]

I saw there was a discussion on reddit about this, including a number of comments supporting this move.

I'm not fully aware of how this affects different groups, but yes, zerocopy was introduced to reduce the amount of unsafe code in rand (though obviously it hasn't replaced all usages of unsafe). So if reverting back to use more unsafe in rand is better for more users I am receptive to making that change — it can be hard to judge reception of such a change.

See also this comment by @joshlf in #1349 and also #1502.

[jswrenn]

I co-maintain zerocopy and lead Rust's Project Safe Transmute — happy to answer questions about either project.

I apologize for any inconvenience this has caused you, but I do believe it's a one-time cost. Zerocopy is a dependency of many major Rust crates, and thus already is a transitive dependency of many non-trivial Rust projects. If zerocopy's use results in more unsafe being concentrated into a single dependency in your project's tree, that means you only need to vet zerocopy — rather than n dependencies all doing their own unsafe transmutes.

We strive to make this vetting process as easy as possible. Zerocopy has extensive internal documentation, and most of our SAFETY comments directly quote from Rust's stdlib or reference documentation. If there's anything we can do to make the vetting process easier, we'd love to hear it.

Stepping back, the use of crates like bytemuck and zerocopy in the Rust ecosystem is incredibly important to Rust's Project Safe Transmute. Functionality requests get funneled to these crates and that, in turn, helps inform the design of the Rust's future support for safer transmutation.

[hpenne]

Let me assure you that I have no doubt that zerocopy is well maintained and of very high quality.

Our situation is that we need to vet our dependencies. This is not a one time cost. Every time we upgrade to a new version of a crate, it needs to be vetted again (a delta review at least). This includes (but is not limited to) doing a code audit/review of all new or modified unsafe code. The fact that zerocopy is well documented and well maintained does not actually help that much in this regard.
This makes the new dependency on zerocopy painful and costly.

The proliferation of dependencies is a challenge for us. rand was a "good crate" in this regard as it has a limited and well defined purpose, and used no dependencies to achieve this. This ended with version 0.9, when zerocopy was introduced. We will now have to do a code audit of a new version zerocopy whenever we upgrade our dependencies. This will have a significant cost for us. It is probably much cheaper for us long term to replace or fork rand somehow to get rid of the dependency on zerocopy.

I suppose the majority of users of rand will be fine with this change. Most/many care little about dependencies at all. However, for the users that are trying to keep the number of dependencies low, or that have to vet their dependencies, the introduction of this dependency on zerocopy will cause pain and higher costs. If a fundamental and extensively used crate like rand does not care about the collateral damage from this, then who will?

If this was done only to remove unsafe code, by delegating this to zerocopy, the please consider fixing/reverting this. Alternatively, if this is only necessary for performance optimization, then please consider putting this performance optimization behind an optional feature so that users that do not need it can opt out.

[jswrenn]

Every time we upgrade to a new version of a crate, it needs to be vetted again (a delta review at least). This includes (but is not limited to) doing a code audit/review of all new or modified unsafe code. The fact that zerocopy is well documented and well maintained does not actually help that much in this regard.

That does sound like it could be painful. However, rand presently uses three APIs from zerocopy: transmute!, as_bytes and as_mut_bytes. The implementation of these APIs is self-contained and does not involve vast swaths of unsafe code. I can't make promises, but we rarely (if ever) update these implementations. We haven't touched transmute! (other than to move it to a different file) in four years.

[mitsuhiko]

I saw there was a discussion on reddit about this, including a number of comments supporting this move.

I understand that reddit is very much in support of this, and seemingly so is a large part of the Rust community. That is disappointing. Unfortunately it means that for plenty of commercial contexts it's very likely that this will create a growing rift in the ecosystem.

It's becoming harder and harder to audit Rust projects because of changes like this. I understand the goal of zerocopy and I can see that it's a great crate. It is however a massive dependency.

In practical terms this will cause a lot of people to move to hand written RNGs or fastrand which I don't think is a good move for this ecosystem.

[benjamin-lieser]

I would say in summary the change to zerocopy was good for most users, but for some it is quite a pain.

I think it is feasible to have zerocopy behind a feature-flag and have safe but possible slow code without it (or fast and unsafe code as alternative). There are only a handful of places which would need to change and be maintained (I found 4, in rand and rand_core). Not sure if the added complexity is worth it, but it is also not too bad if enough people feel strongly about it.

There is also hope this might become unnecessary with better compiler support.

If this is deemed worth it, I can do a PR for it.

[Bluefinger]

In practical terms this will cause a lot of people to move to hand written RNGs or fastrand which I don't think is a good move for this ecosystem.

I think if minimal/no dependencies and zero unsafe is your main focus, then you are probably hand-writing your own RNGs or using fastrand already. But at some point, efforts to reduce unsafe will converge on utilising well-tested/battle-hardened/verified work, and since that effort is particularly hard, folks will want to not have to repeat/retread the same ground over and over. So, they will use what's been provided already by folks who have fully committed into providing said utilities.

And if you are in the position where this is untenable, forking/vendoring is the answer. But I think this is a wrong move to undo efforts to make rand safer just because it invokes a new dependency whose entire purpose is to make things safer. Feature-gating also adds additional maintenance burden which should not be forgotten either. However, I am not a maintainer, so in the end, it will be their call.

[briansmith]

There are only two places in rand that use zerocopy. I believe at least one of them can be easily rewritten to avoid unsafe and avoid zerocopy. I think the other use can probably similarly be rewritten.

The dependency on zerocopy was added during a flurry of PRs that proposed adding zerocopy as a dependency to many crates, including one of mine. In my case, I was able to refactor my code to avoid unsafe and also avoid adding a dependency. I also remember reviewing another project's PR and found that the added zerocopy dependency was also superfluous.

Accordingly, I think spending effort on PRs might be a more useful use of time than the meta-discussion about dependency policies.

[mitsuhiko]

I don't want to drag this out excessively, but I do want to respond to this:

I think if minimal/no dependencies and zero unsafe is your main focus, then you are probably hand-writing your own RNGs or using fastrand already.

In a world where I control all my dependencies, maybe. But that's just unrealistic. For instance rand ends up in a ton of projects today because of the UUID's v4 feature. In some cases you don't even have the option to avoid it, because something else turns it on. At least today you can end up using rand via some other very large well established crates.

[hpenne]

Every time we upgrade to a new version of a crate, it needs to be vetted again (a delta review at least). This includes (but is not limited to) doing a code audit/review of all new or modified unsafe code. The fact that zerocopy is well documented and well maintained does not actually help that much in this regard.

That does sound like it could be painful. However, rand presently uses three APIs from zerocopy: transmute!, as_bytes and as_mut_bytes. The implementation of these APIs is self-contained and does not involve vast swaths of unsafe code. I can't make promises, but we rarely (if ever) update these implementations. We haven't touched transmute! (other than to move it to a different file) in four years.

I'm sure that is all correct, but that does not help an audit process. You see as far as auditing is concerned, this is just hearsay and would have to be verified every time the a new version is audited. It is impractical to do an analysis of the call graph of rand when auditing zerocopy for a project. It is also possible that other crates have started to depend on it since your last audit. In the end, if you depend on it you do not have much choice other than doing a full audit.

[hpenne]

I think an "almost std" crate like rand should have absolutely minimal (preferably zero) dependencies. This should be a top priority, to avoid forcing users of rand to depend on crates that they would otherwise need. Rand is the de facto standard for random. That success comes with responsibilities, and you should not take those lightly. That should mean avoiding taking any dependencies unless they are absolutely necessary, and zerocopy does not seem necessary.

[Bluefinger]

One final response and then I'll leave things here:

In a world where I control all my dependencies, maybe. But that's just unrealistic. For instance rand ends up in a ton of projects today because of the UUID's v4 feature. In some cases you don't even have the option to avoid it, because something else turns it on. At least today you can end up using rand via some other very large well established crates.

And zerocopy tends to appear in a lot of crate dependencies as well, either that or bytemuck. If you are in large enough project territory where rand is likely to appear in your dependency tree, then so is zerocopy and bytemuck. The ecosystem has mostly converged on the two for zero-copy shenanigans and safe transmutes. It's not just any random crate (heh) we are talking about here.

[mitsuhiko]

And zerocopy tends to appear in a lot of crate dependencies as well, either that or bytemuck. If you are in large enough project territory where rand is likely to appear in your dependency tree, then so is zerocopy and bytemuck. The ecosystem has mostly converged on the two for zero-copy shenanigans and safe transmutes. It's not just any random crate (heh) we are talking about here.

You are not wrong, but that is a very recent change. It's very disappointing and frustrating to see that crate show up in critical parts of the ecosystem at the moment for what was previously rather lightweight crates.

I'm not sure where the push towards zerocopy is coming from but it's disappointing. What is however particularly disappointing in the case of rand is that it also uses the derive feature which is much heavier. The other common dependency I have is hashbrown that pulls it in, and it at least does not use derive (See also discussion in tkaitchuck/aHash#180)

Up until the change in rand I did not have any zerocopy-derive in my dependency tree and with it I had plenty of projects without proc-macro2 and syn.