build-dependencies and dependencies should not have features unified

[whitequark]

Consider this real-world example from crc-rs:

[dependencies]
build_const = { version = "0.2", default-features = false }

[build-dependencies]
build_const = "0.2"

The build dependency, of course, needs (and has) the std feature. The runtime dependency does not. Moreover, on my platform, there is no std. However, cargo treats them as the same package, and as a result it is impossible to actually build crc-rs at all.

[vitiral]

wow, I certainly would not have anticipated this (and appear to have not tested it...). Yay me!

[gnzlbg]

Not only dev-dependencies, features of platform-specific dependencies are also unified:

With this Cargo.toml:

[package]
name = "cargo_fubar"
version = "0.1.0"
authors = ["fubar <fubar@fubar.com>"]

[target.'cfg(target_os = "macos")'.dependencies]
mach = "0.1.*"

[dependencies.libc]
version = "0.2"
default-features = false

cargo build --no-default-features --verbose --target x86_64-unknown-linux-gnu outputs:

 Compiling libc v0.2.36
     Running `rustc --crate-name libc /foo/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.36/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=af3eb212fae2904c -C extra-filename=-af3eb212fae2904c --out-dir /foo/projects/sideprojects/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -L dependency=/foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/foo/cargo_fubar/target/debug/deps --cap-lints allow`

Note how I have a platform-specific dependency for MacOSX which should absolutely have nothing to do with linux builds, yet --cfg 'feature="default"' --cfg 'feature="use_std"' is enabled for libc even though [dependencies.libc] and the cargo build --no-default-features explicitly disable all features.

Removing the platform specific dependency and recompiling for linux produces:

Compiling libc v0.2.36
     Running `rustc --crate-name libc /foo/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.36/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=ebd4a22423d44421 -C extra-filename=-ebd4a22423d44421 --out-dir /foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -L dependency=/foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/foo/cargo_fubar/target/debug/deps --cap-lints allow`

Any workarounds?

[gnzlbg]

@alexcrichton I disagree about this being a feature request, I think this is a pretty severe bug in cargo and should be tagged as such.

[gnzlbg]

So I've added a test for this in PR #5007. If nobody beats me to it (and I hope somebody will) I'll either submit a new PR with a fix or add it to that PR (if it doesn't get merged).

[gnzlbg]

So fixing this is very hard, it looks like cargo was designed under the assumption that the features of all dependencies, dev-dependencies, target.dependencies,.... ought to be unified to be able to generate a single dependency graph for build, test, --target, etc.

What I've ended up doing as a workaround is having multiple Cargo.toml files for a single project... I have a Cargo_test.toml, Cargo_build_target_xxx.toml, etc. This sucks, but it work. However, I can't release to crates.io under this setup, because I don't have a single Cargo.toml for all my targets, and adding multiple targets messes the features...

[gnzlbg]

Someone should raise this with the tools team. I don't know how anybody manages to use cargo for targeting any #[no_std] environment.

If your crate happens to use std in any way for, for example, testing or benchmarking, chances are its always being built with std linked in, and forcing other crates to be linked with std as well :/

[vitiral]

@gnzlbg that's an interesting workaround, however it wouldn't fix the issue if your build.rs needed it.

For testing/benchmarking only could you have Cargo.toml be your "released application" and CargoTest.toml be your "test" one?

[SimonSapin]

I think this is related to #4463.

[Ericson2314]

Yeah this is crucial blocker for cross compilation. The solution is a huge refactor where each build.rs generates a new solution space (or at the very least one for each build.rs, one for each build.rs needed by a level-1 build.rs, and so on).

If it helps Haskell's Cabal has some great prior art here.

1. The library, binaries, and build.rs of a package are viewed as separate "components", totally separate nodes in the dependency graph with their own distinct dependencies, but not totally separate nodes in the solution space since they are required to come from the same package.
2. qualified goals are unresolved deps along with just enough provenance information to indicate which simultaneous solution we are solving for. For example, if every build.rs gets independent versions, we could use Vec<Package> for the chain of build.rs, or if we separate build.rs into stages then we just need a counter (corresponding to the length of the chain).

Lastly, this all becomes far more important with public dependencies, when it's required that similar public dependency edges point to the same node. Separately solution spaces are the one way to break those constraints.

Also, as #4866 (comment) sadly demonstrates, the original idea that is a single solution in a Cargo.lock can work for all platforms is probably unworkable. Whereas we can in always solve build.rs separately for consistency, platform-specific crate- and feature- dependencies can "wildly" change the solution space. CC @wycats.

[Ironically, I do think we can eventually simultaneously and cheaply type-check each crate against all possible dependencies (!!!), so perhaps this isn't so bad. Even if the lockfile contains only a few canonical plans for different platforms, we can ensure that all others will at least typecheck!]

[gnzlbg]

@Ericson2314 To really solve this I think that cargo would need to support a dependency graph per profile.

Huge refactor is probably an understatement. We would not only need a backwards compatible mode to support old Cargo.lock files, but we will also probably want to unify the dependency graphs or sub trees of them when possible to avoid downloading and compiling a potentially different version of a dependency per profile.

Also, this will also mean that cargo bench might use a very different dependency graph from cargo build --release, and probably many other things that will need to be worked out.

Also, the solution might be RFC worthy :/

[Ericson2314]

Uhhhh I'm not really sure what profiles are (though I think that goes for most of us), but I wouldn't really expect them to need their own graphs in general? Debug vs release should explicitly be the same thing. system test are really just binaries with a special purpose. The best example would be unit tests since they are turning the primary component (whatever it may be) into a binary.

In otherwords, I think components + optimizations is a much better model than this hopelessly overloaded concept of "profiles".