rustc-dev component recently became a lot bigger

[RalfJung]

I need to download the rustc-dev component quite a lot for Miri development, and noticed that it recently became a lot bigger.

For b1ab3b7, it was 125MB in size.
Now, at 85d089b, it is 205MB in size.

Is that an expected regression? Is there a way to avoid this almost doubling of the size?

[Mark-Simulacrum]

cc @Kobzol @bjorn3, suspected to be due to #101403

[bjorn3]

It looks like we include both llvm bitcode and object code rather than including only llvm bitcode in the rlibs for some reason.

[ehuss]

To avoid the object code, it needs to use -Clinker-plugin-lto. However, what needs object code vs bitcode can be quite complex, so it may not be as simple as just overriding more flags.

[bjorn3]

To avoid the object code, it needs to use -Clinker-plugin-lto.

I though passing -Clto would be enough to disable object code generation, but it seems not.

In any case one way to solve this issue I think would be to ship .rmeta files instead of .rlib files in rustc-dev. All users of rustc-dev should be linking against librustc_driver.so anyway to avoid duplicate symbols when being loaded by rustc or when loading a codegen backend linked against librustc_driver.so. Shipping .rmeta files should even reduce the size of the rustc-dev component to below what it was before #101403 landed.

[RalfJung]

@bjorn3 how would tools like Miri even do that (link against the .so file based on .rmeta files)? And do these .rmeta files even exist on our dist builders or would they have to be generated?

[bjorn3]

Miri links agains librustc_driver.so already. In fact there are no .rlib or .rmeta files generated for rustc_driver as it's crate type is set to dylib. The .rmeta files are already generated as they are used for build pipelining (since a couple of months or so it is no longer possible to disable build pipelining in cargo). All that is necessary is to replace the rule that copies .rlib files to rustc-dev with one copying .rmeta files. Possibly with an assertion that there exists a .rmeta file for every .rlib file that would have been copied previously. Rustc should then automatically pick up the .rmeta files from the sysroot instead of .rlib files and should only error if librustc_driver.so isn't linked in. The error is probably not very helpful though, so maybe we should add a special case for sysroot crates suggesting rustc_driver to be added as dependency.

[bjorn3]

Opened #105609 to fix this.