failed to build archive invalid data encountered

[Mark-Simulacrum]

ea45150 (#59953) appears to introduce some sort of bug in the archive generation, which causes script-servo to fail to build on perf: error: failed to build archive: Invalid data was encountered while parsing the file.

@eddyb, do you think it's worth reverting that PR for now?

[Mark-Simulacrum]

Hm, seems like this has since fixed itself. I'm going to close since it was probably spurious.

[eddyb]

Hmm, that error sounds like a toolchain bug/limitation (it really shouldn't care what we throw in there).
I wonder what the before/after #59953 .rmeta size is for script-servo.

[jdm]

I have encountered this locally building Servo's script crate, as has Servo's CI.

[eddyb]

@jdm @Mark-Simulacrum Could someone provide the .rmeta size (e.g. from cargo check) before and after #59953? (which I was wondering about in #65536 (comment))

Presumably for Servo itself it would have to be before and after a relevant rustup?

[jdm]

Before:

-rw-r--r--  1 jdm  staff   86138724 Nov  6 15:05 target/debug/deps/libscript-c57b17b8b8dc05ef.rmeta

After:

-rw-r--r--  1 jdm  staff  110432968 Nov  6 14:55 target/debug/deps/libscript-d913be2dd3c2bc7a.rmeta

[eddyb]

cc @michaelwoerister That's a 28.2% increase, much larger than what we saw for libcore.

@Mark-Simulacrum Would be interesting to have artifact sizes (for benchmarks themselves) on perf although the damage has been done here.

@jdm Okay that's bad but it doesn't increase the number of bits required to hold the number of bytes, maybe the whole rlib size is closer to that threshold?
That's my only guess so far as to why this is failing.

[jdm]

@eddyb The pre-change rlib was ~530mb and post-change was in the range of ~550mb iirc.

[eddyb]

@jdm Are you sure? I'm talking to @Eijebong and seeing 5.39GiB (older?) rlibs.
Note that there's a hard limit of 10GB for archives.

EDIT: looks like rust.metadata.bin's offset in the rlib, is very close to 4GiB (i.e. 2³² bytes), as it's sandwiched between *.rgcu.o and rcgu.bc.z.

EDIT2: that might have all been irrelevant, looks like the contents of the rmeta are the issue here, trying to create an archive with llvm-ar and just the rmeta fails.

[eddyb]

$ llvm-objdump -all-headers libscript.bad.rmeta | head
libscript.bad.rmeta:       file format COFF-<unknown arch>

architecture: unknown
start address: 0x00000000

So it turns out that our 4-byte 00 00 00 00 prefix for rmeta is valid a COFF header.
We have that so that very old rustc versions (pre-1.10, see #34539) don't succeed loading new rlibs, but maybe it's been long enough? cc @rust-lang/compiler
(we could also maybe rename the rust.metadata.bin file in rlib, which would presumably prevent older compilers from finding the file at all?)

It's also possible we'll save some time if LLVM stops trying to parse our rmeta format as COFF.