ICE: stage2 builds of rustc itself see metadata serialized by a different rustc

[arora-aman]

Error happened while working on updating closure capture analysis in rustc which is part of rustc_typeck.

Code

https://github.com/rust-lang/rust/compare/master...sexxi-goose:use_places_new?expand=1#diff-a7b1a49c68b1a1ce388558eee4f3514aR267

The highlighted line is what changed between the build that broke and the build before it.

Meta

Version reported in build log

rustc 1.48.0-dev running on x86_64-unknown-linux-gnu

Error output

error: internal compiler error: compiler/rustc_middle/src/ty/query/on_disk_cache.rs:422:23: could not decode cached query result: read_option: expected 0 for None or 1 for Some

thread 'rustc' panicked at 'assertion failed: `(left == right)

thread 'rustc' panicked at 'assertion failed: `(left == right)`
  left: `245`,
 right: `0`', 

and similarly one or two more

Complete build log: http://csclub.uwaterloo.ca/~a52arora/ice/log_75e93202-df10-4ea2-bff9-167ff9b968c2
Build log with RUST_BACKTRACE=1: http://csclub.uwaterloo.ca/~a52arora/ice/log_b7ef389e-c9a9-495e-92c1-5f55468705ec

Build directory: http://csclub.uwaterloo.ca/~a52arora/ice/build.tar (Heads up, this is 29G tar, 39G extracted)

[RalfJung]

What was the exact build command that caused the ICE?

If this is with x.py, then I also see this issue every now and then. Locally built rustc all have the "dev" version string, which means it doesn't have to rebuild everything just because you do git commit, but it also means it cannot detect that the ABI changed, so it might try to load old files with the new rustc and of course that will fail. I am not sure if much can be done about this.

[RalfJung]

Some potential duplicates: #74581, #74258, #69639, #60228.

[arora-aman]

What was the exact build command that caused the ICE?

If this is with x.py, then I also see this issue every now and then. Locally built rustc all have the "dev" version string, which means it doesn't have to rebuild everything just because you do git commit, but it also means it cannot detect that the ABI changed, so it might try to load old files with the new rustc and of course that will fail. I am not sure if much can be done about this.

@RalfJung I do x.py build -j50 -i --stage 1.
This happened to me again last night: http://csclub.uwaterloo.ca/~a52arora/rust-builds/log_30d4a7ae-dc48-41c6-a2c3-39043eee22bc

The assertion is fired in the not the same line but the same file, I haven't gotten around looking at the query code to see what failed.

Only thing I worked on since the last x.py clean (maybe 10-15ish builds in, though I could be wrong) was change closure/generator pretty printing (in rustc_middle/src/ty/print/pretty), haven't rebased since.

Though I can see that rustc builds are themselves are more prone incr compile errors, also didn't think git commit had anything to do with the build process, otherthan maybe some version strings.

[RalfJung]

Cc @Mark-Simulacrum

didn't think git commit had anything to do with the build process, otherthan maybe some version strings.

I mentioned git commit because one way to avoid this problem might be to embed the git commit in the version string used for rebuild detection -- but that would be rather annoying in many cases.

[Mark-Simulacrum]

You can set ignore-git to false in config.toml and that'll embed git commit hashes. But I think that must only patch over bugs - it can't be necessary. I doubt it will fully fix this bug.

I suspect the problem is that Cargo doesn't delete incremental state on dependency changes - which means that rustc must be prepared, currently, to see incremental state serialized by a different rustc... ideally, handling that in a way that doesn't just throw everything out, but even just not panicking would probably be enough.

[jyn514]

See also #75849.

[jyn514]

The proper way to fix this is to wipe the build cache whenever the stage1 compiler changes. That will make builds a lot slower, but it's better than ICEs and having to wipe the cache anyway.

I guess we could add a --keep-stage1-cache flag if you're sure you haven't touched metadata - not sure how useful it would be in practice.

@RalfJung said on Zulip:

I think that's a reasonable default.
FWIW, I rarely ever build the stage 1 compiler, and when I do it's to work with external miri and I usually build it once (where I want the cache to be wiped since it's horribly out of date), and then use --keep-stage 0 on all subsequent builds. so with that workflow, wiping the cache on stage1 compiler changes would never make anything slower.

[jyn514]

Wait, I'm confused again. Why does the metadata matter other than the incremental cache? Cargo already invalidates the cache when stage1/bin/rustc is modified, which is why libstd gets rebuilt when you modify the compiler.

Is there anyone not using incremental = true that's been hitting this issue?