network zlib stream error confusion

[ehuss]

Problem
On rust-lang/rust CI, I've been noticing this error happening a lot lately:

    Updating crates.io index
error: failed to get `cc` as a dependency of package `bootstrap v0.0.0 (D:\a\1\s\src\bootstrap)`

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  error inflating zlib stream; class=Zlib (5)

I think the CI has been having network problems in general, but this error in particular seems a little confusing to me. I don't know what would cause this error. I think it might be helpful to understand this error case a little better.

Here is the corresponding error in libgit2.

Should this error class be added to the spurious error list? What can possibly cause this to happen?

Notes

cargo 1.46.0-beta (4f74d9b2a 2020-07-08)

[alexcrichton]

This looks reminiscent of #8258 which was fixed in libgit2/libgit2#5536. Rust 1.45 did not include that PR, but 1.46 (beta) and nightly both have the fix. Do you know if these errors are showing up using Cargo from 1.45? (bootstrap not updated?)

[ehuss]

I'm pretty sure it is the version posted above cargo 1.46.0-beta (4f74d9b2a 2020-07-08)

extracting D:\a\1\s\build\cache\2020-07-16\cargo-beta-i686-pc-windows-msvc.tar.xz

https://dev.azure.com/rust-lang/rust/_build/results?buildId=35201&view=logs&j=339a807e-05eb-50a3-f69a-0d7a3a5b2cef&t=1f87e16f-7721-57ff-dca8-75b52e5f4eac

[alexcrichton]

Bah.

It probably wouldn't hurt to throw this onto the list of spurious errors, but I agree it'd be good to understand. AFAIK zlib inflate errors indicate a corrupt data (e.g. not zlib) getting fed to the zlib inflater. That previous issue was due to a bug in libgit2 (valid zlib stream, invalidly fed), and this may also be that. This'll be really hard to track down though without a reliable reproduction.

Skimming libgit2's commit history I don't see much related to this since that original fix, so I don't think we can get lucky by "simply update the git repo".

[ehuss]

Another instance: https://dev.azure.com/rust-lang/rust/_build/results?buildId=35201&view=logs&j=339a807e-05eb-50a3-f69a-0d7a3a5b2cef&t=1f87e16f-7721-57ff-dca8-75b52e5f4eac

I wonder if there is any significance that these cases happen on Windows i686 (i686-mingw-1 and i686-msvc-1).

[alexcrichton]

Hm the fact that it was retried there and it didn't help anything makes me think that it's not a spurious failure...

[ehuss]

Hm, looking at other instances:
rust-lang/rust#73210 (comment)
https://dev.azure.com/rust-lang/e71b0ddf-dd27-435a-873c-e30f86eea377/_build/results?buildId=35451
Failed in three builders: i686-gnu-opt i686-msvc-1 dist-i686-msvc

rust-lang/rust#74245 (comment)
i686-gnu

rust-lang/rust#74245 (comment)
i686-gnu

rust-lang/rust#74245 (comment)
i686-gnu-opt
i686-msvc-1
dist-i686-mingw

rust-lang/rust#74611 (comment)
i686-mingw-2

rust-lang/rust#74495 (comment)
i686-mingw-2

rust-lang/rust#74520 (comment)
i686-msvc-1

rust-lang/rust#74219 (comment)
i686-mingw-1

Raising priority since this seems important.

[mati865]

It cannot be spurious, it affects probably all i686 platforms.

[ehuss]

Notes from my investigation:

This seems to have gone away for now, though I would anticipate it happening again since it has been coming and going. My most likely hypothesis is that there is some caching layer between azure's network and github's git servers that is misbehaving.

AFAIK, all of the rust-lang/rust jobs start with an empty registry directory (based on my testing with GHA and Azure, they both start with an empty directory, and rust-lang/rust's CI jobs are not configured to cache it). So I don't think it is related to any partial caching, or incremental git updates.

In rust-lang/rust#74689 I attempted to use net.git-fetch-with-cli, but it also failed:

  process didn't exit successfully: `git fetch --force --update-head-ok 'https://github.com/rust-lang/crates.io-index' 'HEAD:refs/remotes/origin/HEAD'` (exit code: 128)
  --- stderr
  error: gnutls_handshake() failed: A TLS fatal alert has been received. while accessing https://github.com/rust-lang/crates.io-index/info/refs

  fatal: HTTP request failed

Interestingly, that failed on several different hosts (dist-i686, dist-x86_64-linux, dist-x86_64-linux-alt).

I'm still really curious why previous failures all seemed to be on the i686 jobs.

During the outage, I was able to easily reproduce on github actions. It didn't really matter which version of Cargo I was using:
1.47: https://github.com/ehuss/test/runs/903771093?check_suite_focus=true
1.46: https://github.com/ehuss/test/runs/903709326?check_suite_focus=true
1.45: https://github.com/ehuss/test/runs/903775146?check_suite_focus=true
1.43: https://github.com/ehuss/test/runs/903798953?check_suite_focus=true

Unfortunately the errors stopped before I could perform more testing.

So I don't think it is related to any recent libgit2 changes (and since it also failed with net.git-fetch-with-cli, I doubt it is related to libgit2 at all) or #8363.

Will revisit this if it ever starts acting up again and I'm around to catch it. Unfortunately there isn't really a good way to be alerted when it does. 😦