Significant performance regression on the `encoding` benchmark

[ecstatic-morse]

#69144 (a rollup) slowed down check builds of the encoding benchmark considerably, see the task-clock measurements. This degradation has been consistent over the last half-dozen perf runs, so it is not spurious. The following candidates have non-trivial changes to the parts of the compiler that are run during a check build.

• parse: merge fn syntax + cleanup item parsing #68728 (parse: merge fn syntax + cleanup item parsing)
• fix lifetime shadowing check in GATs #68938 (fix lifetime shadowing check in GATs) unlikely
• expand: misc cleanups and simplifications #69057 (expand: misc cleanups and simplifications)
• Use HirId in TraitCandidate. #69108 (Use HirId in TraitCandidate.) ruled out by Revert #69108 #69199
• miri: fix exact_div #69126 (miri: fix exact_div) unlikely

I doubt it is #68938 or #69126 so cc @Centril @cjgillot

[Centril]

#69057 is mostly a pure refactoring so it seems unlikely?

[Centril]

Besides changes in https://github.com/rust-lang/rust/pull/68728/files#diff-f5c049612f32b4eda9adb5019ab10a83, nothing pops out in the parsing PR either.

[ecstatic-morse]

Ruled out #69108 via perf results from #69199, so it's likely the big one (#68728) cc @Centril @petrochenkov.

[ecstatic-morse]

Some perf builds around the regression were stuck in the queue when I posted this, so it may not be #69144. The regression occurs in early_lint_checks.

[pnkfelix]

nominating for discussion at T-compiler meeting for prioritization.

[pnkfelix]

discussed at T-compiler meeting. P-high. Removing nomination.

[ecstatic-morse]

A significant portion of the runtime for check builds of encoding is spent doing system calls (around 20% on the latest nightly). I believe this is because error messages are written directly to io::Stderr which does not buffer, which is supported by the perf results in #69227. Although there's a lot of variance, older nightlies appear to spend less time doing system calls than current ones, which could suggest that changes to diagnostics or diagnostics reporting could be the root cause of this issue.

Roughly the same number of errors are reported in old (end of 2019) nightlies as recent ones (~1900 lines output to stderr). However, earlier nightlies use many more calls to write(2, ...) to accomplish this (3500 vs 720). Given this, I would assume that the earlier nightlies would be slower than the current ones on the encoding benchmark, not faster. Perhaps someone else is more knowledgeable.

[ecstatic-morse]

After #69227, the task-clock measurements for encoding-check are back to normal. I think we should close this, although I wish I had a better idea of when/why the regression occurred.