internal: Store token trees in the proc macro server in the representation of the tt crate

[ChayimFriedman2]

And not in the representation of the proc macro bridge. We still do not use the contiguous representation the rest of r-a uses, and prefer a tree with Rcs (this changed from Arc because we don't need thread safety for the bridge) for cheap cloning, but the types are the same.

The reason for that is that conversion to the bridge's types is a lossy conversion, in rustc as well: the internal token tree representation contains details that are lost in this conversion. This is a prerequisite to fix #23088 - the only way to fix it properly is to have an additional kind of token tree for doc comments, like rustc does, as evidenced by the fact that if a proc macro would create this macro (that has a doc comment in its matcher), the doc comment would not be ignored for matching, but if the macro's matcher would have been passed to the proc macro and it wouldn't have touch it (meaning, not even listing the TokenTrees then giving them back), the comment would still be ignored.

It is also required for supporting invisible groups properly for the same reason (there're actually several kinds of invisible delimiters, and the proc macro bridge lossily converts them all into one).

While we're at, I've also tried to untangle the mess of Cargo features of the proc macro server, and reduce the code it contains:

There are now two somewhat-orthogonal axes: the in-rust-tree feature, and the in-ra vs. in-proc-macro-srv feature. The former only decides whether we should import rustc crates from crates.io or from the sysroot. The latter decides what code to enable - the proc macro server doesn't need all the code in tt and proc-macro-api. I also moved the proc macro server's TokenStream from proc-macro-srv into proc-macro-api and inverted their dependency, as was required for this work.

You cannot enable in-proc-macro-srv and disable in-rust-tree (in practice this just disables all proc macro server code since such combination can arise when compiling r-a, also proc-macro-api supports this mode for tests), and you are not supposed to enable both in-ra and in-proc-macro-srv, except for tests. Enabling in-rust-tree and disabling in-proc-macro-srv is possible, though, and done when compiling r-a in-tree. In the future, this mode may use sysroot crates not just for the proc macro server, like was done in the past.

This depends on #23079 to not create merge conflicts for myself.

[Veykril]

This will severely degrade proc-macro performance. We moved over to the bridge intentionally as concat is a super common operation in proc-macro's which our representation does not lend itself well too #21097

[ChayimFriedman2]

No this will not. We do not use the contiguous representation, only the same leaf types.

[rustbot]

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.