Repository navigation
feat(proto): generate packet structs from protocol.json at build time - #971
Merged
Merged
Conversation
The wire format now has one source of truth. `protocol.json` is committed beside the crate and `build.rs` turns the layouts the extractor recovered in full into Rust; nothing about a packet body is transcribed by hand any more, and there is no sync step for the structs to drift from. 176 of the 179 fully-recovered packet classes are generated. The three the generator declines are named, with the reason, in a comment at the top of the file they would have been written into: a custom codec inside a combinator, two fields that would take the same name, and a type that contains itself. None of them is approximated. Adds `hyperion-minecraft-proto-derive`, which derives the existing `Encode`/`Decode` traits from a struct's own fields. `#[proto(max_len)]`, `#[proto(max_count)]` and `#[proto(with = ...)]` carry the limits and the one layout the structural rules do not describe. The tables (packet ids, registries, version) stay committed Rust: they restate data rather than a wire contract, they are what a reader greps, and a stale one is a wrong name in a log line rather than a desynchronised stream. Also fixes a formatting divergence that predates this change: the pipeline's rustfmt never saw the repo's rustfmt.toml, so `cargo fmt` rewrote every committed table and `fmt --check` failed on an untouched tree.
`writeEnum` sends an ordinal and `ClientIntent` sends ids of its own, so a `VarInt` field was as far as the extractor went. Both are readable off the jar: the constants in declaration order, and where the class has one, the `byId` switch that says which number each constant is sent as. Eleven enums come out, `ClientIntent` among them, so `intention` is a `ClientIntent` rather than a number that happens to be 1, 2 or 3. A value the server would reject is now unrepresentable, and an unknown discriminant decodes to `Error::InvalidEnum` instead of to a wrong variant. Two packets that were partial are now complete, because the enum was the only thing under them the extractor could not follow. The decompile scope gains one clause: an enum a packet names. `RecipeBookType` and `RemoteDebugSampleType` carry no `StreamCodec`, so nothing else selected them, and an enum whose constants are missing cannot be resolved at all.
#961 ran a `use_self` fix over the committed `generated/wire.rs` without telling the generator, so `nix flake check`'s staleness check has been failing since: the committed copy says `&'static Self` and the pipeline still produced `&'static Wire`. Emitting `Self` inside the enum is what clippy would leave, so the two agree again. This is the check doing its job -- a hand-edit to a generated file is exactly what it exists to catch.
andrewgazelka
force-pushed
the
feat/proto-codegen
branch
from
July 27, 2026 11:48
4cadfc0 to
3386dca
Compare
Benchmark Results for generalComparing to 3a2d475 |
andrewgazelka
added a commit
that referenced
this pull request
Jul 27, 2026
#971 landed the build-time generator while this was in flight. Login, handshake and status are now generated from protocol.json, so the hand-written ClientboundLoginFinishedPacket added here is gone in favour of the generated one. Configuration follows the generated types where they exist -- select_known_packs, registry_data, update_enabled_features and finish_configuration -- rather than the hand-written duplicates, which shortens the reconciliation that crate's packets/mod.rs describes instead of lengthening it. client_information, custom_payload and update_tags stay hand-written; the generator cannot express them.
andrewgazelka
added a commit
that referenced
this pull request
Jul 27, 2026
…63 pre-play path (#972) Serves Minecraft 26.2 (protocol 776) through `crates/hyperion-minecraft-proto`. A scripted client reaches play state; a vanilla client does not yet stay there, and the reason is spelled out below. ## What changed The four states before play -- handshake, status, login, and the configuration state that did not exist in 1.20.1 -- are now `crates/hyperion/src/net/protocol/`. The valence 763 path was **deleted rather than feature-gated**: a branch that only one of two protocols ever takes is a branch nobody exercises. The transport did not change with the protocol. hyperion frames packets itself and the proxy forwards bytes without reading them, so `net::encoder`, `net::decoder` and `packet_channel` are untouched; one adapter turns a proto-crate `Encode` plus a packet id into a `PacketBundle` and the existing compression and length framing carry it. **Registry contents did not have to be written.** `RegistrySynchronization.packRegistry` writes an element with an empty payload when the client reported the pack that defines it. Offering `minecraft:core:26.2` and getting it back means all 397 elements of all 29 synchronised registries go out as names alone, so `registries.rs` is a name table. The order is `RegistryDataLoader.SYNCHRONIZED_REGISTRIES` and the names are what the vanilla data generator writes; both read off the jar. A client that does not report the core pack is refused with a message saying exactly that. `ClientboundLoginFinishedPacket` was the one packet the proto crate was missing. It was added here by hand and then dropped again when #971 landed mid-flight and generated it; this branch is rebased onto that and uses the generated login, handshake and status structs. Configuration follows the generated types where they exist rather than the hand-written duplicates, which shortens the reconciliation that crate's `packets/mod.rs` describes rather than lengthening it. `client_information`, `custom_payload` and `update_tags` stay hand-written, because the generator cannot express them. ## Verified `nix run .#lint` and `nix run .#test` pass (315 tests). Full transcripts are in `docs/minecraft-26.2-migration.md`. The join, against a real running stack on dedicated ports: ``` [Prober] <- LoginCompression threshold=256 [Prober] <- LoginFinished profile=c07403a5245bf4e124ad10d435771f55 name=Prober (authenticated, NOT yet in the world) [Prober] -> LoginAcknowledged [Prober] <- SelectKnownPacks [('minecraft', 'core', '26.2')] [Prober] -> SelectKnownPacks (accepting all) [Prober] <- RegistryData x29 (397 elements): ... [Prober] <- FinishConfiguration [Prober] -> FinishConfiguration (ack) [Prober] <- Login entity_id=67109860 ** REACHED PLAY STATE ** [Prober] RESULT: reached play state (Login received) ``` `tools/client-26.2.py` is the 776 counterpart to `smash-client.py`. It exists because a player count is not evidence of a join, and on 776 there are two more gates before the world than there were on 763. ## Two bugs this was watched for - **Authenticated but never in the world.** The join now runs off the client's own `finish_configuration` rather than off a skin channel, so a skin that never arrives can no longer keep anyone out. `tests/join_world.rs` still guards the channel that applies skins. - **Skin layers.** 1.20.2 moved `ClientInformation` into configuration, so it is handled there and `DisplayedSkinParts` is set before the player enters the world. The test client caught a third. `hyperion-permission` unicast the command tree the moment a permission group was set, during login. On 763 that landed after `LoginSuccess`, where the client is already reading play packets, so it worked by accident; on 776 the same bytes arrive in the configuration state and are read as whatever configuration packet shares that id. Now gated on the player being in play. ## What a reviewer should not assume works **A vanilla client disconnects immediately after the join packet.** Clientbound play is still valence at 1.20.1 ids -- the same transcript continues `0x1E x4096, 0x24 x2434, 0x4E x1, 0x65 x154`, which are `UnloadChunkS2c`, `ChunkDataS2c`, `ChunkRenderDistanceCenterS2c` and `PlayerListHeaderS2c`. The scripted client counts them and carries on, which is what lets it prove the join; a real client will not. Serverbound play is likewise still decoded on valence ids. So the next piece is the whole of clientbound play, not just terrain. `registries.rs` should also be deleted in favour of generated contents once the proto crate produces them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The wire format now has one source of truth.
protocol.jsonis committedbeside the crate and
build.rsturns the layouts the extractor recovered infull into Rust; nothing about a packet body is transcribed by hand any more,
and there is no sync step for the structs to drift from.
What the generated code looks like
177 of the 180 packet classes the extractor recovered in full are generated.
The three it declines are named, with the reason, in a comment at the top of
the file they would have gone into — a layout that is not fully expressible is
absent, never approximated:
One Java class is one Rust type:
ClientboundKeepAlivePacketservesconfiguration and play, so it is defined once in
packets::commonandre-exported into each state, and a value built for one is the value the other
accepts.
Committed versus generated
protocol.jsonOUT_DIR, viabuild.rsWiretable.rsThe line is what a mistake costs, and whether the projection is total. The
tables are a total function of
protocol.jsonand a committed copy can bechecked by diffing it. The packet structs are a partial projection — the
generator refuses what it cannot express — which is exactly the shape that has
bitten this pipeline before, when an earlier extractor silently truncated
eleven packets and reported them complete. Volume settles the rest: moving
registry.rs's 7,700 lines intoOUT_DIRwould makerg 'minecraft:diamond_sword'miss, for no gain.
The derive
crates/hyperion-minecraft-proto-derivederives the crate's existingEncode/Decodetraits. A packet body is a fixed sequence of fields withnothing between them, so the derive is a loop over the fields in declaration
order; anything that is not that is a compile error rather than a guess.
#[proto(max_len = N)]#[proto(max_count = N)]#[proto(with = path)]path::encodeandpath::decodefor this fieldNamed, tuple and unit structs; borrowed fields; type parameters; fieldless
enums as a
VarIntdiscriminant. Refused, loudly: a variant with fields, morethan one lifetime, a
withbeside a limit.The hand-written handshake, status and login codecs are gone — every one of
their packets is generated.
Enums are enums
writeEnumsendsordinal()andClientIntentsends ids of its own, so anenum field used to come out as a
VarInt. Both are readable off the jar, soeleven enums are now generated, and two packets that were partial became
complete because the enum was the only thing under them the extractor could
not follow.
Ids 1, 2, 3 against ordinals 0, 1, 2 — reading one for the other picks the
wrong intent silently, which is why the
byIdswitch is parsed rather than theposition assumed.
Also fixed
The pipeline's rustfmt never saw the repo's
rustfmt.toml, socargo fmtrewrote every committed table and
fmt --checkfailed on an untouched tree.It now runs the pinned nightly with
--config-path.Gates
The new
protocol.jsonguard was watched failing before it was trusted:editing the committed protocol number to 999 fails the check with a diff
naming the line.
Not done
Written up in
docs/minecraft-26.2-migration.md, and worth reading beforebuilding on this:
and the two disagree.
ClientIntentionPacketwriteshostNameunboundedand reads it with 255. The generated struct is permissive where vanilla is
strict. It cannot desynchronise a stream, but the fix — parsing the private
(FriendlyByteBuf)constructor as a second, independent statement of thelayout and cross-checking it — would also settle
port, which is writteni16and readu16.damage_eventis declined over a defect, not a limit. The extractorinlines a static helper without binding its parameter to the caller's
argument, so two fields are labelled
output.Rebased onto #968, and what that leaves
#968 landed while this was in flight. It hand-wrote configuration and the
play-login sequence against the same decompiled sources, and eleven of its
packets are now also generated —
select_known_packs,registry_data,update_enabled_features,keep_alive,ping,pong,disconnect,finish_configuration,reset_chat,code_of_conduct,accept_code_of_conduct, plusLoginandRespawninplay_login.They are defined twice for now, once flat in the hand-written module and once
under the state's
clientbound/serverboundmodule. The generated one is theone to keep — it also closes #968's own stated gap, since the 64-element cap on
ServerboundSelectKnownPacksis enforced here through theError::ListTooLongthis branch adds. Reconciling means rewriting 725 lines of tests that name the
hand-written types, which is a review of its own, so it is the next change
rather than this one;
src/packets/mod.rssays so in the source.What is not duplicated is the part the generator cannot do:
ClientInformation,CustomPayloadandUpdateTagsall branch on a runtimevalue, and stay hand-written.
One thing found while rebasing
nix flake check's staleness check has been failing on main since #961, whichran a
use_selffix over the committedgenerated/wire.rswithout telling thegenerator. The committed copy said
&'static Selfand the pipeline stillproduced
&'static Wire. Fixed in the generator, which is the check doingexactly what it exists for.