[HIGH] Enforce negotiated HPACK limits

[OskarEichler]

Summary

• Apply SETTINGS_HEADER_TABLE_SIZE to the correct HPACK direction.
• Keep every header-block table-size update within the most recently advertised maximum.
• Enforce SETTINGS_MAX_HEADER_LIST_SIZE while decoding.

Security impact

The decoder's table-size guard could be bypassed by starting a header block with a reduction and following it with an arbitrarily large increase. With a configured 128-byte limit, a focused block changed the effective limit to 1,000,000 bytes; 20 small follow-up blocks then retained 160,810 bytes of dynamic-table data.

Connection SETTINGS handling also applied a peer's header-table setting to the local decoder and an acknowledged local setting to the compressor. A peer could therefore enlarge the memory retained by the decoder, while the receiver's advertised limit did not constrain it.

SETTINGS_MAX_HEADER_LIST_SIZE was recorded but never enforced. A decoder configured for 64 bytes accepted and materialized 100 indexed fields.

This change separates the configured table maximum from the current HPACK table size, fixes the SETTINGS direction, and accounts for RFC 7541's 32-byte per-field overhead before appending decoded fields.

Verification

• rbenv exec bundle exec rake: 448 examples, 0 failures; 41 RuboCop files, no offenses
• Focused 128-byte table-limit bypass now raises CompressionError
• Focused 64-byte header-list overflow now raises ProtocolError
• Focused SETTINGS exchange confirms the peer's 1,000,000-byte value configures the compressor while acknowledged local 128/256-byte limits configure the decoder
• rbenv exec bundle exec rake build
• Targeted Ruby syntax and git diff --check

No repository tests were changed.

Limitations

The repository's HPACK fixture task cannot currently run because its fixtures are absent (tracked separately in issue #200). Live transports, alternate Ruby engines, and sustained memory profiling were not exercised.

Breaking changes

Peers that exceed an advertised HPACK table or header-list bound now receive a connection error. The existing direct Decompressor#table_size= behavior is preserved; a separate internal limit writer tracks negotiated maxima.

[HoneyryderChuck]

wow, it seems this settings has never been enforced, awesome find!

[HoneyryderChuck]

do you need this default and to enforce it, considering that the value is always set in the connection defaults? which case would require this?

[OskarEichler]

The fallback is for direct use of HTTP2::Header::Decompressor.new without a Connection: the compressor specs and HPACK conformance fixtures instantiate it with no connection settings (or only table_size). Connection-created decompressors do receive the connection default, so the fallback is redundant on that path but preserves standalone HPACK usage. It uses the same effectively unlimited default; negotiated/configured limits are still enforced when supplied.

[HoneyryderChuck]

@OskarEichler this needs rebasing (I ended up fixing the issue from the open thread).

[OskarEichler]

Rebased onto current main and adapted the limit checks to the new Settings structure. The redundant standalone default is gone: Decompressor now uses Settings.settings_max_header_list_size directly. The negotiated table maximum remains independent of the current table size and shared encoding settings, including when a context is duplicated. Verified 448 available non-fixture examples, targeted RuboCop, the reduction-then-oversized-update rejection, header-list overflow rejection, and sequential peer/local SETTINGS exchanges. HPACK fixture tests remain unavailable because the fixture data is absent.

[HoneyryderChuck]

@OskarEichler heard you on HPACK tests 👍 I've been fixing that in a separate PR (which was hanging on one of your PRs, perhaps this one). Thx 🙏

[HoneyryderChuck]

just realized this PR is fixing the same as #209 , which looking more compliant.

[HoneyryderChuck]

Merged the changes from #209 , as it was more correct of a fix to the table size changes. Incorporated the remaining delta of this PR into 1e284a8 , namely the enforcement of the headers table size setting.