Repository navigation
Detect truncated compressed body in all decompressors - #1234
Cycloctane wants to merge 3 commits into
Conversation
Merging this PR will not alter performance
Comparing Footnotes
|
There was a problem hiding this comment.
1 issue found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/httpx2/test_decoders.py">
<violation number="1" location="tests/httpx2/test_decoders.py:131">
P3: `zlib.compressobj(...).compress` without a `flush()` never finalizes the gzip stream (no final block, CRC32 or ISIZE trailer), so this fixture does not test truncation of a valid gzip body. The full, untruncated artefact stream still has `eof == False` and now raises `DecodingError("Truncated gzip data")`, so this case cannot distinguish a truncated body from a complete one and would not catch a regression in truncation detection for gzip. Finalize the stream with `.compress(body) + .flush()`, matching `test_gzip`/`test_multi`.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| ("encoding", "compress"), | ||
| [ | ||
| (b"deflate", zlib.compress), | ||
| (b"gzip", zlib.compressobj(wbits=zlib.MAX_WBITS | 16).compress), |
There was a problem hiding this comment.
P3: zlib.compressobj(...).compress without a flush() never finalizes the gzip stream (no final block, CRC32 or ISIZE trailer), so this fixture does not test truncation of a valid gzip body. The full, untruncated artefact stream still has eof == False and now raises DecodingError("Truncated gzip data"), so this case cannot distinguish a truncated body from a complete one and would not catch a regression in truncation detection for gzip. Finalize the stream with .compress(body) + .flush(), matching test_gzip/test_multi.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/httpx2/test_decoders.py, line 131:
<comment>`zlib.compressobj(...).compress` without a `flush()` never finalizes the gzip stream (no final block, CRC32 or ISIZE trailer), so this fixture does not test truncation of a valid gzip body. The full, untruncated artefact stream still has `eof == False` and now raises `DecodingError("Truncated gzip data")`, so this case cannot distinguish a truncated body from a complete one and would not catch a regression in truncation detection for gzip. Finalize the stream with `.compress(body) + .flush()`, matching `test_gzip`/`test_multi`.</comment>
<file context>
@@ -124,6 +124,27 @@ def raw() -> typing.Iterator[bytes]:
+ ("encoding", "compress"),
+ [
+ (b"deflate", zlib.compress),
+ (b"gzip", zlib.compressobj(wbits=zlib.MAX_WBITS | 16).compress),
+ (b"br", brotli.compress),
+ (b"zstd", zstd.compress),
</file context>
There was a problem hiding this comment.
This is used for making truncated data for the test. I don't know how generating truncated data can be a problem when we are generating truncated data.
Summary
Current body decoder only performs EOF checks on certain encoding implementations. When a compressed body is malform or truncated, a
DecodingErrorwill be raised only when using zstd or brotlicffi. For bodies compressed with other encodings (gzip, deflate, brotli), the decoder silently returns incomplete data without any warning or error.This PR applies the checks to other decompressors to make this behavior more consistent. Also adds some improvements to
test_decoderstest suite.Checklist