Skip to content

Detect truncated compressed body in all decompressors - #1234

Open
Cycloctane wants to merge 3 commits into
pydantic:mainfrom
Cycloctane:eof_check
Open

Cycloctane wants to merge 3 commits into
pydantic:mainfrom
Cycloctane:eof_check

Conversation

@Cycloctane

@Cycloctane Cycloctane commented Sep 26, 2026 •

Copy link
Copy Markdown

Summary

Current body decoder only performs EOF checks on certain encoding implementations. When a compressed body is malform or truncated, a DecodingError will be raised only when using zstd or brotlicffi. For bodies compressed with other encodings (gzip, deflate, brotli), the decoder silently returns incomplete data without any warning or error.

This PR applies the checks to other decompressors to make this behavior more consistent. Also adds some improvements to test_decoders test suite.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.

Review in cubic

@codspeed

codspeed Bot commented Sep 26, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 17 untouched benchmarks
⏩ 7 skipped benchmarks1


Comparing Cycloctane:eof_check (5b04596) with main (fc32b34)

Open in CodSpeed

Footnotes

  1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/httpx2/test_decoders.py">

<violation number="1" location="tests/httpx2/test_decoders.py:131">
P3: `zlib.compressobj(...).compress` without a `flush()` never finalizes the gzip stream (no final block, CRC32 or ISIZE trailer), so this fixture does not test truncation of a valid gzip body. The full, untruncated artefact stream still has `eof == False` and now raises `DecodingError("Truncated gzip data")`, so this case cannot distinguish a truncated body from a complete one and would not catch a regression in truncation detection for gzip. Finalize the stream with `.compress(body) + .flush()`, matching `test_gzip`/`test_multi`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

("encoding", "compress"),
[
(b"deflate", zlib.compress),
(b"gzip", zlib.compressobj(wbits=zlib.MAX_WBITS | 16).compress),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: zlib.compressobj(...).compress without a flush() never finalizes the gzip stream (no final block, CRC32 or ISIZE trailer), so this fixture does not test truncation of a valid gzip body. The full, untruncated artefact stream still has eof == False and now raises DecodingError("Truncated gzip data"), so this case cannot distinguish a truncated body from a complete one and would not catch a regression in truncation detection for gzip. Finalize the stream with .compress(body) + .flush(), matching test_gzip/test_multi.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/httpx2/test_decoders.py, line 131:

<comment>`zlib.compressobj(...).compress` without a `flush()` never finalizes the gzip stream (no final block, CRC32 or ISIZE trailer), so this fixture does not test truncation of a valid gzip body. The full, untruncated artefact stream still has `eof == False` and now raises `DecodingError("Truncated gzip data")`, so this case cannot distinguish a truncated body from a complete one and would not catch a regression in truncation detection for gzip. Finalize the stream with `.compress(body) + .flush()`, matching `test_gzip`/`test_multi`.</comment>

<file context>
@@ -124,6 +124,27 @@ def raw() -> typing.Iterator[bytes]:
+    ("encoding", "compress"),
+    [
+        (b"deflate", zlib.compress),
+        (b"gzip", zlib.compressobj(wbits=zlib.MAX_WBITS | 16).compress),
+        (b"br", brotli.compress),
+        (b"zstd", zstd.compress),
</file context>

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used for making truncated data for the test. I don't know how generating truncated data can be a problem when we are generating truncated data.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant