codecs UTF-8 IncrementalDecoder delays acknowledging surrogate range codepoint

[jb2170]

Bug report

Bug description:

Current behavior Python 3.16.0a0:

By the time that an incremental UTF-8 parser reads ed aX or ed bX it should know that the resulting codepoint will be in the surrogate range. But Python's codecs.getincrementaldecoder("UTF-8")() decoder delays raising an exception or emitting U+FFFD ('�') characters until it reads the third byte.

Low surrogates: U+D800 (ed a0 80) to U+DBFF (ed af bf)

High surrogates: U+DC00 (ed b0 80) to U+DFFF (ed bf bf)

"strict" error handling:

import codecs

decoder = codecs.getincrementaldecoder("UTF-8")()

s = decoder.decode(b"\xed\xa0")
print(repr(s))
# ''
s = decoder.decode(b"\x80")
print(repr(s))
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed
#                     in position 0: invalid continuation byte

"replace" error handling:

import codecs

decoder = codecs.getincrementaldecoder("UTF-8")("replace")

s = decoder.decode(b"\xed\xa0")
print(repr(s))
# ''
s = decoder.decode(b"\x80")
print(repr(s))
# '���'

Whereas with normal, non-incremental decoding, it seems that the exception below acknowledges that eg ed a0 can be immediately deduced as invalid. The error message is a bit cryptic, as #67802 points out in general, but ignoring the 'position 0' part, it seems the error is not caused by what would be an 'end of data' error, as can be seen with eg just the ed on its own in the second exception below, but by being in the surrogate range.

b"\xed\xa0".decode()
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed
#                     in position 0: invalid continuation byte

b"\xed".decode()
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed
#                     in position 0: unexpected end of data

The number of U+FFFD ('�'), one per byte in the surrogate bytes, seems consistent with how Python implements 'maximal subpart' decoding, part 3.9.6 of the Unicode (17) standard, but I think the incremental decoder should start emitting them at the second, not third, byte of a surrogate sequence, right?

CPython versions tested on:

3.16, 3.14

Operating systems tested on:

Linux

[jb2170]

Appendix: some test cases of how Python implements maximal-subpart decoding

print(repr(b"a\xc0"            .decode(errors="replace"))) # 'a�'       2-byte overlong U+0000 missing second
print(repr(b"b\xc0\x80"        .decode(errors="replace"))) # 'b��'     2-byte overlong U+0000 full
print(repr(b"c\xe0\x80"        .decode(errors="replace"))) # 'c��'     3-byte overlong U+0000 missing third
print(repr(b"d\xe0\x80\x80"    .decode(errors="replace"))) # 'd���'   3-byte overlong U+0000 full
print(repr(b"e\xf0\x80"        .decode(errors="replace"))) # 'e��'     4-byte overlong U+0000 missing third fourth
print(repr(b"f\xf0\x80\x80"    .decode(errors="replace"))) # 'f���'   4-byte overlong U+0000 missing fourth
print(repr(b"g\xf0\x80\x80\x80".decode(errors="replace"))) # 'g����' 4-byte overlong U+0000 full
print()
print(repr(b"h\xc2"            .decode(errors="replace"))) # 'h�'         2-byte U+00A1 missing second
print(repr(b"i\xc2\xa1"        .decode(errors="replace"))) # 'i¡'          2-byte U+00A1 full
print(repr(b"j\xe0"            .decode(errors="replace"))) # 'j�'         3-byte U+0FFF missing second third; don't know yet if overlong
print(repr(b"k\xe0\xbf"        .decode(errors="replace"))) # 'k�'         3-byte U+0FFF missing third
print(repr(b"l\xe0\xbf\xbf"    .decode(errors="replace"))) # 'l\u0fff'     3-byte U+0FFF full
print(repr(b"m\xe1"            .decode(errors="replace"))) # 'm�'         3-byte U+1000 missing second third; know already not overlong
print(repr(b"n\xe1\x80"        .decode(errors="replace"))) # 'n�'         3-byte U+1000 missing third
print(repr(b"o\xe1\x80\x80"    .decode(errors="replace"))) # 'oက'         3-byte U+1000 full
print(repr(b"p\xf1"            .decode(errors="replace"))) # 'p�'         4-byte U+040000 missing second third fourth
print(repr(b"q\xf1\x80"        .decode(errors="replace"))) # 'q�'         4-byte U+040000 missing third fourth
print(repr(b"r\xf1\x80\x80"    .decode(errors="replace"))) # 'r�'         4-byte U+040000 missing fourth
print(repr(b"s\xf1\x80\x80\x80".decode(errors="replace"))) # 's\U00040000' 4-byte U+040000 full
print()
print(repr(b"t\xed"            .decode(errors="replace"))) # 't�'       3-byte high surrogate U+D800 missing second third
print(repr(b"u\xed\xa0"        .decode(errors="replace"))) # 'u��'     3-byte high surrogate U+D800 missing third
print(repr(b"v\xed\xa0\x80"    .decode(errors="replace"))) # 'v���'   3-byte high surrogate U+D800 full
print()
print(repr(b"w\xed"            .decode(errors="replace"))) # 'w�'      3-byte U+D7FF missing second third
print(repr(b"x\xed\x9f"        .decode(errors="replace"))) # 'x�'      3-byte U+D7FF missing third
print(repr(b"y\xed\x9f\xbf"    .decode(errors="replace"))) # 'y\ud7ff'  3-byte U+D7FF full