Fix XML.unescape rejecting valid whitespace numeric character references#1070
Fix XML.unescape rejecting valid whitespace numeric character references#1070dong0713 wants to merge 1 commit into
Conversation
XML.mustEscape() is shared by both XML.escape() (serialization) and
XMLTokener.unescapeEntity() (deserialization). While the method's Javadoc
and comment quote the W3C XML 1.0 valid-character range
(#x9 | #xA | #xD | [#x20-#xD7FF] | ...), the implementation only checked
[#x20-#xD7FF] in its range clause, omitting #x9/#xA/#xD.
Although the ISO-control clause excluded those three codepoints, the
negated range clause still marked them as 'must escape', so unescapeEntity()
threw JSONException for the XML-allowed control characters TAB, LF and CR.
This broke XML.unescape("&stleary#10;") and XML.toJSONObject("<a>&stleary#10;</a>"),
both of which worked in v20251224 and regressed after stleary#1045 (v20260522).
Align the range clause with the W3C spec by explicitly allowing #x9, #xA
and #xD, matching the comment that was already documented.
Fixes stleary#1059
|
|
What problem does this code solve? Risks Changes to the Existing Behavior Changes to the API Will this require a new release? Should the documentation be updated? Unit Tests Refactoring Review status Starting 3-day comment window |
|
We also discovered the same behavior, and not following the JSON spec, especially around line break characters, break our solution. Can we expect a new release of this library when the fix is merged? |
|
@holubec-petr Yes, there will be a new release when this PR is merged |



Summary
Fixes #1059 —
XML.unescape(" ")andXML.toJSONObject("<a> </a>")throwJSONException: Invalid numeric character reference: in versions after 20251224 (regression introduced by #1045).Root Cause
XML.mustEscape(int cp)is shared by both the serialization path (XML.escape) and the deserialization path (XMLTokener.unescapeEntity). The method's Javadoc and comment correctly quote the W3C XML 1.0 valid-character range:However, the implementation only checked
[#x20-#xD7FF]onwards in its range clause, omitting#x9/#xA/#xD(TAB, LF, CR). Although the ISO-control clause explicitly excluded those three codepoints, the negated range clause (!(...)) still evaluated totruefor them, somustEscape(0xA)returnedtrueandunescapeEntitythrewJSONException.Fix
Align the range clause with the W3C spec by explicitly allowing
cp == 0x9,cp == 0xA, andcp == 0xD, matching the comment that was already documented.Before:
|| !( (cp >= 0x20 && cp <= 0xD7FF) // missing #x9, #xA, #xD || (cp >= 0xE000 && cp <= 0xFFFD) || (cp >= 0x10000 && cp <= 0x10FFFF) )After:
|| !( cp == 0x9 || cp == 0xA || cp == 0xD || (cp >= 0x20 && cp <= 0xD7FF) || (cp >= 0xE000 && cp <= 0xFFFD) || (cp >= 0x10000 && cp <= 0x10FFFF) )Behaviour Change
XML.unescape(" ")JSONException"\n"XML.unescape("	")JSONException"\t"XML.unescape(" ")JSONException"\r"XML.unescape("
")JSONException"\n"XML.toJSONObject("<a> </a>")JSONException{"a":""}XML.unescape(" ")" "(unchanged)" "Testing
testValidWhitespaceNumericEntityUnescape— verifies decimal and hex references for TAB/LF/CR viaXML.unescape()testValidWhitespaceNumericEntityToJSONObject— verifiesXML.toJSONObject()accepts	, , without throwing (regression test for org.json.XML.unescape(" "); throws org.json.JSONException: Invalid numeric character reference: #1059)Fixes #1059