Skip to content

Fix XML.unescape rejecting valid whitespace numeric character references#1070

Open
dong0713 wants to merge 1 commit into
stleary:masterfrom
dong0713:fix/xml-unescape-whitespace-control-chars
Open

Fix XML.unescape rejecting valid whitespace numeric character references#1070
dong0713 wants to merge 1 commit into
stleary:masterfrom
dong0713:fix/xml-unescape-whitespace-control-chars

Conversation

@dong0713

Copy link
Copy Markdown

Summary

Fixes #1059XML.unescape("&#10;") and XML.toJSONObject("<a>&#10;</a>") throw JSONException: Invalid numeric character reference: &#10; in versions after 20251224 (regression introduced by #1045).

Root Cause

XML.mustEscape(int cp) is shared by both the serialization path (XML.escape) and the deserialization path (XMLTokener.unescapeEntity). The method's Javadoc and comment correctly quote the W3C XML 1.0 valid-character range:

#x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

However, the implementation only checked [#x20-#xD7FF] onwards in its range clause, omitting #x9 / #xA / #xD (TAB, LF, CR). Although the ISO-control clause explicitly excluded those three codepoints, the negated range clause (!(...)) still evaluated to true for them, so mustEscape(0xA) returned true and unescapeEntity threw JSONException.

Fix

Align the range clause with the W3C spec by explicitly allowing cp == 0x9, cp == 0xA, and cp == 0xD, matching the comment that was already documented.

Before:

|| !(
    (cp >= 0x20 && cp <= 0xD7FF)   // missing #x9, #xA, #xD
    || (cp >= 0xE000 && cp <= 0xFFFD)
    || (cp >= 0x10000 && cp <= 0x10FFFF)
)

After:

|| !(
    cp == 0x9
    || cp == 0xA
    || cp == 0xD
    || (cp >= 0x20 && cp <= 0xD7FF)
    || (cp >= 0xE000 && cp <= 0xFFFD)
    || (cp >= 0x10000 && cp <= 0x10FFFF)
)

Behaviour Change

Input Before (v20260522+) After
XML.unescape("&#10;") JSONException "\n"
XML.unescape("&#9;") JSONException "\t"
XML.unescape("&#13;") JSONException "\r"
XML.unescape("&#xA;") JSONException "\n"
XML.toJSONObject("<a>&#10;</a>") JSONException {"a":""}
XML.unescape("&#32;") " " (unchanged) " "

Testing

Fixes #1059

XML.mustEscape() is shared by both XML.escape() (serialization) and
XMLTokener.unescapeEntity() (deserialization). While the method's Javadoc
and comment quote the W3C XML 1.0 valid-character range
(#x9 | #xA | #xD | [#x20-#xD7FF] | ...), the implementation only checked
[#x20-#xD7FF] in its range clause, omitting #x9/#xA/#xD.

Although the ISO-control clause excluded those three codepoints, the
negated range clause still marked them as 'must escape', so unescapeEntity()
threw JSONException for the XML-allowed control characters TAB, LF and CR.

This broke XML.unescape("&stleary#10;") and XML.toJSONObject("<a>&stleary#10;</a>"),
both of which worked in v20251224 and regressed after stleary#1045 (v20260522).

Align the range clause with the W3C spec by explicitly allowing #x9, #xA
and #xD, matching the comment that was already documented.

Fixes stleary#1059
@sonarqubecloud

Copy link
Copy Markdown

@stleary

stleary commented Jul 21, 2026

Copy link
Copy Markdown
Owner

What problem does this code solve?
XML.mustEscape() does not handle whitespace codepoints (&#nnn;) correctly for whitespace chars.

Risks
Low

Changes to the Existing Behavior
Codepoints for tab, newline, and carriageReturn will no longer throw an exception

Changes to the API
No

Will this require a new release?
Yes

Should the documentation be updated?
No

Unit Tests
New unit tests were added.

Refactoring
None

Review status
APPROVED

Starting 3-day comment window

@holubec-petr

Copy link
Copy Markdown

We also discovered the same behavior, and not following the JSON spec, especially around line break characters, break our solution.

Can we expect a new release of this library when the fix is merged?

@stleary

stleary commented Jul 22, 2026

Copy link
Copy Markdown
Owner

@holubec-petr Yes, there will be a new release when this PR is merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

org.json.XML.unescape("&#10;"); throws org.json.JSONException: Invalid numeric character reference: &#10;

3 participants