Skip to content

fix(druid): handle 429/503 HTML before JSON parse in DirectDruidClient - #20151

Open
shoemoney wants to merge 1 commit into
apache:masterfrom
shoemoney:fix/broker-429-html
Open

fix(druid): handle 429/503 HTML before JSON parse in DirectDruidClient#20151
shoemoney wants to merge 1 commit into
apache:masterfrom
shoemoney:fix/broker-429-html

Conversation

@shoemoney

Copy link
Copy Markdown

Description

Broker can return HTTP 429 or 503 with an HTML error page (for example from a load balancer or reverse proxy) instead of the expected JSON. DirectDruidClient at 242 enqueues the raw body and later JsonParserIterator tries to parse it as JSON, which fails with a JsonParseException for 0x3c ('<') and masks the real capacity error.

Fix checks response status and Content-Type before enqueue in handleResponse:

  • If status is 429 or 503, throw QueryCapacityExceededException with status and a body preview before any JSON parsing.
  • If Content-Type is text/html or the first non-whitespace byte is '<' instead of '{' or '[', fail fast with a meaningful QueryInterruptedException preview instead of a generic JsonParseException.

This preserves the correct exception type for capacity errors and avoids HTML being misreported as a JSON parse failure.

Testing

  • RED to GREEN verified locally
  • DirectDruidClientTest 12 tests pass before and after the change
  • Formatter blast radius limited to DirectDruidClient.java

Release note

Fix broker HTML 429/503 responses being masked as JsonParseException in DirectDruidClient.


Key changed/added classes in this PR
  • DirectDruidClient

This PR has:

  • been self-reviewed

Fix verified RED->GREEN. Broker masks 429/503 HTML as JsonParseException 0x3c at DirectDruidClient.java:242
break;
}
}
if (statusCode == 429 || statusCode == 503) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this if block should be placed in front of above if block which detects whether the response is html body

}
throw QueryCapacityExceededException.withErrorMessageAndResolvedHost(msg);
}
if (isHtmlContentType || isHtmlBody) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should re-organise the new code into two methods which follow current checkTotalBytesLimit and checkQueryTimeout, so in this handleResponse it looks like sth like:

...
checkQueryTimeout();
checkStatusCode(); // which checks 429 and 503
checkHtmlResponse();
checkTotalBytesLimit(response.getContent().readableBytes());
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants