Parse rfc6532 addresses by chibenwa · Pull Request #3130 · apache/james-project

chibenwa · 2026-08-23T07:55:01Z

No description provided.

…ing and output.

This changes the handling of some noncompliant IMAP clients, which James would not tolerate before.

Application-layer code often assumes that addresses can be compared using String.equalsIgnoreCase(), and some also uses regular expressions or substring matching on addresses. This commit provides addresses to upper-layer code in their UTF8 form, so that kind of code continues to work. This might also have security implications: If upper-layer code can be confused about whether two addresses are the same, that sounds as if an attacker could exploit the confusion. This change should block the possibility.

RFC 6532 says we SHOULD do this and JAMES is generally very careful, so I did this as well.

…etc. This should not make a difference, but a sufficiently inventive attacker might combine it with something to confuse some code...

…that domain.

…PUTF8.

Once the server advertises UTF8=ACCEPT, angus-mail enables it and then takes its supportsUtf8() branch in IMAPProtocol.search(), which passes a null charset down to Argument.writeString(String) and encodes the search strings with ASCIIUtility.getBytes() -- one truncated byte per char. "天天向上" reaches the server as 29 29 11 0A, so the criterion is destroyed client-side before it hits the wire and no server-side decoding can recover it. Same in angus-mail 2.0.5, and there is no property to keep the client from enabling the capability. Send the RFC 6855 wire form ourselves instead: UTF-8 octets in literals, plus a quoted-string variant. These fail against consumeLiteral()'s former US-ASCII decoding, so they cover the gap they were written for.

…a 5x slow down)

…UTF8

chibenwa · 2026-09-03T08:57:48Z

(
Simple master rebase, no changes.
The ci failed with build compilation.
I suppose this is due to recent master changes causing compilation issue atop this branch: conflicts but without merge conflict.
)

chibenwa · 2026-09-07T18:12:15Z

Please address your comments @Arsnael

chibenwa · 2026-09-08T09:58:51Z

@arnt no objection with that one?

Also may I ask for a ICLA ? That's a major contribution, as per ASF rule we would need IP clearance.

https://www.apache.org/licenses/contributor-agreements.html

Thanks by advance

arnt · 2026-09-08T12:26:49Z

I started the day with opening tabs for my open PRs, which are many and the James PR is big ;) The only reason I could write the james patches was that several colleagues were travelling so I had unexpected peace.

Not done with the PR catching-upping, today's my first pass since June. Still have a few tabs to read and close, and James is one.

I'll send the corporate CLA to the legal team at work; it should not be a problem (apart from the GPG invocations, which I am afraid the legal team will leave to me).

I need lunch now, will review this after lunch.

arnt · 2026-09-08T14:11:51Z

@@ -84,24 +101,59 @@ void receivedMessagesShouldContainDeliveredToHeaders() throws Exception {

    @Test
    void receivedMessagesShouldContainDeliveredToHeadersI8N() throws Exception {


Is that i8n instead of i18n?

arnt · 2026-09-08T14:27:18Z

+     * mime4j then falls back to us-ascii. Only the RFC 3798 form honours the
+     * declared charset.
+     */
+    private static String reportCharset(Entity entity) {


Just a comment. The code is correct, I think.

There a chance that an RFC will appear sometime in 2027/8 that deprecates message/global and says to use message/rfc822 for everything. Ditto text/globalheaders or whatever it was.

I think generating message/global is good, for now, but on the parse side I might lean towards accepting UTF8 in message/rfc822 already. Not 100% sure. I might be strict there too, it depends on the code.

Thanks for the notice. We will follow upcoming RFCs when published I presume.

Yes.

I think my brain has found an opinion: at this point it's best to always follow RFC 6533 when generating, and mostly when parsing (unless that's too complicated and requires too much code).

arnt · 2026-09-08T14:39:49Z

+    }
+
+    @Test
+    void asciiEmailWithSmtpUtf8ShouldBeReceived() throws Exception {


What does this test?

Simply that we can use ascii addresses with SMTPUTF8

Maybe not the most useful of a test but defnitly harmless.

arnt · 2026-09-08T15:01:22Z

+        // Left alone for the ASCII envelopes that make up the bulk of the
+        // traffic, so they neither change behaviour nor trip Angus' "allowutf8
+        // set but server doesn't advertise SMTPUTF8" log line. The pool clears
+        // the property again when the session is passivated.


Good point.

Looking at all this, I wonder why the tests passed.

arnt · 2026-09-08T15:14:24Z

+     *
+     * UTF-8 is handled at {@link TestIMAPClient}'s own boundary instead: see
+     * {@link TestIMAPClient#asOctets(String)} and {@link TestIMAPClient#asText(String)}.
+     */


I suppose it has to be like that. IMAP must read octets and decode to chars in a context-sensitive manner. While I find it difficult to name the octets "ISO-8859-1", given the legacy I suppose it makes sense.

Well, working around our own test dependencies o_o

arnt · 2026-09-08T15:21:07Z

Lovely work. I really am puzzled that the test suite was green after my work, before yours @chibenwa

Thanks!

chibenwa · 2026-09-08T17:10:00Z

You are very welcome!
Thanks for embarking us in this journey, I'm pretty sure we would not have been able to do it by our own.

chibenwa mentioned this pull request Aug 23, 2026

JAMES-4133 Add support for unicode adddresses as defined in RFC6532. #2728

Draft

chibenwa force-pushed the parse-rfc6532-addresses branch 3 times, most recently from e673a0c to 2a4b7ca Compare August 31, 2026 08:39

chibenwa and others added 26 commits September 3, 2026 10:56

JAMES-4133 POC RFC-6532 Support for i8n emails

297a692

JAMES-4133 Add support for unicode adddresses as defined in RFC6532.

99b4351

JAMES-4133 Use UTF-8 as the charset supported atop SMTP protocol

2898199

JAMES-4133 UTF-8 support for james-server-core

ee5a945

JAMES-4133 Integration tests for UTF-8 mail reception

35388bc

JAMES-4133 Provide ENABLE UTF8=ACCEPT (doing nothing so far).

4f5a748

JAMES-4133 Use UTF8 for mailbox names instead of mUTF7, both for pars…

d00bce4

…ing and output.

JAMES-4133 Send UTF8 quoted-strings if enabled, and accept them anyway.

6a596a5

This changes the handling of some noncompliant IMAP clients, which James would not tolerate before.

JAMES-4133 Add support for RFC 6533 (utf8 addresses in DSNs/MDNs).

7daf9c8

JAMES-4133 Add SMTP server support for RFC 6531.

ac3a06e

JAMES-4133 Add support for RFC 6531 (SMTPUTF8) to remote delivery.

094b12c

JAMES-4133 Normalise addresses to NFC, per RFC 6532 3.1.

a9f998d

RFC 6532 says we SHOULD do this and JAMES is generally very careful, so I did this as well.

JAMES-4133 Check for and reject addresses containing lone surrogates …

751d190

…etc. This should not make a difference, but a sufficiently inventive attacker might combine it with something to confuse some code...

JAMES-4133 Name the malformed domain name in the error message about …

288c8ae

…that domain.

JAMES-4133 Generate the correct WITH keywords when James receives SMT…

0eaeeac

…PUTF8.

JAMES-4133 Adapt MDN regarding RFC 6533 implicit use of UTF-8

3ed274f

JAMES-4133 SMTP should support SMTPUTF8 by default

f525bed

JAMES-4133 JMH benchmark for mail address parsing

ebc881d

JAMES-4133 Performance: optimize for ascii address (normalization is …

ff604d5

…a 5x slow down)

JAMES-4133 Make utf8Accepted a prime citizen of IMAP session

d417129

JAMES-4133 Leverage MailAddres::isAscii in more places

d96c1ed

JAMES-4133 Remote Delivery integration tests and fixes regarding SMTP…

4d9056b

…UTF8

JAMES-4133 Adapt SMTPUTF8 within LMTP too

4fa65ba

[FIX] AddDeliveredToHeaderTest

1e8e1e4

chibenwa force-pushed the parse-rfc6532-addresses branch from 2a4b7ca to 1e8e1e4 Compare September 3, 2026 08:56

chibenwa added 2 commits September 3, 2026 10:58

JAMES-4133 Master rebase compilation issue

1a27dca

JAMES-4133 Bump mock-smtp-server to version 0.8

478cf7b

Arsnael reviewed Sep 7, 2026

View reviewed changes

Comment thread mailbox/opensearch/src/test/resources/eml/cve-2024-23184.eml Outdated

Comment thread core/src/test/java/org/apache/james/core/DomainTest.java Outdated

JAMES-4133 Little fixes

a96f85d

Arsnael approved these changes Sep 8, 2026

View reviewed changes

quantranhong1999 approved these changes Sep 8, 2026

View reviewed changes

arnt reviewed Sep 8, 2026

View reviewed changes

JAMES-4133 Fix typo in test name

fe5e6c5

Arsnael approved these changes Sep 9, 2026

View reviewed changes

		@@ -84,24 +101,59 @@ void receivedMessagesShouldContainDeliveredToHeaders() throws Exception {

		@Test
		void receivedMessagesShouldContainDeliveredToHeadersI8N() throws Exception {

Uh oh!

Conversation

chibenwa commented Aug 23, 2026

Uh oh!

chibenwa commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

chibenwa commented Sep 7, 2026

Uh oh!

chibenwa commented Sep 8, 2026

Uh oh!

arnt commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnt commented Sep 8, 2026

Uh oh!

chibenwa commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants