Skip to content

fix(auth): count policy wildcards in UTF-16 code units, as IAM does - #581

Open
LeeroyHannigan wants to merge 4 commits into
mainfrom
fix/wildcard-question-mark
Open

LeeroyHannigan wants to merge 4 commits into
mainfrom
fix/wildcard-question-mark

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #575.

Summary

IAM counts what ? and * match in UTF-16 code units. wildcard_match_impl counted bytes of the UTF-8 encoding, so a value with a character outside ASCII matched differently from IAM in both directions. Each IAM answer below was measured with the policy simulator, in a Resource pattern:

Pattern Value IAM main at 744ea291 This PR
user-? user-é match no match match
user-?? user-é no match match no match
*??bc 日bc no match match no match
?.json 😀.json no match no match no match
??.json 😀.json match no match match
*??bc 😀bc match match match

The matcher serves Action, NotAction, Resource and NotResource and the StringLike, StringNotLike, ArnLike and ArnNotLike operators. A pattern with one ? per UTF-16 unit of such a value did not match it, and a pattern with one ? per byte did, so an Allow or a Deny that used ? could decide the opposite of IAM. With #471, ArnEquals and ArnNotEquals use the same matcher.

  • wildcard_match_impl matches ASCII input as bytes, as before, and other input as UTF-16 code units, with the same greedy algorithm (match_units, over u8 or u16). Only input outside ASCII is converted, so ASCII matching does not allocate.
  • docs/manuals/07-upgrade-manual.md gains a behaviour change entry.

The first commit counted code points; the second replaces that with UTF-16 code units after the simulator showed that 😀 takes ??.

Testing

  • IAM's answers, measured with simulate-custom-policy: one call per case with retries off, ASCII policy documents, and the characters only in resource ARNs and context values. In a Resource pattern, ? matched é and 日 and ?? did not; ??? did not match 日; against 😀 (U+1F600), ? did not match, ?? did and ???? did not; *??bc did not match 日bc and matched 😀bc; *?bc matched 😀bc. In a StringLike condition, p??/* matched p😀/x and p?/* did not.
  • Two tests in matcher/utf16_tests.rs pin those answers. Negative controls on the committed tree: matching every value as bytes (the behaviour on main) fails both tests, and counting code points instead of UTF-16 code units fails the two 😀 assertions.
  • An exhaustive check, not committed, compared wildcard_match and wildcard_match_ignore_case with a recursive UTF-16 matcher on every pattern and value of up to four symbols from a, A, é, 日, 😀, 😁, * and ?: 43,823,522 comparisons, no difference.
  • cargo test -p extenddb-auth: 221 unit tests and 3 doctests pass. cargo fmt --all --check and cargo clippy -p extenddb-auth --all-targets -- -D warnings are clean.
  • With fix(auth): match IAM wildcards in ArnEquals and ArnNotEquals #471's head (a80313ee), the code merges cleanly. Both PRs add a section at the same place in docs/manuals/07-upgrade-manual.md, and keeping both sections resolves that. On the merged tree, cargo test -p extenddb-auth passes 252 unit tests and 3 doctests.

Behaviour change for existing policies

A pattern that uses ? against a value outside ASCII can change its decision after the upgrade. docs/manuals/07-upgrade-manual.md has the entry. ASCII values evaluate as before, and no catalog migration is needed.

Checklist

  • Tests added for every measured behaviour
  • cargo fmt, cargo clippy clean (with -W clippy::pedantic, the one warning in matcher.rs is in is_valid_cidr, unchanged here)
  • Docs updated (docs/manuals/07-upgrade-manual.md)
  • No catalog migration

wildcard_match_impl advanced `?` by one byte and backtracked a `*` by
one byte, so a value with a multibyte UTF-8 character matched
differently from IAM: `user-?` did not match `user-é`, `user-??` did,
and `*??bc` matched `日bc`.

`?` now takes the whole UTF-8 sequence at the value position, and a
star backtracks by whole characters. IAM's one-character rule was
measured with the policy simulator.

Fixes #575
The IAM policy simulator counts what `?` and `*` match in UTF-16 code
units. A character outside the Basic Multilingual Plane is two units:
against a value holding U+1F600, `?` did not match, `??` did and `????`
did not, and `*??bc` matched U+1F600 followed by `bc`. A one-unit
character such as `é` or `日` takes one `?`.

wildcard_match_impl matches ASCII input as bytes, as before, and other
input as UTF-16 code units, with the same greedy algorithm.
Add a behaviour change entry: `?` and `*` in policy patterns count
UTF-16 code units, as in IAM, where they counted UTF-8 bytes, so a
pattern that uses `?` against a value outside ASCII can change its
decision after the upgrade.
With the two UTF-16 wildcard tests, matcher.rs passed 600 lines. The tests
move unchanged into the child module matcher/utf16_tests.rs, and the file
is 597 lines.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Policy wildcard ? matches one byte instead of one character

3 participants