Support one-way SPIFFE compatibility with legacy identities

[bellatrix007]

Description

Builds on the SPIFFE support merged in #142. This branch is rebased onto branch-3.6; this PR's diff contains only the compatibility changes described below, not the already-merged implementation.

Preserve certificate-derived client ID strings while supporting one-way SPIFFE-to-legacy application-name compatibility in URI-domain mappings, direct znode ACLs, and legacy superuser-ID selection. Eligible SPIFFE applications can match bare application names or formatted legacy service-principal targets; legacy SAN/DN clients retain their original pre-PR matching behavior.

getClientId(...) returns an immutable ClientIdentity with the original ID and its extraction type: SPIFFE_V1_WL, SPIFFE_V1_WORKLOAD, SPIFFE_V2, LEGACY_SAN, or SUBJECT_DN. The original SPIFFE/configured SAN/Subject DN extraction order and strings are unchanged; certificate-derived IDs are not normalized or rewritten.

Successful authentication retains this pair on the connection, separately from AuthInfo. Authentication failure clears it. Direct ACL checks therefore use authenticated type information without reparsing certificates per request or changing stored identity strings.

Each request snapshots the authenticated identity. Followers and observers carry it to the leader using a versioned, reserved zookeeper-internal-x509 entry in the existing quorum packet's authinfo vector. The receiver strictly decodes and removes that entry, restoring the identity on Request even when no client connection exists locally. The entry is transport-only: it is not added to connection AuthInfo, accepted as a client authentication scheme, or stored in znode ACLs. Invalid versions/types and duplicate metadata are rejected.

ACL preparation, eager authorization and read processing pass request identity through the provider context. The same direct legacy ACL match therefore applies to leader-local requests and forwarded writes, including setData, create, delete, setACL and multi operations. Sync requests and observer relays preserve the context as well.

URI-domain matching

Lookup order:

1. Return an exact client-ID mapping if present.
2. Otherwise, collect domains from the existing slash-aligned prefix mappings. Any matching prefix takes precedence over legacy compatibility, including an explicitly empty prefix mapping.
3. Only when both lookups miss, compare an eligible SPIFFE application's name with bare application-name or formatted legacy service-principal mapping keys.

Eligible identities:

Extraction type	Original client ID	Name used only for compatibility
SPIFFE_V1_WL	kafka	kafka
SPIFFE_V1_WORKLOAD	application/example-mp/kafka	kafka
SPIFFE_V2	application/example-mp/kafka	kafka
Either application type, with a tag	application/example-mp/kafka/cluster-a	kafka

Application paths must be exactly application/<mp>/<app>[/<tag>], with nonempty segments. Product-only paths such as application/example-mp, missing app segments, arbitrary v2 paths, and user/group/airflow/workload paths do not gain legacy service-principal compatibility. The product or tag is never used in place of the app name.

Bare names (kafka), truncated names (servicePrincipal(kafka), closed names (servicePrincipal(kafka)), and complete service-principal URNs are supported. Compatible legacy keys contribute a union of their domain sets only on fallback. Legacy SAN/DN and string-only lookup retain their existing behavior.

Bare compatibility targets must match ^[A-Za-z0-9][A-Za-z0-9._-]*$. For example, application/example-mp/kafka can match kafka in mappings, ACLs or superuser selection. DN/URN-shaped targets such as CN=admin and urn:example:admin are not interpreted as bare application names. Existing exact matching remains valid: a v1/wl or legacy SAN identity extracted as kafka still matches a stored kafka exactly. A full application-path target is never shortened.

The existing znode can remain:

/zookeeper/uri-domain-map/kafka/servicePrincipal(kafka

A legacy SAN ID servicePrincipal(kafka matches its exact key. A SPIFFE v1/wl ID kafka, or a v1/v2 ID application/example-mp/kafka, can reach the same domain through the compatibility fallback without renaming that legacy mapping. A bare mapping such as /zookeeper/uri-domain-map/broker-access/kafka can also grant broker-access to a full SPIFFE application named kafka when exact and prefix lookup miss.

Legacy compatibility compares app names only; it does not distinguish products or tags. Full-path or prefix mappings take precedence when product/tag-specific domain selection is configured. The full authenticated identity is retained.

Direct znode ACL matching

Authenticated client identity	Matching direct ACL
(SPIFFE_V1_WL, "kafka")	x509:kafka (exact), or x509:servicePrincipal(kafka
SPIFFE v1/v2 application/example-mp/kafka[/cluster-a]	x509:kafka or x509:servicePrincipal(kafka through compatibility; its full original ID remains an exact match
(LEGACY_SAN, "kafka")	x509:kafka (existing exact match)
(LEGACY_SAN, "servicePrincipal(kafka")	x509:servicePrincipal(kafka (existing exact match)

Closed and full-URN service-principal ACL forms are supported as well. An application identity is not rewritten into a bare ID. Legacy IDs do not gain reverse aliases to bare or full product/tag-qualified application ACLs.

A bare ACL ID can also denote a domain. A client explicitly mapped into domain kafka can still match x509:kafka by exact domain identity. That existing grant is independent of application-name compatibility.

LegacyServicePrincipalMatcher shares the rules between domain lookup and direct ACL checks. The optional request-context-aware AuthenticationProvider.matches(...) overload is forwarded by the existing provider wrapper; other providers retain their original string matcher. Both X509 providers use the authenticated identity pair.

The candidate AuthInfo ID must equal the original certificate-derived ID. A mapped domain cannot be reinterpreted as another service principal. Ordinary x509 identities and creator ACL expansion rules are unchanged; no synthetic x509 aliases are added. Superuser role markers are described below.

When the original client identity is selected for a creator ACL, its string is stored unchanged: v1/wl kafka remains bare, while v1/v2 application/example-mp/kafka remains a full application ID. Matching a legacy ACL does not rewrite that ACL or add another entry. ACL entries remain independent grants: a matching bare ALL entry is not restricted by a separate formatted READ entry.

User/group/metadata principal names, untyped identities, unrelated application names, and Subject DN identities do not gain compatibility access. Matching is case-sensitive, and normal ACL permission checks remain enforced.

Always-on legacy superuser-ID compatibility

Compatibility is unconditional for both zookeeper.X509AuthenticationProvider.superUser and zookeeper.X509ZNodeGroupAclProvider.superUserId selection. Exact configured client-ID matches always take precedence. Otherwise, eligible SPIFFE v1/wl or complete v1/v2 application identities can match a configured bare application name or formatted legacy service principal using the same compatibility rules. Multiple compatible configured IDs select a stable marker.

There is no enable/disable flag. The former zookeeper.ssl.x509.legacySuperUserCompatibilityEnabled property is not read, even if a leftover setting specifies false. An actual matching superuser configuration is still required; this does not make every SPIFFE client a superuser.

The authenticated certificate-derived identity is unchanged. The super AuthInfo entry uses the matched configured ID:

Authenticated ID:       application/example-mp/kafka
Configured superuser:   servicePrincipal(kafka
Super AuthInfo marker:  super:servicePrincipal(kafka

Keeping the configured marker is important: PrepRequestProcessor.fixupACL() still recognizes the connection as an explicit superuser rather than a cross-domain component, so it does not accidentally replace that superuser's requested ACLs under the group provider's automatic ACL policy.

Only forward SPIFFE-to-legacy compatibility is added for superuser selection, just as for domain and direct-ACL fallback. User/group/airflow/arbitrary-v2/DN identities do not gain superuser aliases, and mapped domain names are not substituted for the authenticated identity. Existing exact legacy SAN/DN superuser settings remain valid, including bare and structured ID strings.

Exact URI-domain and direct-ACL matching, as well as existing cross-domain grants, are unchanged.

Privilege scope: legacy app names do not distinguish products or tags. Configuring a bare or formatted legacy superuser grants full superuser privileges to all eligible trusted identities with that app name. Use exact full application IDs instead of legacy app names when product/tag-specific privileges are required. Treat superuser configuration as a startup setting and restart servers or reconnect clients when changing it; it is not an immediate revocation switch for already authenticated connections.

No workflow, dependency, quorum-TLS, or unrelated refactoring changes are included.

Tests

• Local code review completed
• Original identity strings and extraction types covered with real certificates and existing SAN/DN cases
• Complete application identities, optional tags, missing app segments, and non-application types covered
• Exact/prefix precedence, prefix-domain union, empty prefix mappings, and legacy fallback covered
• Real-certificate authentication exercises mapping deletion/recreation while retaining the full ID
• Direct ACL parity exercised through ZooKeeperServer.checkACL and both X509 providers' matching paths
• Permission denials, principal-type boundaries, mapped-domain isolation, missing authentication context, failed-authentication clearing, unchanged creator ACLs, and superuser behavior covered
• Both providers' unconditional superuser compatibility, exact precedence, stable configured markers, type/name denials, and original identity preservation covered
• Compatibility works with the removed property absent, false or true; bare and formatted superuser targets are covered
• Explicit-superuser ACL handling distinguished from cross-domain ACL replacement; certificate trust rejection and mapped-domain non-promotion retained
• Restricted bare-target matching and grammar/type/name denials in ACL/domain/superuser paths, literal DN/URN-shaped SPIFFE app segments, unchanged formatted service-principal parsing, and exact v1/wl and legacy SAN/DN behavior covered
• Request identity snapshots, actual Learner/Jute/LearnerHandler round-trips, observer relays and sync context covered
• Missing, malformed and duplicate quorum metadata, unchanged ordinary AuthInfo, and rejection of the reserved client-auth/ACL scheme covered

The mapping fixture uses the existing temporary-directory helper to isolate its test database.

Focused compatibility test command:

mvn -q -pl zookeeper-server -am \
  -Dtest=X509DirectAclTest,X509ZNodeGroupAclProviderTest,ZkClientUriDomainMappingHelperTest \
  -DfailIfNoTests=false -Dsurefire-forkcount=1 package

Results: 63 tests passed across these three affected suites, with no failures, errors, or skipped tests. The single-fork setting avoids the existing provider fixture's intermittent parallel port-binding conflicts.

A separate local harness, not included in this PR, exercised 30 expected operation outcomes, 16 exact stored-ACL assertions and 24 data/version assertions across two standalone TLS group-provider profiles using disposable CA-signed certificates. With setX509ClientIdAsAcl=true and storeAuthedClientId false/true, Grestin-style and SPIFFE clients each created and updated their own znode, then updated it from a fresh TLS connection. Stored identities remained unchanged. A SPIFFE client could update the formatted legacy ACL-protected znode; reverse legacy access to the full SPIFFE-ID ACL and unrelated-client access were denied. No mappings or application superuser grants were present. Both local servers were stopped afterward.

That standalone integration run does not exercise bare-name targets or quorum forwarding. Bare-name matching and forwarded ACL comparisons are covered by the targeted Java suites; broader live integration coverage was not rerun for the bare-name restoration.

These local tests do not cover production issuer integration, automatic certificate hot-reload/revocation, mixed-version upgrades, quorum-to-quorum TLS, or load testing.

Known inherited gaps

The broader review reproduced behaviors inherited from the base: deleting the last nested mapping leaf can cause its parent to become a broader prefix grant; the basic X509 provider's DN-only ACL validator rejects explicit non-DN ACL authoring even though existing ACL matching succeeds; and explicit group-provider superusers cannot expand CREATOR_ALL_ACL when only a super marker is present. These limitations are not fixed by the request-context forwarding change.

An unauthorized write relayed from an observer with eager ACL checks disabled to a follower/ObserverMaster with eager ACL checks enabled also triggers an inherited null-connection error and critical request-processor failure. This was reproduced with both providers; it is not fixed here.

Changes that Break Backward Compatibility (Optional)

Both public getClientId(...) overloads return ClientIdentity rather than String. In-repository callers are updated; external callers must recompile and use .getId() for the previous string result.

The request-context-aware matches(...) method is a default interface overload, preserving existing authentication-provider implementations. String-only domain lookup retains its original behavior.

The quorum Jute schema is unchanged, but both the client-facing peer and the leader must support the identity context for forwarded compatibility matches. Missing context never enables an alias; do not rely on this compatibility during a mixed-version rollout. Existing exact identities and ordinary domain/superuser AuthInfo retain their matching behavior.

The intended authorization change is one-way compatibility from eligible SPIFFE application identities to equivalent bare application names or formatted legacy service-principal names. Bare compatibility targets use the restricted grammar documented above. Certificate validation, raw extracted IDs, stored AuthInfo/ACL formats, normal ACL permission checks, and quorum configuration are unchanged. Superuser-ID compatibility is always enabled: existing bare or formatted legacy superuser settings also authorize eligible SPIFFE applications with the same application name. Exact matching and original pre-PR legacy-client matching remain supported.

For the domain provider, direct principal ACL compatibility applies when the original client ID is present in AuthInfo under the existing configuration; domain-only AuthInfo is not promoted to a principal.

Documentation (Optional)

The typed identity result, lookup order, application-name requirements, direct-ACL rules, and compatibility boundaries are documented above and in the implementation. Unconditional superuser matching, its privilege scope, marker semantics, request-context forwarding and mixed-version caveats are documented in zookeeperAdmin.md.

Generated with GitHub Copilot CLI

[arkmish]

Checking approval

Test completed

[laxman-ch]

Historically (For grestin as well), the certificate parsing implementation is duplicated here due to open source limitation. This duplication is incorrect and can cause inconsistencies. We need to make this pluggable and integrate this with interservice auth.

With plugin implementation this duplication will be eliminated.

Follow up ticket: https://linkedin.atlassian.net/browse/CICP-51582