DRILL-8548: Integrate Apache Ranger authorization for Drill by shfshihuafeng · Pull Request #3056 · apache/drill

shfshihuafeng · 2026-07-22T03:10:21Z

DRILL-8548: Integrate Apache Ranger authorization for Drill

Description

Motivation

Apache Drill is a federated query engine that concurrently accesses heterogeneous data sources such as relational databases (MySQL), file systems (local/HDFS), and search engines (Elasticsearch). However, each data source has its own independent permission model with varying granularities – relational databases support schema/table/column level controls, file systems often lack column‑level privileges, and Elasticsearch only provides index‑level authorization. This forces administrators to maintain separate permission schemes for each data source individually, resulting in cumbersome configuration, high operational overhead, and the inability to achieve cross‑source unified auditing. Therefore, it is necessary to build a unified, fine‑grained permission governance system for Drill, which is primarily embodied as follows:

Unified Policy Management: All data sources accessible via Drill can be governed through a single Ranger console. A single Ranger policy applies simultaneously across multiple heterogeneous sources, eliminating the need for per‑storage‑system configuration and greatly simplifying administrative complexity.
Column‑Level Fine‑Grained Access Control: Permissions can be restricted down to individual columns within a table, satisfying the need to isolate sensitive fields (e.g., ID numbers, salaries, phone numbers) and achieving true field‑level security protection.
Consistent Enforcement at the Federation Layer: Permission checks are performed uniformly during Drill's toRel phase (before logical plan generation), ensuring that the same set of policies takes effect regardless of whether the query is a single‑table scan, a cross‑source JOIN, or a nested subquery. There is no possibility of bypassing Ranger through other storage plugins, guaranteeing complete integrity of mandatory enforcement.
Audit & Compliance Readiness: Every access decision is recorded and pushed into Ranger's audit pipeline, providing a complete audit trail of "who queried what and when". This natively supports compliance reporting requirements.

Introduction

To address the resource hierarchy discrepancies among different data sources (e.g., relational databases support database/table/column levels, whereas file systems lack the Schema concept), I designed a four‑level logical resource model (DataSource → Schema → Table → Column) that uniformly maps the structures of heterogeneous data sources onto this hierarchy, thereby providing a unified resource addressing foundation for Ranger policy configuration.

This PR introduces Apache Ranger as a pluggable authorization framework for Drill, enabling centralized table-level and column-level access control for Drill queries. It is a substantial feature spanning three layers: a new drill-ranger module (authorization plugin + Ranger Admin service plugin), integration hooks in exec/java-exec, and distribution packaging. The design follows Drill's existing AccessAuthorizer SPI and Calcite's RelShuttle mechanism so that column-level checks happen in the toRel phase before physical planning.

Documentation

This PR ships a developer-facing quick start guide, RangerAuthorization.md, covering the end-to-end architecture, configuration, deployment, and authorization behavior of the Ranger integration. It is intended both for contributors extending/debugging the integration and for operators who want to understand the column-level authorization flow.

Testing

`drill-ranger-service` (1 file, 28 cases, JUnit 5 + Mockito)

RangerServiceDrillTest — buildBaseUrl normalization, escapeSql SQL-injection guard, firstHint/extractFirstColumnValues JSON parsing, lookupResource/validateConfig edge cases.

`exec/java-exec` (4 files, 27 cases, JUnit 4 + Mockito, extends `BaseTest`)

AccessAuthorizerFactoryTest — config-driven reflective loading, caching, failure modes.
NoOpAccessAuthorizerTest — fail-open contract.
RangerAccessAuthorizerTest — delegation to DrillAccessControl with DrillAccessType enum normalization (null/unknown → SELECT).
DrillCalciteCatalogReaderTest — getDefaultSchemaByDataSource package-private static helper.

All tests pass on Windows with -DforkCount=0 (the default forkCount=1 fails with CreateProcess error=5 on locked-down Windows hosts).

pjfanning · 2026-07-22T08:52:29Z

+      </plugin>
+      <plugin>
+        <artifactId>maven-shade-plugin</artifactId>
+        <version>3.2.4</version>


this version is very old - 3.6.2 exists

@pjfanning Sorry for the confusion – dependency-reduced-pom.xml was a leftover from the shade plugin and was accidentally committed in the initial commit. It will be removed.

pjfanning · 2026-07-22T08:54:47Z

+    <dependency>
+      <groupId>io.netty</groupId>
+      <artifactId>netty-handler</artifactId>
+      <version>4.1.118.Final</version>


very old - security contingent too

We have this in the main pom.xml.

<netty.version>4.1.135.Final</netty.version>

You should be able to omit the version here and with other other netty jars (below).

pjfanning · 2026-07-22T08:55:27Z

+    <dependency>
+      <groupId>org.apache.hadoop</groupId>
+      <artifactId>hadoop-common</artifactId>
+      <version>3.3.4</version>


again - you should not be including versions here - the version control should be in the main pom.xml

pjfanning · 2026-07-22T08:55:58Z

+    <dependency>
+      <groupId>org.slf4j</groupId>
+      <artifactId>slf4j-api</artifactId>
+      <version>2.0.6</version>


no versions here - and why do you keep using ancient versions?

pjfanning · 2026-07-22T08:56:34Z

+    <dependency>
+      <groupId>ch.qos.logback</groupId>
+      <artifactId>logback-classic</artifactId>
+      <version>1.5.25</version>


all these version numbers should be in main pom.xml

pjfanning · 2026-07-22T08:57:40Z

+    <dependency>
+      <groupId>com.sun.jersey</groupId>
+      <artifactId>jersey-client</artifactId>
+      <version>1.19.4</version>


this jar version is not maintained and riddled with security issues

Thanks for flagging this issue in the review — it's an important one. Two points in response:

Constrained by Ranger 2.8.0 (the highest resolvable release). Ranger 2.8.0's RangerAdminRESTClient / RangerRESTClient hard-code the com.sun.jersey.api.client.* API, so the plugin must ship Jersey 1.x client/core. Ranger 3.x migrated to Jersey 2.x, but Ranger 3.0.0 artifacts cannot be resolved from the configured repositories, so 2.8.0 is effectively the latest release we can consume right now. Upgrading Ranger is the only path to drop Jersey 1.x.
Tracked as a follow-up, not blocking this PR. I've left a SECURITY NOTE block in drill-ranger/drill-ranger-plugin/pom.xml marking this for when Ranger is upgraded.
Separately, I'll take a closer look at the Ranger source later to see whether there's a way to avoid shipping the Jersey 1.x client classes entirely. Do you have any other suggestions or a better approach in mind?

Thanks for flagging this. You're right — Jersey 1.19.4 is unmaintained and carries known CVEs. I have reviewed the Ranger code and reworked the approach ,and would like to validate the approach before pushing the changes.

Why not simply upgrade to Jersey 2.35

Ranger 2.8.0 ships RangerAdminJersey2RESTClient (in ranger-knox-plugin), which is built on Jersey 2.x (javax.ws.rs.* API + org.glassfish.jersey.* implementation). However, Drill's own REST server depends on Jersey 3.1.9 (jakarta.ws.rs.* API + org.glassfish.jersey.* implementation).

Jersey 2.x and 3.x share the same org.glassfish.jersey.* package namespace but are binary-incompatible due to the Jakarta EE 8→9 namespace migration (javax.ws.rs.* → jakarta.ws.rs.*). Putting both on the same classpath causes ClassCastException / NoSuchMethodError at runtime. So Jersey 2.35 cannot sit on Drill's main classpath alongside Jersey 3.1.9.

Isolation approach

We use Ranger's built-in RangerPluginClassLoader — a child-first (parent-last) classloader that isolates the plugin's Jersey 2.x dependencies from the Drillbit's main classpath:

Jersey 2.35 and its transitive deps (HK2 2.6.1, jakarta.ws.rs-api:2.1.6, jakarta.annotation-api:1.3.5) are placed in a dedicated directory jars/ranger-drill-plugin-impl/, loaded only by RangerPluginClassLoader.
The Drillbit main classpath (jars/3rdparty/ + jars/classb/) keeps only Jersey 3.1.9; Jersey 2.x jars are excluded via version-pinned assembly rules and Maven dependency exclusions.
RangerAccessAuthorizer (in drill-java-exec) loads DrillAccessControl reflectively through the isolated classloader, switching the TCCL (Thread Context ClassLoader) around each invocation. This avoids any compile-scope dependency from drill-java-exec on Jersey 2.x.
ranger-plugin-classloader is the only Ranger artifact on the main classpath — it must be there to bootstrap the isolated classloader

Is this classloader-isolation approach acceptable to the community? If so, I'll push the implementation.

if 2 can be made to work, that seems good to me

@pjfanning I'll go with option 2, implement it, run thorough tests, and then submit the PR

cgivre · 2026-07-22T16:25:06Z

@shfshihuafeng Do you use the Drill slack channel? I had a question for you that does not pertain to this pull request.

shfshihuafeng · 2026-07-23T03:09:23Z

@shfshihuafeng Do you use the Drill slack channel? I had a question for you that does not pertain to this pull request.
@cgivre I haven't joined the Drill Slack channel yet, since Slack is not accessible within mainland China. If convenient, email me at shfshihuafeng@163.com/shfshihuafeng@outlook.com

cgivre · 2026-07-23T13:00:02Z

@shfshihuafeng Can you please rebase to latest master. There is a fix for flaky Splunk tests that were breaking the CI.

shfshihuafeng · 2026-07-23T14:36:16Z

@shfshihuafeng Do you use the Drill slack channel? I had a question for you that does not pertain to this pull request.

done

cgivre · 2026-07-24T15:09:52Z

@shfshihuafeng Do you use the Drill slack channel? I had a question for you that does not pertain to this pull request.
@cgivre I haven't joined the Drill Slack channel yet, since Slack is not accessible within mainland China. If convenient, email me at shfshihuafeng@163.com/shfshihuafeng@outlook.com

I sent you an email.

shfshihuafeng · 2026-07-24T23:08:03Z

@shfshihuafeng Do you use the Drill slack channel? I had a question for you that does not pertain to this pull request.
@cgivre I haven't joined the Drill Slack channel yet, since Slack is not accessible within mainland China. If convenient, email me at shfshihuafeng@163.com/shfshihuafeng@outlook.com

I sent you an email.

@cgivre Thank you for your email – I received it. I’m glad to accept the your invitation. I had already replied to your previous invitation email a few days ago, but I'm not sure if you received it. Could you please confirm? I will resend my reply right now

shfshihuafeng · 2026-07-26T13:33:56Z

@shfshihuafeng Do you use the Drill slack channel? I had a question for you that does not pertain to this pull request.
@cgivre I haven't joined the Drill Slack channel yet, since Slack is not accessible within mainland China. If convenient, email me at shfshihuafeng@163.com/shfshihuafeng@outlook.com

I sent you an email.

@cgivre Thank you for your email – I received it. I’m glad to accept the your invitation. I had already replied to your previous invitation email a few days ago, but I'm not sure if you received it. Could you please confirm? I will resend my reply right now

@cgivre I have resent the email, but I still haven't heard back from you, so I would like to confirm whether you received my previous email.

pjfanning reviewed Jul 22, 2026

View reviewed changes

cgivre assigned shfshihuafeng Jul 22, 2026

cgivre added enhancement PRs that add a new functionality to Drill doc-impacting PRs that affect the documentation security major-update labels Jul 22, 2026

shfshihuafeng force-pushed the Drill-8548 branch from 6d7d66c to 4cf3cd7 Compare July 23, 2026 03:44

shfshihuafeng force-pushed the Drill-8548 branch from 4cf3cd7 to 59ab5cb Compare July 23, 2026 14:19

DRILL-8548: Integrate Apache Ranger authorization for Drill

5c46a5d

shfshihuafeng force-pushed the Drill-8548 branch from 59ab5cb to 5c46a5d Compare July 23, 2026 15:37

Uh oh!

Conversation

shfshihuafeng commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DRILL-8548: Integrate Apache Ranger authorization for Drill

Description

Motivation

Introduction

Documentation

Testing

drill-ranger-service (1 file, 28 cases, JUnit 5 + Mockito)

exec/java-exec (4 files, 27 cases, JUnit 4 + Mockito, extends BaseTest)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shfshihuafeng Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgivre commented Jul 22, 2026

Uh oh!

shfshihuafeng commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgivre commented Jul 23, 2026

Uh oh!

shfshihuafeng commented Jul 23, 2026

Uh oh!

cgivre commented Jul 24, 2026

Uh oh!

shfshihuafeng commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shfshihuafeng commented Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shfshihuafeng commented Jul 22, 2026 •

edited

Loading

`drill-ranger-service` (1 file, 28 cases, JUnit 5 + Mockito)

`exec/java-exec` (4 files, 27 cases, JUnit 4 + Mockito, extends `BaseTest`)

shfshihuafeng Jul 28, 2026 •

edited

Loading

shfshihuafeng commented Jul 23, 2026 •

edited

Loading

shfshihuafeng commented Jul 24, 2026 •

edited

Loading