feat(services/hdfs): add copy support by hfutatzhanghb · Pull Request #8332 · apache/opendal

hfutatzhanghb · 2026-09-22T04:04:02Z

Which issue does this PR close?

Closes #.

Rationale for this change

HDFS currently reports copy as unsupported. Enabling native Operator::copy for the HDFS service makes same-cluster file duplication consistent with other filesystem-like backends such as fs.

hdrs 0.3.3 exposes Client::copy_file, which wraps libhdfs hdfsCopy / Hadoop FileUtil.copy with overwrite=true. This PR uses that API instead of streaming bytes through hdrs::AsyncFile.

What changes are included in this PR?

Bump hdrs from 0.3.2 to 0.3.3, which also pulls hdfs-sys 0.3.0 → 0.3.1 (JNI attach/detach tracking, cargo:rustc-link-arg=-Wl,-rpath on non-Windows, and dropping bindgen static_flag(true))
Enable the HDFS copy capability
Implement hdfs_copy() that:
- rejects directory sources (FileUtil.copy would recurse)
- rejects directory destinations (FileUtil.checkDest would rewrite to dst/<srcName>)
- creates missing parent directories (copy_file requires the destination parent to exist)
- deletes an existing destination file first; hdfsCopy has also been verified to overwrite natively
- runs copy_file on spawn_blocking so the data-path JNI copy does not block the async worker
Wire Service::copy to oio::OneShotCopier::new_with so temporary IO errors remain retryable
Mark copy as supported in service docs

Are there any user-facing changes?

Yes. HDFS operators that previously got Unsupported from copy can now copy files. Capability discovery will report copy: true.

Breaking changes

AI Usage Statement

Harness: Cursor
Model: Cursor Grok 4.6
Effort: default
Role: Drafted the HDFS copy implementation, updated it to use hdrs 0.3.3 copy_file, and applied review follow-ups. Behavior tests against a live HDFS cluster were not run on this machine.

Enable HDFS copy via OneShotCopier by streaming bytes through hdrs AsyncFile APIs, creating parent directories and overwriting existing destination files when needed.

Bump hdrs to 0.3.3 and replace the AsyncFile streaming copy with native Client::copy_file.

Move hdrs copy_file onto spawn_blocking, drop the pre-delete now that hdfsCopy overwrites, and use OneShotCopier::new_with so temporary IO errors can be retried.

Restore remove_file for existing copy destinations and document that hdfsCopy has been verified to overwrite natively.

hfutatzhanghb · 2026-09-23T02:24:40Z

Hi, @Xuanwo @erickguan . Have self-reviewed by different coding agents. Could you please review another time when have free time? Thanks very much!!!

erickguan

Please try to understand your code or your agentic code pipeline. Above all, what you want to achieve for the project besides code.

Supporting copy with hdfs, solid use case. But there are some issues I don't know what you want to achieve:

Do you want copy atomicity? Does HDFS support it?
Linking another crate's code for explanation is okay. Though I suggest documenting critical assumptions and decisions.

erickguan · 2026-09-23T14:13:39Z

+
+    pub async fn hdfs_copy(&self, from: &str, to: &str) -> Result<Metadata> {
+        let from_path = build_rooted_abs_path(&self.root, from);
+        // FileUtil.copy recurses when the source is a directory.


What does this mean?

OpenDAL defines this operation as file-to-file copy. Hadoop FileUtil.copy also
accepts directory sources and recursively copies their contents, which would
produce behavior outside OpenDAL's copy contract.

I will rewrite the comment to describe the OpenDAL constraint directly:

// OpenDAL copy is file-to-file only. Reject directory sources before
// Hadoop can recursively copy their contents.

erickguan · 2026-09-23T14:21:31Z

+        let to_path = build_rooted_abs_path(&self.root, to);
+        match self.client.metadata(&to_path) {
+            Ok(meta) => {
+                // FileUtil.checkDest rewrites a directory destination to dst/<srcName>


Why do we mention FileUtil.checkDest here?

We do not need to mention the internal FileUtil.checkDest implementation here.

The intended constraint is that OpenDAL treats to as the exact destination
file path. If to is an existing directory, Hadoop may copy the source under
that directory instead, while OpenDAL should return IsADirectory.

I will rewrite the comment as:

// OpenDAL treats to as the exact destination file path. Reject an
// existing directory instead of copying the source into it.

erickguan · 2026-09-23T14:22:51Z

+                // hdfsCopy has been verified to overwrite natively via
+                // FileUtil.copy(..., overwrite=true).
+                self.client
+                    .remove_file(&to_path)
+                    .map_err(new_std_io_error)?;
+            }


This comment really contradicts the code.

You are right. The comment is incorrect and contradicts the implementation.

The hdrs copy_file binding calls hdfsCopy, and this path does not pass an
overwrite=true argument. The existing destination is removed explicitly to
implement OpenDAL's default overwrite semantics.

I will remove the incorrect claim and document the actual behavior:

// hdfsCopy does not replace an existing destination, so remove the
// destination first to implement OpenDAL's overwrite semantics.
//
// This replacement is not atomic. If the copy fails after removal, the
// destination may be missing or incomplete.

hfutatzhanghb · 2026-09-23T16:01:23Z

Please try to understand your code or your agentic code pipeline. Above all, what you want to achieve for the project besides code.

Supporting copy with hdfs, solid use case. But there are some issues I don't know what you want to achieve:

Do you want copy atomicity? Does HDFS support it?

Linking another crate's code for explanation is okay. Though I suggest documenting critical assumptions and decisions.

Thanks for the guidance.

The motivating use case is Lance's opt-in native copy path for immutable data
and index files stored in the same HDFS filesystem. Lance copies files to new,
unpublished paths and validates the destination size before publishing the new
metadata. Dataset visibility and concurrent commit safety are handled separately
by an atomic rename-if-not-exists of the manifest.

Therefore, this copy operation does not require atomicity. HDFS hdfsCopy /
FileUtil.copy does not provide atomic copy semantics either. In particular,
implementing OpenDAL's overwrite behavior by removing an existing destination
before copying is a multi-step operation: if the copy fails, the destination
may be missing or incomplete. I will document this limitation explicitly.

The existing-destination removal is needed to implement OpenDAL's default copy
contract, where the destination is overwritten. It is not intended to provide
atomic replacement.

I will also rewrite the comments around directory handling in terms of OpenDAL's
contract instead of FileUtil implementation details, and remove the incorrect
claim that hdfsCopy is called with overwrite=true.

erickguan

Nice, thank you!

hfutatzhanghb · 2026-09-24T05:42:28Z

Nice, thank you!

Thanks for your very valuable suggestions!

erickguan · 2026-09-24T06:22:00Z

Happy to help.

hfutatzhanghb requested a review from Xuanwo as a code owner September 22, 2026 04:04

github-actions Bot added releases-note/feat The PR implements a new feature or has a title that begins with "feat" services/hdfs size:M This PR changes 30-99 lines, ignoring generated files. labels Sep 22, 2026

hfutatzhanghb marked this pull request as draft September 22, 2026 05:49

hfutatzhanghb marked this pull request as ready for review September 22, 2026 10:26

zhanghaobo@kanzhun.com added 4 commits September 22, 2026 21:18

feat(services/hdfs): add copy support

8100b0c

Enable HDFS copy via OneShotCopier by streaming bytes through hdrs AsyncFile APIs, creating parent directories and overwriting existing destination files when needed.

feat(services/hdfs): copy via hdrs copy_file

2751e9c

Bump hdrs to 0.3.3 and replace the AsyncFile streaming copy with native Client::copy_file.

fix(services/hdfs): run copy_file off the async worker

124a383

Move hdrs copy_file onto spawn_blocking, drop the pre-delete now that hdfsCopy overwrites, and use OneShotCopier::new_with so temporary IO errors can be retried.

fix(services/hdfs): keep destination delete before copy

0327d2f

Restore remove_file for existing copy destinations and document that hdfsCopy has been verified to overwrite natively.

hfutatzhanghb force-pushed the feat/hdfs-copy-support branch from eaaa7d9 to 0327d2f Compare September 22, 2026 13:18

erickguan reviewed Sep 23, 2026

View reviewed changes

docs(services/hdfs): clarify copy semantics

8c6648d

github-actions Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Sep 23, 2026

ci: retrigger checks

afc9ff2

erickguan approved these changes Sep 24, 2026

View reviewed changes

erickguan merged commit 53610f8 into apache:main Sep 24, 2026
382 checks passed

Navigation Menu

feat(services/hdfs): add copy support - #8332

feat(services/hdfs): add copy support#8332
erickguan merged 6 commits into
apache:mainfrom
hfutatzhanghb:feat/hdfs-copy-support

hfutatzhanghb commented Sep 22, 2026 •

edited

Loading

Uh oh!

hfutatzhanghb commented Sep 23, 2026

Uh oh!

erickguan left a comment

Uh oh!

Uh oh!

erickguan Sep 23, 2026

Uh oh!

hfutatzhanghb Sep 23, 2026

Uh oh!

erickguan Sep 23, 2026

Uh oh!

hfutatzhanghb Sep 23, 2026

Uh oh!

erickguan Sep 23, 2026

Uh oh!

hfutatzhanghb Sep 23, 2026

Uh oh!

hfutatzhanghb commented Sep 23, 2026

Uh oh!

erickguan left a comment

Uh oh!

hfutatzhanghb commented Sep 24, 2026

Uh oh!

Uh oh!

erickguan commented Sep 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

hfutatzhanghb commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Breaking changes

AI Usage Statement

Uh oh!

hfutatzhanghb commented Sep 23, 2026

Uh oh!

erickguan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erickguan Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

erickguan Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

erickguan Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb commented Sep 23, 2026

Uh oh!

erickguan left a comment

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb commented Sep 24, 2026

Uh oh!

Uh oh!

erickguan commented Sep 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hfutatzhanghb commented Sep 22, 2026 •

edited

Loading