Core, Spark: Clean up uncommitted files when a staged table is aborted

[wombatu-kun]

Summary

StagedSparkTable.abortStagedChanges() was an empty // TODO: clean up. Spark calls it to roll back an atomic CTAS/RTAS, and it is also used by the snapshot/migrate actions. Iceberg's internal transaction cleanup only runs when commitTransaction() is actually invoked and then fails. When a failure happens after the staged write but before commitStagedChanges() is called (for example, SnapshotTableSparkAction/MigrateTableSparkAction failing while importing data, or an interrupted CTAS), nothing cleaned the manifest list and manifests already written into the uncommitted transaction. For a staged CREATE the table is never registered, so those orphans have no table metadata pointing at them and are unreachable by removeOrphanFiles - they leak permanently.

Changes

• Add a best-effort default void abortTransaction() to the Transaction API. The default is a documented no-op, so existing implementations are unaffected.
• Override it in BaseTransaction to run the existing cleanUp() (cleanAllUpdates() + deleteUncommittedFiles()) - the same cleanup Iceberg already performs when a create/replace transaction's own commit fails.
• Delegate it in CommitCallbackTransaction; the post-commit callback is intentionally not run on abort.
• Wire StagedSparkTable.abortStagedChanges() to transaction.abortTransaction() in spark/v4.1. The Spark-side change is identical for 3.5 and 4.0; a backport PR will follow.
• Add the corresponding java.method.addedToInterface entry to .palantir/revapi.yml.

A no-op default (rather than throwing) is used because abortTransaction() runs from catch/finally blocks where a secondary exception would mask the original failure. The underlying deletion is already best-effort and idempotent (CatalogUtil.deleteFile swallows NotFoundException, cleanAllUpdates() suppresses failures), so calling abort after a failed commitStagedChanges() is safe.

Out of scope

Executor-written data files in the write-succeeds-then-commit-fails path are not deleted here. That matches Iceberg's existing create-transaction behavior, and those files are handled by SparkWrite.abort() on write-job failure.

Testing

• New abort case in core TestCreateTransaction (runs across all format-version templates): stages a create transaction, performs an append, asserts the manifest and manifest list exist, calls abortTransaction(), asserts they are removed and the table is not created, and verifies a second abortTransaction() does not throw.
• New TestStagedSparkTable in spark/v4.1: stages a CREATE via SparkCatalog.stageCreate, routes an append into the staged transaction, calls abortStagedChanges(), and asserts the uncommitted manifest/manifest-list files are deleted and the table is not created. The session-catalog parameterization is skipped (it produces RollbackStagedTable, not StagedSparkTable). 3 configs (hive/hadoop/rest) pass, 1 skipped, 0 failures.
• ./gradlew revApiCheck, spotlessApply -DallModules, and the tests above all pass.

---

AI Disclosure

• Model: Claude Opus 4.7
• Platform/Tool: Claude Code
• Human Oversight: fully reviewed
• Prompt Summary: Implement StagedSparkTable.abortStagedChanges, which was an empty "// TODO: clean up", so aborted staged tables clean up their uncommitted files.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[wombatu-kun]

not stale

[wombatu-kun]

@RussellSpitzer @rdblue this has been open since May 18 with no review (it already collected a stale-bot warning). CI is green (56/56) and it still merges cleanly on current main, so it is only waiting on a reviewer.

StagedSparkTable.abortStagedChanges() is an empty // TODO: clean up, so a failed atomic CTAS/RTAS leaks the manifests and manifest list already written into the uncommitted transaction. For a staged CREATE the table is never registered, so removeOrphanFiles cannot reach those files and they leak permanently. The fix adds a best-effort no-op Transaction.abortTransaction() default and wires it up.