Skip to content

fix(gdal): rename sidecar files when overwriting a GDAL dataset - #874

Merged
Maxxen merged 1 commit into
duckdb:v1.5-variegatafrom
dajiaohuang:fix/859-gdal-copy-sidecar-tmp-files
Sep 22, 2026
Merged

Maxxen merged 1 commit into
duckdb:v1.5-variegatafrom
dajiaohuang:fix/859-gdal-copy-sidecar-tmp-files

Conversation

@dajiaohuang

Copy link
Copy Markdown

Fixes #859.

Problem

Copying over an existing file writes to a tmp_* path, and only the main file is renamed back once the copy is finalized. Drivers that spread one dataset over several files therefore leave the remaining files behind under their temporary name, while the new main file keeps being paired with the leftovers of the previous export:

COPY shapes TO 'data.shp' WITH (FORMAT GDAL, DRIVER 'ESRI Shapefile');
DELETE FROM shapes WHERE id = 2;
COPY shapes TO 'data.shp' WITH (FORMAT GDAL, DRIVER 'ESRI Shapefile');
-- data.cpg data.dbf data.shp data.shx tmp_data.cpg tmp_data.dbf tmp_data.shx

The re-exported dataset is then read through the sidecars of the previous export. On DuckDB 1.5.5 the one-row export above still returns two rows, and other shapes of the same mismatch fail with IO Error: Error in fread() reading object ... from .shp file.

Fix

The GDAL copy function now records the path the dataset was created at, looks up the files belonging to that dataset while it is still open, and renames the ones still carrying the temporary name alongside the main file. The main file is still renamed by DuckDB itself, and components a driver reports without having created them are skipped, so single-file drivers are unaffected.

Validation

Reproduced on DuckDB 1.5.5 (Windows, x86_64) with the shipped spatial build: before the change the two-step export leaves tmp_overwrite.dbf and tmp_overwrite.shx behind and ST_Read returns 2 rows. test/sql/gdal/st_write_overwrite.test asserts both the absence of leftover temporary files and the correct row count.

I could not compile the extension locally (no CMake/vcpkg/GDAL toolchain on this machine), so that test has only been run against the unpatched build, where it fails on both assertions; the source change itself has not been compiled here.

Copying over an existing file writes to a temporary "tmp_*" path, and only
the main file is renamed back once the copy is finalized. Drivers that spread
a single dataset over several files (such as the .shx, .dbf and .prj sidecars
of an ESRI Shapefile) therefore left those files behind under their temporary
name, while the new main file kept being paired with the sidecars of the
previous export. Rename them together with the main file.
@Maxxen
Maxxen self-requested a review September 22, 2026 13:25
@Maxxen

Maxxen commented Sep 22, 2026

Copy link
Copy Markdown
Member

Thanks!

@Maxxen
Maxxen merged commit 04270fe into duckdb:v1.5-variegata Sep 22, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Overwriting a Shapefile with GDAL COPY leaves tmp sidecars and corrupts the dataset

2 participants