Repository navigation
Conversation
DuckDB 1.5 introduced CRS metadata on the GEOMETRY type (e.g.,
GEOMETRY('EPSG:4326')). This change makes that CRS survive a full
round trip through PostgreSQL/PostGIS via postgres_scanner.
Write side (DuckDB → PostGIS):
- WriteGeometry detects CRS on the LogicalType and writes EWKB
instead of plain WKB so PostGIS receives the SRID:
WKB: [byte_order:1] [type:4] [payload...]
EWKB: [byte_order:1] [type|0x20000000:4] [srid:4] [payload...]
Falls back to plain WKB when no CRS is set, preserving the
previous behavior. (Uses GeoType::HasCRS / GeoType::GetCRS — the
earlier draft mistakenly referenced LogicalType::GetCRS, which
does not exist on DuckDB 1.5.)
Read side (PostGIS → DuckDB):
- TypeToLogicalType decodes the SRID from PostGIS's column-level
typmod for `geometry(TYPE, SRID)` columns and returns
LogicalType::GEOMETRY("EPSG:N") instead of plain GEOMETRY().
The typmod layout is:
bits 0 : has-M
bits 1 : has-Z
bits 2-7 : geometry type code (POINT=1, LINESTRING=2, ...)
bits 8-29 : SRID
Untyped `geometry` columns carry typmod -1 and fall back to plain
GEOMETRY (no CRS), matching the previous behavior.
Before: DuckDB GEOMETRY('EPSG:4326') → PostGIS geometry(SRID=0)
PostGIS geometry(POINT,4326) → DuckDB GEOMETRY (no CRS)
After: DuckDB GEOMETRY('EPSG:4326') → PostGIS geometry(SRID=4326)
PostGIS geometry(POINT,4326) → DuckDB GEOMETRY('EPSG:4326')
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added the EWKB write path and typmod read decode,
but on CREATE TABLE AS the CRS was being stripped before reaching
either side, so the resulting PostGIS column landed as plain
`geometry` (typmod -1) and the read-back lost the SRID.
Two strip points in this file:
1. ToPostgresType (called by AddCastToPostgresTypes during CTAS
planning) returned LogicalType::GEOMETRY() unconditionally,
dropping CRS before the projection that feeds the new physical
plan. Now returns the input unchanged for GEOMETRY so CRS
survives the cast/projection.
2. TypeToString returned plain "GEOMETRY" for any GEOMETRY type and
would have been short-circuited anyway by the HasAlias() early
return (GEOMETRY('EPSG:N') carries an alias of "GEOMETRY"). The
new branch runs *before* HasAlias, detects CRS via
GeoType::HasCRS, and emits `geometry(Geometry, N)` so PostGIS
records the SRID in pg_attribute.atttypmod and the existing
read-side typmod decode picks it up.
Verified end-to-end through the streamability harness in
duckdb-warehouse-transfer: postgres -> oracle and postgres -> snowflake
now round-trip GEOMETRY('EPSG:3857') with byte-perfect WKB and CRS
preserved at the destination, with no plan-level materialization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix error message for Postgres connection failure
first attempt to enable use_information_schema for ATTACH
…alog-load-race fix: catalog cache loading race
This is a follow-up to duckdb#476, it extends the `VARCHAR` fallback logic for types, that are defined in a schema that is different from the attached schema. Testing: new test added that covers inaccessible schema type. Ref: duckdb#497
…alogset-scan-uaf fix: prevent catalog scan use-after-free
…exes-constraint-type-assertion fix: avoid crash by setting default index constraint type
This PR extends the `SCHEMA` option for `ATTACH` allowing it additionally to take a list of schemas: ```sql ATTACH '...' AS p (TYPE POSTGRES, SCHEMA ['s1', 's2', 's3']) ``` Testing: new test added that covers visibility for multiple schemas. Closes: duckdb#498
…-return-type.patch
CONNECT on Redshift failed unless the user manually did SET pg_use_text_protocol=true, while ATTACH + USE worked. DuckDB rewrites a CONNECT passthrough statement into postgres_query() (PostgresCatalog::RemoteExecute). Its bind, PGQueryBind, never called PostgresScanFunction::PrepareBind - which is where every server capability decision is made, including the Redshift opt-out from the binary COPY protocol. As a result the CONNECT path always wrapped the query in COPY (...) TO STDOUT (FORMAT "binary"), which Redshift does not support. Instead of duplicating the Redshift check on each bind path, make the protocol a property of the attached database: - add a use_text_protocol ATTACH option, accepting TRUE / FALSE / 'auto' (the default), stored on the PostgresCatalog - PostgresCatalog::UseTextProtocol() is now the single place resolving the protocol for a database: an explicit option wins, 'auto' falls back to the pg_use_text_protocol setting or to Redshift auto-detection - PGQueryBind calls PrepareBind, so the CONNECT path resolves it as well Also resolve the protocol at the start of PrepareBind: it was set after the ctid scan and max_threads decisions that depend on it, which happened to be harmless only because Redshift reports major version 8.
Apply the text protocol opt-out on the postgres_query (CONNECT) path
Since duckdb/duckdb#23539: ```sql SELECT ARRAY['a', NULL] = ARRAY['a', NULL]; -- Postgres: t (true) -- DuckDB, post-#23539: NULL ``` so removing the NULL carrying value
…alog-cache-staleness fix: detect Postgres table catalog staleness
Put the default application_name into URI connection strings instead of in front of them
The query passed to `postgres_query()` function was being wrapped into a subquery with an explicit list of columns. This wrapping is necessary in some cases when a Postgres-side `CAST` is required for particular column. But it can be omitted in most of the usual cases. This PR enables the wrapping of the user query only when necessry. Ref: duckdblabs/duckdb-internal#11178
Queries that return columns with same names currently cannot be run
through `postgres_query()`, because DuckDB requires table functions to
return distinct column names:
```sql
postgres=# SELECT 41, 42;
?column? | ?column?
----------+----------
41 | 42
(1 row)
```
This PR adds deduplication to the `postgres_query()` column names, the
example above is changed to: `?column?`, `?column?_1`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR changes the RDS IAM secrets creation by resolving the template on `ATTACH` time before `SECRET_STORAGE_TABLE` is registered. Fixes: duckdb#633
Use fallback_application_name for the default DuckDB application name
When the `postgres_query()` call is being bound, the specified query is prepared in remote Postgres database using `PQprepare`. This PR implements support for reusing that prepared statement when it is possible: - when the query is not modified before the execution in Postgres: - `COPY` is not used - `query_mode=PLAIN_QUERY_BINARY` or `query_mode=PLAIN_QUERY_TEXT` is used, or query parameters are used - resulting columns are not reorderd - no resulting columns require Postrges-side cast to `VARCHAR` (required for arrays) Additionally the usage of `shred_ptr<OwnedPostgresConnection>` is removed completely, as this usage was considerably complicating the connection lifetime logic in conjunction with a connection pool.
This PR impelements the extension-side support for the `RemotePushdownOptimizer` (duckdb/duckdb#22914, duckdb/duckdb#23135). Queries and statements that only reference tables of a single attached Postgres database are executed in Postgres as a whole (instead of scanning the tables and executing the query in DuckDB), when every construct they use can be translated to Postgres SQL with the same semantics. This covers e.g. joins, aggregates, window functions, CTEs, set operations, subqueries, many scalar functions, `UPDATE` / `DELETE` / `INSERT` (including `RETURNING` and `ON CONFLICT`), `CREATE TABLE ... AS`, `CREATE VIEW` and most `DROP` / `ALTER TABLE` statements. A pushed down query is executed as a single Postgres query over a single connection. Queries that are plain scans of a single table are instead executed through the regular table scan, which reads the table in parallel over multiple connections (by ranges of the `ctid`). A query is a plain scan when it reads a single table, selects only columns (or `*`), filters only with comparisons of columns with constants, `IN` lists and `IS [NOT] NULL` combined with `AND` / `OR`, and has no `ORDER BY` or `OFFSET` (a `LIMIT` is fine). For example, this query is still executed with a parallel scan: ```sql SELECT id, customer, amount FROM postgres_db.orders WHERE status = 'open' AND amount > 100 LIMIT 10; ``` Any other construct makes the query a candidate for pushdown, including seemingly simple ones such as a computed column, a `LIKE` filter or an `ORDER BY`. Remote pushdown is enabled by default. It can be disabled for newly attached databases (the setting is only read at `ATTACH` time): ```sql SET pg_enable_remote_pushdown = FALSE; ```
Redshift supports special kind of views created with: ```sql WITH NO SCHEMA BINDING ``` This PR adds support for correctly listing these views by fetching them from `pg_get_late_binding_view_cols()`. There are no changes to ordinary Postgres logic. Fixes: duckdb#641
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v8 calls apt-key, which the image no longer ships. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
action-setup-postgres v9 no longer installs Postgres 14. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR improves support for the following Redshift types - `GEOGRAPHY` - `GEOMETRY` - `HLLSKETCH` - `SUPER` - `VARBYTE` - `BLOB` - `INTERVAL` Logic for ordinary Postgres types is kept the same.
Pass query parameters to statements without result columns
This PR enables the binary protocol (`query_mode=PLAIN_QUERY_BINARY`) by default with Redshift servers. Binary protocol has less overhead, it supports most of Redshift resulting columns (with `HLLSKETCH` being an exception). Protocol is changed to text one automatically when `HLLSKETCH` column is present. There are no changes for ordinary Postgres servers.
… altertable-v2.0-cyanoptera # Conflicts: # .github/workflows/IntegrationTests.yml # duckdb
…tera Track our DuckDB fork's altertable-v2.0-cyanoptera branch and pin it to 03bbf5a, so this fork builds and tests against the DuckDB we ship. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LogicalExtensionOperator::GetExtensionName() throws by default. Optimizer hooks of other extensions (match_recognize) call it on every extension operator in a plan, so CREATE INDEX on an attached Postgres database failed when one was loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ension-name Name the extension of the CREATE INDEX operator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking PR for our
altertable-v2.0-cyanopterabranch, kept as a draft so its CI shows the branch status. It is not meant to be merged.🤖 Generated with Claude Code