Skip to content

Altertable v2.0 cyanoptera - #3

Draft
redox wants to merge 174 commits into
mainfrom
altertable-v2.0-cyanoptera
Draft

redox wants to merge 174 commits into
mainfrom
altertable-v2.0-cyanoptera

Conversation

@redox

@redox redox commented Nov 5, 2026

Copy link
Copy Markdown
Member

Tracking PR for our altertable-v2.0-cyanoptera branch, kept as a draft so its CI shows the branch status. It is not meant to be merged.

🤖 Generated with Claude Code

jatorre and others added 30 commits April 25, 2026 09:00
DuckDB 1.5 introduced CRS metadata on the GEOMETRY type (e.g.,
GEOMETRY('EPSG:4326')). This change makes that CRS survive a full
round trip through PostgreSQL/PostGIS via postgres_scanner.

Write side (DuckDB → PostGIS):
- WriteGeometry detects CRS on the LogicalType and writes EWKB
  instead of plain WKB so PostGIS receives the SRID:
    WKB:  [byte_order:1] [type:4]                    [payload...]
    EWKB: [byte_order:1] [type|0x20000000:4] [srid:4] [payload...]
  Falls back to plain WKB when no CRS is set, preserving the
  previous behavior. (Uses GeoType::HasCRS / GeoType::GetCRS — the
  earlier draft mistakenly referenced LogicalType::GetCRS, which
  does not exist on DuckDB 1.5.)

Read side (PostGIS → DuckDB):
- TypeToLogicalType decodes the SRID from PostGIS's column-level
  typmod for `geometry(TYPE, SRID)` columns and returns
  LogicalType::GEOMETRY("EPSG:N") instead of plain GEOMETRY().
  The typmod layout is:
      bits  0    : has-M
      bits  1    : has-Z
      bits  2-7  : geometry type code (POINT=1, LINESTRING=2, ...)
      bits  8-29 : SRID
  Untyped `geometry` columns carry typmod -1 and fall back to plain
  GEOMETRY (no CRS), matching the previous behavior.

Before: DuckDB GEOMETRY('EPSG:4326') → PostGIS geometry(SRID=0)
        PostGIS geometry(POINT,4326)  → DuckDB GEOMETRY (no CRS)
After:  DuckDB GEOMETRY('EPSG:4326') → PostGIS geometry(SRID=4326)
        PostGIS geometry(POINT,4326)  → DuckDB GEOMETRY('EPSG:4326')

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added the EWKB write path and typmod read decode,
but on CREATE TABLE AS the CRS was being stripped before reaching
either side, so the resulting PostGIS column landed as plain
`geometry` (typmod -1) and the read-back lost the SRID.

Two strip points in this file:

  1. ToPostgresType (called by AddCastToPostgresTypes during CTAS
     planning) returned LogicalType::GEOMETRY() unconditionally,
     dropping CRS before the projection that feeds the new physical
     plan.  Now returns the input unchanged for GEOMETRY so CRS
     survives the cast/projection.

  2. TypeToString returned plain "GEOMETRY" for any GEOMETRY type and
     would have been short-circuited anyway by the HasAlias() early
     return (GEOMETRY('EPSG:N') carries an alias of "GEOMETRY").  The
     new branch runs *before* HasAlias, detects CRS via
     GeoType::HasCRS, and emits `geometry(Geometry, N)` so PostGIS
     records the SRID in pg_attribute.atttypmod and the existing
     read-side typmod decode picks it up.

Verified end-to-end through the streamability harness in
duckdb-warehouse-transfer: postgres -> oracle and postgres -> snowflake
now round-trip GEOMETRY('EPSG:3857') with byte-perfect WKB and CRS
preserved at the destination, with no plan-level materialization.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix error message for Postgres connection failure
first attempt to enable use_information_schema for ATTACH
…alog-load-race

fix: catalog cache loading race
This is a follow-up to duckdb#476, it extends the `VARCHAR` fallback logic for
types, that are defined in a schema that is different from the attached
schema.

Testing: new test added that covers inaccessible schema type.

Ref: duckdb#497
…alogset-scan-uaf

fix: prevent catalog scan use-after-free
…exes-constraint-type-assertion

fix: avoid crash by setting default index constraint type
This PR extends the `SCHEMA` option for `ATTACH` allowing it
additionally to take a list of schemas:

```sql
ATTACH '...' AS p (TYPE POSTGRES, SCHEMA ['s1', 's2', 's3'])
```

Testing: new test added that covers visibility for multiple schemas.

Closes: duckdb#498
CONNECT on Redshift failed unless the user manually did
SET pg_use_text_protocol=true, while ATTACH + USE worked.

DuckDB rewrites a CONNECT passthrough statement into postgres_query()
(PostgresCatalog::RemoteExecute). Its bind, PGQueryBind, never called
PostgresScanFunction::PrepareBind - which is where every server capability
decision is made, including the Redshift opt-out from the binary COPY
protocol. As a result the CONNECT path always wrapped the query in
COPY (...) TO STDOUT (FORMAT "binary"), which Redshift does not support.

Instead of duplicating the Redshift check on each bind path, make the
protocol a property of the attached database:

- add a use_text_protocol ATTACH option, accepting TRUE / FALSE / 'auto'
  (the default), stored on the PostgresCatalog
- PostgresCatalog::UseTextProtocol() is now the single place resolving the
  protocol for a database: an explicit option wins, 'auto' falls back to the
  pg_use_text_protocol setting or to Redshift auto-detection
- PGQueryBind calls PrepareBind, so the CONNECT path resolves it as well

Also resolve the protocol at the start of PrepareBind: it was set after the
ctid scan and max_threads decisions that depend on it, which happened to be
harmless only because Redshift reports major version 8.
Apply the text protocol opt-out on the postgres_query (CONNECT) path
Since duckdb/duckdb#23539:
```sql
  SELECT ARRAY['a', NULL] = ARRAY['a', NULL];
  -- Postgres:                  t     (true)
  -- DuckDB, post-#23539:       NULL
```

so removing the NULL carrying value
…alog-cache-staleness

fix: detect Postgres table catalog staleness
staticlibs and others added 30 commits October 15, 2026 07:37
Put the default application_name into URI connection strings instead of in front of them
The query passed to `postgres_query()` function was being wrapped into
a subquery with an explicit list of columns. This wrapping is necessary
in some cases when a Postgres-side `CAST` is required for particular
column. But it can be omitted in most of the usual cases.

This PR enables the wrapping of the user query only when necessry.

Ref: duckdblabs/duckdb-internal#11178
Queries that return columns with same names currently cannot be run
through `postgres_query()`, because DuckDB requires table functions to
return distinct column names:

```sql
postgres=# SELECT 41, 42;
 ?column? | ?column?
----------+----------
       41 |       42
(1 row)
```

This PR adds deduplication to the `postgres_query()` column names, the
example above is changed to: `?column?`, `?column?_1`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR changes the RDS IAM secrets creation by resolving the template
on `ATTACH` time before `SECRET_STORAGE_TABLE` is registered.

Fixes: duckdb#633
Use fallback_application_name for the default DuckDB application name
When the `postgres_query()` call is being bound, the specified query is
prepared in remote Postgres database using `PQprepare`. This PR
implements support for reusing that prepared statement when it is
possible:

 - when the query is not modified before the execution in Postgres:
   - `COPY` is not used - `query_mode=PLAIN_QUERY_BINARY` or
   `query_mode=PLAIN_QUERY_TEXT` is used, or query parameters are used
   - resulting columns are not reorderd
   - no resulting columns require Postrges-side cast to `VARCHAR`
   (required for arrays)

Additionally the usage of `shred_ptr<OwnedPostgresConnection>` is
removed completely, as this usage was considerably complicating the
connection lifetime logic in conjunction with a connection pool.
This PR impelements the extension-side support for the
`RemotePushdownOptimizer` (duckdb/duckdb#22914, duckdb/duckdb#23135).

Queries and statements that only reference tables of a single attached
Postgres database are executed in Postgres as a whole (instead of
scanning the tables and executing the query in DuckDB), when every
construct they use can be translated to Postgres SQL with the same
semantics. This covers e.g. joins, aggregates, window functions, CTEs,
set operations, subqueries, many scalar functions, `UPDATE` / `DELETE`
/ `INSERT` (including `RETURNING` and `ON CONFLICT`),
`CREATE TABLE ... AS`, `CREATE VIEW` and most `DROP` / `ALTER TABLE`
statements.

A pushed down query is executed as a single Postgres query over a single
connection. Queries that are plain scans of a single table are instead
executed through the regular table scan, which reads the table in
parallel over multiple connections (by ranges of the `ctid`). A query
is a plain scan when it reads a single table, selects only columns
(or `*`), filters only with comparisons of columns with constants,
`IN` lists and `IS [NOT] NULL` combined with `AND` / `OR`, and has no
`ORDER BY` or `OFFSET` (a `LIMIT` is fine). For example, this query is
still executed with a parallel scan:

```sql
SELECT id, customer, amount FROM postgres_db.orders WHERE status = 'open' AND amount > 100 LIMIT 10;
```

Any other construct makes the query a candidate for pushdown, including
seemingly simple ones such as a computed column, a `LIKE` filter or an
`ORDER BY`.

Remote pushdown is enabled by default. It can be disabled for newly
attached databases (the setting is only read at `ATTACH` time):

```sql
SET pg_enable_remote_pushdown = FALSE;
```
Redshift supports special kind of views created with:

```sql
WITH NO SCHEMA BINDING
```

This PR adds support for correctly listing these views by fetching
them from `pg_get_late_binding_view_cols()`.

There are no changes to ordinary Postgres logic.

Fixes: duckdb#641
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v8 calls apt-key, which the image no longer ships.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
action-setup-postgres v9 no longer installs Postgres 14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR improves support for the following Redshift types

 - `GEOGRAPHY`
 - `GEOMETRY`
 - `HLLSKETCH`
 - `SUPER`
 - `VARBYTE`
 - `BLOB`
 - `INTERVAL`

Logic for ordinary Postgres types is kept the same.
Pass query parameters to statements without result columns
This PR enables the binary protocol (`query_mode=PLAIN_QUERY_BINARY`)
by default with Redshift servers. Binary protocol has less overhead,
it supports most of Redshift resulting columns (with `HLLSKETCH`
being an exception). Protocol is changed to text one automatically when
`HLLSKETCH` column is present.

There are no changes for ordinary Postgres servers.
… altertable-v2.0-cyanoptera

# Conflicts:
#	.github/workflows/IntegrationTests.yml
#	duckdb
…tera

Track our DuckDB fork's altertable-v2.0-cyanoptera branch and pin it to
03bbf5a, so this fork builds and tests against the DuckDB we ship.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LogicalExtensionOperator::GetExtensionName() throws by default. Optimizer
hooks of other extensions (match_recognize) call it on every extension
operator in a plan, so CREATE INDEX on an attached Postgres database
failed when one was loaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ension-name

Name the extension of the CREATE INDEX operator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.