Skip to content

Observability: pxf_stat_activity and pxf_cancel_backend - #124

Draft
ostinru wants to merge 3 commits into
apache:mainfrom
ostinru:pxf_stat_activity
Draft

Observability: pxf_stat_activity and pxf_cancel_backend#124
ostinru wants to merge 3 commits into
apache:mainfrom
ostinru:pxf_stat_activity

Conversation

@ostinru

@ostinru ostinru commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Add pxf_stat_activity

Add pxf_stat_activity to help DBAs find out what happening inside PXF server.

postgres=# select * from pxf_stat_activity ;
-[ RECORD 1 ]-+---------------------------------------------------------
segment_id    | 1
session_id    | 57997
command_count | 8
xid           | 57997-0000000008
operation     | READ_BRIDGE
usename       | gpadmin
server        | ostinru-test
profile       | s3:parquet
schema_name   | public
table_name    | test_blob_read
data_source   | ostinru-test/topics/cuscom.email.created.v1/2026-05-29/*
query_start   | 2026-06-23 16:16:48.318+03
pxf_host      | example.com

Add pxf_cancel_backend and pxf_interrupt_backend

Add pxf_cancel_backend and pxf_interrupt_backend to allow killing running queries.

Update tests

  1. Update pxfExtension tests - use testconatiners.
  2. Explicitly define extension version. So, there is no need in sql file manipulations.

@ostinru
ostinru requested a review from leborchuk July 10, 2026 18:35
@ostinru

ostinru commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

Moved test-related code to separate PR #125 in order to keep PR focused.

Comment thread fdw/pxf_stat_activity.c
elog(DEBUG2, "pxf_stat_activity: segment %d requesting %s",
GpIdentity.segindex, uri.data);

handle = churl_init_download(uri.data, headers);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we check here that there are no curl errors and everything was smoothly inited?

hawq code https://github.com/apache/hawq/blob/master/src/backend/access/external/pxfutils.c#L122C3-L122C57 contains call churl_read_check_connectivity(client_context->handle); I suggest do the similar check

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. We do check churl_read_check_connectivity(handle) after consuming the response body, but I agree it is better to keep the same pattern as PxfBridgeImportStart() and validate connectivity immediately after churl_init_download().

@ostinru
ostinru force-pushed the pxf_stat_activity branch from 24eb226 to 691ac21 Compare July 13, 2026 21:41
ostinru added 2 commits August 5, 2026 17:24
# Conflicts:
#	.github/workflows/pxf-ci.yml
#	automation/src/test/java/org/apache/cloudberry/pxf/automation/features/extension/PxfExtensionTest.java
#	automation/src/test/java/org/apache/cloudberry/pxf/automation/features/extension/PxfFdwExtensionTest.java
#	ci/docker/pxf-cbdb-dev/common/script/run_tests.sh
@ostinru
ostinru force-pushed the pxf_stat_activity branch from 691ac21 to fe63f6b Compare August 5, 2026 09:24
@ostinru

ostinru commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

I have tests this PR in order to detect any password leakage to pxf_stat_activity:

db2=#  CREATE EXTERNAL TABLE ext_big_table_100gb (...)
  LOCATION (
      'pxf://public.big_table_100gb?PROFILE=Jdbc&JDBC_DRIVER=org.postgresql.Driver&DB_URL=jdbc:postgresql://pg.example.com:5432/db2&USER=ostinru&PASS=SECRET'
  )
  FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import');


postgres=# select * from pxf_stat_activity ;
-[ RECORD 1 ]-+---------------------------------------------------
segment_id    | 0
session_id    | 112562
command_count | 72
xid           | 112562-0000000072
operation     | READ_BRIDGE
usename       | gpadmin
server        | default
profile       | jdbc
schema_name   | public
table_name    | ext_big_table_100gb
data_source   | public.big_table_100gb
query_start   | 2026-07-31 20:01:50.385+03
pxf_host      | gp.example.com

Also, secret in jdbc URL:

pxf://big_table_100gb?PROFILE=Jdbc&JDBC_DRIVER=com.mysql.jdbc.Driver&DB_URL=jdbc:mysql://ostinru:SECRET@mysql.example.com:3306/db1

postgres=# select * from pxf_stat_activity ;
-[ RECORD 1 ]-+---------------------------------------------------
segment_id    | 1
session_id    | 153617
command_count | 16
xid           | 153617-0000000016
operation     | READ_BRIDGE
usename       | gpadmin
server        | default
profile       | jdbc
schema_name   | public
table_name    | ext_mysql_big_table_100gb
data_source   | big_table_100gb
query_start   | 2026-07-31 21:34:39.361+03
pxf_host      | gp.example.com

with S3:

CREATE EXTERNAL TABLE test_blob_read_with_password (...)
  LOCATION (
      'pxf://ostinru/topics/cuscom.email.created.v1/2026-05-29/*?PROFILE=s3:parquet&SERVER=ostinru&accesskey=SECRET&secretkey=SECRET'
  )
  FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import');

postgres=# select * from pxf_stat_activity ;
-[ RECORD 1 ]-+---------------------------------------------------------
segment_id    | 1
session_id    | 153617
command_count | 99
xid           | 153617-0000000099
operation     | READ_BRIDGE
usename       | gpadmin
server        | ostinru
profile       | s3:parquet
schema_name   | public
table_name    | test_blob_read_with_password
data_source   | ostinru/topics/cuscom.email.created.v1/2026-05-29/*
query_start   | 2026-07-31 21:48:03.509+03
pxf_host      | gp.example.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants