Add table-model query support to the Grafana data source plugin#114
Conversation
The Grafana data source plugin only understood the tree model: both
QueryEditor modes build root.* paths and the backend calls
/grafana/v1/query/expression, which has no table-model equivalent, so
table-model users could not visualize their data through the plugin.
This adds a third QueryEditor mode, "SQL: Table Model", that runs standard
SQL against a database and renders the result. It reuses IoTDB's existing
table REST endpoint (POST /rest/table/v1/query) rather than adding a new
server API: the backend sends {database, sql} and turns the row-major
QueryDataSet into a Grafana frame with one typed, nullable field per
column, driven by the response's data_types so column types are exact
instead of guessed. The body is decoded with UseNumber so INT64 values
beyond 2^53 keep their precision.
In the default Time series format, rows are sorted by the first TIMESTAMP
column and a long-shaped result (time + tag columns + value columns) is
pivoted into one labeled series per tag combination, so a multi-device
query renders as separate lines; the Table format preserves the server's
row order. A $__timeFilter(col) / $__timeFrom / $__timeTo macro set
expands to the panel's range, accepting the $__timeFrom() / $__timeTo()
function forms as well. Because the table REST endpoint returns TIMESTAMP
values (and compares integer time literals) in the server's configured
timestamp_precision -- unlike the tree-model /grafana/v1 endpoints, which
normalize to milliseconds server-side -- a new data source "time
precision" option (ms default / us / ns) scales both the macro bounds and
the TIMESTAMP rendering, so the mode also works on us/ns servers.
Follow-up work for the same issue: a database -> table -> column browser,
a visual query builder, a default row limit to bound large SELECTs, and
the Grafana plugin SDK / Go backend modernization.
Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| this.setState({ | ||
| isDropDownList: this.props.query.isDropDownList, | ||
| sqlType: this.props.query.sqlType, | ||
| database: this.props.query.database ?? '', | ||
| sql: this.props.query.sql ?? '', | ||
| format: this.props.query.format ?? tableFormats[0], | ||
| }); |
There was a problem hiding this comment.
Pull request overview
This PR adds table-model SQL query support to the IoTDB Grafana data source plugin by introducing a new “SQL: Table Model” editor mode and a Go backend implementation that queries IoTDB’s /rest/table/v1/query endpoint, with timestamp-precision-aware macro expansion and typed/nullable result frames.
Changes:
- Add a new QueryEditor mode (“SQL: Table Model”) with database + SQL inputs and a result format selector (Time series vs Table).
- Implement a Go backend table-model query path that expands Grafana time macros, parses row-major
QueryDataSetresponses withUseNumber, and builds typed Grafana frames (including optional long-to-wide pivot). - Add a datasource setting for
timestampPrecision(ms/us/ns) and document the new mode/macros/precision behavior in the README.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| connectors/grafana-plugin/src/types.ts | Extends query/options types to carry table-model SQL/database/format and timestamp precision setting. |
| connectors/grafana-plugin/src/QueryEditor.tsx | Adds “SQL: Table Model” UI mode (database + SQL editor + format). |
| connectors/grafana-plugin/src/datasource.ts | Expands template variables for table-model sql and database. |
| connectors/grafana-plugin/src/ConfigEditor.tsx | Adds a “time precision” datasource setting (ms/us/ns). |
| connectors/grafana-plugin/README.md | Documents the new table-model mode, macros, format behavior, and timestamp precision option. |
| connectors/grafana-plugin/pkg/plugin/table_query.go | Implements backend table-model querying, macro expansion, typed frame building, and pivoting. |
| connectors/grafana-plugin/pkg/plugin/table_query_test.go | Adds unit tests covering macro expansion, typing/nulls, precision, sorting, pivoting, and error paths. |
| connectors/grafana-plugin/pkg/plugin/plugin.go | Wires new query fields + timestamp precision into the backend query flow and instance model. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Thanks for the PR! Two requests before we move forward with the review:
The frontend part of this PR (the new QueryEditor mode, macros, format/precision options) looks like it can stay mostly as-is regardless of which transport the backend uses. |
Per review: query table-model data through apache/iotdb-client-go (v2.0.8, TableSession) on the RPC port instead of POST /rest/table/v1/query. - A lazily created, per-datasource TableSessionPool serves the queries; a new "rpc address" datasource option sets the endpoint, defaulting to the URL's host with the default RPC port 6667. Tree-model queries and the health check keep using the REST service for now. - Values arrive as typed Go values instead of JSON, removing the UseNumber/json.Number coercion layer; INT64 precision is inherent, and the client converts TIMESTAMP/DATE values with the server-reported timestamp precision, so the short-lived "time precision" datasource option is removed again. - The time macros now expand to ISO 8601 UTC timestamp literals (e.g. 2020-09-13T12:26:40.000+00:00) rather than epoch-ms integers, which the server parses correctly under any configured timestamp_precision. - The per-query database is applied with USE on the pooled session; DATE renders as yyyy-MM-dd and BLOB as 0x-prefixed hex, matching the REST transport's output. - The server-side REST row cap (rest_query_default_row_size_limit) no longer applies; the README now recommends a LIMIT clause for wide scans. - Frame building, the Time series/Table formats, the long-to-wide pivot and the row sorting are unchanged; the fetch path sits behind a narrow result-set interface so it stays unit-testable. 13 unit tests, go vet and tsc --noEmit pass. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
|
Thanks @CritasWang @HTHou — done: the table-model backend now goes through apache/iotdb-client-go v2.0.8 ( Notes on what the switch changed:
On the toolchain: I'll take the |
|
Thanks for the quick turnaround on the native-client switch — the new backend reads well, and deferring the toolchain migration (plus the remaining frontend bot notes) to the follow-up PR is exactly what we agreed. Two things before this can be merged: 1.
if s.session.config.Database != s.sessionPool.config.Database && s.sessionPool.config.Database != "" {
err := s.session.ExecuteNonQueryStatement("use " + s.sessionPool.config.Database)The pool created in Either fix works for me:
2. Please smoke-test against a live server before merge The PR description notes the new path "has not yet been smoke-tested against a live server", and the 13 unit tests all sit behind the mocked result-set interface — the actual RPC path (connect, auth, |
Per review: queryTableModel runs USE <database> on a pooled session, and PooledTableSession.Close() only restores the database when the pool has one configured — which this pool does not — so a session's USE state survived release. A query with an empty database field could then nondeterministically inherit whichever database a previous panel had selected on that session. Make the database required for the table-model mode: verifyQuery rejects an empty database, the editor marks the field required, and the README documents it. Every query therefore always runs USE for its own database on the session it checked out, so leftover session state can never influence a result regardless of pool scheduling. Fully-qualified database.table references keep working and take precedence over the session database. Verified against a live IoTDB 2.0.8: alternating panels on two databases sharing one session pool each saw exactly their own data across repeated rounds. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
|
Both points addressed: 1. USE state leak — fixed by making the database required. 2. Live smoke test — ran the new mode end-to-end against a real IoTDB 2.0.8 (
One honest caveat: this drove the backend query path directly rather than a rendered Grafana panel — the current |
|
Both points look good to me. The The smoke-test caveat (backend path only, not through a rendered Grafana panel) is acceptable given the agreed toolchain situation. The scenarios you covered — multi-database pool sharing, all-types round-trip including INT64 above 2^53, ISO-8601 macro expansion on a live server, error surfacing — are exactly what matters for this PR. The in-Grafana screenshot can follow with the toolchain-migration PR as discussed. LGTM from my side. Waiting for CI on the latest commit to go green, then this is ready to merge. |
What
Adds table-model query support to the IoTDB Grafana data source plugin — the first slice of #18258.
The plugin previously only spoke the tree model: both QueryEditor modes build
root.*paths and the backend calls/grafana/v1/query/expression, which has no table-model equivalent, so table-model users could not visualize their data.How
TableSession) against the RPC port, served by a lazily created per-datasourceTableSessionPool. A new optionalrpc addressdatasource setting configures the endpoint (defaults to the URL's host with port 6667). Tree-model queries and the health check keep using the REST service for now.Time seriesformat, rows are sorted by the first TIMESTAMP column and a long-shaped result (time + tag columns + value columns) is pivoted with the SDK'sdata.LongToWide, turning tag columns into field labels —SELECT time, device_id, temperature FROM tdraws one line per device instead of one interleaved series. TheTableformat returns rows untouched (preserves a user'sORDER BY). If the pivot cannot apply (e.g. null timestamps), the plain frame is returned.$__timeFilter(col)/$__timeFrom/$__timeTomacros expand to ISO 8601 UTC timestamp literals for the panel's range, which the server parses under any configuredtimestamp_precision; the function forms$__timeFrom()/$__timeTo()familiar from Grafana's SQL data sources are accepted too, and$__timeFilterallows one level of nested parentheses.LIMITrecommendation for wide scans.Testing
go build,go vet, andgo test ./pkg/plugin(13 unit tests) pass — covering macro expansion (incl. function forms and nested parentheses), the result-set fetch path (behind a narrow interface, incl. nulls and error propagation), per-type conversion incl. DATE/BLOB rendering and INT64 precision, the long-to-wide pivot for multi-device results (incl. unsorted input and the null-timestamp fallback), Table-format row-order preservation, and RPC endpoint resolution.tsc --noEmitagainst the@grafanatypes.TableSession,SessionDataSettyped getters, server-reported time precision) and the server's relational planner (ignoreTimestampsemantics). It has not yet been smoke-tested against a live server.Follow-ups (same issue)
The
@grafana/toolkit→@grafana/create-plugintoolchain migration (next PR, as discussed), a database → table → column browser, a visual query builder, frontend Jest coverage for the new mode, path-picker performance, and moving the tree-model path and health check to the native client as part of the broader modernization.