fix(kernel): Thrift-parity fixes + GetArrowBatches + U2M scopes (TIMESTAMP, DDL, User-Agent, VOID, Arrow batches, U2M scopes, auth logging)

[mani-mathur-arch]

What

SEA-via-kernel↔Thrift parity fixes surfaced by the Go comparator (two database/sql connections against the same warehouse, differing only in useKernel), plus the public Arrow batch API on the kernel path and an auth-logging cleanup. The parity fixes are kernel-path-only divergences from the default Thrift path and are entirely Go-side (no kernel/C-ABI change).

Kernel ↔ Thrift result/parity fixes

• TIMESTAMP out-of-nanosecond-range wrapping. The kernel Rows path converted arrow timestamps with arrow's ToTime, which forms an int64-nanosecond intermediate (value * unit-multiplier). For a microsecond TIMESTAMP — Databricks' wire unit — that overflows int64 outside ~1678–2262 and silently wraps a valid instant to a wrong one: TIMESTAMP '0001-01-01' scanned back as 1754-08-30, '9999-12-31' as 1816-03-30. Now converts via the unit-specific time.Unix/UnixMilli/UnixMicro constructors, which split into (seconds, nanoseconds) and represent the full 0001–9999 range. The default Thrift path is unaffected — it receives TIMESTAMP as a preformatted server string and never runs this multiply.

• DDL affected-rows reported as -1 instead of 0. The C ABI returns -1 for "not applicable / unknown" affected-row counts (DDL, SELECT, or a warehouse that doesn't surface the counter — the kernel's num_modified_rows Option<i64> None). The Thrift path reports 0 in those cases, so Result.RowsAffected() diverged: a CREATE/DROP SCHEMA returned -1 on the kernel backend vs 0 on Thrift. The -1 sentinel is now folded to 0; a real DML count (>= 0) passes through unchanged.

• User-Agent not forwarded on the kernel path. The kernel backend never forwarded the driver's composed User-Agent, so the kernel's built-in UA leaked through and query history misattributed SEA-path queries. Now forwards the same UA the Thrift path sends via set_custom_header, so both backends are attributed alike. Default-on; WithUserAgentEntry still customizes it.

• VOID/NULL columns reported as NULL instead of STRING. The kernel mapped an arrow.NULL column to DatabaseTypeName="NULL"/nil scan type, but the server stringifies VOID over the Thrift wire — verified live, even a bare SELECT NULL reports STRING_TYPE, same as it does for intervals. So sql.ColumnType diverged between backends for any null/void column. Now maps arrow.NULL to STRING (unbounded length, string scan type), matching Thrift. Also corrects two pure-Go parity tests that had pinned NULL_TYPE — that expectation was derived from the enum name, not captured live, and the live server disagrees.

Public Arrow batch API on the kernel path

• GetArrowBatches not exposed on the kernel path. kernelRows implemented only driver.Rows + the ColumnType* interfaces, so the public batch API — conn.Raw(...) → rows.(rows.Rows).GetArrowBatches(ctx) — failed its type assertion on the kernel backend while succeeding on Thrift. It now implements GetArrowBatches by wrapping the existing zero-copy next_batch pull (prefetch one batch so HasNext is exact; transfer each record's ownership to the caller so the row scanner and Close can't double-release). GetArrowIPCStreams is rejected with ErrNotSupportedByKernel — the kernel C ABI exports Arrow via the C Data Interface, not IPC bytes; callers should use GetArrowBatches (or the Thrift backend for IPC streams). A compile-time rows.Rows assertion guards the gap so it can't silently regress.

U2M OAuth scope parity

• U2M forwarded no scopes, so the kernel used its own default set. The kernel U2M path forwarded only the client id, so the kernel applied all-apis + offline_access. The Thrift path requests offline_access + sql (AWS/GCP) or offline_access + <tenant>/user_impersonation (Azure) via oauth.GetScopes. A workspace whose built-in databricks-sql-connector client isn't granted all-apis rejects the kernel's request with access_denied - Scopes 'all-apis' are not assigned. resolveKernelAuth now populates Auth.Scopes from the same oauth.GetScopes the Thrift path uses, so both backends authorize identically. The setAuth → set_auth_u2m wiring already forwarded Scopes; only the population was missing. No kernel/C-ABI change.

Auth logging

• Auth logs bypassed the driver's configured log level. The M2M, U2M, OAuth-config, and token-provider paths logged through the global zerolog/log logger (default level Debug), so lines like "databricks OAuth token fetched successfully" printed as raw JSON regardless of WithLogLevel — on both backends. Routed every one of them through the driver's logger package so they honor the configured level and use the driver's format; Error/Warn lines stay visible at the default Warn, Debug/Info gate off. Auth logic is untouched. This closes the last global-zerolog call sites in the driver, so DATABRICKS_LOG_LEVEL=debug now uniformly controls auth logging too.

Testing

• TestScanCellTimestampOutOfNanoRange — pins both TIMESTAMP repro endpoints (0001-01-01, 9999-12-31) plus the int64-ns boundary years; existing timestamp unit/location tests still pass.
• TestNormalizeAffectedRows — table test for the DDL sentinel-normalization rule, in a pure (untagged) normalizeAffectedRows so it runs under the default CGO_ENABLED=0 build.
• TestColumnTypeInfoFor / TestColumnTypeInfoMatchesThriftMapping / TestColumnTypeInfoScanTypeCoversScanner — the VOID→STRING mapping and its two cross-checks against the Thrift functions; all green.
• GetArrowBatches verified live against staging on the kernel backend via a Conn.Raw + rows.Rows type-assertion harness: 100,000 rows over 32 zero-copy arrow.Record batches, row-count and schema checks pass; the compile-time rows.Rows assertion pins the interface. GetArrowIPCStreams returns the ErrNotSupportedByKernel sentinel.
• U2M scope parity — TestValidateKernelConfig and TestResolveKernelAuthRealAuthenticators (fake + real U2M authenticator) now assert the resolved Auth.Scopes equal oauth.GetScopes(host); TestSetAuthByMode's "U2M full" case still pins the set_auth_u2m scope/port marshalling.
• Auth logging — ./auth/... package tests green after the logger swaps; no import cycle (the logger package depends only on zerolog + go-isatty).
• Verified on both build paths: default pure-Go (CGO_ENABLED=0) and tagged cgo (-tags databricks_kernel, linking the pinned kernel .a) — build + unit tests green on each.
• Comparator run (databricks-driver-test Go comparator, Thrift vs SEA): the pre-fix run was at 13 diffs vs driver main's 41; the latest run against this branch head is at 9. The 5 dt_interval_column metadata diffs (databaseTypeName=""/scanType=*interface{} on the kernel path vs STRING/string on Thrift, across 5 connection configs — ColumnTypeInfoFor and the value scanner lacked arrow.INTERVAL_DAY_TIME) are fixed in this PR (map INTERVAL_DAY_TIME → STRING for both metadata and value scanning) and are confirmed gone in the latest run. Of the remaining 9:
  • 8 are not driver bugs: NaN-vs-NaN comparator equality artifact (×3, both sides NaN), non-deterministic EXPLAIN plan node IDs (same plan), by-design volume-staging rejection (ErrNotSupportedByKernel), raw-fetch_arrow schema/value shape (×2 — see below), and a CREATE PROCEDURE sqlState detail on a permission-gated statement (both sides fail; only the error detail differs).
  • 1 new STATEMENT_DDL diff (deterministic, under investigation): DESCRIBE TABLE on the SEA/kernel side returns two extra trailing rows Thrift does not — # Clustering Information and # col_name section markers. The driver does no DESCRIBE post-processing (it scans server rows verbatim), so this is server/backend output, not a driver-rendering bug; the Python comparator does not surface it because its config carries result_set_filters (skip_rows) while the Go comparator config currently has result_set_filters: []. Caveat/open item: it reproduces across runs (not flaky) and was reportedly not present before recent changes; no mechanism has been found by which this PR's changes (arrow interval/decimal metadata, RowsAffected, auth logging, GetArrowBatches) could alter DESCRIBE row count, so causation is unresolved. Likely resolution is a comparator-side skip_rows filter (in databricks-driver-test, not this repo), pending a direct Thrift-vs-kernel DESCRIBE reproduction.
  • Decimal on the raw-fetch_arrow path (part of the residual, and shared root with intervals): the kernel returns native decimal128 / native intervals while Thrift's server pre-stringifies to utf8; the Go driver also formats decimals per-cell via decimalfmt.ExactString on the row-scan path. On the Arrow-batch path native types are arguably the desired behavior, so this is a tradeoff, not a bug. Cross-driver note: Python avoids the equivalent diff by setting the kernel's intervals_as_string. Tracked as a separate follow-up (kernel ResultConfig decimal_as_string, needs a kernel change + KERNEL_REV bump) — PECOBLR-3838.

Stacked on mani/sea-kernel-new-items (#412).

Co-authored-by: Isaac

[msrathore-db]

make this change in kernel not here

[mani-mathur-arch]

changing this in the kernel will break the other drivers. other drivers have different specs for each "not applicable" row counts (python has -1, node has none). Go thrift has 0 for the database/sql spec.

[msrathore-db]

please verify from query history

[mani-mathur-arch]

verified, the behavior is sound and matches how the python driver passes its user agent and what the query history shows

[msrathore-db]

can you please confirm this NULL to string in other drivers too?

[mani-mathur-arch]

The other drivers (node and python) also have this gap, the kernel returns nulls as void but thrift returns as string. i think this is an intentional design decision. verified by the python comparator, node one is currently broken due to dependency differences. I can remove it if that should be taken up as a separate fix for all the drivers or at the kernel level

[msrathore-db]

Please confirm the similar issues in other drivers over kernel like node and python so we can decide the location of these fixes

[msrathore-db]

Verify other drivers