dxt

Data eXecution & Transformation
A native, dbt-project-compatible transformation engine.

Prerelease — not production ready. The initial Linux SQL scope has passed its acceptance gates. Evaluate your own workloads and operational requirements before adopting dxt in production.

dxt loads dbt projects, compiles typed Jinja and SQL, executes resource graphs and writes dbt artifacts through a native Zig binary. The initial execution scope is SQL models on DuckDB and PostgreSQL. Python model files remain visible in discovery, parsing, selection and compilation; selecting them for execution fails before warehouse writes. Authored Python is never run.

Native semantic queries, typed SQL analysis, versioned environments and governed cross-database execution are also implemented. They have namespaced commands and artifacts alongside the dbt-compatible interface.

SQL macros can query databases through native sessions, and CSV seeds load through each adapter's native lifecycle with typed data and rollback on failure.

Compatibility evidence targets dbt Core 1.10.5, dbt-duckdb 1.9.6, dbt-postgres 1.9.1, MetricFlow 0.208.1 and semantic interfaces 0.9.0. The validation snapshot records the accepted initial candidate, its exact source and the public CI evidence. The compatibility matrix defines supported behavior and intentional differences. Each later candidate requires its own applicable checks; the initial results do not establish universal dbt or adapter certification. The project is independent of dbt Labs.

Quick Start

Install the newest published Linux or macOS release, including prereleases:

curl -fsSL https://raw.githubusercontent.com/sabino/dxt/main/scripts/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
dxt version

The installer chooses your OS/CPU, checks SHA256 and installs into ~/.local/bin without sudo. Use sh -s -- --version v0.0.0 after the pipe to choose a specific published version. You can also download and verify a matching archive from GitHub Releases. Actions creates these downloads automatically for validated version tags and published releases. macOS archives bundle both database clients; Linux archives use the external drivers described below. macOS downloads remain experimental, and Windows binaries are not yet available. Until the first release is published, use a source build. See download instructions.

On macOS, build a standalone executable from source:

./scripts/build_macos.sh
export PATH="$PWD/zig-out/bin:$PATH"
dxt version

The helper uses checksum-verified Zig 0.16.0 and Zig-managed native build dependencies. It bundles DuckDB and the PostgreSQL client with TLS into dxt; running this binary requires no separately installed database libraries, Homebrew packages, Python, dbt or MetricFlow. Apple's Command Line Tools/macOS SDK are needed to build, and macOS supplies the system/C++ runtime libraries. Install the tools with xcode-select --install if needed. Apple Silicon source builds and downloads are experimental; macOS does not yet have complete Core compatibility certification.

On Linux, the existing build/release contract uses external native drivers:

zig build -Doptimize=ReleaseSafe
export PATH="$PWD/zig-out/bin:$PATH"

Install DuckDB 1.4.2's native library so libduckdb.so is discoverable, or select it with DXT_DUCKDB_LIBRARY. PostgreSQL execution uses libpq. See Getting started for platform requirements, standalone build options and explicit driver paths.

Create and build a local DuckDB project:

export DXT_DUCKDB_BACKEND=native
dxt init demo
dxt debug --project-dir demo --profiles-dir demo
dxt build --project-dir demo --profiles-dir demo --threads 2
dxt docs generate --project-dir demo --profiles-dir demo --static
dxt docs serve --project-dir demo --profiles-dir demo --port 8080 --no-browser

The scaffold includes a DuckDB profile, seed, model and data tests. Generated artifacts are written to demo/target; static docs include an offline static_index.html. Stop the docs server with Ctrl+C.

Existing projects retain their dbt project files, profiles, SQL and packages. Use --project-dir, --profiles-dir, --profile and --target as needed. PostgreSQL uses a dbt type: postgres profile and native libpq, discovered as libpq.so.5 on Linux or libpq.5.dylib/libpq.dylib on macOS, or bundled in the standalone build. DXT_POSTGRES_LIBRARY explicitly selects an external library. Your PostgreSQL server remains external and user-provided; no server is included or installed. Dependency fetching/extraction uses system git, curl and tar for the corresponding package transports. Ordinary run/build with installed packages does not need those tools. Bundled DuckDB overrides must match its pinned version; use a dynamic build when evaluating a newer library. Upgrade notes record the Zig 0.17 and DuckDB 1.5.6 candidates.

Implemented Surface

Area	Native implementation
Project and resources	Shared YAML/config precedence, custom database/schema/alias naming, packages, versions, groups/access, models, analyses, seeds, sources, SQL/YAML snapshots, macros, docs, exposures, data/unit tests and semantic resources.
Compiler	Typed expressions and containers, string formatting, macro arguments/returns, dispatch, bundled upstream SQL macros, native re/datetime/pytz/itertools providers, Relation/Column context, database-backed queries/statements and adapter introspection.
Execution	Native DuckDB/libpq sessions, dependency workers, ephemeral ancestry, unit-test gates, durable errors/skips, fail-fast cancellation, transactions and retry.
Materializations	Table/view, enforced contracts/constraints, authored SQL materializations, hooks/grants/persisted docs, adapter-specific incremental strategies and schema changes, microbatch, seeds, snapshots and clone views; PostgreSQL materialized views and DuckDB local external/table functions.
Profiles	Native DuckDB configuration, settings, attachments, secrets, connection lifetime and retry policies; PostgreSQL connection profiles through libpq.
Selection and integration	Graph/YAML selectors, indirect selection, state comparisons, defer/favor-state, effective command/env options, structured logs, real debug/init/operations, docs and freshness.
Artifacts and caches	Manifest v12, Run Results v6, Catalog v1, Sources v3, semantic manifests, native parse/relation/SQL caches and the embedded dbt docs application.
SQL analysis	Native dialect grammars, typed logical IR, column lineage, source diagnostics, explain output and dependency-aware invalidation.
Metrics	Simple, derived, ratio, cumulative and conversion planning, entity/grain checks, time spines/windows, saved queries and transactional exports.
Cross-database	Named connections, source reduction/pushdown, typed movement, retained stages, incremental watermarks, policy/budget guards, locks, recovery and adaptive task retries.
Environments	Immutable model versions, isolated environment views, physical reuse, interval/backfill accounting, audits, promotion and rollback.

These rows describe implemented capabilities within the pinned compatibility contract. The replacement roadmap distinguishes the delivered initial scope, production rollout work and later implementation.

System Map

flowchart LR
    Files[dbt project and profiles] --> Graph[Native loader and resource graph]
    Graph --> Select[Selectors, state and defer]
    Select --> Compile[Typed Jinja and SQL compiler]
    Compile --> Workers[Dependency workers]
    Workers --> DuckDB[(Native DuckDB)]
    Workers --> PG[(Native PostgreSQL)]
    Graph --> Plans[SQL, metrics, environments and movement plans]
    Plans --> Workers
    Graph --> Artifacts[dbt artifacts and docs]
    Workers --> Artifacts

Loading

Documentation

Document	Purpose
Getting started	macOS standalone builds, first project and platform/dependency requirements.
Primer	Runnable workflows and command families.
Compatibility	Versioned support, evidence and explicit boundaries.
Replacement roadmap	Delivered scope, operational work and future implementation.
Validation snapshot	Accepted source identity, test counts and public CI evidence.
Architecture	Native modules, execution flow and dependencies.
Releases	Linux x86_64/ARM archives, notices, checksums and installation checks.
Performance	Correctness-aware cold/warm measurement and budgets.
Changelog	Unreleased implementation changes and earlier history.
ExecPlan	Active integration and validation work.
Agent rules	Runtime, planning and public-safety requirements.
Multi-agent workflow	Isolated branch/worktree ownership and convergence.
Agent OS, protocols, GitHub Projects	Repository coordination and developer automation.

Development And Verification

Python requirements are developer-only oracle and fixture dependencies. Use CPython 3.12 for the canonical Core comparisons:

python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-dev.txt -r requirements-oracle.txt
zig build
zig build test
pytest -q tests/test_usability_scheduler.py
python scripts/check_runtime_boundary.py
python scripts/check_public_safety.py

Native database and browser fixtures are required for the complete suite. CI fixture setup documents the pinned dependencies; the portable PostgreSQL helper can use installed native tools through DXT_POSTGRES_BIN. Canonical CI distributes the complete suite over eight isolated runners per architecture and audits every case identity; see parallel validation commands. Run pytest -q for integrated validation and python scripts/validate_dbt_artifacts.py <artifact.json> for complete upstream artifact schemas.

DuckDB warehouse execution requires its native library and held sessions. The retained CLI autocommit query helper has separate developer conformance coverage; forcing DXT_DUCKDB_BACKEND=cli does not enable full transformation commands.

CI configures native tests/safety, full compatibility fixtures, all six public Jaffle steps, an unchanged PostgreSQL dbt-utils project, performance checks and actual Linux x86_64/ARM installation tests. Release jobs extract the real checksum-validated archive and exercise both adapters with PATH empty. These configured gates must pass on the final candidate before claiming release readiness; earlier focused successes do not replace that final run.

Python 3.11 jobs run developer/native CLI checks. The complete canonical Core suite runs under Python 3.12 on both Linux architectures.

Optional native coverage artifacts are produced by the Coverage workflow. They supplement the CLI, warehouse and artifact comparisons rather than replacing compatibility gates.