Repository navigation
Conversation
The top-level `config:` block in profiles.yml has been deprecated since
dbt 1.8, and the dbt Fusion engine rejects it outright ("Unexpected
'config' key in profiles.yml"), so nothing in integration_test_project
could even parse on Fusion.
Its two settings move to tox `setenv` as DBT_SEND_ANONYMOUS_USAGE_STATS
and DBT_USE_COLORS, which every dbt version reads.
Test-harness only: no change to the package.
dbt Core only fires on-run-end for commands that execute nodes. The dbt Fusion engine also fires it for `dbt compile` and `dbt show`, including the previews the dbt VS Code extension runs. On Fusion every one of those uploaded a phantom `model..inline` row (empty package_name, materialization 'inline') plus an extra invocation, and spent ~6s on inserts. upload_results now returns early when invocation_args_dict.which is 'compile' or 'show', so both engines record the same invocations. No-op on Core, where the hook never runs for those commands.
The dbt Fusion engine deprecates generic-test arguments at the top level
of a test definition ("will error post preview"); they must sit under
`arguments:`. dbt Core only reads that form from 1.10 without the
root-project `require_generic_test_arguments_property` flag, which a
package cannot set for its consumers. Verified on 1.10, 1.11 and 1.12 with
the flag unset: compiled tests carry the real bounds and values.
So the support floor moves from 1.3 to 1.10:
- require-dbt-version: [">=1.10.0", "<3.0.0"]
- the 11 `is_between` / `accepted_values` tests in models/*.yml nest
their args under `arguments:`
- tox envs and release-matrix pins below 1.10 removed; SQL Server pin
moves 1_9_0 -> 1_10_0 (dbt-sqlserver now has 1.10-1.12 releases);
disabled Spark env moves to dbt-spark 1.10
- docs publishing installs dbt-snowflake 1.11 instead of 1.9, which
would now fail the version check
- pyproject.toml raises adapter floors and pins dbt-core >=1.10,<2 so
the host venv no longer resolves dbt-core 1.8.9
1.3-1.8 were already unverifiable in CI (Python 3.12 toolchain, no
microbatch), and dbt-snowflake < 1.5 can't use the inline key-pair auth
the tests now require.
BREAKING CHANGE: dbt < 1.10 is no longer supported. Consumers on older
dbt should pin dbt_artifacts 2.11.1. Release as 3.0.0.
Fusion is a standalone binary, not a PyPI package, so it can't go through uv/tox. scripts/fusion/ runs it in a pinned Docker image (FUSION_VERSION=2.0.6) against integration_test_project, writing to a separate `..._fusion_...` schema so it never collides with Core runs: - check.sh: Fusion parse with --warn-error. Fusion's parse never connects to the warehouse, so it runs on dummy credentials and is safe for the no-secrets PR tier. Fails if a deprecated form is reintroduced. - test.sh <warehouse>: the same two-pass build as the tox envs. Snowflake only; Postgres/Trino are experimental in Fusion and SQL Server/Spark are unsupported. - dbtf.sh: run any Fusion command; sources the repo-root .env and forwards variables by name so the multi-line private key survives. CI: `fusion-parse` in all three tiers, `integration-snowflake-fusion` in Tier 2 and 3, keeping each tier a superset of the one below. The tox.ini Fusion placeholder now points at scripts/fusion/. README documents Fusion support: Snowflake integration-tested, BigQuery and Databricks parse-only, and the compile/show hook difference that upload_results now handles. Verified locally: Fusion 2.0.6 on Snowflake 58/58 then 129/129.
With the dbt floor at 1.10, the unpinned integration_sqlserver env now resolves dbt-sqlserver 1.12, which made pyodbc an optional extra. The test profile uses the ODBC driver (pyodbc backend), so the Tier 1 job failed before running anything: The legacy `pyodbc` backend was requested, but the optional dependency `pyodbc` is not installed. Install dbt-sqlserver[pyodbc] in the tox env and in pyproject.toml (the host venv also resolves 1.12). The pinned 1_10_0 env is unchanged; 1.10 still ships pyodbc as a hard dependency.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dbt Fusion (dbt 2.x) compatibility; raise dbt floor to 1.10
Summary
Makes the package parse and run cleanly on the dbt Fusion engine, and adds
Fusion to CI so it stays that way. To get there, the minimum supported dbt
version moves from 1.3 to 1.10. That is a breaking change, so this should
ship as 3.0.0.
On Fusion 2.0.6, before this PR:
integration_test_projectdidn't parse at all (deprecatedconfig:block inprofiles.ymlis a hard error).deprecations (args at the top level instead of under
arguments:).dbt compileanddbt show(including VS Code extension previews)fired
on-run-endand uploaded a phantommodel..inlinerow plus an extrainvocation.
The upload macros themselves needed no changes. One sampled row per source
table on Fusion has the same shape as on Core (timing,
adapter_response,rows_affected,thread_id, checksums, freshness, exposure owners).Breaking change
require-dbt-versionis now[">=1.10.0", "<3.0.0"].Fusion requires generic-test arguments under
arguments:. dbt Core only readsthat form from 1.10 without the
require_generic_test_arguments_propertyflag,and a package can't set that flag for its consumers. 1.3–1.8 were already out
of CI (Python 3.12 toolchain, no microbatch), and dbt-snowflake < 1.5 can't use
the inline key-pair auth the tests now require.
Consumers on dbt < 1.10 should pin
dbt_artifactsto2.11.1. The README saysso.
Changes, by commit
test:drop deprecated config block from integration profiles.ymlMoves
send_anonymous_usage_stats/use_colorsto toxsetenv.Test harness only.
fix:skip upload_results for compile and showReturns early when
invocation_args_dict.whichiscompileorshow.No-op on Core, where the hook never fires for those commands.
feat!:require dbt >= 1.10 and nest generic test args under argumentsrequire-dbt-versionfloor → 1.10is_between/accepted_valuestests nest their args underarguments:1_9_0→1_10_0; disabled Spark env → dbt-spark 1.10publish_docs_on_release.yml: dbt-snowflake 1.9 → 1.11 (1.9 would failthe new version check at release time)
pyproject.toml: adapter floors raised,dbt-core>=1.10,<2added; thehost venv was resolving dbt-core 1.8.9
ci:run the dbt Fusion engine in Docker and test the package on itscripts/fusion/: pinned Docker image (FUSION_VERSION=2.0.6), no hostinstall.
check.sh(parse,--warn-error, no secrets),test.sh snowflake(two-pass build, same as tox),
dbtf.sh(ad-hoc commands). Writes to aseparate
..._fusion_...schema.fusion-parsein Tiers 1, 2 and 3;integration-snowflake-fusionin Tiers 2 and 3. Each tier stays a superset of the one below.
scripts/ci/README.md: Fusion scripts.Fusion adapter coverage
Testing
All runs on the final tree.
--warn-errorcleanAlso checked:
arguments:tests with the real bounds and values (e.g.
success_rate < 0 or success_rate > 1),so they aren't passing vacuously.
check.shfails when one test is put back in the old top-level form.compileandshowupload nothing, andrunstill uploads.
Trino and SQL Server weren't run locally on this branch; Tier 1 covers them.
Reviewer notes
.github/,scripts/,tox.ini,pyproject.toml,uv.lock,dbt_project.yml.uv.lockis large (about +1,200 lines) because raising the adapter floorspulls dbt-core 1.12 and its dependency tree into the host venv.
fusion (parse)runs on this PR. The Fusion Snowflake build runs aftermerge (Tier 2) and on release candidates (Tier 3).
release-candidate/3.0.0branch.Follow-ups (not in this PR)
database is required for databricks relation. Fusion needscatalog:in the test profile. DatabricksCI is stubbed, so this waits for its reactivation.
credentials mounted into the container.
1_10_0. Newer minors arecovered by the unpinned "latest" envs.
3.0.0 is tagged.
dbt_version = 2.0.0ininvocationswhile the binary is 2.0.6, and itsinvocation_argskeys differ from Core's. Both are stored as-is.