Continuous integration and continuous deployment for dbt transforms analytics engineering from manual, error-prone deployments into an automated workflow where every code change is tested before merging and deployed reliably to production. This guide covers CI/CD architecture for dbt projects, including slim CI, environment management, and deployment orchestration patterns.
Continuous integration and continuous deployment for dbt brings software engineering practices to analytics engineering — automated testing, environment parity, and reliable deployment replacing manual, undocumented, error-prone processes. A mature dbt CI/CD implementation means that every code change is automatically tested before merging and deployed to production through a reproducible, version-controlled process rather than a developer running dbt run from their laptop.
The Case for dbt CI/CD
Without CI/CD, dbt projects accumulate problems:
**Untested merges** — changes are merged to the main branch based on peer review of SQL logic rather than verified test results. A schema change in an upstream model that breaks downstream tests is discovered in production, not before.
**Environment drift** — developers run models against dev environments that diverge from production over time. Logic that works in dev fails in production because the data is different, the permissions are different, or the configuration is different.
**Manual deployments** — deploying changes to production requires a developer to pull the latest code and run dbt manually, introducing human error and creating a dependency on the person who knows the deployment process.
**No rollback capability** — when a deployment causes failures, reverting requires re-running a previous version of the code — which may or may not produce the same results as the original run, depending on source data changes.
dbt CI/CD addresses all of these with automation.
Environment Architecture
A standard dbt environment architecture for CI/CD:
**Development** — each developer has their own dbt schema in the warehouse (typically named after the developer's username). Model changes are built and tested here before creating a PR. Development environments use the same source data as production but write to developer-namespaced schemas to avoid polluting production.
**CI** — a shared environment used for automated testing of PRs. When a PR is opened, the CI job builds and tests the modified models in the CI environment. The CI environment uses a recent snapshot of production data (or production data directly, with appropriate access controls) to catch issues that would not surface against synthetic dev data.
**Production** — the environment where certified, merged models run on the production schedule. Production deployments are triggered by merges to the main branch or by a scheduled production job.
The schema naming convention that enables per-developer environments in dbt:
In profiles.yml (or dbt Cloud environment configuration):
schema: "dbt_{{ env_var('DBT_USER', 'default') }}"
Each developer's environment variable sets their schema name, producing separate schemas for each developer without any coordination overhead.
Slim CI: Only Testing What Changed
Running the full dbt project on every PR is expensive and slow on large projects — a project with 500 models does not need to rebuild 500 models to test a change to 3 models.
dbt's slim CI approach uses the --select state:modified+ flag in combination with a comparison manifest:
dbt build --select state:modified+ --defer --state ./prod-manifest
The --defer flag tells dbt to use the production environment for models that are not being rebuilt in CI. If model A feeds model B, and the PR changes only model B, dbt builds model B in CI but defers model A to the production environment's results. This means model B's CI run uses production data for its upstream inputs without needing to rebuild model A.
The comparison manifest (produced by dbt ls or dbt compile in the production environment) defines the baseline that --state compares against to identify modified models.
The result: CI build time proportional to change size, not project size. A change to 3 models builds and tests 3 models plus their downstream dependents — not 500.
CI Pipeline Configuration
A GitHub Actions workflow for dbt CI:
name: dbt CI
on: [pull_request]
jobs:
dbt_ci:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install dbt
run: pip install dbt-snowflake
- name: Download production manifest
run: aws s3 cp s3://your-bucket/prod/manifest.json ./prod-manifest/manifest.json
- name: dbt build (modified models only)
run: dbt build --select state:modified+ --defer --state ./prod-manifest --target ci
env:
DBT_SNOWFLAKE_ACCOUNT: "value from secrets.SNOWFLAKE_ACCOUNT"
DBT_SNOWFLAKE_USER: "value from secrets.SNOWFLAKE_CI_USER"
DBT_SNOWFLAKE_PASSWORD: "value from secrets.SNOWFLAKE_CI_PASSWORD"
The production manifest is downloaded from a storage location where the production job uploads it after each successful run. This ensures the CI comparison is always against the current production state.
Production Deployment
Production deployments are triggered by merges to the main branch. The production job:
1. Checks out the merged code.
2. Runs dbt build (or dbt run + dbt test) against the production warehouse.
3. On success, uploads the produced manifest.json to the storage location used by CI.
4. On failure, sends an alert to the data team.
Production job scheduling depends on data freshness requirements. Most dbt production jobs run on a schedule — nightly, every few hours, or triggered by upstream data availability signals. Orchestrators like Airflow, Prefect, Dagster, and dbt Cloud's native scheduling all support dbt job orchestration.
dbt Cloud vs Self-Hosted CI/CD
dbt Cloud provides managed CI/CD with built-in support for slim CI, environment management, and deployment scheduling without the infrastructure overhead:
**dbt Cloud CI** — automatically runs slim CI when PRs are opened, using a configured CI environment. The production manifest is managed automatically.
**dbt Cloud jobs** — production, staging, and other scheduled runs configured through the UI, with native alerting and run history.
**dbt Cloud advantages** — minimal infrastructure setup, integrated with dbt's cloud IDE, managed execution environment.
**Self-hosted CI/CD advantages** — full control over the execution environment, integration with existing CI/CD infrastructure (GitHub Actions, GitLab CI, Jenkins), and no dependency on dbt Cloud's pricing model.
For teams already using GitHub Actions or GitLab CI for other projects, extending those pipelines for dbt is straightforward. For teams new to CI/CD or wanting minimal setup, dbt Cloud is the faster path.
Secrets and Credential Management
dbt CI/CD requires warehouse credentials to be available in the CI environment without committing them to the repository:
**GitHub Actions secrets** — environment variables stored in GitHub's encrypted secrets store, injected into CI jobs. The standard approach for GitHub-hosted CI.
**AWS Secrets Manager / HashiCorp Vault** — for more complex credential management requirements, credentials fetched from a secrets manager at job start.
**Service account credentials** — CI and production jobs should use dedicated service accounts with the minimum privileges required: read access to source schemas and write access to the target (dbt output) schemas. Not developer credentials.
The service account used for CI should be different from the service account used for production, so that CI runs cannot modify production schemas.
Our data architecture practice designs dbt project architectures and CI/CD pipelines for enterprise analytics teams — contact us to discuss analytics engineering infrastructure for your organisation.
A former Microsoft data architect audits your data foundation, identifies your top priorities, and sends you a written plan. Free. No pitch.
Book a Call →