DuckDB: iterate on your laptop, for free
Same project, different engine. Check the logic locally in seconds, keep Snowflake for what only Snowflake can answer.
Watch Alice iterate. Change a join, dbt run, wait for the warehouse, read the result, fix a filter, run again. Every loop costs credits and a minute of waiting, and nine times out of ten she's checking something that has nothing to do with Snowflake: does the join keep the right rows, is the grain (step 3: what one row means) still one order, does the case statement handle nulls.
DuckDB is a database that runs inside the program using it, here dbt itself. No server, no account, no credentials. The entire warehouse is one file sitting next to your project, and it's fast on millions of rows.
The trick: one extra target
You don't copy the project. You run the same project, same models, same macros, same tests, pointed at a different engine. An adapter is dbt's plugin for one database. You have dbt-snowflake; here you need dbt-duckdb, which brings DuckDB along. It's already in requirements-dev.txt (step 11). Then add a second personal target:
# ~/. my_project: target: userland outputs: userland: # Snowflake sandbox, unchanged from step 14 # ... local: # DuckDB on the laptop type: duckdb path: analytics.dbt/ profiles. yml, Alice's two personal targets duckdb # one file, created where you run dbt threads: 4
No login, just a file. Run dbt from the project root, and add *.duckdb to .gitignore. With no database: or schema: key, dbt-duckdb names the database after the file (analytics) and uses the schema main. Your macros from steps 5 and 14 then route every model to analytics.main:
# the whole workflow change is one flag dbt run --target local # seconds, zero credits dbt test --target local # data tests + unit tests, same definitions
Feeding it: where the local data comes from
DuckDB starts empty, and your sources need tables to read: one CSV per source table. Two kinds, in two folders:
- Hand-made rows, in
sample_data/, committed. Type a dozen rows per file, or generate them with a Python library like Faker. Include a null, a refund, a duplicate. CI uses the same files. - A real extract, in
local_data/, never committed (add it to.gitignore). Run a smallSELECTin a Snowsight worksheet and use Download results. Real shapes, real oddities. Leave personal columns out and follow your company's data rules.
A short script loads either folder. The duckdb Python package came with dbt-duckdb:
# load_local. import sys import duckdb folder = sys.argv[1] if len(py, at the project root: each CSV becomes a table sys. con = duckdb.argv) > 1 else "sample_ data" connect("analytics. con.execute("create schema if not exists public") # same name as RAW_duckdb" )DB. for table in ["orders", "customers"]: con.execute( f"create or replace table public.PUBLIC {table} as " f"select * from read_csv( ) con.close()'{folder}/ {table}. csv')"
python load_local.py loads the hand-made rows, python load_local.py local_data your extract. Why not dbt seed? Seeds belong to the project, so every target would load them, production included.
One catch: your sources point at RAW_DB, which doesn't exist inside the DuckDB file. Make the source's database follow the target:
# models/staging/ sources: - name: raw database: "{{ target._ sources. yml database if target.type == 'duckdb' else 'RAW_DB' }}" schema: PUBLIC # DuckDB ignores case: PUBLIC finds public tables: - name: orders - name: customers
On DuckDB that reads analytics.public.orders. The volume is tiny on purpose: you're checking logic, not performance.
What a local run does and doesn't prove
DuckDB answers this
ref() resolves, build order is right.Only Snowflake answers this
--exclude.So the rule is simple: catch on the laptop everything the laptop can catch, and let the warehouse be the last checkpoint instead of the first.
The same trick in CI
Here's step 11's pipeline, grown up, in its three GitLab stages. While a merge request into pre_prod is open, test runs the sqlfmt, dbt-score and DuckDB jobs side by side, with no warehouse. Then build builds the change on Snowflake with target pre_prod (step 10's state:modified+ command). deploy runs after the merge.
The DuckDB job runs on the CI runner, the throwaway machine GitLab starts for each job. Copy step 12's score job (stage, image, DBT_PROFILES_DIR: ci, rules) and swap in this script. No credentials, no credits:
# the DuckDB job's script, using the local target in ci/profiles. pip install -r requirements-dev.yml txt dbt deps # install packages (step 6) python load_local. dbt run --target local --empty # every model, zero rows (dbt 1.8+) dbt test --target local --select test_py # sample_ data/ , committed type: unit
--empty reads no rows, but the source tables must exist, for their columns, so the load runs first. Broken SQL and failing unit tests die here, and only code that passed reaches the build stage:

