Running dbt with Cosmos: one Airflow task per model
Don't hand-write one Airflow task per model. Cosmos generates the whole DAG from dbt.
You could write an Airflow task for every dbt model by hand, but that breaks the moment Alice adds a model, and it throws away everything dbt already knows about order and tests. Cosmos (the astronomer-cosmos package) reads your dbt project and builds the Airflow DAG for you: one task per model, in the same order as your ref() graph, with tests wired in.
The orchestration problem (before the tool)
Start with the need. Thomas wants Airflow to run Alice's dbt project every morning. The obvious way is one task that shells out to dbt run. It works, and it's a black box:
One dbt run task
A task per model
That's the problem Cosmos solves: it turns the opaque single task into a graph where every model is a first-class Airflow task. One picture says it, your dbt folder goes in, an Airflow DAG with the same shape comes out:
- model_hotel_1.sql
- model_hotel_2.sql
- model_booking_1.sql
- model_booking_2.sql
- + their ref() calls
- + tests in schema.yml
Airflow DAG
model_hotel_1run ✓ → test ✓model_hotel_2run ✓ → test ✓model_booking_1run ✓ → test ✓model_booking_2run ✓ → test ✓every model = its run, then its own data-quality tests
Look at the small print inside each task: run, then test. Data quality is baked into the graph itself, every model runs and is immediately checked by its own schema.yml tests before its children may start. Everything below is how.
dbt ls and caches the result, which is fine for most projects. Past a few hundred models, have CI build manifest.json and point Cosmos at it: ProjectConfig(manifest_path=...) plus RenderConfig(load_method=LoadMode.DBT_MANIFEST) , no dbt call at parse time.The whole DAG in a few lines
# dags/analytics_ from cosmos import DbtDag, ProjectConfig, ProfileConfig from cosmos._ ismafrost_ _ dbt. py profiles import Snowflake from datetime import datetime profile_Private Key Pem Profile Mapping config = ProfileConfig( # the pro profile, built at run time profile_name="my_project", target_name="pro", profile_mapping= ) dbt_dag = DbtDag( project_Snowflake Private Key Pem Profile Mapping( conn_ id="snowflake_ pro" ),config= profile_ProjectConfig("/ usr/ ),local/ airflow/ dbt/ my_ project" config= schedule="0 6 * * *", # every day at 06:00 start_profile_ config, date= dag_id="analytics_datetime(2024, 1, 1), _ , # team_ismafrost_ _ dbt" _ tags=["owner:domain_ _ type, step 8 analytics" , "domain:ismafrost" , "criticality:high" ], )
This is where the pro target from part 2 actually lives. ProfileConfig turns the snowflake_pro connection (the platform's service account and its key) into a dbt profile at run time, so there's no profiles.yml for pro anywhere: not in the repo, not on a laptop. A pre-prod DAG uses the same code with its own connection and target_name="pre_prod".
Every model, then its data quality tests
A generated model isn't "run the whole project". It's a small task group of two scoped tasks, and the second one is the part that matters most: build the model, then immediately run the data quality tests attached to it. Those are the tests Alice wrote in part 2, unique, not_null, accepted_values, relationships, sitting in the YAML next to the model.
# what Cosmos actually executes for the stg_orders group dbt run --select stg_orders # 1. build the table dbt test --select stg_orders # 2. its data quality tests, right away # both green → children may start # test red → the test task fails, everything downstream is held
That's test_, the default and the one you want. The alternative, AFTER_ALL, builds the entire project first and tests at the end, which means bad rows have already flowed into every mart before anyone finds out.
Concretely, for the pizzeria's staging layer, that second command runs everything Alice declared in YAML:
# models/staging/ - name: stg_orders columns: - name: order_id data_tests: [unique, not_null] # grain intact? - name: channel data_tests: - accepted__ models. yml: what dbt test --select stg_orders executes values: arguments: values: [dine_in, delivery] - name: customer_id data_tests: - relationships: arguments: to: ref('stg_ field: customer_idcustomers')
Four checks, run on the rows that were just built, before a single downstream model starts. It's the same safety net as part 2, except now it fires every morning at six instead of when someone remembers.
So a duplicate order id doesn't just get reported, it stops the branch it poisons:
stg_orders ✗run ✓ · test ✗ (dupes found)fct_ordersheld: upstream failedstg_customers ✓run ✓ · test ✓dim_customers ✓its branch was cleanCompare that with the single dbt run task: one duplicate order id there would have failed the entire morning for every team. Here, the customers branch shipped on time, the orders branch is one click to retry once the dupe is fixed.
Render only part of your project: select with RenderConfig
By default a DbtDag turns your whole project into tasks. Often you want a DAG for just one area, one mart and everything it needs. Pass a RenderConfig with select (or exclude), using dbt's own graph selectors.
from cosmos import DbtDag, ProjectConfig, RenderConfig finance_dag = DbtDag( project_config= profile_ProjectConfig("/ usr/ ),local/ airflow/ dbt/ my_ project" config= render_profile_ config, config= select=["+fct_RenderConfig( orders" ], # fct_orders and everything it depends on ), schedule="0 6 * * *", dag_id="finance__ , tags=["owner:orders_ _ dbt" finance" , "domain:orders" , "criticality:high" ], )
The + is dbt's graph operator, and the side it sits on is the whole trick:
Put + on the left for upstream (everything the model needs), on the right for downstream (everything that needs the model), or both sides for the full neighbourhood: +fct_orders+.
dbt run --select +fct_orders on your laptop builds the exact set Cosmos renders into the DAG. You can also select by "tag:finance" or "path:models/marts", or flip it with exclude=[...]. Learn them once, use them both places.Run a single dbt command: the Local operators
Sometimes you don't want a task per model, you want one Airflow task that runs dbt run and nothing more. Cosmos ships an operator per dbt command. In local execution mode (the simplest one, where dbt is installed in the same environment as Airflow) those are DbtRunLocalOperator, DbtTestLocalOperator, DbtSeedLocalOperator and the rest.
from cosmos.operators. run_marts = DbtRunLocalOperator( task_id="run_marts", project_dir="/local import DbtRunLocalOperator usr/ , profile_local/ airflow/ dbt/ my_ project" config= select="+fct_profile_ config, orders" , # the same selectors work here too )
Reach for a local operator when you want one clean step inside a bigger DAG (load → this single dbt run → refresh a dashboard). Reach for DbtDag when you want a task per model and full per-model visibility. Same engine underneath, you're just choosing the granularity.
astronomer-cosmos to your Astro project's requirements.txt, then astro dev restart. Put the dbt project in a dbt/ folder at the project root (it lands in /usr/local/airflow/dbt/, the layout the multi-team lesson uses) and point ProjectConfig at it. Cosmos recommends installing dbt itself in a separate virtualenv in the Dockerfile, so its dependencies never fight Airflow's; you then pass its path as dbt_executable_path (through ExecutionConfig on a DbtDag, directly on an operator, as the canary in step 5 does).The full picture: SQL to scheduled DAG
This is the whole stack, end to end: Alice writes SQL in dbt, Cosmos turns the project into an Airflow DAG, and Astronomer runs that DAG every morning, retries failures and alerts on breakage. Nobody babysits a pipeline again.

