Course contents

The Analytics Engineer’s Field GuideAirflow 1: the ideaStep 1: Why dbt needs a platform

Why dbt needs a platform

Part 3 · Airflow · step 1

Why dbt needs a platform around it

Thomas's turn. Here's what gets built around dbt, and which piece this part is.

· · ·

dbt is an engine. A powerful one: feed it SQL and it turns raw tables into a modelled warehouse. Alice spent the whole of part 2 building that engine.

An engine alone can't move a car.

dbt

the engine

powerful for transforming data, with SQL

SO WHO…?

?…starts it every morning, retries it, tells you it broke?
?…reviews the change, tests it and puts it in production?
?…creates the databases, warehouses and roles it needs?
?…decides how all of it works, the same way, for every team?

so we build the car

around the engine

ONE ANSWER EACH

Airflow

the drivetrain

decides when the run starts, in what order, and what happens when it breaks

← this whole part

GitLab

the logbook

versions the code, runs the tests, ships the deploy: what changed, by whom, why

← the CI lessons

Terraform

the plumbing

declares the databases, warehouses and roles as code, so every environment matches

← the infrastructure

Platform team

the pit crew

sets the standards, builds the tooling and keeps it reliable for every team

← Alice, Thomas, Ismail
dbt gives the power. The platform around it gives the motion.one way of working · automation end to end · data products a business can rely on

Each card answers one of the four questions, and it's worth seeing them side by side before we pick one apart:

Part
What it takes care of
So that…
Airflow
Starts the run every morning, in the right order, as the platform's service account, and watches it. A task fails, it retries. It fails again, someone gets told.
the tables are fresh at 07:00 whether or not anyone opened a laptop
GitLab
Holds every model, DAG and macro in a repository, so a change is a merge request someone reviews.
nobody edits production directly, and history explains every choice
GitLab CI
Checks that merge request automatically (format, tests on DuckDB, then a pre_prod build on Snowflake) and deploys it once it's green.
nobody deploys by hand, and broken code never reaches the schedule
Terraform
Declares databases, warehouses, roles and grants as code instead of clicks in a console.
dev, pre-prod and prod match, and "who granted this?" is a diff
Platform team
Sets the conventions, builds the shared tooling, and keeps the whole thing standing.
every team gets the same setup instead of inventing its own

Read the middle column again and one thing stands out: GitLab and Terraform answer where the code and the infrastructure come from. Neither of them makes anything happen at six in the morning.

· · ·

So which part are we actually building?

Line the questions up against the parts and one of them is left over:

dbt can't…
Answered by
Covered
create the databases, warehouses and roles it runs on
Terraform
outside this guide
get reviewed, tested and deployed
GitLab
the CI lessons
schedule its own runs, retry, alert
Airflow
this part, starting now

So: the infrastructure exists, the code is in a repository, and the models are correct. The only thing missing is something that makes them run every morning without Alice. That's the drivetrain, and it comes in two pieces, the tool and the place it runs.

Next: what those two pieces are, and why people keep confusing them.

Lesson quiz

  1. Which of these does dbt on its own NOT do?

    1. Run data tests right after the models they cover
    2. Turn a SELECT into a table or view in Snowflake
    3. Run models in the order their ref() calls imply
    4. Start itself every morning and retry when a run fails
  2. Databases, warehouses, roles and grants are declared as code. Which piece of the platform does that?

    1. Terraform
    2. Airflow
    3. dbt
    4. GitLab CI
  3. Finance needs a new warehouse, and an analyst offers to click it into existence in Snowsight because it's faster. In this platform, where should the change go?

    1. Into a dbt macro run once with the pro target
    2. Into an Airflow DAG that creates it at 06:00
    3. Into Snowsight, then documented on the wiki
    4. Into Terraform, as code someone reviews