Course contents

The Analytics Engineer’s Field GuideAirflow 1: the ideaStep 2: From cron to Airflow

From cron to Airflow

Part 3 · Airflow · step 2

Airflow: when it runs, in what order, what if it breaks

A Python file says what should happen and when. Airflow does the rest, every day, and tells you when it doesn't.

· · ·

Thomas's first week

Thomas wrote a cron job. One line on a small server in the corner of the office: 0 6 * * * dbt run. It worked for eleven days.

  • Day 12. Snowflake dropped the connection for four seconds. dbt failed, cron shrugged, and nothing rebuilt. Nobody found out until 11am.
  • Day 19. The loader ran late, so dbt built on yesterday's rows. Everything reported success. The dashboards were quietly wrong for a day and a half.
  • Day 26. Somebody asked which model broke on the 14th. The logs had rolled over. There were no logs.
  • Day 33. The little server needed a reboot. Nobody knew it existed, so nobody restarted the cron. Three days of no data.

None of that is a new problem. Retrying a flaky task, refusing to run before the data has landed, keeping the history of what ran and when: this is exactly what Airflow was built to do, and Thomas gets all of it by writing a Python file instead of a cron line. The fourth day is a different problem, and it belongs to Astronomer, the next lesson.

People say the two names in the same breath, which makes them sound like alternatives. They're not. One is the tool, the other is where the tool lives:

Where Airflow sits

your DAG files
Python, in the repo · you write these
Airflow
reads them, decides when · in what order · what if it fails
Astronomer
runs Airflow itself · servers, upgrades, scaling
dbt + Snowflakewhere the work actually happens · Airflow only asks
each layer holds the one above it · none of them transform a single row

Read it bottom to top: Astronomer keeps Airflow alive, Airflow reads your files, your files say what should happen and when. And to the right, at arm's length, the things that do the actual work.

What Airflow does for us

In our case, Airflow runs Alice's dbt project every morning and watches what happens. You write a Python file that says run this, then this, then this, every day at six, and Airflow takes care of it from then on, whether Thomas is at his desk or on holiday. It runs dbt as the platform's service account, through a connection stored in Airflow, never with anyone's laptop credentials.

Three decisions, and only three:

Airflow decides
Which means, concretely
WHEN
06:00 every day, or once the data has landed, or when another pipeline has produced its data.
WHAT ORDER
Load before transform, transform before the alert. If step 2 fails, step 3 never runs.
WHAT IF
Retry three times, wait five minutes between tries, then page someone. Keep the log either way.
What Airflow is not
It's not a data engine. It never touches your rows. The SQL runs in Snowflake, dbt does the transforming, and Airflow gives the order and watches what happens. Every design choice in this part follows from that split.
· · ·

The four things Thomas stops building himself

Every one of them was a day in that first week.

Schedule

Run dbt every morning at 06:00, no human watching.

Retry

Snowflake slow, connection dropped? Try again instead of failing the morning.

Alert

Tell Thomas the moment something truly fails, not a manager at 9am.

Coordinate

Run dbt after ingestion and before the dashboard refresh. In order, every time.

· · ·

Before Airflow, and after

What the same platform looks like on both sides of the migration.

✗ Without a shared orchestrator
Cron jobs scattered everywhereSpread across multiple tools with no coordination, several orchestrators running at once.
No monitoringFailures go unnoticed until someone complains about a dashboard.
Scattered documentation and communicationNobody knows where a job lives, what it depends on, or who owns it.
✓ With Airflow on Astronomer
One scheduler for everythingEvery run logged, monitored, alertable, in one place.
Retries, timeouts, alerts built inBreakage pages you, not a manager at 9am.
Clear dependenciesThe graph declares what runs after what, no guessing.
Relaunch in 2 clicksRed box, clear, re-run. No SSH into a forgotten machine.
Local test before pushingastro dev start runs the whole platform on your laptop first.

One thing is still missing from this picture. Airflow is software, and software has to run somewhere. That's the next lesson.

Lesson quiz

  1. On day 19 the loader ran late, and Thomas's 06:00 cron job built dbt on yesterday's rows and reported success. What was missing?

    1. A bigger warehouse so dbt finished sooner
    2. Log retention long enough to trace which run used stale data
    3. A dependency on the data landing, not on the clock
    4. A retry policy on the dbt command
  2. Which statement about Airflow is true?

    1. It gives the orders and watches; the SQL runs in Snowflake
    2. It pulls the rows into workers and transforms them in Python
    3. It replaces dbt by running the SQL files in the right order
    4. It stores the pipeline's intermediate tables between tasks
  3. Thomas runs Airflow on his own server. After an OS update the scheduler doesn't come back, and nothing runs until he notices at noon. In the lesson's stack, which piece takes that problem away?

    1. dbt, which reruns the models that were missed
    2. Astronomer, which keeps Airflow itself running
    3. A sensor that waits for the scheduler to return
    4. Airflow's task retries, set on every task