Course contents

The Analytics Engineer’s Field GuideAirflow 2: writing DAGsStep 4: Writing your first DAG

Writing your first DAG

Part 3 · Airflow · step 4

Writing your first DAG: tasks, operators, connections

The whole vocabulary, one real file, and what happens when the clock hits six.

· · ·

Steps 1 to 3 were why and where. This is how: four words, one Python file, and three things to know about running it. Nothing here is advanced, and everything later in this part is built on it.

The four words of Airflow

DAG, task, operator, scheduler: the whole vocabulary.

Everything in Airflow is a DAG, and the scary acronym is actually a simple promise:

  • Graph: your workflow is boxes (tasks) connected by arrows.
  • Directed: the arrows have a direction, "run A, then B".
  • Acyclic: no loops allowed, so a workflow always finishes instead of chasing its own tail.

In practice a DAG is one Python file. Four words cover most of the tool:

DAG

A Python file that defines your workflow: which tasks run, in what order, on what schedule. The unit you deploy.

Task

One unit of work inside a DAG: run a dbt model, query Snowflake, send a Teams message, refresh a dashboard.

Operator

A template for a kind of task. There are operators for SQL, dbt, Python, HTTP, email and dozens more.

Scheduler

The engine that watches the clock. It reads every DAG, decides what is due, and hands tasks to workers. You never call it, it is always running.

Gina asks for one thing

The pizzeria's revenue table is refreshed by a Snowflake procedure. Gina opens her dashboard at seven every morning with a coffee, and until now Thomas has been running that refresh himself, from his laptop, when he remembers. Last Tuesday he was on a train.

So the ask is small and precise: every day at 06:00, refresh the revenue table, and if it fails, tell someone. No dashboards to build, no data to move. Just make it happen without Thomas.

That's a DAG, and it's about fifteen lines. Before the code, here are the four decisions Thomas has to make, and the answer he chose for each:

The question
The answer in this file
When does it run?
Every day at 06:00. Written "0 6 * * *", which is cron: minute, hour, day, month, weekday.
Since when does it exist?
1 January 2024. That's start_date, and it doesn't mean "run now", it means the schedule starts counting from there.
Should it replay the past?
No. catchup=False: switched on in 2026, it skips two years of missed days and waits for the next 06:00. Deployed at 15:00, its first run is tomorrow morning, trigger it by hand if you need today's. It's the Airflow 3 default, written out so nobody has to remember that.
What does it actually do?
One task: call a Snowflake procedure, using a saved connection called snowflake_pro.
# a tiny DAG: call a Snowflake procedure every day at 06:00
from airflow.sdk import dag   # Airflow 3: the public API lives in airflow.sdk
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
from datetime import datetime

@dag(schedule="0 6 * * *", start_date=datetime(2024,1,1), catchup=False, tags=["owner:finance", "domain:revenue", "criticality:high"])
def finance__revenue__daily():   # team__domain__type, the naming rule of part 3, step 8
    SQLExecuteQueryOperator(
        task_id="refresh_revenue",
        conn_id="snowflake_pro",
        sql="CALL ANALYTICS.FINANCE.REFRESH_REVENUE()",
    )

finance__revenue__daily()

Three things to notice, and they're the whole model:

  • The DAG is the function. @dag on top of finance__revenue__daily() is what makes this file a workflow. The last line calls it, which registers it with Airflow.
  • The task is inside. Anything you create in that function becomes a step Airflow can run, retry and show you.
  • No credentials anywhere. conn_id="snowflake_pro" points at a connection stored in Airflow, which is what keeps secrets out of the repository.

When a DAG has several tasks, you order them with >>, read as "then": load >> transform >> notify. That guaranteed order is the whole point of the G in DAG.

What actually happens at 06:00

The hand-off from clock to green box, step by step.

Nobody presses a button. The pieces hand off to each other:

Scheduler wakes
"it's 06:00, finance__revenue__daily is due"
→
A worker picks it up
runs refresh_revenue
→
Snowflake executes
the CALL runs
→
UI turns green
or red, and retries fire
fails? Airflow retries on its own, then alerts · succeeds? nobody even had to look

One DAG, many runs

The two words on every screen of the UI: DAG Run and Task Instance.

The same DAG runs every day, so the UI needs a name for "today's execution" and a name for "this step inside today's execution". That is all these two terms are.

A DAG Run is one execution of the whole DAG, for one date. Think of the DAG as a recipe: this morning's cooking of it is a DAG Run. Yesterday's was another. The recipe stays the same, the cooking sessions stack up, one per day.

A Task Instance is one step of the recipe, as cooked this morning: refresh_revenue, inside today's run, currently on try 2. It has its own state, its own log, its own retry counter.

The question you're asking
Where to look
"Did this morning run?"
The DAG Run: one green or red bar per day.
"Which step broke, and on which try?"
The Task Instance: one box per step, inside that day.
Reading the UI in ten seconds
The grid view is the whole dashboard: columns are DAG Runs (one per day), rows are tasks, each cell a Task Instance. Green means done, red means failed, click a red cell and open Logs. That gesture, column → cell → log, is 90% of what you'll ever do in the interface.
· · ·

Two ways to write a task

The task in that DAG came from an operator: a ready-made step you fill in with parameters. SQLExecuteQueryOperator already knows how to connect, authenticate, run SQL and report back, so you supply three values instead of fifty lines. Airflow ships operators for all the common moves.

For your own logic there's the TaskFlow API: put @task on an ordinary function and Airflow turns it into a task.

from airflow.sdk import dag, task   # Airflow 3: decorators come from airflow.sdk

@task
def count_orders() -> int:
    # any Python you like: an API call, a check, a computation
    return 18342                     # the return value is handed to the next task

@task
def check_volume(n: int):
    if n < 1000:
        raise ValueError(f"Only {n} orders loaded, that's suspicious")

check_volume(count_orders())         # inside a @dag function: calling them wires the dependency

Two functions, two tasks. count_orders returns a number and check_volume raises if that number looks wrong, which fails the task and stops the DAG. Notice what you didn't write: no operator class, no task_id, no dependency line. Calling check_volume(count_orders()) hands the value over (through XCom, Airflow's small-value store) and tells Airflow that counting runs first.

The two styles, side by side

You'll meet both in real code, because the classic style is still everywhere:

Classic operators
TaskFlow (@task)
What you write
Instantiate a class, fill in parameters
A plain Python function with a decorator
A custom step
PythonOperator(task_id=…, python_callable=fn)
@task above fn, that's all
Naming the task
You pass task_id by hand
The function name becomes the task name
Passing a value along
xcom_push then xcom_pull, by key
return it, take it as an argument
Declaring the order
Wire it yourself with >>
Calling the functions wires it
Best for
Ready-made moves: run SQL, wait for a file, call an API, trigger dbt
Your own logic: checks, computations, anything with an if

It isn't a choice you make once. The two mix freely inside a single DAG, and a normal file has both: SQLExecuteQueryOperator for the warehouse call, @task for the sanity check that follows it.

If "decorator" is a new word
It's just Python's syntax for wrapping a function in extra behaviour. @task wraps yours with "run me as an Airflow task", and @dag wraps a function with "this whole thing is a workflow".
· · ·

Where the credentials live

The conn_id="snowflake_pro" from the operator above points at a Connection: account, user and secret stored encrypted in Airflow, never in your repo. Create it once in the UI (Admin → Connections) or as an environment variable in your deployment (AIRFLOW_CONN_SNOWFLAKE_PRO). Test it before relying on it: the form's Test button and airflow connections test snowflake_pro both work once an admin enables test_connection, which is off by default for security. Swap the same DAG between pre_prod and pro by swapping the connection, the code doesn't change.

This is also where dbt's pro target lives. Production runs under the platform's service account, and its key sits in this connection, not in anyone's ~/.dbt/profiles.yml. In step 6, Cosmos's ProfileConfig turns the connection into the dbt profile at run time.

DAG file
conn_id="snowflake_pro"
→
Airflow secret store
encrypted credentials
→
Snowflake
authenticated session
the repo only ever contains the name, never the secret
That's the foundation
A DAG is a Python file. A task is a step in it. An operator is a ready-made step. A connection is a named credential. Everything in the rest of this part, dbt in Airflow, sensors, retries, team standards, is a variation on those four ideas.

Lesson quiz

  1. What does the A in DAG guarantee?

    1. No cycles, so a workflow always reaches an end
    2. Automatic retries for every failed task
    3. Asynchronous tasks that never block each other
    4. All tasks run in parallel whenever possible
  2. A DAG has start_date=2024-01-01 and catchup=False, and you deploy it in 2026. What does catchup=False prevent?

    1. Running the DAG at all until its next start_date comes round
    2. Retrying a task that already failed on a previous day
    3. Creating a run for every missed day since January 2024
    4. Two runs of the same DAG overlapping on the same day
  3. Thomas wants the same DAG file to run against pre_prod in a test Deployment and against pro in production. Following the lesson, what should differ between the two Deployments?

    1. The connection behind conn_id, set in each Deployment
    2. A second DAG file holding the pre_prod credentials
    3. An if on the hostname inside the DAG file
    4. A profiles.yml per Deployment, committed to the repo