Course contents

The Analytics Engineer’s Field GuideAirflow 1: the ideaChallenge: Airflow 1

Challenge: Airflow 1

Challenge: Airflow 1

Three situations about who does what when a morning pipeline misbehaves.

Challenge questions

  1. On Airflow 3, Thomas deploys a DAG at 15:00 with schedule="0 6 * * *", start_date=datetime(2024, 1, 1) and catchup=False. Finance needs today's numbers. What happens?

    1. It runs at once for today's 06:00 tick, which it missed
    2. Its first run is tomorrow at 06:00; trigger today's by hand
    3. It replays every day since January 2024, then runs
    4. It waits for start_date's anniversary, then runs daily
  2. Thomas creates connection snowflake_pro on a fresh Airflow 3 install and wants to check it before the 06:00 run, but the Test button is unavailable and airflow connections test refuses too. Why?

    1. The DAG that uses it has to be unpaused first
    2. The connection is broken and has to be recreated
    3. Testing is off by default until an admin enables it
    4. Only AIRFLOW_CONN_ environment variables can be tested
  3. Your dev Deployment hibernates at 20:00. At 19:30 you start a backfill that needs an hour. What happens at 20:00, and how do you avoid it?

    1. The task finishes on its worker, and only new tasks wait until the morning
    2. It's killed and marked failed, so set a wake override or start earlier
    3. The task pauses, then picks up where it stopped when the Deployment wakes
    4. Production takes over the task, since both Deployments share the DAG code and the connection