Course contents

The Analytics Engineer’s Field GuideAirflow 2: writing DAGsChallenge: Airflow 2

Challenge: Airflow 2

Challenge: Airflow 2

Three situations where a DAG that looks fine misbehaves in production.

Challenge questions

  1. Twenty SqlSensor tasks and twenty S3KeySensor tasks each wait up to two hours every morning. Thomas wants the waiting to tie up as few worker slots as possible. What's the best setup?

    1. mode="poke" with poke_interval=1800 on all forty
    2. deferrable=True on the S3 sensors, reschedule on the SqlSensors
    3. deferrable=True on all forty, handing every wait to the triggerer
    4. mode="reschedule" on all forty, which never holds a slot
  2. process_country.expand(country=countries) runs for 11 countries with retries=2. FR fails twice on a blip, then succeeds; DE fails on all three tries. What's the state of the run?

    1. FR succeeded on its third try; only DE has failed
    2. The mapped task failed as one unit, so all 11 rerun
    3. FR is failed too, since its first two tries failed
    4. All eleven reran each time FR or DE failed
  3. pick_mode chooses a full refresh on Mondays, using datetime.now(). Monday's run failed on a Snowflake outage, and on Tuesday Thomas clears the whole run. What happens, and what's the fix?

    1. It takes the incremental path, so read the run's logical_date instead of now()
    2. It can't run, since catchup=False blocks past dates, so set catchup=True
    3. It full-refreshes, because a cleared run restores the clock from its first try
    4. Both paths run, since a branch can't decide on a cleared run, so add a sensor