Course contents

The Analytics Engineer’s Field GuideAirflow 3: running it for a teamStep 8: Rules a machine checks

Rules a machine checks

Part 3 · Airflow · step 8

Standards that hold: rules a machine can check

A platform with N teams survives on conventions a machine can check. Names, tags, owners, enforced at merge, visible in the UI.

· · ·

The previous lesson built the chassis. This one keeps fifteen teams from shipping chaos into it, and the answer is boring on purpose: a naming rule, a tag rule, and a CI job that says no.

Two levers, and why a machine pulls them

  • Custom operators. The platform team ships tested, documented building blocks (a dbt operator, a Snowflake operator, a Tableau operator), so feature teams write SQL instead of reinventing orchestration plumbing.
  • Naming and tagging rules, checked by CI. Every DAG is called team__domain__type and carries the tags owner, domain and criticality. A commit that doesn't conform gets its merge blocked.

The second one is this lesson. It sounds bureaucratic until you open an Airflow UI with three hundred DAGs in it and try to answer "which of these is ours, and which one matters at 3am?"

The rules, at a glance

DAG naming structure
team__domain__type
↑ who owns it  ·  ↑ what it's about  ·  ↑ what it does
Required DAG tags
owner domain criticality
missing one = not deployable
Enforced by CI/CD
commit→ validation→ ✓ allow merge✗ block merge

Two commits, two fates

✗ NON-CONFORMING · MERGE BLOCKED
daily_sales_update
No team, no type, and its tags name three owners at once. Six months on, nobody would know whose it is, so CI blocks it at the door.
✓ COMPLIANT · MERGE APPROVED
marketing__leads__ingestion
Team, domain and type in the name, tags owner:marketing, domain:growth, criticality:high. Anyone can route an incident in five seconds.

What the Airflow UI looks like once the rule holds

DAG
Owner (team)
Schedule
● sales__kpi__aggregation
aggregate and publish daily sales KPIs
sales
@daily
● marketing__leads__ingestion
ingest marketing leads from source systems
marketing
@hourly
● revenue__mart__transformation
build and refresh the revenue 360 mart
revenue
@daily
● loyalty__dq__validation
run data quality checks on critical datasets
loyalty
@daily

Sort three hundred DAGs by name and they group by team for free. Filter one tag and you have a team's whole estate. That's what a naming rule buys: findability at 3am.

The job that says no

The check is a short script in the shared CI templates. It loads every DAG the way Airflow would, lists every broken rule, and fails the pipeline if there's one:

# ci/check_dag_standards.py, run on every merge request
import re, sys
from airflow.models import DagBag

NAME = re.compile(r"^[a-z0-9]+__[a-z0-9_]+__[a-z]+$")   # team__domain__type
REQUIRED = {"owner", "domain", "criticality"}

bag = DagBag("dags/", include_examples=False)
errors = [f"{path}: import error" for path in bag.import_errors]
for dag_id, dag in bag.dags.items():
    if not NAME.match(dag_id):
        errors.append(f"{dag_id}: name must be team__domain__type")
    if missing := REQUIRED - {t.split(":")[0] for t in dag.tags}:
        errors.append(f"{dag_id}: missing tags {sorted(missing)}")

print("\n".join(errors) or "all DAGs conform")
sys.exit(1 if errors else 0)          # non-zero = merge blocked
The mental model
Independence at the edges (a repo per team), consistency at the center (one core, one template, one pipeline, rules a machine checks), and nothing ships without someone deciding it should.

Lesson quiz

  1. Why enforce DAG naming and tags in CI instead of a wiki page?

    1. CI checks cost less to run each month than keeping a wiki page up to date
    2. Airflow refuses to load any DAG whose name doesn't follow the platform's declared naming convention
    3. A rule a pipeline checks holds by construction; a written one lasts until the first deadline
    4. A wiki page can't describe rules as precise as the team__domain__type pattern
  2. What does the name marketing__leads__ingestion tell you before you open the DAG?

    1. The domain, the criticality, and the warehouse
    2. The owner, the schedule, and the criticality
    3. The owning team, the domain, and what the DAG does
    4. The team, the Snowflake database, and the schema
  3. A DAG has tags=["owner:finance", "domain:revenue", "high"]. What does the lesson's check_dag_standards.py report for it?

    1. missing tags ['owner', 'domain']
    2. Nothing, since three tags are present
    3. An error that high isn't a valid level
    4. missing tags ['criticality']