Course contents

The Analytics Engineer’s Field GuideAirflow 3: running it for a teamStep 7: Scaling to many teams

Scaling to many teams

Part 3 · Airflow · step 7

Scaling to many teams: one core, one repo each

Every team keeps its own repo. One core project assembles them all and ships to a single Airflow. Nothing deploys by accident.

· · ·

Cosmos solved one team's DAG. Now zoom out: five teams, five dbt projects, five sets of DAGs, all landing on one Astronomer cluster. How do you organise the code?

Three ways to arrange it, two of them hurt

Shape
What it means
Why it hurts (or doesn't)
One monorepo
Everyone merges into the same tree.
Every team's mistake is everyone's incident, and one CODEOWNERS file rules the world.
One cluster per team
Five separate Airflows.
Five bills, five upgrade cycles, five things to keep alive.
Core + submodules
Each team keeps its own repo. A core project pulls them all in.
Own repo, own reviews, own pace, and one deployable unit at the end.

What a submodule actually is

  • A bookmark, not a copy. A Git submodule mounts one repo inside another at a pinned commit. The outer repo records "include project X, exactly at commit abc123".
  • The code stays where it lives. The finance team owns its repo, its merge requests and its release rhythm, and the core pulls in only its dbt/ and dags/ folders.
  • The core decides what ships. Bumping a submodule pointer is an explicit commit in the core repo, so a team's change reaches production when the core says so.
  • What deploys is reproducible. Five pinned commits describe the whole assembly, exactly, and you can check out that assembly again a month later.

The alternative worth naming: publishing each project as a versioned package works too, but it adds a release ceremony most data teams don't want. Submodules sit in the middle.

The shape of the platform

Follow the arrows: a template stamps out projects, projects plug into the core as submodules, and the core assembles them into the Airflow that runs on Astronomer (each project's CI triggers the deploy, as shown below). The green box around all of them is the part that saves the most time:

templateCopier, stamps repos

astro-ci-cd-pipelinesthe CI, written once · every repo inside includes it

Project_1own repo · own team
Project_2own repo · own team

⋮

Project_Nown repo · own team
Core projectpulls only dags/ + dbt/pins one commit per project
ASTRONOMERone cluster
a new team gets a stamped repo and the pipeline it needs · nobody writes a CI file from scratch

Everything inside that green boundary shares one CI, and that is the reason a new team is productive on day one. astro-ci-cd-pipelines holds the pipeline definitions, lint SQL, run dbt on DuckDB, run it on Snowflake, check the naming rules, deploy to Astronomer, and every project includes them rather than copying them:

  • The engineer writes almost no CI. A stamped repo arrives with a .gitlab-ci.yml that include:s the shared templates. A few variables (project name, dbt target) and the pipeline works.
  • The core uses the same templates. Assembling submodules and deploying to Astronomer are jobs in that same repo, so there's one definition of "how we ship", not one per team.
  • A fix reaches everyone. Improve a job, or bump a dbt version, in the shared repo and every project picks it up on its next pipeline. No sweep through fifteen YAML files.
  • Standards travel with it. The naming and tagging checks from the next lesson live in those templates, so the rules are enforced everywhere by default rather than by memory.

How the assembly works

The core repo has exactly two branches of its own, pre_prod and pro, and each one deploys straight to its matching Astronomer environment. Which branch you push to core is what decides where it goes live:

  • Project 1…N are submodules of the core GitLab project, the CI/CD pipeline pulls each one's dags and dbt folders into the core repo, nothing else from those projects gets copied over.
  • The core ends up holding one combined view: all submodules' dags plus all their dbt models, ready to ship as a single unit.
  • Each feature project maintains its own pre_prod and prod branches, and the core pulls in only the matching branch from each, so pre_prod code can never leak into the production cluster.
Project 1pre_prodpro
Project 2pre_prodpro
matching branch only
CORE projectpre_prodpro
Airflow · pre_prodAstronomer deploymentonly pre_prod code
Airflow · proAstronomer deploymentonly pro code
colour = branch · the core pulls only the matching branch from each project

One set of requirements for everyone

A classic multi-team failure is dependency drift: team A on dbt 1.9, team B on 1.10, and the shared cluster can only run one. The submodule contract kills it at the source:

  • Core pulls only dags/ and dbt/ from each submodule.
  • Submodules' own include/, Dockerfile, requirements.txt: ignored entirely.
  • Core has one shared include/, Dockerfile, requirements.txt, the ones that actually ship.
  • Same dependencies, same environment, no drift between teams.
core-project/# the only thing Astronomer ever sees ├─ dags/# assembled from every submodule │  ├─ project_1/ project_2/ …# one folder per team ├─ dbt/# all teams' dbt models, side by side ├─ include/# SHARED, core's own ├─ Dockerfile# SHARED, one image for everyone └─ requirements.txt# SHARED, one dependency set

Three ways to deploy, one decision

When code lands on core, the pipeline asks a single question, which files changed?, and picks the cheapest deploy that's still correct (a -f in CI scripts is --force, which deploys even when the DAG parse check fails, not a flag for the Deployment ID):

command
what it does
takes
astro deploy $DEPLOYMENT_ID
Full deploy. Rebuilds the entire Docker image (Dockerfile, requirements.txt, include/) and redeploys it to Astronomer. Needed whenever the environment itself changed.
~5 min
astro deploy $DEPLOYMENT_ID --dags
DAG-only deploy. Skips the image rebuild and syncs only the dags/ folder into the running Deployment. Airflow re-scans that folder, so new DAGs show up on the next scan. No restart, no rebuild.
~1 min
astro dbt deploy $DEPLOYMENT_ID
dbt-only deploy. Pushes the dbt/ project on its own and mounts it at /usr/local/airflow/dbt/<project>, the path Cosmos reads. --dags never ships it, because it isn't under dags/.
~1 min
Push to core
gitlab
→
Which files changed?
the pipeline decides
→
Full deploy
Dockerfile / reqs · ~5 min
only dags/ or dbt/ touched → the matching fast deploy, ~1 minute, the track you'll take 9 times out of 10

New project in one command: the Copier template

Nobody hand-builds a repo on this platform. A template project holds one Copier template for all new projects:

  • Copier generates a repo instantly: same structure, dags/, dbt/, tests, .gitlab-ci.yml, CI/CD already wired.
  • Copier also updates existing projects, pushing template changes to everyone at once, standards propagate instead of rotting.
  • The template has one branch per Astro Runtime version (named like astro/3.0, astro/3.1…). Picking a branch when generating locks the project to that Runtime, and an upgrade later just means moving to the next branch.

One pipeline, every team: what a project's CI file looks like

The shared astro-ci-cd-pipelines project from the diagram is tagged per version, like a library. A team pins a tag, and an upgrade is a one-line bump instead of a rewrite:

# .gitlab-ci.yml in a feature project, the whole deploy
include:
  - project: 'data-platform/astro-ci-cd-pipelines'
    ref: 'v0.0.2'                    # pinned tag: upgrades are a one-line bump
    file: 'one_project_push_astro.yml'

astro_deploy_pre_prod:
  extends: .astro_deploy                 # the shared job, from the include
  variables:
    ASTRO_API_TOKEN:    $GITLAB_ASTRO_PPD_API_TOKEN
    DEPLOYMENT_ID:      $GITLAB_ASTRO_PPD_DEPLOYMENT_ID
    ASTRO_WORKSPACE_ID: $GITLAB_ASTRO_PPD_WORKSPACE_ID
    environment: "pre_prod"
  rules:
    - if: '$CI_COMMIT_BRANCH == "pre_prod"'   # fires only on that branch

Onboarding a team: one YAML file

Even project creation is declarative. One YAML file lists every project name, the single source of truth, and a CI/CD pipeline diffs desired state against reality:

# projects.yaml, the whole platform roster
projects:
  - name: sales
  - name: marketing
  - name: finance
  - name: customer_360   # ← add a line, get a project
  • Add a name: CI/CD creates the repo from the template and adds the submodule to all environments.
  • Remove a name: CI/CD deletes the project and removes the submodule everywhere.
  • No manual repo setup, no manual submodule commands per team. Declarative list in, correct core project state out, automatically.
What's still missing
The chassis exists: repos assembled, one pipeline, one deploy, and the pro credentials in exactly one place, the Airflow connections of the pro Deployment that Cosmos's ProfileConfig reads. What keeps N teams from shipping chaos into it, names, tags, ownership, enforced at merge, is the next lesson.

Lesson quiz

  1. How do N team projects end up on one Astronomer cluster?

    1. Each team deploys its own repo straight to the shared cluster once it's reviewed
    2. Each is a submodule of a core repo, which takes only dags/ and dbt/ and deploys
    3. All teams commit to one monorepo, with a shared CODEOWNERS file for reviews
    4. Each team gets a separate Astronomer cluster and deploys its repo there
  2. A push to core changes only files under dags/. What does the pipeline run?

    1. astro deploy --dags, syncing dags/ with no image rebuild
    2. astro deploy -f, the short flag for a DAG-only sync
    3. A scheduler restart so Airflow picks up the new files
    4. astro deploy, rebuilding the whole image to be safe
  3. The finance team adds a Python library to the requirements.txt in its own repo, and its DAG imports it. After the core deploys, the DAG fails with ModuleNotFoundError. Why?

    1. The DAG-only deploy skipped it; rerun with --dags --force
    2. Each submodule needs its own Astronomer Deployment
    3. The core ignores team requirements; add it to the core's
    4. Cosmos strips imports out of DAGs that it generates