Scaling to many teams: one core, one repo each
Every team keeps its own repo. One core project assembles them all and ships to a single Airflow. Nothing deploys by accident.
Cosmos solved one team's DAG. Now zoom out: five teams, five dbt projects, five sets of DAGs, all landing on one Astronomer cluster. How do you organise the code?
Three ways to arrange it, two of them hurt
What a submodule actually is
- A bookmark, not a copy. A Git submodule mounts one repo inside another at a pinned commit. The outer repo records "include project X, exactly at commit abc123".
- The code stays where it lives. The finance team owns its repo, its merge requests and its release rhythm, and the core pulls in only its
dbt/anddags/folders. - The core decides what ships. Bumping a submodule pointer is an explicit commit in the core repo, so a team's change reaches production when the core says so.
- What deploys is reproducible. Five pinned commits describe the whole assembly, exactly, and you can check out that assembly again a month later.
The alternative worth naming: publishing each project as a versioned package works too, but it adds a release ceremony most data teams don't want. Submodules sit in the middle.
The shape of the platform
Follow the arrows: a template stamps out projects, projects plug into the core as submodules, and the core assembles them into the Airflow that runs on Astronomer (each project's CI triggers the deploy, as shown below). The green box around all of them is the part that saves the most time:
astro-ci-cd-pipelinesthe CI, written once · every repo inside includes it
Project_1own repo · own teamProject_2own repo · own team⋮
Project_Nown repo · own teamEverything inside that green boundary shares one CI, and that is the reason a new team is productive on day one. astro-ci-cd-pipelines holds the pipeline definitions, lint SQL, run dbt on DuckDB, run it on Snowflake, check the naming rules, deploy to Astronomer, and every project includes them rather than copying them:
- The engineer writes almost no CI. A stamped repo arrives with a
.gitlab-ci.ymlthatinclude:s the shared templates. A few variables (project name, dbt target) and the pipeline works. - The core uses the same templates. Assembling submodules and deploying to Astronomer are jobs in that same repo, so there's one definition of "how we ship", not one per team.
- A fix reaches everyone. Improve a job, or bump a dbt version, in the shared repo and every project picks it up on its next pipeline. No sweep through fifteen YAML files.
- Standards travel with it. The naming and tagging checks from the next lesson live in those templates, so the rules are enforced everywhere by default rather than by memory.
How the assembly works
The core repo has exactly two branches of its own, pre_prod and pro, and each one deploys straight to its matching Astronomer environment. Which branch you push to core is what decides where it goes live:
- Project 1…N are submodules of the core GitLab project, the CI/CD pipeline pulls each one's
dagsanddbtfolders into the core repo, nothing else from those projects gets copied over. - The core ends up holding one combined view: all submodules' dags plus all their dbt models, ready to ship as a single unit.
- Each feature project maintains its own
pre_prodandprodbranches, and the core pulls in only the matching branch from each, so pre_prod code can never leak into the production cluster.
One set of requirements for everyone
A classic multi-team failure is dependency drift: team A on dbt 1.9, team B on 1.10, and the shared cluster can only run one. The submodule contract kills it at the source:
- Core pulls only
dags/anddbt/from each submodule. - Submodules' own
include/,Dockerfile,requirements.txt: ignored entirely. - Core has one shared
include/,Dockerfile,requirements.txt, the ones that actually ship. - Same dependencies, same environment, no drift between teams.
Three ways to deploy, one decision
When code lands on core, the pipeline asks a single question, which files changed?, and picks the cheapest deploy that's still correct (a -f in CI scripts is --force, which deploys even when the DAG parse check fails, not a flag for the Deployment ID):
astro deploy $DEPLOYMENT_IDastro deploy $DEPLOYMENT_ID --dagsdags/ folder into the running Deployment. Airflow re-scans that folder, so new DAGs show up on the next scan. No restart, no rebuild.astro dbt deploy $DEPLOYMENT_IDdbt/ project on its own and mounts it at /usr/local/airflow/dbt/<project>, the path Cosmos reads. --dags never ships it, because it isn't under dags/.New project in one command: the Copier template
Nobody hand-builds a repo on this platform. A template project holds one Copier template for all new projects:
- Copier generates a repo instantly: same structure,
dags/,dbt/, tests,.gitlab-ci.yml, CI/CD already wired. - Copier also updates existing projects, pushing template changes to everyone at once, standards propagate instead of rotting.
- The template has one branch per Astro Runtime version (named like
astro/3.0,astro/3.1…). Picking a branch when generating locks the project to that Runtime, and an upgrade later just means moving to the next branch.
One pipeline, every team: what a project's CI file looks like
The shared astro-ci-cd-pipelines project from the diagram is tagged per version, like a library. A team pins a tag, and an upgrade is a one-line bump instead of a rewrite:
# .gitlab-ci. include: - project: 'data-platform/yml in a feature project, the whole deploy astro-ci-cd-pipelines' ref: 'v0.0.2' # pinned tag: upgrades are a one-line bump file: 'one_project_ astro_push_ astro. yml' deploy_ : extends: .pre_ prod astro_ variables: ASTRO_deploy # the shared job, from the include API_ : $GITLAB_TOKEN ASTRO_ DEPLOYMENT_PPD_ API_ TOKEN ID : $GITLAB_ASTRO_ ASTRO_PPD_ DEPLOYMENT_ ID WORKSPACE_ : $GITLAB_ID ASTRO_ environment: "pre_prod" rules: - if: '$CI_PPD_ WORKSPACE_ ID COMMIT_ # fires only on that branchBRANCH == "pre_prod"'
Onboarding a team: one YAML file
Even project creation is declarative. One YAML file lists every project name, the single source of truth, and a CI/CD pipeline diffs desired state against reality:
# projects.yaml, the whole platform roster projects: - name: sales - name: marketing - name: finance - name: customer_360 # ← add a line, get a project
- Add a name: CI/CD creates the repo from the template and adds the submodule to all environments.
- Remove a name: CI/CD deletes the project and removes the submodule everywhere.
- No manual repo setup, no manual submodule commands per team. Declarative list in, correct core project state out, automatically.
pro credentials in exactly one place, the Airflow connections of the pro Deployment that Cosmos's ProfileConfig reads. What keeps N teams from shipping chaos into it, names, tags, ownership, enforced at merge, is the next lesson.
