Course contents

The Analytics Engineer’s Field GuideAirflow 1: the ideaStep 3: Managed Airflow: Astronomer

Managed Airflow: Astronomer

Part 3 · Airflow · step 3

Astronomer: managed Airflow, and what it saves you

Somebody has to keep Airflow itself alive. Astronomer is the answer to that question, and it costs you one command.

· · ·

The problem: Airflow has to live somewhere

Airflow is software, and software has to run somewhere. Left to yourself that means a scheduler process, a metadata database, workers, an API server for the UI, and someone patching all of it. Thomas's day-33 problem was exactly this: a machine nobody was watching.

Astronomer runs that for you. Same Airflow, same Python files, same UI, except the servers are somebody else's problem. It's the trade you already made with Snowflake: you could install a database on a machine you patch yourself, and instead you never think about the machine.

What you get in practice:

  • It's always up. Scheduler, database, workers and API server are hosted, monitored, patched and upgraded. No reboot takes your pipelines down.
  • It scales on its own. Twenty DAGs one month, two hundred the next, and you don't size a server for it.
  • Local development is one command. astro dev start runs Airflow in Docker on your laptop, on the same Runtime image as production, so you test a DAG before anyone sees it.
  • Deployment is one command. astro deploy ships your project, and in this platform GitLab CI runs it for you after the merge.
  • Support and guardrails. Sensible defaults, an upgrade path that doesn't involve a migration weekend, and people to call when Airflow itself misbehaves.
The one-line version
Airflow decides when your models run and what happens when they don't. Astronomer makes sure Airflow itself is up, patched and scaled, so that decision never depends on a server nobody is watching.

Everything in this guide runs on Astronomer, and nothing you learn here would change if it didn't.

The four pieces you actually touch

Astronomer's offer comes as four named pieces, and you'll hear all four names:

Astro CLI

Local development and deploys. One command spins up Airflow on your laptop with Docker, and DAG changes apply as you save.

Astro

The hosted platform, on AWS, GCP or Azure. Each environment is a Deployment, with auto-scaling workers and no infrastructure to maintain.

Astro Runtime

The Airflow Docker image Astronomer builds, tests and versions, with common providers pre-installed and a published support lifecycle.

Astro Observe

Data observability on top of Airflow: lineage, and freshness or delivery SLAs on the data products a business relies on.

Installing it, and running Airflow on your laptop

Docker Desktop has to be running first, because the CLI starts the whole platform as containers. Then it's four commands:

# 1. install the CLI
brew install astro                        # macOS
# Linux:   curl -sSL install.astronomer.io | sudo bash -s
# Windows: winget install -e --id Astronomer.Astro
astro version

# 2. create a project (dags/, include/, requirements.txt, Dockerfile)
mkdir pizzeria-airflow && cd pizzeria-airflow
astro dev init

# 3. start it locally: every Airflow component + a Postgres metadata DB
astro dev start
#    → http://localhost:8080   (login details print in the terminal)
#    drop a .py file into dags/ and it appears, no restart needed

# 4. the ones you'll use daily
astro dev logs -f                         # follow the Airflow logs
astro dev restart                         # after changing requirements.txt, packages.txt or the Dockerfile
astro dev bash                            # a shell inside the container
astro dev stop                            # give the laptop its RAM back

Adding a Python package (say astronomer-cosmos for the dbt lesson) means one line in requirements.txt and an astro dev restart. When it works locally, astro deploy sends the same project to the hosted Deployment.

Hibernation: switch a whole environment off

A Deployment with Development Mode turned on can be hibernated. Scheduler, API server and workers scale to zero and the Deployment stops consuming compute; its configuration is kept, and it wakes up as it was. While it sleeps, nothing runs: no DAGs, no UI, no deploys, and a task still running when it falls asleep is marked failed.

That matters because most environments aren't needed most of the time:

  • Dev environments overnight and at weekends. A dev Deployment that only runs while people are working can sleep two thirds of the week.
  • On a schedule you set. Define wake hours in the Astro UI (calendar or cron, in UTC) and Astronomer applies them without anyone remembering.
  • Or on demand. Override the schedule from the UI, the API or astro deployment wake-up when someone needs to test outside the window.
  • Spin environments up and down freely. A Deployment per team or per experiment gets cheap when the idle ones sleep, so you create the environment you need instead of sharing one.
The habit worth forming
Production stays awake (a Deployment without Development Mode can't hibernate anyway). Every dev Deployment gets a hibernation schedule the day it's created, and you stop paying for machines nobody is using.

What you'd be operating without it

Here are the parts behind that first list, the ones you'd otherwise run yourself:

Scheduler
decides what runs
+
Workers
run the tasks
+
Metadata DB
history, state
+
API server
the UI (Airflow 2's webserver)
Astronomer runs and scales all four for you
· · ·

The knobs you still get to turn

Astronomer runs these four for you. The one knob you still choose is how the workers are housed: the executor.

The executor: how tasks actually run

Executor
What it does
Where you meet it
LocalExecutor
Runs tasks as subprocesses inside the scheduler.
Your laptop: it's what astro dev start uses. Astro Deployments don't offer it.
Astro executor
Astronomer's own: worker agents pull tasks from the API server.
The default for new Airflow 3 Deployments on Astro.
CeleryExecutor
A pool of long-running workers pulls tasks off a queue.
Steady high throughput on a shared worker image.
KubernetesExecutor
One fresh pod per task, with its own resources.
Per-task compute isolation, at the cost of pod-startup latency.

The real choice is isolation. Celery and the Astro executor share each worker across many tasks (fast, but a heavy task can starve the small ones unless you split them into worker queues), Kubernetes gives each task its own pod and resource limits (isolated, but every task pays startup time). A dbt run that needs 8 GB shouldn't share a worker with fifty tiny sensors.

CeleryExecutor · one shared worker

dbt buildeats 8 GB of the worker
50 sensors, squeezed
the heavy task starves the small ones

KubernetesExecutor · one pod per task

dbt buildits pod: 8 GB · 2 CPU
spark job16 GB · 4 CPU
sensor · 0.1 CPU
each declares its appetite, crashes alone, leaves no residue
same three workloads, two ways to house them

Why KubernetesExecutor is popular on data platforms: a dbt build, a Spark submit and a tiny sensor have wildly different appetites, and one pod per task means each declares its own CPU and memory. The trade is a few seconds of pod startup per task. Here's its whole life:

Task is due
scheduler queues it
→
Fresh pod spawns
its own CPU + memory
→
Task runs inside
isolated from everything
→
Pod dies
state saved, nothing left

That's the machine: a scheduler that watches the clock, workers that do the work, an executor that decides how they're housed, and Astronomer running all of it. Next lesson: the tasks themselves.

Lesson quiz

  1. What does Astronomer run on your behalf?

    1. Airflow's scheduler, workers, metadata database and API server
    2. Your Snowflake warehouses, resized automatically on demand
    3. Your DAG code, after converting it into Astronomer's own format
    4. Your dbt models, on Astronomer's own compute instead of Snowflake
  2. A dbt task needing 8 GB shouldn't share a worker with fifty tiny sensors. Which executor on an Astro Deployment gives each task its own resources?

    1. LocalExecutor, running tasks as subprocesses
    2. KubernetesExecutor, one fresh pod per task
    3. The Astro executor, whose agents share each worker
    4. CeleryExecutor, with its long-running worker pool
  3. To cut costs, a teammate proposes a hibernation schedule on the production Deployment too, from 22:00 to 05:00. What's the catch?

    1. It can't: hibernation needs Development Mode
    2. It hibernates, but keeps scheduled DAGs running
    3. Each night's hibernation wipes its configuration
    4. None; production hibernates exactly like dev