Course contents

The Analytics Engineer’s Field Guidedbt 2: make it trustworthyStep 9: Docs & lineage

Docs & lineage

Part 2 · dbt · step 9

Docs and lineage: a map of your project

dbt builds you a documentation site and a clickable lineage graph, for free.

· · ·

Alice's project has grown to dozens of models, and a new teammate just joined asking the question every analyst eventually asks: "where does this number actually come from?" Every ref() and source() she wrote already describes how the project fits together. dbt turns that into an interactive docs site and a lineage graph, so the new joiner can see the whole project in minutes instead of asking around.

Describe your models

Add descriptions in YAML. They show up in the docs site next to each model and column.

# models/marts/_models.yml
models:
  - name: fct_orders
    description: "One row per order, with amount in EUR and customer country."
    columns:
      - name: amount_eur
        description: "Order total, converted from cents."

doc() blocks: definitions longer than a sentence

A YAML string is fine for one line. Some definitions need a paragraph, and squeezed into YAML they make the file unreadable. Write them in a markdown file instead. {% docs name %} opens a named block of text, {% enddocs %} closes it, and doc('name') pulls it in:

# models/marts/_docs.md  (any .md name under models/ works)
{% docs amount_eur_definition %}
Order total in euros, converted from cents in stg_orders.
Refunded orders are included: filter on status to drop them.
{% enddocs %}
# models/marts/_models.yml, referencing the block
      - name: amount_eur
        description: "{{ doc('amount_eur_definition') }}"

Here doc() sits inside a description, which is free text, so it needs {{ }}. Every model carrying the column can reference the same block: one definition, no copies drifting apart.

persist_docs: push descriptions into Snowflake itself

By default, everything you wrote lives only in the dbt docs site. Snowflake can store a free-text COMMENT on every table and column. Two lines in dbt_project.yml make dbt copy your descriptions there, for tables and views (an ephemeral model has no object to comment on):

# dbt_project.yml  (+ marks a config, not a folder; unrelated to --select's +)
models:
  my_project:
    +persist_docs:
      relation: true    # comment on the table itself
      columns: true     # comment on every column

Now anyone browsing in Snowsight (Snowflake's web interface), or any tool reading INFORMATION_SCHEMA (the views that list every table and column), sees the descriptions. The documentation travels with the data.

Generate and serve the docs

# build the docs: writes target/manifest.json + target/catalog.json
dbt docs generate

# open the site at http://localhost:8080, on your machine only
dbt docs serve

manifest.json is the project's structure: every model, ref and test, with its compiled SQL. Any dbt command rewrites it. catalog.json is the warehouse's view: real columns and types. To get it, dbt docs generate queries Snowflake, so it needs your connection and a few seconds of warehouse time. The site shows both: what you declared and what actually exists. The new teammate can't see your localhost. Usually an automated job (CI or Airflow) regenerates the site after every merge and publishes it on an internal web page.

The lineage graph

The graph is drawn straight from your ref() and source() calls. Click any model to open its page. You get its description and columns, with types and tests. You get its SQL, raw and compiled (the same compiled SQL as target/compiled/, step 2). And you get its parents and children. That page answers "what is this?" and "can I trust it?". The lineage tab answers "what breaks if I touch it?"

raw.orders
raw.customers
stg_orders
stg_customers
fct_orders
revenue_dashboard
finance_report

Exposures: put the dashboards on the graph

Look at the right edge of that graph: revenue_dashboard and finance_report. dbt can't see inside your BI tool, so those endpoints only appear if you declare them. An exposure is a YAML entry that says "this dashboard depends on these models":

# models/marts/_exposures.yml
exposures:
  - name: revenue_dashboard       # the id, used in selectors
    label: "Revenue dashboard"     # the display name in the docs
    type: dashboard             # or notebook, analysis, ml, application
    maturity: high              # high, medium or low: a label, nothing more
    url: https://bi.ismafrost.com/dashboards/revenue
    description: "Weekly revenue, by country and channel."
    depends_on:
      - ref('fct_orders')
    owner:
      name: Finance team
      email: finance@ismafrost.com

Two payoffs. The lineage graph now ends at the things people actually look at, so "what breaks if I change fct_orders?" includes the dashboards. And selection understands it: dbt build --select +exposure:revenue_dashboard builds and tests every model upstream of that dashboard (the exposure itself has nothing to build). Run it before a risky change ships. dbt can't check an exposure against the BI tool, so the dashboard's owner updates the YAML when the dashboard changes, in a normal pull request.

Why it's worth it
Docs and lineage cost you almost nothing: you already wrote the refs. In return, "where does this number come from?" stops being a meeting and becomes a click.

Lesson quiz

  1. Where does dbt's lineage graph come from?

    1. The ref() and source() calls in your models
    2. The folder structure you create under models/
    3. A dependency list you maintain in dbt_project.yml
    4. Snowflake's access history for the queries dbt ran
  2. dbt docs generate writes two JSON files the site is built from. What does each hold?

    1. manifest.json: the project's structure; catalog.json: the real columns and types
    2. manifest.json: the real column types; catalog.json: models, refs and tests
    3. catalog.json: your YAML descriptions; manifest.json: the rendered lineage image
    4. sources.json: the freshness results; run_results.json: the test outcomes
  3. The revenue dashboard is declared as exposure revenue_dashboard. Before a risky change ships, Alice wants to build and test every model that dashboard depends on, and nothing else. Which command?

    1. dbt build --select exposure:revenue_dashboard+
    2. dbt build --select fct_orders+
    3. dbt docs generate --select +exposure:revenue_dashboard
    4. dbt build --select +exposure:revenue_dashboard