Scenario
Every business decision, compliance review and AI initiative depends on people being able to trust and understand their data. When that understanding lives only in the heads of the engineers who built the pipelines, onboarding slows, governance stalls and AI tools give weaker answers. Picture a domain team that owns a Snowflake schema with several hundred tables, all built and tested in dbt. The pipelines run every night, and the people who built them know what each table holds. Then a new analyst joins, or a data steward starts a review, or someone asks an AI assistant a question about the data. The dbt YAML files describe almost nothing, so everyone has to ask around.
This is the situation that led us to build the utility described in this article. The team did not lack the will to document their data. They lacked the time, because writing descriptions for thousands of columns by hand is slow, repetitive work that always loses to the next delivery. We built a Streamlit app that drafts the descriptions with Snowflake Cortex AI and hands the team a ready-to-use dbt YAML file.
Why Documentation Matters
Descriptions in a data model are easy to treat as a nice-to-have. In practice, three things depend on them:
- Governance. A steward cannot review, classify or take ownership of a column that nobody has described. A description is the first sign that a dataset is understood and managed.
- Discoverability. Analysts look for data by what it means, not by what it is called. A column named AMT_3 tells them nothing, and a missing description sends them to a colleague.
- AI readiness. Semantic layers and tools such as Cortex Analyst read descriptions to decide which table or column answers a question. Without that context, the answers get weaker and harder to trust.
Our approach at GrowthArc is human-first, AI-assisted. We let Snowflake Cortex AI write the first draft, because that is the part that does not scale by hand, and we keep people in charge of the final text. The developer reviews the draft, the domain steward corrects it, and the result lives in dbt YAML under version control like any other code. The rest of this article shows how the utility does that.
The Problem
A dbt project is only as useful to other people as its descriptions allow. On the schema we built this for, hundreds of tables and thousands of columns were waiting for a description in the YAML files.
Writing all of this by hand is not realistic at this scale, and sending machine-written text to production without a check is not acceptable either. The requirements came down to three points:
- Hundreds of tables with thousands of columns need descriptions in dbt YAML files
- Writing descriptions manually is not feasible at scale
- Descriptions need an initial validation before going to production
The Solution
We built the Metadata YAML Generator on three pieces of the stack we already used: Snowflake Cortex AI to draft the descriptions, Streamlit in Snowflake for the app that developers work in, and dbt as the home of the final YAML.
For a domain developer the outcome is simple. They pick a database, a schema and the tables, click Generate, and download a dbt YAML file that is ready to review and commit. They do not write prompts, call functions or merge files by hand.
Two design choices make it safe to run again and again. Anything a developer has already written is kept as it is, and anything the AI has written before is cached and reused.
Here is what the tool delivers:
- A Streamlit in Snowflake app: Metadata YAML Generator
- Select the database, schema, and tables → click Generate → download a ready-to-use dbt YAML
- AI fills in missing descriptions, existing descriptions are preserved
- Descriptions are cached — subsequent runs are faster (no repeated AI calls)
How It Works
High-Level Flow
The utility works in three stages. First, the developer uses the app to choose the database, schema and tables to document, and can upload the YAML file they already have. Second, for every table and column, the stored procedure behind the app decides where the description should come from: the uploaded YAML, a history table of earlier AI output, or a fresh call to Cortex AI. Third, the developer downloads the finished YAML, reviews it, places it in the dbt project and commits it, and CI/CD carries it to QA and production like any other change. The diagram below shows the flow.
Priority Logic (Description Resolution)
Whatever a person has already written is never touched. The AI only fills the gaps, and what it writes is saved in a history table. That also has a cost benefit: a column is sent to Cortex AI once, and every later run reuses the stored description instead of making another AI call. Running the tool again on a large schema therefore adds little cost, and reviewers are not asked to re-read descriptions that changed for no reason.
When the tool processes each column, it checks in this order:
Batch Processing
Calling an AI function for every column in a large schema takes time, and one long-running request can hit a timeout. To avoid this, the app splits the work into batches.
When a schema has ten or more tables, the app processes them in batches of ten, one batch after another. A schema with 45 tables, for example, runs as five batches. A progress bar shows which batch is running, so the developer can see that work is moving and how far along it is.
Cached descriptions help here as well. Tables processed on an earlier run are read from the history table, so a repeat run spends its time only on tables that are new or changed.
The developer does not manage any of this. Whatever the size of the schema, the steps stay the same: select the tables, click Generate and wait for the progress bar to finish.
Using the App
Below is the Streamlit application interface.
In the app, the developer selects a database, schema, and tables. The tool then generates AI-powered descriptions and produces a dbt-compatible YAML file ready for review and commit.
Field values shown above (database, schema, table counts) are placeholders and will differ in each environment.
What the Output Looks Like
The tool merges AI-generated descriptions into the team’s existing YAML structure. This means:
- Descriptions written by the team → Preserved as-is
- Tests, tags, meta, and other configurations → Preserved untouched
- Only empty/missing descriptions → Filled by AI
- New tables/columns not in the YAML → Appended with AI descriptions
Developers never need to merge the output with the existing file by hand. They upload current YAML, and the tool returns a complete, enriched version with everything intact.
Example — before and after (the “# ←” notes are annotations for this article; the tool does not write them):
# YOUR UPLOADED YAML (input):
version: 2
models:
- name: account
description: "" # ← empty, AI will fill
columns:
- name: accountid
description: "Primary key" # ← already set, preserved
tests:
- not_null
- unique
- name: revenue
description: "" # ← empty, AI will fill
# GENERATED OUTPUT:
version: 2
models:
- name: account
description: Stores customer account records including company details and territory assignments.
columns:
- name: accountid
description: "Primary key" # ← YOUR description kept
tests:
- not_null # ← YOUR tests kept
- unique
- name: revenue
description: Annual revenue amount associated with the account in USD.Corrections & Review Process
After generating the YAML and doing an initial validation, the developer shares it with the domain owner or business data steward for review. They know the business context best.
Correction Flow
Review happens in two stages. The developer first does a quick validation of the generated YAML and looks for anything clearly wrong. The file then goes to the domain owner or business data steward, who knows what the data means in practice. If the steward finds a description that needs to change, they raise a JIRA ticket with the correction. The developer applies it directly in the YAML file and commits the final version to the dbt repo. Because the fix is made in the YAML and not in the tool, it is versioned and reviewable like any other change, and the next run keeps it (see Key Points on Corrections).
Key Points on Corrections
- Corrections are made directly in the YAML file: edit the description text, commit, done
- Once corrected in the YAML, the next run with that YAML uploaded preserves the corrections (Priority 1)
- The tool never overwrites descriptions that already exist in the uploaded YAML
- FORCE mode is the only exception: it ignores the uploaded YAML and the history table and regenerates every description, including ones people have written. Use it sparingly, ideally one table at a time
Where the Tool Helps
| Scenario | What to do |
| New schema / first time | Select all tables, no YAML upload, click Generate |
| Added new tables to existing schema | Upload the current YAML, select all tables → only new tables get AI descriptions |
| Table structure changed (new columns) | Upload current YAML, select that table → new columns get descriptions, existing ones stay |
| Want fresh descriptions for everything | Check “FORCE mode”, select tables, Generate (this also replaces human-written descriptions) |
| Corrections received from steward | Edit the YAML file directly, commit to repo |
Where It Runs
Behind the app sit a single stored procedure, a history table that caches AI-generated descriptions across runs, a stage for the input and output YAML files, and a run log for auditing. All of it lives in a dedicated development schema. Only the reviewed YAML moves on to QA and production, through the dbt repo.
For the descriptions to appear as table and column comments in Snowflake, the dbt project needs persist_docs: {relation: true, columns: true} enabled. Without it, the descriptions show up in dbt docs but not in Snowflake itself.
