# How to Build Reliable Data Pipelines with Delta Live Tables and Medallion Architecture

> Delta Live Tables simplifies reliable pipelines in Databricks with Medallion Architecture. Tutorial with Bronze, Silver, Gold, and data quality.

Source: https://www.erathos.com/en/blog/reliable-data-pipelines-delta-live-tables-medallion
Em português: https://www.erathos.com/blog/reliable-data-pipelines-delta-live-tables-medallion
Published: 2026-01-22
Category: Tutorials

![Data Pipelines with Delta Live Tables and Medallion Architecture](https://cms-media.erathos.com/vKAGhFXBvQ70iQPAMBq5APLtcuE-1.png)

## Creating Your First Data Pipeline with Medallion Architecture and Delta Live Tables (DLT)

Learn how to build a robust data pipeline using DLT, the Medallion architecture, and governance via Unity Catalog in Databricks.

[Delta Live Tables (DLT)](https://www.databricks.com/product/delta-live-tables) is the declarative framework from Databricks for building reliable, scalable, and observable data pipelines. Unlike traditional approaches based on scheduled notebooks or isolated SQL scripts, DLT was designed to solve a common problem in analytical environments: **the operational complexity of maintaining consistent data pipelines over time**.

In data engineering projects, it's common to start with simple transformations using Delta tables and Databricks jobs. However, as data volume grows and pipelines become more critical, challenges arise such as dependency control, error handling, data quality validation, schema evolution, and end-to-end pipeline observability.

This is the context where DLT becomes the most suitable choice. By adopting a declarative approach, the data engineer describes **the desired state of the data**, for example, which tables should exist, their quality expectations, and their dependencies, while Databricks automatically handles orchestration, infrastructure management, monitoring, and recovery in case of failures.

Recently integrated into Databricks Lakeflow as part of _Declarative Pipelines_, DLT stands out as the ideal solution for continuous or scheduled pipelines requiring reliability, governance, and ease of maintenance. It does not completely replace other ways of creating tables in Databricks, but it shines in scenarios where predictability, data quality, and observability are core requirements.

In this article, we explore how to use Delta Live Tables to build well-structured data pipelines, demonstrating in practice how it can be integrated with external ingestion processes and used as the central transformation layer within Databricks.

### Unity Catalog and Unified Governance

Unlike legacy architectures that relied on DBFS, modern pipelines operate under [Unity Catalog (UC)](https://docs.databricks.com/en/data-governance/unity-catalog/index.html). UC provides:

- **Unified Governance:** Centralized access control and automatic data lineage.
- **Volumes vs. Tables:** Unity Catalog differentiates raw files (Volumes) from processed data registered as managed tables.
- **Isolation:** Ease of separating `dev`, `staging`, and `prod` environments within the same metastore.

## What is the Medallion Architecture?

![hUMLKuHlolcc7pKsDKBVCuj0Ebo.png](https://cms-media.erathos.com/hUMLKuHlolcc7pKsDKBVCuj0Ebo-1.png)

The [Medallion architecture ](/blog/arquitetura-medalhao-guiacompleto)describes a series of data layers denoting the quality of data stored in the Lakehouse:

- **Bronze:** The landing zone. Data is kept in its raw format, allowing reprocessing if necessary.
- **Silver:** Data is cleansed, normalized, and validated. Here, we apply _Expectations_ (quality rules).
- **Gold:** The final layer, featuring aggregated data ready for consumption by BI analysts and Machine Learning models.

> By using DLT within the Lakeflow ecosystem, you get native observability: Databricks automatically generates the lineage graph and monitors pipeline health without requiring you to configure external tools.

### Why use Delta Live Tables?

- **Infrastructure Management:** Databricks automatically scales compute resources.
- **Native Data Quality:** Define Expectations to prevent corrupted data from reaching downstream layers.
- **Automatic Lineage:** Visualize how data flows from source to final consumption.
- **Streaming and Batch Support:** Process data in real time or in batches using the same SQL or Python syntax.

> For this guide, we will use SQL, which is the most common language for analytical transformations in Databricks, but DLT also fully supports Python.

## Prerequisites

To get the most out of this tutorial, make sure you understand:

- SQL basics.
- The concept of [Medallion Architecture](https://www.databricks.com/glossary/medallion-architecture).
- Basic navigation in the Databricks Workspace.

## Prerequisites: Preparing the Data Source (MongoDB)

Before starting our data pipeline in Databricks, we need an operational data source. In this tutorial, we will use the **MongoDB Atlas Free Tier** as our source database, simulating a real-world transactional data scenario.

**Creating a MongoDB Free Tier Cluster**

To create a free cluster in MongoDB, follow the official MongoDB tutorial:

- Deploy a Free Tier Cluster:
[https://www.mongodb.com/pt-br/docs/atlas/tutorial/deploy-free-tier-cluster/](https://www.mongodb.com/pt-br/docs/atlas/tutorial/deploy-free-tier-cluster/)
- Getting Started Guide:
[https://www.mongodb.com/pt-br/docs/atlas/getting-started/](https://www.mongodb.com/pt-br/docs/atlas/getting-started/)

After creating the cluster, make sure to:

- Create a database user
- Allow network access for your IP (or allow access from any IP for testing purposes)
- Copy the connection string

## Step 1: Preparing the Source Data

In this guide, the source data **is not accessed directly by Databricks** via native connectors or Auto Loader. Instead, we use **Erathos** as the ingestion layer, simulating a real modern data architecture scenario where ingestion and transformation are well-defined, separate responsibilities.

[Erathos](https://erathos.com/) is a data ingestion tool that lets you connect to different sources (databases, APIs, and external systems) and load this data directly into your Lakehouse, abstracting the complexity of:

- Authentication
- Incremental extraction
- Scheduling
- Monitoring
- Writing to Delta Lake

### Connecting MongoDB to Erathos

Erathos features a native connector for MongoDB Atlas.

- MongoDB connector documentation:
👉 [https://docs.erathos.com/connectors/databases/mongodb](https://docs.erathos.com/connectors/databases/mongodb)
- Creating and managing connections:
👉 [https://docs.erathos.com/platform/connections](https://docs.erathos.com/platform/connections)

In this step, you will:

1. Create a connection to MongoDB Atlas using your connection string
2. Select the desired collection
3. Define the synchronization mode (full or incremental)

### Configuring Databricks as the Destination

After configuring the source, we set Databricks as the destination for our data.

Erathos offers direct integration with **Databricks + Unity Catalog**, ensuring that the ingested data arrives already governed in the Lakehouse.

- Official documentation:
👉 [https://docs.erathos.com/destinations/databricks](https://docs.erathos.com/destinations/databricks)

In this step, you will:

1. Provide your Databricks workspace details
2. Select the destination catalog and schema
3. Persist the data as Delta tables

### Ingestion Result

For this tutorial, Erathos was used to:

- Connect to a MongoDB database (demo)
- Ingest the collection:
- `theaters` (1,564 records)
- Persist the data as **Delta tables** in Databricks, within a schema governed by **Unity Catalog**

From this point on, all data processing and transformation will be handled exclusively via **Delta Live Tables**, keeping the focus of this article on the transformation and data quality layer.

### Governance and Data Organization

Before starting the DLT pipeline, it is essential to ensure that the **target Schema** already exists in Unity Catalog. DLT strictly follows this hierarchy:

`Catalog > Schema > Table`

Without a pre-created schema, the pipeline won't be able to properly register the metadata or expose the tables for external consumption.

> Note: In a real-world scenario, Erathos could be ingesting data from transactional databases, APIs, or third-party systems, dropping it directly into a governed catalog in Databricks.

## Ingestion Strategies: Batch vs. Streaming

Before coding our first layer, it's important to understand how DLT consumes different types of sources. Whether data is ingested via Erathos, Auto Loader, or another mechanism, DLT supports both batch and streaming processing.

1. **Auto Loader**: Used to ingest raw files incrementally using `cloud_files`. This is the most common approach. You point to a folder in cloud storage (S3, ADLS, GCS) or a **Unity Catalog Volume**.
- **How it works:** Databricks monitors the arrival of new files (JSON, CSV, Parquet) and processes only the new ones.
- **Code Example:**

`CREATE OR REFRESH STREAMING TABLE taxi_raw_bronze`
`AS SELECT * FROM cloud_files("/Volumes/main/default/my_volume/raw_data/", "json");`

1. **Via Delta Table (Stream from Table)**: If your source is already a Delta Table (rather than raw files), it needs to support change tracking (Change Data Feed).
- **How it works:** You read the table as a continuous stream.
- **Code Example:**

`CREATE OR REFRESH STREAMING TABLE bronze_table`
`AS SELECT * FROM STREAM(catalog.schema.source_delta_table);`

> **Implementation Note:** In this guide, the `theaters` data was ingested beforehand via Erathos and persisted as static Delta tables. For this reason, we will use the `LIVE TABLE` command (batch). Even so, the Medallion architecture remains the same and can easily be adapted for incremental or streaming sources.

## Technical Prerequisites in Databricks

- Databricks Workspace with **Unity Catalog** enabled.
- Permissions to create DLT pipelines and write to a schema in your catalog.

## Step 2: Create the Transformation Notebook

**1.** In your Workspace, click **New** > **Notebook**. **2.** Name it `dlt_medallion_pipeline`. **3.** Make sure the default language is set to **SQL**.

Table definitions in Delta Live Tables are declarative and live in SQL or Python notebooks. Each `CREATE OR REFRESH LIVE TABLE` command describes _what_ the table should look like, while Databricks automatically manages _how_ data is processed, versioned, and optimized inside the pipeline.

## Bronze Layer

The Bronze layer represents the entry point for data in our Lakehouse. In this scenario, the data has already been ingested into Databricks using **Erathos**, which handles extraction and incremental synchronization from the source database.

The primary goal of Bronze is **fidelity**: we capture data with minimal transformations, preserving the original format, including JSON fields to ensure traceability and allow future reprocessing.

`CREATE OR REFRESH LIVE TABLE theaters_bronze`
`COMMENT "Bronze layer: raw theaters data from Erathos"`
`AS`
`SELECT`
`  _id,`
`  theater_id,`
`  location,`
`  _erathos_execution_id,`
`  _erathos_synced_at`
`FROM erathos_db.erathos_db.theaters`

## Silver Layer

The Silver layer is where the technical complexity increases. Here, we perform **data structuring**, apply **data quality rules**, and **normalize complex fields** (such as JSONs), converting raw data into a reliable, analytical format.

The `location` field is stored at the source as a JSON string. In the Silver layer, we use the `FROM_JSON` function to convert this field into a typed structure (`STRUCT`), allowing direct access to address attributes and geographic coordinates, as well as applying quality rules to this data.

`CREATE OR REFRESH LIVE TABLE theaters_cleaned_silver (`

`  CONSTRAINT valid_theater_id`
`    EXPECT (theater_id IS NOT NULL)`
`    ON VIOLATION FAIL UPDATE,`

`  CONSTRAINT valid_state`
`    EXPECT (state IS NOT NULL)`
`    ON VIOLATION DROP ROW,`

`  CONSTRAINT valid_coordinates`
`    EXPECT (`
`      latitude IS NOT NULL`
`      AND longitude IS NOT NULL`
`    )`

`)`
`COMMENT "Silver layer: cleaned and structured theaters data"`
`AS`
`SELECT`
`  _id,`
`  theater_id,`

`  location_struct.address.street1 AS street,`
`  location_struct.address.city    AS city,`
`  location_struct.address.state   AS state,`
`  location_struct.address.zipcode AS zipcode,`

`  location_struct.geo.coordinates[0] AS longitude,`
`  location_struct.geo.coordinates[1] AS latitude,`

`  _erathos_synced_at AS ingested_at`

`FROM (`
`  SELECT`
`    *,`
`    FROM_JSON(`
`      location,`
`      'STRUCT<`
`         address: STRUCT<`
`           street1: STRING,`
`           city: STRING,`
`           state: STRING,`
`           zipcode: STRING`
`         >,`
`         geo: STRUCT<`
`           type: STRING,`
`           coordinates: ARRAY<DOUBLE>`
`         >`
`       >'`
`    ) AS location_struct`
`  FROM LIVE.theaters_bronze`
`);`

**Expectation Severity Levels:**

- **EXPECT:** Only generates metrics. Ideal for understanding data cleanliness issues without stopping the pipeline.
- **DROP ROW:** Ensures the Silver layer contains only trusted data.
- **FAIL UPDATE:** Halts processing. Critical to ensure that calculations like tax or payment processing are never run with null values.

> Expectations also automatically generate metrics within the pipeline, enabling you to track data quality over time.

## Gold Layer

The Gold layer is optimized for final consumption, offering stable, simple, and high-performance tables for analytics, BI, and downstream applications.

`CREATE OR REFRESH LIVE TABLE theaters_gold`
`COMMENT "Gold layer: theaters dimension table"`
`AS`
`SELECT`
`  theater_id,`
`  street,`
`  city,`
`  state,`
`  zipcode,`
`  latitude,`
`  longitude`
`FROM LIVE.theaters_cleaned_silver;`

**Autonomous Maintenance (Vacuum and Optimize):**

Unlike standard Spark tables, DLT automatically manages [OPTIMIZE](https://docs.databricks.com/en/delta/optimize.html) (small file compaction) and [VACUUM](https://docs.databricks.com/en/delta/vacuum.html) (old file cleanup). This ensures that Gold layer queries remain highly performant, even with multiple incremental updates over time.

## Step 3: Configure the Pipeline in Unity Catalog

Now that the code is ready, we need to create the Pipeline object to run it.

- In the sidebar, click **Jobs & Pipelines**.
- From the "Create New" menu, select **ETL pipeline** (Build ETL pipelines using SQL and Python).
- In the configuration screen, you must **provide a catalog and schema** in the upper right corner to bind your tables and logs to Unity Catalog.
- Select **Add existing assets** to link the notebook created in Step 2.

> Note on Modernization: When selecting your code, Databricks might display a "Legacy configuration" warning. This happens because Lakeflow favors raw code files (.sql) to support DevOps best practices. For this tutorial, we will stick with the Notebook format for easier immediate data visualization, but in large-scale production environments, migrating to Workspace Files is recommended.

## Step 4: Execute and Validate

1. On your Pipeline screen, click **Run pipeline**.
2. Databricks will spin up a cluster, and you will see the lineage graph **(DAG)** render on the screen.
3. Track progress: the graph will display the row count moving from Bronze to Gold.

If any row violates the **theater\_id** IS NOT NULL rule defined in the Silver layer, DLT will drop that record and log the metric in the quality dashboard.

## Conclusion

You have successfully implemented a robust data pipeline utilizing the **Medallion Architecture** and **Delta Live Tables** within the **Databricks Lakeflow** ecosystem. By following this guide, you have established a solid foundation for modern data engineering, ensuring:

- **Intelligent Ingestion**: Understanding the flexibility between processing via **Auto Loader** for raw files and ingesting existing Delta tables.
- **Active Governance**: Implementing **Expectations** to guarantee only high-quality data reaches consumer layers, reducing debugging time.
- **Native Performance**: An architecture that benefits from automated maintenance like **Optimize** and **Vacuum**, ensuring fast queries in the Gold layer without manual overhead.
- **Lineage and Transparency**: Through **Unity Catalog**, your pipeline now features automatic data lineage, making auditing and compliance much easier.

By integrating an ingestion tool like Erathos with Delta Live Tables, we cleanly separate ingestion and transformation responsibilities, resulting in simpler, more governable, and highly scalable pipelines.