How to Build Reliable Data Pipelines with Delta Live Tables and Medallion Architecture

Creating Your First Data Pipeline with Medallion Architecture and Delta Live Tables (DLT)

Learn how to build a robust data pipeline using DLT, the Medallion architecture, and governance via Unity Catalog in Databricks.

Delta Live Tables (DLT) is the declarative framework from Databricks for building reliable, scalable, and observable data pipelines. Unlike traditional approaches based on scheduled notebooks or isolated SQL scripts, DLT was designed to solve a common problem in analytical environments: the operational complexity of maintaining consistent data pipelines over time.

In data engineering projects, it's common to start with simple transformations using Delta tables and Databricks jobs. However, as data volume grows and pipelines become more critical, challenges arise such as dependency control, error handling, data quality validation, schema evolution, and end-to-end pipeline observability.

This is the context where DLT becomes the most suitable choice. By adopting a declarative approach, the data engineer describes the desired state of the data, for example, which tables should exist, their quality expectations, and their dependencies, while Databricks automatically handles orchestration, infrastructure management, monitoring, and recovery in case of failures.

Recently integrated into Databricks Lakeflow as part of Declarative Pipelines, DLT stands out as the ideal solution for continuous or scheduled pipelines requiring reliability, governance, and ease of maintenance. It does not completely replace other ways of creating tables in Databricks, but it shines in scenarios where predictability, data quality, and observability are core requirements.

In this article, we explore how to use Delta Live Tables to build well-structured data pipelines, demonstrating in practice how it can be integrated with external ingestion processes and used as the central transformation layer within Databricks.

Unity Catalog and Unified Governance

Unlike legacy architectures that relied on DBFS, modern pipelines operate under Unity Catalog (UC). UC provides:

• Unified Governance: Centralized access control and automatic data lineage.
• Volumes vs. Tables: Unity Catalog differentiates raw files (Volumes) from processed data registered as managed tables.
• Isolation: Ease of separating dev, staging, and prod environments within the same metastore.

What is the Medallion Architecture?

The Medallion architecture describes a series of data layers denoting the quality of data stored in the Lakehouse:

• Bronze: The landing zone. Data is kept in its raw format, allowing reprocessing if necessary.
• Silver: Data is cleansed, normalized, and validated. Here, we apply Expectations (quality rules).
• Gold: The final layer, featuring aggregated data ready for consumption by BI analysts and Machine Learning models.

By using DLT within the Lakeflow ecosystem, you get native observability: Databricks automatically generates the lineage graph and monitors pipeline health without requiring you to configure external tools.

Why use Delta Live Tables?

• Infrastructure Management: Databricks automatically scales compute resources.
• Native Data Quality: Define Expectations to prevent corrupted data from reaching downstream layers.
• Automatic Lineage: Visualize how data flows from source to final consumption.
• Streaming and Batch Support: Process data in real time or in batches using the same SQL or Python syntax.

For this guide, we will use SQL, which is the most common language for analytical transformations in Databricks, but DLT also fully supports Python.

Prerequisites

To get the most out of this tutorial, make sure you understand:

• SQL basics.
• The concept of Medallion Architecture.
• Basic navigation in the Databricks Workspace.

Prerequisites: Preparing the Data Source (MongoDB)

Before starting our data pipeline in Databricks, we need an operational data source. In this tutorial, we will use the MongoDB Atlas Free Tier as our source database, simulating a real-world transactional data scenario.

Creating a MongoDB Free Tier Cluster

To create a free cluster in MongoDB, follow the official MongoDB tutorial:

• Deploy a Free Tier Cluster:
  https://www.mongodb.com/pt-br/docs/atlas/tutorial/deploy-free-tier-cluster/
• Getting Started Guide:
  https://www.mongodb.com/pt-br/docs/atlas/getting-started/

After creating the cluster, make sure to:

• Create a database user
• Allow network access for your IP (or allow access from any IP for testing purposes)
• Copy the connection string

Step 1: Preparing the Source Data

In this guide, the source data is not accessed directly by Databricks via native connectors or Auto Loader. Instead, we use Erathos as the ingestion layer, simulating a real modern data architecture scenario where ingestion and transformation are well-defined, separate responsibilities.

Erathos is a data ingestion tool that lets you connect to different sources (databases, APIs, and external systems) and load this data directly into your Lakehouse, abstracting the complexity of:

• Authentication
• Incremental extraction
• Scheduling
• Monitoring
• Writing to Delta Lake

Connecting MongoDB to Erathos

Erathos features a native connector for MongoDB Atlas.

• MongoDB connector documentation:
  👉 https://docs.erathos.com/connectors/databases/mongodb
• Creating and managing connections:
  👉 https://docs.erathos.com/platform/connections

In this step, you will:

1. Create a connection to MongoDB Atlas using your connection string
2. Select the desired collection
3. Define the synchronization mode (full or incremental)

Configuring Databricks as the Destination

After configuring the source, we set Databricks as the destination for our data.

Erathos offers direct integration with Databricks + Unity Catalog, ensuring that the ingested data arrives already governed in the Lakehouse.

• Official documentation:
  👉 https://docs.erathos.com/destinations/databricks

In this step, you will:

1. Provide your Databricks workspace details
2. Select the destination catalog and schema
3. Persist the data as Delta tables

Ingestion Result

For this tutorial, Erathos was used to:

• Connect to a MongoDB database (demo)
• Ingest the collection:
• • theaters (1,564 records)
• Persist the data as Delta tables in Databricks, within a schema governed by Unity Catalog

From this point on, all data processing and transformation will be handled exclusively via Delta Live Tables, keeping the focus of this article on the transformation and data quality layer.

Governance and Data Organization

Before starting the DLT pipeline, it is essential to ensure that the target Schema already exists in Unity Catalog. DLT strictly follows this hierarchy:

Catalog > Schema > Table

Without a pre-created schema, the pipeline won't be able to properly register the metadata or expose the tables for external consumption.

Note: In a real-world scenario, Erathos could be ingesting data from transactional databases, APIs, or third-party systems, dropping it directly into a governed catalog in Databricks.

Ingestion Strategies: Batch vs. Streaming

Before coding our first layer, it's important to understand how DLT consumes different types of sources. Whether data is ingested via Erathos, Auto Loader, or another mechanism, DLT supports both batch and streaming processing.

1. Auto Loader: Used to ingest raw files incrementally using cloud_files. This is the most common approach. You point to a folder in cloud storage (S3, ADLS, GCS) or a Unity Catalog Volume.
2. • How it works: Databricks monitors the arrival of new files (JSON, CSV, Parquet) and processes only the new ones.
   • Code Example:

CREATE OR REFRESH STREAMING TABLE taxi_raw_bronze
AS SELECT * FROM cloud_files("/Volumes/main/default/my_volume/raw_data/", "json");

1. Via Delta Table (Stream from Table): If your source is already a Delta Table (rather than raw files), it needs to support change tracking (Change Data Feed).
2. • How it works: You read the table as a continuous stream.
   • Code Example:

CREATE OR REFRESH STREAMING TABLE bronze_table
AS SELECT * FROM STREAM(catalog.schema.source_delta_table);

Implementation Note: In this guide, the theaters data was ingested beforehand via Erathos and persisted as static Delta tables. For this reason, we will use the LIVE TABLE command (batch). Even so, the Medallion architecture remains the same and can easily be adapted for incremental or streaming sources.

Technical Prerequisites in Databricks

• Databricks Workspace with Unity Catalog enabled.
• Permissions to create DLT pipelines and write to a schema in your catalog.

Step 2: Create the Transformation Notebook

1. In your Workspace, click New > Notebook. 2. Name it dlt_medallion_pipeline. 3. Make sure the default language is set to SQL.

Table definitions in Delta Live Tables are declarative and live in SQL or Python notebooks. Each CREATE OR REFRESH LIVE TABLE command describes what the table should look like, while Databricks automatically manages how data is processed, versioned, and optimized inside the pipeline.

Bronze Layer

The Bronze layer represents the entry point for data in our Lakehouse. In this scenario, the data has already been ingested into Databricks using Erathos, which handles extraction and incremental synchronization from the source database.

The primary goal of Bronze is fidelity: we capture data with minimal transformations, preserving the original format, including JSON fields to ensure traceability and allow future reprocessing.

CREATE OR REFRESH LIVE TABLE theaters_bronze
COMMENT "Bronze layer: raw theaters data from Erathos"
AS
SELECT
_id,
theater_id,
location,
_erathos_execution_id,
_erathos_synced_at
FROM erathos_db.erathos_db.theaters

Silver Layer

The Silver layer is where the technical complexity increases. Here, we perform data structuring, apply data quality rules, and normalize complex fields (such as JSONs), converting raw data into a reliable, analytical format.

The location field is stored at the source as a JSON string. In the Silver layer, we use the FROM_JSON function to convert this field into a typed structure (STRUCT), allowing direct access to address attributes and geographic coordinates, as well as applying quality rules to this data.

CREATE OR REFRESH LIVE TABLE theaters_cleaned_silver (

CONSTRAINT valid_theater_id
EXPECT (theater_id IS NOT NULL)
ON VIOLATION FAIL UPDATE,

CONSTRAINT valid_state
EXPECT (state IS NOT NULL)
ON VIOLATION DROP ROW,

CONSTRAINT valid_coordinates
EXPECT (
latitude IS NOT NULL
AND longitude IS NOT NULL
)

)
COMMENT "Silver layer: cleaned and structured theaters data"
AS
SELECT
_id,
theater_id,

location_struct.address.street1 AS street,
location_struct.address.city AS city,
location_struct.address.state AS state,
location_struct.address.zipcode AS zipcode,

location_struct.geo.coordinates[0] AS longitude,
location_struct.geo.coordinates[1] AS latitude,

_erathos_synced_at AS ingested_at

FROM (
SELECT
*,
FROM_JSON(
location,
'STRUCT<
address: STRUCT<
street1: STRING,
city: STRING,
state: STRING,
zipcode: STRING
>,
geo: STRUCT<
type: STRING,
coordinates: ARRAY<DOUBLE>
>
>'
) AS location_struct
FROM LIVE.theaters_bronze
);

Expectation Severity Levels:

• EXPECT: Only generates metrics. Ideal for understanding data cleanliness issues without stopping the pipeline.
• DROP ROW: Ensures the Silver layer contains only trusted data.
• FAIL UPDATE: Halts processing. Critical to ensure that calculations like tax or payment processing are never run with null values.

Expectations also automatically generate metrics within the pipeline, enabling you to track data quality over time.

Gold Layer

The Gold layer is optimized for final consumption, offering stable, simple, and high-performance tables for analytics, BI, and downstream applications.

CREATE OR REFRESH LIVE TABLE theaters_gold
COMMENT "Gold layer: theaters dimension table"
AS
SELECT
theater_id,
street,
city,
state,
zipcode,
latitude,
longitude
FROM LIVE.theaters_cleaned_silver;

Autonomous Maintenance (Vacuum and Optimize):

Unlike standard Spark tables, DLT automatically manages OPTIMIZE (small file compaction) and VACUUM (old file cleanup). This ensures that Gold layer queries remain highly performant, even with multiple incremental updates over time.

Step 3: Configure the Pipeline in Unity Catalog

Now that the code is ready, we need to create the Pipeline object to run it.

• In the sidebar, click Jobs & Pipelines.
• From the "Create New" menu, select ETL pipeline (Build ETL pipelines using SQL and Python).
• In the configuration screen, you must provide a catalog and schema in the upper right corner to bind your tables and logs to Unity Catalog.
• Select Add existing assets to link the notebook created in Step 2.

Note on Modernization: When selecting your code, Databricks might display a "Legacy configuration" warning. This happens because Lakeflow favors raw code files (.sql) to support DevOps best practices. For this tutorial, we will stick with the Notebook format for easier immediate data visualization, but in large-scale production environments, migrating to Workspace Files is recommended.

Step 4: Execute and Validate

1. On your Pipeline screen, click Run pipeline.
2. Databricks will spin up a cluster, and you will see the lineage graph (DAG) render on the screen.
3. Track progress: the graph will display the row count moving from Bronze to Gold.

If any row violates the theater_id IS NOT NULL rule defined in the Silver layer, DLT will drop that record and log the metric in the quality dashboard.

Conclusion

You have successfully implemented a robust data pipeline utilizing the Medallion Architecture and Delta Live Tables within the Databricks Lakeflow ecosystem. By following this guide, you have established a solid foundation for modern data engineering, ensuring:

• Intelligent Ingestion: Understanding the flexibility between processing via Auto Loader for raw files and ingesting existing Delta tables.
• Active Governance: Implementing Expectations to guarantee only high-quality data reaches consumer layers, reducing debugging time.
• Native Performance: An architecture that benefits from automated maintenance like Optimize and Vacuum, ensuring fast queries in the Gold layer without manual overhead.
• Lineage and Transparency: Through Unity Catalog, your pipeline now features automatic data lineage, making auditing and compliance much easier.

By integrating an ingestion tool like Erathos with Delta Live Tables, we cleanly separate ingestion and transformation responsibilities, resulting in simpler, more governable, and highly scalable pipelines.