Back to blog

Data Pipeline Monitoring: A 6-Step Guide with Alerts

Found out a pipeline broke from a stale report? Set up data pipeline monitoring in 6 steps, with failure alerts and freshness checks.

Data pipeline monitoring.png

Monday, five minutes before the results meeting. Someone opens the sales dashboard and notices the numbers are from last Thursday. Nobody was warned. The job failed silently, the API token expired, or the machine running the script rebooted, and the report kept looking up to date.

This scenario is very common in companies that started using data before they had a formal data team. Ingestion starts as a script written by whoever had some spare time, turns into five scripts running on VMs, and by the time anyone notices, it's critical infrastructure that nobody monitors.

Pipelines breaking is normal. APIs change, credentials expire, sources go down. The problem is when leadership is the first to find out, not the team that looks after the data. This guide shows how to set up data pipeline monitoring that flips that order.

Why data pipelines fail silently

Most ingestion failures don't produce an error that the people using the data can see:

  • Expired credential or token: the API call returns 401, and the script logs the error in a file nobody reads.
  • Schema change at the source: a field is renamed and the load fails, or worse, it loads with the column empty.
  • Rate limit: the API cuts the extraction halfway, and the job finishes "successfully" with half the records.
  • Homegrown infrastructure: the VM reboots, the cron doesn't come back, and nothing signals that the schedule stopped.

In all of these cases, the dashboard keeps loading. It just shows old data.

Step 1: inventory your sources and who depends on them

Before setting up any alert, list every data source that feeds the warehouse. For each one, record:

  • Source (e.g., HubSpot, the finance system, the production database)
  • Destination (table or dataset in the warehouse)
  • Current update frequency
  • Who owns the pipeline
  • Which reports and teams use that data

This inventory shows the real size of the problem. It's common to find pipelines that nobody remembers creating, or sources that are critical for finance running on someone's personal script.

Step 2: define how often each dataset really needs to update

Not all data needs to be refreshed every hour. The CRM that feeds the sales team may need hourly updates. A finance system checked at month-end close can run once a day. Sources that don't support incremental loads (no cursor on an updated date) may make more sense as a weekly refresh.

Setting this expected frequency per source is what turns "is the data late?" into a question with an objective answer. Without that agreement, there's no such thing as late, only guesswork.

Step 3: set up failure alerts in the channel your team already uses

An alert that depends on someone opening a screen isn't an alert. The failure needs to reach the place where the team already works: Slack, email, Discord, or whatever incident tool you use.

At a minimum, the alert should say which pipeline failed, when, and what the error was. With that, whoever receives it can decide in seconds whether it's something transient or whether they need to act.

If ingestion runs on a dedicated tool, this usually comes built in. In Erathos, for example, any job that fails triggers a notification by Slack, email, or Discord, with each connector's run history available for investigation. If you already have an observability stack, the API lets you send these events there.

Step 4: monitor data freshness, not just errors

This is the step most teams forget. A job can finish with a success status and still bring in zero rows, or bring in data from the wrong period.

To cover this, monitor freshness in the warehouse itself:

  • Compare the highest updated_at value (or equivalent) in each table with the frequency defined in step 2.
  • If you use dbt, set up source freshness with warn_after and error_after limits per source.
  • Track the number of rows loaded per run. A sudden drop usually points to a problem at the source.

A green job doesn't mean fresh data. It's the two signals together that give you confidence.

Step 5: move ingestion off homegrown code

Your own scripts may work at first, but the whole layer of retries, pagination, incremental control, rate limit handling, and alerting is on whoever wrote them. And whoever wrote them is almost always busy with another priority.

It's worth doing the math: how much time does the team spend maintaining connectors and investigating failures, instead of modeling data and working with the business teams? Building a robust connector from scratch usually takes weeks. Keeping it running is a cost that never ends.

Erathos has more than 100 ready-made connectors (CRMs, ERPs, paid media, and databases like PostgreSQL, MySQL, and MongoDB), delivering to BigQuery, Databricks, Redshift, and S3. When a source isn't in the catalog, the team builds it with you, including connections to internal APIs. You can try it for 14 days, no credit card required.

Step 6: create a response routine

An alert without an owner becomes noise. Define:

  1. Who receives it: one person or a channel with a clear owner.
  2. First diagnosis: did the next run go through normally? If so, it was a transient failure.
  3. When to escalate: if the failure repeats, investigate the source or contact the tool's vendor.
  4. How to communicate: if critical data is going to be late, warn the teams that use it before the meeting, not after.

Document this flow. It's what keeps monitoring working when the person who set it up goes on vacation.

In practice: how Convenia stopped "finding out from the report"

Convenia, an HR management platform serving more than 3,000 companies, didn't have a formal data team. Data work fell to the RevOps team, and pipelines ran loose on VMs with no visibility at all. At least once a week, someone found out that some data was broken right when they needed it for a meeting.

When Bernardo Cardozo joined to build the data team, Erathos was his first hire. Today Convenia centralizes HubSpot, Conta Azul, Google Ads, Facebook Ads, MongoDB, Vindi, and its internal accounting API in BigQuery, with HubSpot updating every hour and failure alerts in Slack.

"Today, whenever anything goes wrong, a notification lands in Slack for me. I check whether the next job run went through, and if I can't figure out why, I talk directly to the Erathos team."
Bernardo Cardozo, Data Coordinator at Convenia

The results:

  • About 15 minutes to get a new source from the catalog flowing into BigQuery, versus an estimated month to build it from scratch.
  • 3 people handling all ingestion, modeling, and analytics for a 150-employee company, with the team's time redirected from pipeline maintenance to governance.
  • About R$147,000 in net savings per year, roughly 6 times the amount invested in Erathos.*

*Estimate based on the cost of a data engineer dedicated to ingestion (average market salary plus Brazilian CLT employment costs) over 12 months, compared with the annual investment in Erathos.

Read the full Convenia case study

Get started now

Monitoring your data pipelines starts with moving ingestion out of the places nobody can see.