# Data Observability: Breaking the cycle of firefighting > Data observability uses logs, traces, and metrics to detect anomalies before your users do. How tools like dbt and Erathos work in practice. Source: https://www.erathos.com/en/blog/data-observability-breaking-the-firefighting-cycle Em português: https://www.erathos.com/blog/data-observability-breaking-the-firefighting-cycle Published: 2024-09-10 Category: Governance & Observability ![Illustration for the article: Data Observability: Breaking the Firefighting Cycle](https://cms-media.erathos.com/LnjKE5l2xLVEUXQekrWrGUrBQs-1.png) Let the data professional who has never received a message like this cast the first stone! ![wtv1nks8LJ2jCBggJFxswMi6Us.png](https://cms-media.erathos.com/wtv1nks8LJ2jCBggJFxswMi6Us-1.png) First of all, I want to explain to you why this is a bigger problem than you think… When we receive a message like this, we are spending the most scarce resource that data teams have: **trust**. Imagine that the commercial team manager (let's call him João) let us know that his department's main dashboard is broken. João probably has a lot on his plate, and consequently, his schedule is very tight. When he accesses any data product, he doesn't just need the information that the dashboard in question should present, but he also needs to take some action based on that information. After all, that's what being data-driven is all about, right? Taking decisions and actions based on data. There are actions that can be taken without data input, but others, such as paying commissions to the sales team, will likely be blocked until the data team resolves the issue. See that in this hypothetical situation, besides leaving João "stranded," we abruptly interrupted his workflow, causing friction that could get much worse during the resolution of this ticket, but that's a topic for another article. Cases like João's are recurrent. According to a study by Dun & Bradstreet ([The Past, Present and Future of Data](https://www.dnb.co.uk/content/dam/english/dnb-data-insight/DNB_Past_Present_and_Future_of_Data_Report.pdf)), 42% of companies have already faced issues with inconsistent data. There are countless reasons that lead to data issues, usually grouped into compliance, business requirements, human error, misinterpretation, and data drift. It is important to understand these issues as something that comes with data projects, not incompetence of the team or the developed solution. [This article by Google](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf) even points out that there is a 92% prevalence of a data issue occurring in a project. What makes dealing with these situations difficult is that, by itself, a dataset does not generate **exceptions**. No matter how wrong this dataset is, the inconsistency will only be discovered when it is used. Andy Petrella, author of the book _Fundamentals of Data Observability_, goes even further by calling data a "silent killer," since situations like this result in a general slowdown, destruction of trust, and an increase in stress, anger, and anxiety, without any warning alert. ## What is data observability in practice The goal of observability, and of this specific article on data, is to inform an observer about the status of a system. To do this, we make use of three main components: **logs**, **traces**, and **metrics**, tracking the execution of the various stages of a data pipeline and generating alerts if any rule is violated. In the previous example, when we were notified that the commercial dashboard had inconsistencies, to solve the problem we would have to explore every step of the data architecture until we identified the error. Is it in the dashboard? In the data warehouse? Did the ELT run correctly? There are many steps to explore, but if we use observability instruments, we can identify these inconsistencies even before our users notice. This directly connects with [data lineage](%252Fblog%252Fdata-lineage-o-que-%25C3%25A9-e-para-que-serve): observability warns you that something broke, lineage shows you exactly where and what was affected. Imagine that you received an alert that there is an inconsistency in the `orders_raw` table and, as a result, all downstream steps will not be updated. Based on this notification, we can warn the team that the sales dashboard is outdated, in addition to being able to give an estimate for the fix. These are stressful situations avoided purely because we received an alert about the error in the data pipeline. ## The golden rule of alerts The main rule that must always be followed when implementing alerts is that, if an alert is generated, action must be taken. Let whoever has never muted a Slack notification channel cast the first stone (we're running out of stones already, hahaha). Following this rule ensures that alerts are not seen as a negative thing by the team, but rather as a critical information channel for the data operation. Alerts that no one acts upon become noise, and noise is the first step for the team to start ignoring all alerts, including the ones that actually matter. ## Tools to implement observability There are several tools that can be used to facilitate the implementation of observability in data pipelines: **dbt** has several features like schema and data tests, as well as unit tests (available from version 1.8), which validate the logic of a transformation even before it runs against real data. **Great Expectations** allows the building of data contracts and generates an alert if any of these contracts are violated, acting as a validation layer between pipeline stages. This type of instrumentation is what separates a team that discovers the problem when the client complains from a team that discovers the problem before anyone else notices. ## How this works in practice within Erathos At Erathos, this observability layer comes out-of-the-box for any of the more than 80 available connectors, with no extra configuration needed: **Consolidated jobs view.** All ingestion pipelines in a single table, with status, frequency, processed volume, and recent activity. Hover over a recent run and see the reason for the error instantly, without having to open the job. **Full execution history.** Every run is recorded, with processed rows, duration, and status, and you can investigate the full details of any execution right on the screen, without opening an external log. **Alerts with full context.** When a job fails, the notification arrives on Slack, Discord, or email with the job, the error, and the duration, without needing to open the tool to understand what happened. And you can re-run the job or mute the alert directly from the notification, without leaving the channel. As Fernando R., Head of Data at one of Erathos' clients, sums up: "with no worries about data ingestion, the team focused on strategy and not on maintaining pipelines. Deploying a new connector takes minutes and the reliability is huge." ## Observability and governance go hand in hand Observability is not an isolated practice, it is part of the same effort that supports good [data governance](/blog/governanca-de-dados): knowing what is happening with the data, when something goes off track, and having a trail to investigate quickly, instead of reconstructing the data flow from memory with every incident. ## Frequently asked questions about data observability **What is the difference between observability and data monitoring?** Monitoring usually focuses on specific metrics and point alerts. Observability is broader: it combines logs, traces, and metrics to give complete visibility of what is happening at each stage of the pipeline, allowing you to investigate the root cause, not just know that something went wrong. **Do I need a dedicated observability tool, or are dbt and Great Expectations enough?** It depends on the complexity of your stack. For smaller teams, dbt tests and data contracts in Great Expectations cover a good part of the problem. Larger teams, with more complex pipelines, usually benefit from dedicated end-to-end data observability tools. **Why does my team ignore the alerts we set up?** Usually because alerts that don't trigger action turn into noise. The golden rule is simple: if an alert doesn't require any real action when it fires, it shouldn't exist, or it needs to be recalibrated. **Does data observability also cover the ingestion layer?** Yes, and it's one of the most important layers, because an ingestion problem propagates to all subsequent steps. Managed ingestion tools, like Erathos, already deliver this native observability layer out-of-the-box. ## Conclusion We hope this content has helped in some way. If you want to know more about how Erathos delivers native observability right from the ingestion layer, [create your free account](https://app.erathos.com/signup?slug=blog&button=cta&utm_campaign=observabilidade_de_dados) and try the platform.