Revolutionize Data Automation with Efficient Integrations

The evolution of data workflow automation is here! Introducing the Erathos API, an innovative tool that provides flexibility and efficiency for teams and developers looking to optimize their data pipelines. With robust integrations such as Apache Airflow, dbt, and Databricks, the Erathos API lets you automate workflows, simplify operations, and scale processes with ease.

What is the Erathos API?

The Erathos API allows developers to interact programmatically with the Erathos platform. From running jobs to real-time notifications via webhooks, the API simplifies the management of complex data workflows.

Key Features:

• Job Execution: Automate tasks such as data warehouse updates or chaining multiple processes.

• Webhooks: Receive real-time notifications when specific events occur, with detailed execution metadata.

• Secure Variable and Secret Management: Store credentials and other sensitive values securely.

Why use the Erathos API?

Data-driven companies rely on efficient workflows. The Erathos API enhances your operation by integrating seamlessly with leading tools like Airflow, dbt, and Databricks.

Use Case: Orchestration with Apache Airflow

The integration between Erathos and Airflow provides flexibility to automate complex processes across both platforms. Check out two powerful scenarios:

1. Trigger DAGs in Airflow after a Job runs in Erathos

Automate DAG execution in Airflow based on completed events in Erathos, leveraging metadata to dynamically configure processes.

Webhook Example:

{
    "description": "Disparo de DAG no Airflow",
    "jobs": ["id_do_job"],
    "method": "POST",
    "url": "<https://airflow-servidor/api/v1/dags/${{erathos.JOB_ID}>}/dagRuns",
    "header": {
        "Authorization": "Bearer ${{secrets.TOKEN_AIRFLOW}}"
    },
    "body": {
        "conf": {
            "status_do_job": "${{erathos.STATUS}}",
            "linhas_processadas": "${{erathos.ROWS}}"
        }
    }
}

2. Use Airflow to trigger Job execution in Erathos

Unlike the previous scenario, Airflow can also be configured to start a job execution directly on the Erathos platform. This is useful for including Erathos jobs as part of a larger pipeline managed by Airflow.

Airflow Operator Example:

You can use the HttpOperator operator to send a POST request to the Erathos API, triggering a job programmatically.

from airflow.providers.http.operators.http import HttpOperator

workspace_id = "<WORKSPACE_ID>"
job_id = "<JOB_ID>"
api_key = "<ERATHOS_API_KEY>"

execute_erathos_job = HttpOperator(
    task_id='execute_erathos_job',
    http_conn_id='erathos_api',  # Configurado nas conexões do Airflow
    endpoint=f'/developers/workspaces/{workspace_id}/jobs/{job_id}/run/',
    method='POST',
    headers={
        "Authorization": f"Api-Key {api_key}",
        "Content-Type": "application/json"
    },
    request_params={
        "force_full_refresh": False
    },
    response_check=lambda response: response.status_code == 201,
    dag=dag
)

In this example, the operator triggers a job in Erathos, enabling Airflow to manage and orchestrate the entire pipeline, including steps executed on the Erathos platform.

Use Case: Data Transformations with dbt

Integrate your dbt workflows with Erathos:

• Trigger dbt models after a pipeline runs, ensuring transformations are aligned with the most recent data.

• Use metadata to customize dbt commands, such as selecting specific schemas or tables.

Use Case: Analytics at Scale with Databricks

Leverage the power of Databricks for large-scale analytics:

• Automatically update Delta tables after a job completes.

• Dynamically configure Databricks notebooks based on metadata, such as TABLE_NAME and SCHEMA_NAME.

Example Flow:

1. Trigger a Databricks notebook after a job runs.

2. Update machine learning models or generate insights with the latest pipeline data.

How to Get Started

Ready to transform your data workflows? Follow these steps:

1. Generate your API Key:

   1. See the Authentication section in the API Documentation to create your API key.

2. Get the Workspace ID:

   1. Use the List Workspaces endpoint to find the ID of the workspace you want to automate.

3. Select the Jobs:

   1. Identify job IDs with the List Jobs endpoint. These jobs will be used to trigger webhooks.

4. Create and Register Webhooks:

   1. Customize the webhook JSON template to integrate with tools like Airflow, dbt, and Databricks.

Conclusion

The Erathos API Beta is a powerful addition for data-driven organizations. With robust integrations with tools like Airflow, dbt, and Databricks, the API delivers automation, optimization, and scalability for your workflows.

Join the beta program today to explore all these capabilities. Visit the API Documentation and discover how the Erathos API can transform your data operations.