Erathos API: Learn more about this new platform feature and how it can benefit your data team.

Revolutionize Data Automation with Efficient Integrations

The evolution of data workflow automation has arrived! Introducing the Erathos API, an innovative tool that offers flexibility and efficiency for teams and developers looking to optimize their data pipelines. With robust integrations like Apache Airflow, dbt, and Databricks, the Erathos API allows you to automate workflows, simplify operations, and scale processes with ease.

What is the Erathos API?

The Erathos API allows developers to programmatically interact with the Erathos platform. From running jobs to real-time notifications via webhooks, the API simplifies the management of complex data workflows.

Key Features:

• Job Execution: Automate tasks such as data warehouse refreshes or chaining multiple processes together.
• Webhooks: Receive real-time notifications when specific events occur, complete with detailed execution metadata.
• Secure Variable and Secret Management: Store credentials and other sensitive values securely.

Why use the Erathos API?

Data-driven companies rely on efficient workflows. The Erathos API powers your operation by easily integrating with leading tools like Airflow, dbt, and Databricks.

Use Case: Orchestration with Apache Airflow

The integration between Erathos and Airflow offers the flexibility to automate complex processes across both platforms. Check out two powerful scenarios:

1. Trigger DAGs in Airflow after a Job runs in Erathos

Automate Airflow DAG execution based on completed events in Erathos, leveraging metadata to dynamically configure processes.

Webhook Example:

{
"description": "Trigger Airflow DAG",
"jobs": ["job_id"],
"method": "POST",
"url": "<https://airflow-server/api/v1/dags/${{erathos.JOB_ID}>}/dagRuns",
"header": {
"Authorization": "Bearer ${{secrets.TOKEN_AIRFLOW}}"
},
"body": {
"conf": {
"job_status": "${{erathos.STATUS}}",
"rows_processed": "${{erathos.ROWS}}"
}
}
}

2. Use Airflow to trigger a Job execution in Erathos

Conversely, Airflow can also be configured to trigger job execution directly within the Erathos platform. This is useful for including Erathos jobs as part of a larger pipeline managed by Airflow.

Airflow Operator Example:

You can use the HttpOperator to send a POST request to the Erathos API, triggering a job programmatically.

from airflow.providers.http.operators.http import HttpOperator

workspace_id = "<WORKSPACE_ID>"
job_id = "<JOB_ID>"
api_key = "<ERATHOS_API_KEY>"

execute_erathos_job = HttpOperator(
task_id='execute_erathos_job',
http_conn_id='erathos_api', # Configured in Airflow connections
endpoint=f'/developers/workspaces/{workspace_id}/jobs/{job_id}/run/',
method='POST',
headers={
"Authorization": f"Api-Key {api_key}",
"Content-Type": "application/json"
},
request_params={
"force_full_refresh": False
},
response_check=lambda response: response.status_code == 201,
dag=dag
)

In this example, the operator triggers a job in Erathos, enabling Airflow to manage and orchestrate the entire pipeline, including steps executed on the Erathos platform.

Use Case: Data Transformations with dbt

Integrate your dbt workflows with Erathos:

• Trigger dbt models after a pipeline execution, ensuring your transformations run against the freshest data.
• Use metadata to customize dbt commands, such as selecting specific schemas or tables.

Use Case: Analytics at Scale with Databricks

Leverage the power of Databricks for large-scale analytics:

• Automatically refresh Delta tables once a job completes.
• Dynamically configure Databricks notebooks based on metadata, such as TABLE_NAME and SCHEMA_NAME.

Example Flow:

1. Trigger a Databricks notebook following a job execution.
2. Update machine learning models or generate insights with the latest data from the pipeline.

Getting Started

Ready to transform your data workflows? Follow these steps:

1. Generate your API Key:
2. 1. Refer to the Authentication section in the API Documentation to create your API key.
2. Get the Workspace ID:
3. 1. Use the List Workspaces endpoint to find the ID of the workspace you want to automate.
3. Select the Jobs:
4. 1. Identify the job IDs using the List Jobs endpoint. These jobs will be used to trigger the webhooks.
4. Create and Register Webhooks:
5. 1. Customize the JSON webhook template to integrate with tools like Airflow, dbt, and Databricks.

Conclusion

The Erathos API Beta is a powerful addition for data-driven organizations. With robust integrations into tools like Airflow, dbt, and Databricks, the API delivers automation, optimization, and scalability to your workflows.

Join the beta program today to explore all of these features. Visit the API Documentation and discover how the Erathos API can transform your data operations.