Erathos + dbt Cloud: How to Orchestrate ELT Pipelines with Webhooks (Complete Guide)

In modern data projects, separating responsibilities between ingestion and transformation has become a fundamental practice. Erathos solves the problem of extracting and loading data (EL) from various sources into the data warehouse, while dbt specializes in transforming (T) this data into reliable analytical models.

This separation, however, makes us wonder: how do we orchestrate these processes efficiently? How do we ensure that dbt only runs transformations after new data arrives? How do we make Erathos react to failures in critical dbt transformations?

The Erathos Orchestration API solves this problem by allowing the creation of event-driven pipelines. With webhooks, workflows are automatically triggered by real-world events such as completed ingestions, failures, or variations in data volume, reducing reliance on fixed schedules that don't always reflect the current scenario.

The architecture we will implement in this tutorial represents a modern data engineering pattern:

1. Erathos ingests transactional data from operational systems (PostgreSQL, MongoDB, APIs) into BigQuery
2. Webhooks automatically trigger dbt Cloud jobs when new loads are completed
3. dbt Cloud transforms raw data into analytical models (facts and dimensions)
4. Reverse orchestration allows dbt to also control executions in Erathos when necessary

The Problem: Decoupled Orchestration

Before implementing the solution, it is important to understand the problems it solves:

Scenario 1: Disconnected Fixed Schedules

Situation: Erathos is scheduled to run at 2:00 AM, and dbt at 3:00 AM. If the Erathos ingestion fails or is delayed, dbt will process outdated data or fail due to a lack of new data.

Solution: dbt runs only when the Erathos ingestion completes successfully and after at least 1 row has been transferred.

Scenario 2: Lack of Traceability

Situation: A dashboard displays incorrect data. There is no clarity on which step of the pipeline failed or is outdated.

Solution: Each dbt run receives metadata from the Erathos execution (execution_id, number of rows, affected tables), allowing complete tracking.

Scenario 3: Resource Waste

Situation: dbt runs periodically even when there is no new data, unnecessarily consuming BigQuery credits.

Solution: Webhooks trigger only when there is new data (ROWS > 0), optimizing resource utilization.

Solution Architecture

The implemented workflow follows this structure:

Prerequisites

This tutorial assumes familiarity with:

• Basic dbt concepts (models, sources, refs)
• Navigating the dbt interface
• REST APIs and webhooks concepts
• Basic structure of data warehouses (facts and dimensions)

Requirements

Erathos Environment

• Active workspace
• Connected data source (PostgreSQL, MongoDB, API, etc.)
• Ingestion job configured and tested
• Destination configured for BigQuery

dbt Cloud Environment

• dbt project connected to BigQuery
• Models already created and tested
• Job configured in dbt Cloud

BigQuery

• Active project in Google Cloud
• Created datasets:
• • raw_data (raw data from Erathos)
  • staging (intermediate layer of dbt)
  • analytics (final layer - facts and dimensions)

Step 0: Gathering Required Information

Before starting, you will need to gather some information. Use this table to organize it:

Step 1: Configure Authentication Credentials

API integration requires authentication on both platforms. Credentials will be securely stored using Erathos's Variables and Secrets system.

Generate Erathos API Key

1. Go to Settings > User in Erathos
2. In the API Key section, click Create
3. IMPORTANT: The key is displayed only once. Copy and store it in a secure location
4. Optionally, set an expiration date

API Key Example:

erathos_abc123456

Generating dbt Cloud Personal Token

1. Access your account in dbt Cloud
2. Navigate to Account Settings > API Access
3. Click on Create Service Token (or Personal Token)
4. Copy the generated token
5. Also copy the Account ID (you will need it)

Example Configuration:

Account ID: 12345
Personal Token: dbt_abc12345

Create Variables and Secrets in Erathos

Erathos allows you to centralize frequently used values (such as URLs and credentials) in a key-value format.

• Variables: for non-sensitive values that can be reused in webhooks and other processes.
• Secrets: for sensitive data, such as credentials, which cannot be viewed after creation, ensuring greater security.

Creating the Variable dbt_account_id

Via API:

curl -X POST <https://api.erathos.com/developers/workspaces/{workspace_id}/variables/> \\
-H "Authorization: Api-Key {your_erathos_api_key}" \\
-H "Content-Type: application/json" \\
-d '{
"name": "dbt_account_id",
"value": "12345"
}'

Creating the Secret dbt_personal_token

Via API:

curl -X POST <https://api.erathos.com/developers/workspaces/{workspace_id}/secrets/> \\
-H "Authorization: Api-Key {your_erathos_api_key}" \\
-H "Content-Type: application/json" \\
-d '{
"name": "dbt_personal_token",
"value": "dbt_a1b2c3d4e5f6..."
}'

Step 2: Identify Required IDs

Before creating webhooks, we need to collect the unique identifiers of the resources we will orchestrate.

Obtaining Erathos Job ID

Each data connection in Erathos is represented by a Job. The UUID of this job is required for configuration.

Via UI:

1. Navigate to Jobs
2. Locate the desired job (e.g., orders coming from PostgreSQL)
3. Copy the Job ID from the URL or the UI

Via API:

curl -X GET "<https://api.erathos.com/developers/workspaces/{workspace_id}/jobs/>" \\
-H "Authorization: Api-Key {your_erathos_api_key}"

Response (example):

{
"count": 3,
"results": [
{
"_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"name": "orders",
"schema_name": "raw_data",
"connector_name": "PostgreSQL",
"is_active": true
},
{
"_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
"name": "products",
"schema_name": "raw_data",
"connector_name": "PostgreSQL",
"is_active": true
}
]
}

Note the IDs:

• Job orders: a1b2c3d4-e5f6-7890-abcd-ef1234567890
• Job products: b2c3d4e5-f6a7-8901-bcde-f12345678901

Obtaining dbt Cloud Job ID

In dbt Cloud, each scheduled or ad-hoc run is configured as a Job. We need the numeric ID of this job.

Via UI:

1. In dbt Cloud, navigate to Deploy > Jobs
2. Click on the job you want to trigger (e.g., "Daily Analytics Refresh")
3. The Job ID will be in the URL: https://cloud.getdbt.com/deploy/{account_id}/projects/{project_id}/jobs/{job_id}

Via API:

curl -X GET "<https://cloud.getdbt.com/api/v2/accounts/{account_id}/jobs/>" \\
-H "Authorization: Bearer {dbt_personal_token}"

Job ID Example: 54321

Step 3: Erathos Triggering dbt Cloud

After Erathos finishes ingesting data into BigQuery, it automatically triggers a job in dbt Cloud to transform it.

Understanding the Flow

Erathos Job (orders) finishes successfully
↓
Webhook triggers
↓
POST to dbt Cloud API
↓
dbt Job starts automatically
↓
Transformations run (staging → facts → dims)

Anatomy of the Webhook

A webhook in Erathos contains:

1. description: Description to identify the webhook
2. is_active: Whether it is active or not
3. jobs: List of Erathos Job IDs that should trigger the webhook
4. method: HTTP request method (POST, GET, DELETE, etc.)
5. url: Destination endpoint (dbt Cloud API)
6. header: HTTP Headers (authentication, content-type)
7. body: Request body (data sent to dbt)
8. rules: Conditions that must be met for the webhook to trigger

Creating the Webhook: Erathos → dbt Cloud

curl -X POST "<https://api.erathos.com/developers/workspaces/{workspace_id}/orchestration/webhooks/>" \\
-H "Authorization: Api-Key {your_erathos_api_key}" \\
-H "Content-Type: application/json" \\
-d '{
"description": "Trigger dbt Cloud job after successful orders ingestion",
"is_active": true,
"method": "POST",
"url": "<https://cloud.getdbt.com/api/v2/accounts/${{variables.dbt_account_id}>}/jobs/54321/run/",
"header": {
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": "Bearer ${{secrets.dbt_personal_token}}"
},
"body": {
"cause": "Triggered by Erathos after successful ingestion of ${{erathos.TABLE_NAME}} | Execution ID: ${{erathos.EXECUTION_ID}} | Rows: ${{erathos.ROWS}}"
},
"rules": [
{
"variable_name": "STATUS",
"operation": "EQUAL",
"value": "FINISHED"
},
{
"variable_name": "ROWS",
"operation": "GREATER_THAN",
"value": "0"
},
{
"variable_name": "NESTED_TABLE",
"operation": "EQUAL",
"value": "false"
}
],
"jobs": [
"a1b2c3d4-e5f6-7890-abcd-ef1234567890"
]
}'

Breaking Down the Components

Dynamic URL

"url": "<https://cloud.getdbt.com/api/v2/accounts/${{variables.dbt_account_id}>}/jobs/54321/run/"

• ${{variables.dbt_account_id}}: Automatically replaced by the value of the variable created in Step 1
• 54321: dbt Cloud job ID obtained in Step 2
• /run/: dbt Cloud API endpoint to trigger runs

Authentication Headers

"header": {
"Authorization": "Bearer ${{secrets.dbt_personal_token}}"
}

The token is securely referenced using the Secrets system.

Body with Metadata

"body": {
"cause": "Triggered by Erathos after successful ingestion of ${{erathos.TABLE_NAME}}"
}

• ${{erathos.TABLE_NAME}}: Ingested table name (e.g., "orders")
• ${{erathos.EXECUTION_ID}}: Erathos unique execution UUID
• ${{erathos.ROWS}}: Number of rows transferred

Result in dbt Cloud: When the dbt job runs, you will see in the UI:

Cause: Triggered by Erathos after successful ingestion of orders | Execution ID: abc-123-def | Rows: 1547

This information simplifies troubleshooting, allowing you to trace which Erathos run triggered each dbt run.

Rules: Smart Conditions

The rules determine when the webhook should trigger.

"rules": [
{
"variable_name": "STATUS",
"operation": "EQUAL",
"value": "FINISHED"
},
{
"variable_name": "ROWS",
"operation": "GREATER_THAN",
"value": "0"
},
{
"variable_name": "NESTED_TABLE",
"operation": "EQUAL",
"value": "false"
}
]

Translating into logic:

IF (
STATUS == "FINISHED" AND
ROWS > 0 AND
NESTED_TABLE == false
) THEN {
trigger webhook
}

Why these rules?

1. STATUS == "FINISHED": Triggers only if the ingestion succeeded (does not trigger on failure)
2. ROWS > 0: Avoids processing empty runs (saves BigQuery credits)
3. NESTED_TABLE == false: For API connectors with multiple nested tables, this ensures only the main table triggers the webhook (prevents multiple triggers)

Testing the Webhook

After creating the webhook, test it manually:

1. In Erathos: Manually trigger the job orders
2. Monitor: The run must complete successfully and transfer at least 1 row
3. Check dbt Cloud: Navigate to Deploy > Run History
4. Validate: You should see a new run with the custom cause

Expected log example:

Run #4523
Triggered at: 2026-01-20 14:32:15 UTC
Cause: Triggered by Erathos after successful ingestion of orders | Execution ID: abc-123 | Rows: 1547
Status: Success
Duration: 2m 34s

Step 4: Monitoring and Troubleshooting

Viewing Created Webhooks

To list all webhooks configured in the workspace:

curl -X GET "<https://api.erathos.com/developers/workspaces/{workspace_id}/orchestration/webhooks/>" \\
-H "Authorization: Api-Key {erathos_api_key}"

Response:

{
"count": 2,
"results": [
{
"_id": "webhook-uuid-123",
"description": "Trigger dbt Cloud job after successful orders ingestion",
"is_active": true,
"method": "POST",
"url": "<https://cloud.getdbt.com/api/v2/accounts/12345/jobs/54321/run/>",
"jobs": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"]
},
{
"_id": "webhook-uuid-456",
"description": "Run job_b after job_a completes successfully",
"is_active": true,
"method": "POST",
"jobs": ["{job_a_id}"]
}
]
}

Temporarily Disabling a Webhook

If you need to temporarily pause orchestration (e.g., during dbt Cloud maintenance):

curl -X PUT "<https://api.erathos.com/developers/workspaces/{workspace_id}/orchestration/webhooks/{webhook_id}/>" \\
-H "Authorization: Api-Key {erathos_api_key}" \\
-H "Content-Type: application/json" \\
-d '{
"is_active": false
}'

Conclusion

You have successfully implemented an orchestration architecture between Erathos and dbt Cloud, creating truly event-driven and intelligent data pipelines.

• Declarative Automation: Webhooks that automatically react to real-world events (completed ingestions, data volumes, times), eliminating the need for complex orchestrators.
• Full Traceability: Erathos metadata flows into dbt Cloud, allowing you to debug failures and understand the history of each run.
• Operational Efficiency: Rules ensure that transformations only run when actually necessary, saving BigQuery credits and processing time.
• Architectural Flexibility: The same API supports integrations with Airflow, Prefect, custom systems, and Erathos's own internal orchestration.

References

• Erathos API Official Documentation: https://docs.erathos.com/api
• dbt Cloud API Documentation: https://docs.getdbt.com/dbt-cloud/api-v2
• Erathos - Integration with dbt Cloud: https://docs.erathos.com/orchestration/dbt-cloud
• Erathos - Variables and Secrets: https://docs.erathos.com/platform/variables-and-secrets
• dbt Cloud - Webhooks (Outbound): https://docs.getdbt.com/docs/deploy/webhooks