# Erathos — Full Product Context

> Erathos is a managed data ingestion platform (ELT) that connects data sources to data warehouses using pre-built connectors, with granular per-table pipeline control and full execution observability.

## What problem Erathos solves

For most data teams, ingestion is the most fragile and least visible part of the data stack. Data engineers spend 12–24 hours per week fixing broken pipelines instead of building value. Data quality issues only surface at decision-making time. Pipelines break when schemas change in source systems. Debugging takes hours because context is missing when something fails.

When ingestion breaks, everything downstream becomes unreliable.

Erathos handles extraction and loading (EL) so data teams operate with reliability and agility without managing infrastructure, authentication, or rate limits over time.

## Core concepts

### Connections
A connection is the bridge between a data source and a data warehouse. Each connection can manage one or multiple jobs. Jobs define specific data extraction and loading tasks — typically one job per table or API endpoint.

### Jobs
Each job represents a single data stream: one table from a database, or one endpoint from a SaaS API. Jobs are configured independently, meaning schedule, sync type, retry strategy, and block window are all set at the job level — not at the connection level.

### Sync types
Erathos supports the following sync strategies, configurable per job:

**Full Refresh** — complete read and load of the source table, overwriting all values in the destination. Always available.

**Partial Overwrite** — incremental sync using a cursor column (updated_at, created_at, or an auto-incremental ID). New records are inserted; updated records overwrite outdated ones. Recommended default for most use cases.

**Partial Append** — incremental sync where all records are inserted, resulting in updated records being duplicated (both original and new values present).

**Partial Versioned** — similar to Partial Append, but each record is marked with an indicator identifying whether it is the most recent version.

For partial syncs, prerequisites are: a primary key must exist, and a date/datetime column must be available as cursor. For API connectors, Erathos maps and applies the optimal cursor automatically. For database connectors, the cursor is set manually.

Multiple cursors can be combined — for example, `created_at OR updated_at > last_loaded_value` when `updated_at` is null on record creation.

**Enable Full On** — an advanced parameter for partial syncs that schedules periodic full refreshes to resolve discrepancies between source and destination.

### Sync schedule
Each job has an independent schedule defined by two parameters:

- **Base hour**: the anchor time of day for the first execution
- **Frequency**: how often the job repeats after the base hour

Available frequencies: manually, weekly, daily, every 12 hours, every 6 hours, every 3 hours, every hour, every 30 minutes, every 5 minutes.

### Block Window
Prevents ingestion updates during specific time windows per job. Useful for protecting downstream processes (dbt runs, BI refreshes) from running against partially loaded data. When Block Window is active and "Allow manual executions" is disabled, API calls to trigger that job return a 400 response.

### Custom retry strategy
Defines how many times Erathos retries a failed job execution before surfacing an error, and the interval between retries. Configured per connector.

### Custom backfill
Applies filters to reload only a subset of historical data, avoiding full reloads of large tables. Useful when only a specific date range or partition needs to be reprocessed.

### Column Anonymization
Masks or anonymizes specific columns during ingestion before data reaches the destination warehouse.

## Observability

### Runs history
Full execution history accessible from the Runs page or from the Runs tab of any connection. Filterable by job name, connection, execution status (success, failure), and date range. Each execution shows logs, timestamps, row counts, metrics, and errors.

### Alert integrations
Real-time notifications when a job fails, before the issue propagates downstream. Available channels:

- **Email**
- **Slack** — supports actionable notifications: retry failed jobs, jump to execution details, or mute notifications for a specific period, directly from Slack
- **Discord** — same actionable notifications as Slack

Configured in Settings → Communications.

## API (Beta)

The Erathos API enables programmatic interaction with the platform. Two primary use cases:

**Trigger jobs in Erathos** — external orchestration tools (Airflow, Dagster, Prefect) call the Erathos API to start job executions as part of a broader pipeline.

**Trigger jobs outside Erathos** — at the end of a job execution, Erathos makes a callback request to a specified endpoint, allowing downstream systems to react (e.g., start a dbt run, notify a monitoring system).

API endpoints cover: Jobs, Orchestration, Secrets, Variables, Workspaces.

Authentication, variables, secrets, and rules are documented in the Concepts section of the API docs.

**Job dependency behavior**: for API connectors, some endpoints depend on a primary endpoint (e.g., a "user details" endpoint depends on a "list users" endpoint). Dependent jobs cannot run independently — running the primary job automatically triggers its dependents.

## Destinations

- BigQuery
- Redshift
- Databricks
- PostgreSQL
- Azure SQL Server
- S3 Iceberg

## Get started

- Sign up (free, no credit card required): https://app.erathos.com/signup
- Documentation: https://docs.erathos.com
- Book a demo: https://calendly.com/erathos-gelsonbagetti/erathos
- Contact: contato@erathos.com