How to build a custom API connector in minutes, with an AI agent and without writing Python

Every data team has that one API

Every data team has at least one API that no ingestion tool covers. It might be an internal system, a niche SaaS in your industry, or a vendor that only your company uses.

The Erathos Custom Connector solves exactly this case. You describe the API in a few YAML files, point Erathos to the GitHub repository, and the connector shows up in the catalog like any other. Now, with the agent skill we included in the template, you don't even need to write the YAMLs by hand.

In this post, we walk through the full flow with a real case: we built a connector for the Salvy API, from scratch to data in BigQuery.

The problem: the script nobody wants to maintain

When there is no ready-made connector, the usual fallback is a Python script running on a cron, a Cloud Function, or an Airflow DAG. It works on day one. The cost shows up later:

• Pagination, retries, and rate limits reimplemented in every script, each one differently.
• Silent failures. The API returns a 404 or changes a field, the job "passes", and the table ends up empty or incomplete without anyone noticing.
• Zero observability. No run history, no alerts, no idea how many rows came in on each sync.
• Concentrated knowledge. Only the person who wrote it understands the code. When that person leaves, the pipeline becomes a black box.

The result is that the niche API, precisely the one that most sets your business apart, becomes the weakest link in your data stack.

What the Custom Connector is

The Custom Connector is a connector defined in declarative code that lives in your GitHub repository. Erathos watches that repository and rebuilds the connector image on every push.

The project starts from the erathos-data/custom-api-connector template. In it, each API endpoint is a YAML file, and a _default.yml holds what all of them share: base URL, authentication, HTTP codes, and the validation endpoint. The template already covers:

• Authentication: bearer token and OAuth2, among others.
• Pagination: by page, offset, token, time, and GraphQL, with an example of each in the repo.
• Dependent endpoints: parent and child endpoints, such as /accounts/{id}/messages.
• Field mapping, variable placeholders, and custom fields.
• A validator (validate_connector.py) that checks the project before deployment.

Nobody writes Python. And once the first build passes, the connector appears in the Erathos catalog with the Custom badge and works like any native connector: same job configuration, same sync types, same monitoring.

Because the connector is code in Git, it inherits what the data team already uses: pull request review, change history, and rollback by commit.

The skill: describe the API, the agent writes the connector

What's new is that the template now ships with an agent skill in .claude/skills/custom-api-connector. It teaches a coding agent, such as Claude Code, to build connectors following the Erathos standard.

The idea came up during a test before launch. The person testing the GitHub connector suggested adding a ready-to-use prompt for Claude to the documentation. We went one step further and packaged that knowledge as a skill.

In practice, you open the repository in the agent and ask in plain language, for example: "connect the Acme API to Erathos, syncing orders and customers incrementally". From there, the agent:

1. Reads the API documentation or OpenAPI spec.
2. Asks about anything the documentation leaves unclear.
3. Makes real calls to the API, when credentials are available, to confirm the response format.
4. Generates the YAMLs: _default.yml and one file per endpoint.
5. Runs the validator.
6. Documents where each credential should be configured.

The important point: the skill doesn't hide its decisions. It explains every non-obvious choice in the YAML and flags anything not yet confirmed against the real API.

Step by step: a connector for the Salvy API

For this example, we used the Salvy API v2, at the suggestion of Hudo, Salvy's CTO. The entire flow, from fork to data in BigQuery, fits in a few minutes of video.

Step 1: fork the template

Fork erathos-data/custom-api-connector into your account or organization. If you already have a repository, just copy the .claude/skills/ folder into it.

Step 2: ask the agent for the connector

Clone the repository, open it in Claude Code, and send a short prompt:

crie um custom-api-connector para a API da Salvy:
https://docs.salvy.com.br/api-reference/v2/introduction
usando a skill do repo

(In English: "create a custom-api-connector for the Salvy API: [link] using the repo's skill".)

The agent read the documentation, mapped the endpoints, and returned this summary:

More interesting than the table are the decisions it explained on its own:

• Validation endpoint. v2 has no /me, so it used /virtual-phone-accounts/area-codes, the cheapest authenticated GET, which returns 401 with an invalid key.
• 404 is not ignored globally. Only in sms_messages, where a row canceled between the parent call and the child call is expected. In listings, a 404 means a wrong path, and silencing it would produce zero-row syncs with no error.
• includeArchived: "true" on assets. The API excludes archived items by default, which would make equipment disappear from the table without a trace.
• customFields and detections as JSON, because they are structures whose format changes without notice.
• Child endpoint cost made explicit: N rows × M pages per sync, with the cursor keeping M close to 1 after the first load.

It also checked all 41 fields against the official OpenAPI spec and marked everything as # UNVERIFIED, because it transcribed them from the spec and not from real HTTP responses. Its recommendation was to run a curl with a sandbox key before merging.

Step 3: commit and pull request

We asked the agent to commit to a branch and open the PR. It created the salvy-v2-connector branch with 13 files (4 new endpoints, a rewritten _default.yml, template examples removed) and a PR description listing the decisions above. The team reviews it like any other code.

Step 4: connect the repository in Erathos

Go to Connections → New Connection → Select datasource and click Custom connector via GitHub. The first time, you install the Erathos GitHub App and authorize the repository (you need to be an admin of the account or organization).

Then enter the repository in owner/repo format, the source name as it will appear in the catalog, and, if you like, an icon. The first build may take a few minutes. Watched repositories are listed under Settings → Workspace → Repositories.

Step 5: create the connection and configure the jobs

Salvy shows up in the catalog with the Custom badge. From here on, it's the same flow as any connector:

1. Create the connection and enter the API key.
2. Choose the destination schema (in the video, salvy).
3. In Manage Jobs, enable the endpoints and set frequency, sync type, and schedule.
4. Run the first sync.

In BigQuery, the salvy_raw dataset appears with the area_codes, assets, and virtual_phone_accounts tables, already including the Erathos control columns: _erathos_execution_id, _erathos_metadata, and _erathos_synced_at.

After deployment: native-connector control and observability

The real gain isn't just building fast. It's that the custom connector runs on the same platform as the native ones, with the same level of control:

• Jobs per endpoint. Each endpoint becomes a job with its own frequency, sync type (full refresh or incremental), and schedule. Child endpoints show up with a Parent Job trigger.
• Run history. On the Jobs screen, each recent run shows as success or failure, side by side with GitHub, database, and SaaS connectors.
• Row-level traceability. The _erathos_execution_id and _erathos_synced_at columns tell you which run and when each record came from.
• Versioned changes. Did the API change? You (or the agent) adjust the YAML, open a PR, merge it, and Erathos rebuilds the connector on its own.

In practice, that niche API stops being an isolated script and gets the same observability as the rest of the stack.

When to use it (and when not to)

Use the Custom Connector when:

• The source is a REST or GraphQL API with no ready-made connector in the catalog.
• It's an internal API at your company.
• You want to version and review ingestion logic as code.

Choose another path when:

• A native connector already exists for the source. Check the catalog.
• The source isn't an HTTP API, such as a database or files.
• You don't want to maintain a repository. In that case, use Request a connector on the source selection screen.

A good practice with the skill: treat whatever the agent marks as # UNVERIFIED as a real to-do. Before merging, run a call with sandbox credentials to confirm the response format.

Get started now

With the Custom Connector, the API no tool covers gets the same control and observability as native connectors. With the skill, the path from "I need this data" to "the table is in the warehouse" is now a conversation with the agent and a pull request.

• Watch the video with the full Salvy flow: [insert video link]
• Read the documentation: Custom Connectors
• Fork the template: erathos-data/custom-api-connector
• Create your free account at Erathos, no credit card required.