Custom API Connectors as Code: Any API, Straight From GitHub

Chances are you have some Python script in your stack somewhere that connects to some API that no ingestion tool out there supports: Your company's internal orders API, some niche SaaS tool the sales team loves, or some regional system with no connector in any catalog. This script runs on some cron job or cloud function, until one day it doesn’t.

With the Erathos Custom Connector, you write some YAML files in your own GitHub repository to describe the API and then push to your GitHub repo. Erathos then runs your Custom Connector as if it was any other connector.

The homemade script nobody wants to maintain

Homemade API scripts often suffer from several problems:

• No retries. A single failed request (like a timeout or rate-limit) will stop the script entirely, meaning you'll be a day behind on updates until someone notices.
• No alerting. If the script fails, nobody will be notified. You'll only find out when something looks wrong on a dashboard.
• Credentials embedded in the script. Your API key will be hardcoded in the script or in a config file alongside it.
• No provenance. There's no way to know which run of the script loaded which rows, or when a given record was last updated.
• No resilience against API changes. If the API changes (maybe a field gets renamed, or the pagination style changes), you'll have to dig back into a script that you haven't touched in months.
• Single-point-of-failure. Only one person will know how to run or modify the script, so if they're on vacation or leave the company, you're in trouble.

Each new API usually means a brand-new script, which means reimplementing pagination, retries and rate limiting each time.

What is an Erathos Custom Connector?

An Erathos Custom Connector is a connector you describe in declarative YAML files, stored in your own GitHub repository, without having to write any Python code. Erathos then watches the repository and rebuilds the connector on every push.

Once the initial build is complete, your connector is available in the datasource catalog, with a Custom badge, and can be connected in exactly the same way as a built-in connector. You choose the endpoints, the schedule and the sync type, and the data is piped into your warehouse.

The YAML defines three things: the location of the API's data, how to page through it, and the names of the fields. Erathos then takes care of the running, retrying, and keeping run history. From the template repository: "You will not write any Python."

How the YAML notation describes an API

Plan: walk through _default.yml, one endpoint file, fields and the three auth types with a realistic example of each file.

Pagination, incremental sync, dependent endpoints and GraphQL

Most APIs do not provide all the results in a single response, and a full daily reload of tables will get slow. To avoid this, we can use the notation shown below in the endpoint file.

Four pagination types

The pagination block under response takes one of four types:

The <page> placeholder is used to pass the current offset/page number/token value to the next request. This is replaced with <timepage_start> and <timepage_end> when using time pagination. In both cases, if the total count path is not provided for the offset or page pagination types, the connector will continue to request more pages until an empty page is returned. Token pagination is the standard way GraphQL APIs paginate. The GraphQL docs consider cursor-based pagination as "the most powerful" approach to the general patterns because the offset approach is subject to change if new records are inserted between requests.

Incremental sync with cursor fields

Now, the cursor: true property is used for incremental sync. It tells Erathos to store the highest value seen for that field, and to insert that value into subsequent requests wherever the field name is used as a placeholder. In orders.yml we have updated_since: "<updated_at>" which makes sure we only fetch the orders that have been updated since the last sync. The cursor_timestamp_format in _default.yml defines how that date should be formatted.

Note that the placeholder must match the name of the field exactly. So, if we had a field named updatedAt, we would use <updatedAt> as the placeholder, not <updated_at>.

Also note that multiple fields can be used as the cursor. This is called a composite cursor and can be useful, for example, if updated_at can be empty on new records and you want to use both created_at and updated_at as the cursor. For incremental sync on the Erathos side, we require both a primary key and a date cursor. This is another reason why we set primary_key above.

Dependent endpoints

Some data only comes from a detail call: you list orders, then fetch the items of each order. A child endpoint declares one parent endpoint and one field from it:

dependency:
endpoint: orders
field: id
as: order_id
request:
method: GET
endpoint: "/orders/<order_id>/items"
response:
path: [data]
primary_key: [id]
fields:
- name: id
type: string
- name: sku
type: string
- name: quantity
type: integer

For each record returned by the orders endpoint, Erathos then calls order_items, substituting <order_id> with the corresponding order id. One endpoint can only depend on one parent through one field, and child endpoints cost one request per parent record (this can be kept small after the first load by using a cursor on the parent). For example, with 3 orders, the child endpoint makes 3 calls:

GraphQL

A GraphQL endpoint uses query instead of params. You paste the query as-is, with no need to escape the braces. This trimmed version of the template's example, based on Linear's issues API, pages with a token and filters by a cursor:

request:
method: POST
endpoint: "/graphql"
query: >
{
issues(first: 250 <page> filter: {updatedAt: {gte: "<updatedAt>"}}) {
pageInfo { hasNextPage endCursor }
nodes { id title updatedAt state { name } }
}
}
response:
path: [data, issues, nodes]
primary_key: [id]
pagination:
type: token
path: [data, issues, pageInfo, endCursor]
fields:
- name: id
type: string
- name: title
type: string
nullable: true
- name: updatedAt
type: timestamp
cursor: true
cursor_timestamp_format: "%Y-%m-%dT%H:%M:%S.000Z"
- name: state.name
type: string
nullable: true

The response path steps into the GraphQL envelope, and the pagination path points to endCursor (which is the field the GraphQL connection model uses to indicate the next page).

From repository to connector

Before you push, the template's validator checks the folder. It needs PyYAML and runs from the repository root:

pip install pyyaml
python validate_connector.py .

It flags missing required keys, unknown keys, bad field types, placeholders that don't match anything and YAML that doesn't parse. With url_base removed from _default.yml, it stops with exit code 1:

Validating connector YAML in: ./acme_broken

_default.yml
[ERROR] request.url_base: missing required field

Summary: 1 error(s), 0 warning(s) across 1 file(s).

Note that the validator only checks if the shapes match. A response.path that points at a key the API never returns will still pass the validator. Such a connector would then sync zero rows. Thus, before trusting a new endpoint, test it on a real response.

To connect your repository, go through the following steps as outlined in the Erathos docs:

1. Make sure you have admin access on the GitHub account or organization that owns the repository.
2. In Erathos, go to Connections → New Connection → Select datasource, or Settings → Workspace → Repositories, and click Connect a custom connector via GitHub.
3. In GitHub, Erathos will ask you to install the Erathos GitHub App. Choose the account or organization you want to install the app on, and select the repositories Erathos should be able to read.
4. After installing the app in GitHub, the Watch repository dialog opens in Erathos. In the dialog, enter the repository in the form of owner/repo (e.g. your-org/your-connector) and click Watch repository.
5. Erathos then starts the first build. It can take a few minutes, and the Repositories page refreshes until the build is complete.
6. Once the build is successful, the connector is listed in the datasource catalog and has a Custom badge.

Organization owners can install apps on the organization, and you can choose to make the app accessible only to Only select repositories instead of all repositories on the account or organization. If you are a member of an organization and do not have the rights to install apps on the organization, the request will be forwarded to an organization owner.

Each push to the watched repository will trigger a rebuild of the connector image. Thus, every push to the watched repository is a deploy. This means that a typo merged on a Friday afternoon will be deployed with the next build, so using the validator and going through a pull request review might be worth the extra minute.

The overall flow from YAML files to rows in your warehouse is:
Under Settings → Workspace → Repositories, you can see a list of repositories you watch, as well as the branch. This is also where you can remove them. However, once you click "Remove," the connector stops rebuilding and syncing immediately, and there is no way to undo that on the page.

Why keep connectors as code

The connector is a collection of files in your repository, so it’s treated like other code in your project.

• Review. All changes to endpoints are done via a pull request. Other team members can see that someone’s changed orders.yml from page to token pagination before that change is deployed.
• History. Git tracks changes to fields, as well as who changed them and when. If a column starts to come back empty, run git log orders.yml to see what changed.
• Rollback. Since every push to the repository triggers a rebuild of the connector, a rollback of a bad change is as simple as running git revert on the change and then running git push to deploy the previous version of the connector.

Because the connector is no longer a script that someone wrote, anyone on the team can understand the connector after a few minutes of reading the YAML files.

What you get after deploy

If the build is successful, you can create a connection from the connector with the Custom badge, inputting the credentials you defined, selecting a destination schema and enabling the endpoints you want, in the Manage Jobs section. Then, it will work as any native connector:

• One job per endpoint, with its own frequency, schedule and sync type. Child endpoints will be listed with a Parent Job trigger and will only run after the successful execution of its parent job.
• Full refresh or incremental syncs: Full refresh will load the entire endpoint in every run. Incremental (partial) types will only load the records after the cursor and require the primary_key and the cursor field defined in the YAML file.
• Run history: Every run will be tracked as a success or a failure, along with all the other connectors. Logs, timestamps, metrics and errors are available in the runs page for each run.
• Row-level traceability: Control columns, such as _erathos_execution_id and _erathos_synced_at will be appended to every table loaded. This allows you to track any row back to the run that loaded it and the time it was loaded.

The Erathos control page has more details on the options available at the job level, such as triggering jobs through API, running them from Airflow or Dagster, retry strategies, etc.

Building the files with an AI agent

We have also created an AI agent skill for this template repository. This skill will read the API’s documentation and then ask you any questions that are left open. It will write the YAML files for you and run the validator. It will also add a # UNVERIFIED comment for each line that it could not confirm from the docs. This means that you can then focus on testing these lines first. We have written a blog post about this AI agent skill, in which we go through the skill step-by-step and create a full connector for one real API.

Limits of the YAML notation

The notation for custom connectors covers static header or token auth and OAuth2, the four pagination types, and responses you can reach with a flat path list. It does not, however, cover all possible API structures. For example, some APIs use authentication methods other than static headers, tokens, and OAuth2, such as signing each request. Similarly, some APIs use pagination methods other than offset, page, token, and time. APIs may also return responses with complex structures that cannot be reached with a simple list of keys or must be merged from multiple places in a single response. Finally, dependencies between endpoints may be more complex than a single child endpoint depending on a single field of a single parent endpoint. In all of these cases, the custom connector template recommends stopping to ask rather than attempting to shoehorn it into the closest field in the template.

Custom connectors are also not intended for databases, files, or other sources that have their own native Erathos connectors. These native connectors are maintained by Erathos.

Finally, one setup detail that is easy to miss is that the example files in the template are valid endpoints, so any example_*.yml file that is not deleted before the next push will be created as a live endpoint.

Custom Connector, native connector or homemade script?

Here’s a quick guide to help you choose between those options:

• Use a native connector if your source already has one. You can browse the Erathos connector catalog and request a connector there in case your source doesn’t already have one and you don’t want to maintain a repository yourself.
• Use a Custom Connector if your source is an HTTP API with JSON responses, doesn’t have a native connector, and can be described with the notation (auth, pagination, response shape). Examples are internal APIs, niche SaaS tools, and regional systems.
• Use a script if your API uses an auth, pagination, or response shape that isn’t covered by the notation, if your source isn’t an HTTP API, or if you need to run a one-off pull that you will never repeat.

Note that Custom Connector is available on the Pro plan.

FAQ

Do I need to know Python to build a Custom Connector?

No, you’re just writing YAML files with the API described, and there’s a line in the template that says explicitly you won’t write any Python code. The only Python code is running the validator script, which is one command.

Where are the API credentials stored?

Credentials never go in your repository. The YAML file contains only placeholders like <api_key>. When you create a connection in Erathos, you’ll enter the actual values for these placeholders, and secret values will be masked in the UI.

What happens if the API changes?

After making the necessary changes in the YAML file for the endpoint affected by the API update, we open a pull request and merge the changes into the connector. The push to the connector triggers a rebuild and the next sync run will show up in the run history as a failure if it didn’t succeed.

Can I use a Custom Connector with a GraphQL API?

Yes. A GraphQL endpoint puts the query in request.query. POST is used for GraphQL, and paging is most commonly done using the token type that points to the endCursor field.

How do I roll back a bad change?

Revert the commit and push. Every push to the watched repository triggers a rebuild of the connector, and reverting will restore the previous version in the next build.

Try it on your own API

Test the API behind the most annoying script you have. Copy the template, write down one endpoint, run the validator, watch the repository. Try Erathos for free for 14 days. You’ll get the Pro version of Erathos during your trial period. No credit card required.