GitHub Connector: the pipeline you don’t need to maintain
Managed connector to sync GitHub repositories, members, issues, pull requests, and commits to your warehouse. Create your account and try it now.
Bring repositories, members, issues, pull_requests, and commits into your warehouse without worrying about pagination, rate limits, alerts, or schema evolution.
The problem with hand-built GitHub pipelines
Every data team, at some point, gets the same question: how are we measuring engineering productivity? The answer almost always runs through the data already living in GitHub: pull requests, commits, issues, org members. What looks like a simple ELT use case actually hides a classic SaaS API ingestion problem. The pipeline works until the day it doesn't, and nobody notices.
Building from scratch is tempting. The GitHub API is public, well documented, and any engineer can pull a PR listing in just a few lines. The cost shows up later, and it's not in the code you write in the first sprint. It's in everything you end up maintaining forever.
Every endpoint relevant for analytics paginates differently, and the logic you wrote once has to be revisited when GitHub changes a default. Rate limits are shared across operations and can drop without warning when you scale to a large org, so you end up implementing exponential backoff, ETag caching, and some mechanism to spread out the extraction window. Schemas change: new fields appear, others become nullable. Your DAG in Airflow keeps reporting success while delivering less data than it should.
And there's the part nobody remembers when it's time to estimate: alerts. When a run fails, someone needs to know. When the record count drops 80% compared to the previous week, someone needs to know. When the PR count goes to zero for an active repository, someone needs to know. Building and maintaining that observability logic (not the product's, the pipeline's) is a platform project in itself.
The worst-case scenario isn't the pipeline that breaks. It's the pipeline that silently degrades, and the metric that reaches the CTO's dashboard is already wrong.
What you can do with GitHub data in your warehouse
Before talking about the connector, it's worth spelling out what you gain when this data lands modeled alongside the rest of your analytics stack.
Engineering analytics and DORA metrics. Pull request lead time (from opened_at to merged_at), throughput by repository, average review time, distribution of changes across authors. Combine that with CI/CD deploy data and you can calculate the four DORA metrics (deployment frequency, lead time for changes, change failure rate, and MTTR) without relying on an external tool that charges a license fee.
Review bottleneck detection. By joining pull_requests with members, you can quickly see who is carrying the review queue and where changes are getting stuck the longest. It's the kind of management input data teams could never deliver when the data lived only in the SaaS tool.
Bug hotspots and quality. By combining issues with commits by repository, you can identify where churn is highest and where issue resolution time is longest. The kind of analysis that, until now, required an analyst digging through manual reports.
Audit, compliance, and onboarding. Knowing who joined and left the org, who has access to which repositories, and combining that with the HR system in the warehouse. Useful for teams that need to prove controls for SOC 2 or ISO 27001.
Combining with the rest of the stack. The real win shows up when GitHub data sits next to Linear or Jira, the CRM, and product data. That's when questions like "how much engineering time went to enterprise customers last quarter?" stop being a spreadsheet and become a dbt model maintained by the team.
Why outsource ingestion to Erathos
The premise of the connector is simple: maintaining the ingestion layer shouldn't be your data team's responsibility. Pagination, rate limit, retry, schema evolution, failure alerts, degradation alerts, backfill. All of that is the responsibility of whoever operates the ingestion platform, not a technical decision for your analytics engineer.
That's exactly what the connector delivers. You generate the token, connect the org, and from that point on:
End-to-end visibility for every run. How long each extraction took, how many records came from each endpoint, which windows were processed, and where retries happened. This helps both with finding the root cause of a bad metric and with answering the product team when they ask why the number changed.
Alerts configured out of the box. Failures, volume drops, and delayed windows are detected and sent through the alerting integrations your team already uses. You don't have to write that code.
Reprocessing as a supported operation. When you need to reprocess a specific window, whether because the model changed or because you received a data correction from the source, that's a button, not an improvised SQL exercise.
On top of that, correct pagination, rate limit management, schema evolution, and backfill are handled by the platform. The team focuses on the data model, not the plumbing.
What's available in the connector
The connector delivers five endpoints ready to be materialized in your destination warehouse:
repositories: org repository catalogmembers: org membersissues: open and closed issuespull_requests: pull requests with review and merge metadatacommits: commits from synchronized repositories
The destinations currently supported are BigQuery, Redshift, PostgreSQL, SQL Server, Databricks, and Amazon S3.
Authentication is done with a GitHub Personal Access Token (classic). The token needs two scopes: repo (to read repository data) and read:org (to read org members). The step-by-step process is described in the connector documentation.
Get started now
Create your Erathos account and test the connector with your repositories. In just a few minutes, with the token and the org name, you'll see the first data landing in the warehouse, with no pipeline code to write, maintain, or monitor.
Engineering generates data every day. It makes little sense for that data to stay locked in a SaaS, outside your model, or worse, in a homemade pipeline that will cost you attention every month forever.
Bring repositories, members, issues, pull_requests, and commits into your warehouse without worrying about pagination, rate limits, alerts, or schema evolution.
The problem with hand-built GitHub pipelines
Every data team, at some point, gets the same question: how are we measuring engineering productivity? The answer almost always runs through the data already living in GitHub: pull requests, commits, issues, org members. What looks like a simple ELT use case actually hides a classic SaaS API ingestion problem. The pipeline works until the day it doesn't, and nobody notices.
Building from scratch is tempting. The GitHub API is public, well documented, and any engineer can pull a PR listing in just a few lines. The cost shows up later, and it's not in the code you write in the first sprint. It's in everything you end up maintaining forever.
Every endpoint relevant for analytics paginates differently, and the logic you wrote once has to be revisited when GitHub changes a default. Rate limits are shared across operations and can drop without warning when you scale to a large org, so you end up implementing exponential backoff, ETag caching, and some mechanism to spread out the extraction window. Schemas change: new fields appear, others become nullable. Your DAG in Airflow keeps reporting success while delivering less data than it should.
And there's the part nobody remembers when it's time to estimate: alerts. When a run fails, someone needs to know. When the record count drops 80% compared to the previous week, someone needs to know. When the PR count goes to zero for an active repository, someone needs to know. Building and maintaining that observability logic (not the product's, the pipeline's) is a platform project in itself.
The worst-case scenario isn't the pipeline that breaks. It's the pipeline that silently degrades, and the metric that reaches the CTO's dashboard is already wrong.
What you can do with GitHub data in your warehouse
Before talking about the connector, it's worth spelling out what you gain when this data lands modeled alongside the rest of your analytics stack.
Engineering analytics and DORA metrics. Pull request lead time (from opened_at to merged_at), throughput by repository, average review time, distribution of changes across authors. Combine that with CI/CD deploy data and you can calculate the four DORA metrics (deployment frequency, lead time for changes, change failure rate, and MTTR) without relying on an external tool that charges a license fee.
Review bottleneck detection. By joining pull_requests with members, you can quickly see who is carrying the review queue and where changes are getting stuck the longest. It's the kind of management input data teams could never deliver when the data lived only in the SaaS tool.
Bug hotspots and quality. By combining issues with commits by repository, you can identify where churn is highest and where issue resolution time is longest. The kind of analysis that, until now, required an analyst digging through manual reports.
Audit, compliance, and onboarding. Knowing who joined and left the org, who has access to which repositories, and combining that with the HR system in the warehouse. Useful for teams that need to prove controls for SOC 2 or ISO 27001.
Combining with the rest of the stack. The real win shows up when GitHub data sits next to Linear or Jira, the CRM, and product data. That's when questions like "how much engineering time went to enterprise customers last quarter?" stop being a spreadsheet and become a dbt model maintained by the team.
Why outsource ingestion to Erathos
The premise of the connector is simple: maintaining the ingestion layer shouldn't be your data team's responsibility. Pagination, rate limit, retry, schema evolution, failure alerts, degradation alerts, backfill. All of that is the responsibility of whoever operates the ingestion platform, not a technical decision for your analytics engineer.
That's exactly what the connector delivers. You generate the token, connect the org, and from that point on:
End-to-end visibility for every run. How long each extraction took, how many records came from each endpoint, which windows were processed, and where retries happened. This helps both with finding the root cause of a bad metric and with answering the product team when they ask why the number changed.
Alerts configured out of the box. Failures, volume drops, and delayed windows are detected and sent through the alerting integrations your team already uses. You don't have to write that code.
Reprocessing as a supported operation. When you need to reprocess a specific window, whether because the model changed or because you received a data correction from the source, that's a button, not an improvised SQL exercise.
On top of that, correct pagination, rate limit management, schema evolution, and backfill are handled by the platform. The team focuses on the data model, not the plumbing.
What's available in the connector
The connector delivers five endpoints ready to be materialized in your destination warehouse:
repositories: org repository catalogmembers: org membersissues: open and closed issuespull_requests: pull requests with review and merge metadatacommits: commits from synchronized repositories
The destinations currently supported are BigQuery, Redshift, PostgreSQL, SQL Server, Databricks, and Amazon S3.
Authentication is done with a GitHub Personal Access Token (classic). The token needs two scopes: repo (to read repository data) and read:org (to read org members). The step-by-step process is described in the connector documentation.
Get started now
Create your Erathos account and test the connector with your repositories. In just a few minutes, with the token and the org name, you'll see the first data landing in the warehouse, with no pipeline code to write, maintain, or monitor.
Engineering generates data every day. It makes little sense for that data to stay locked in a SaaS, outside your model, or worse, in a homemade pipeline that will cost you attention every month forever.