Confluence to data warehouse: automatic sync for spaces, pages, and tasks
Sync Confluence spaces, pages, labels, and tasks to BigQuery, Redshift, or PostgreSQL with the new Erathos Confluence connector.
Erathos launches a managed connector for Confluence. Sixty-three endpoints available, ten destinations supported, zero pipeline code to write or maintain.
Confluence exposes a REST API authenticated with an Atlassian API token, which makes a self-built extraction look cheap at first glance. In practice, it means writing and maintaining a custom pipeline to chase cursor pagination, rate limits that vary by method, and page bodies that arrive as nested JSON in Atlassian Document Format. Erathos resolves this integration as a managed service, with no custom code for your team to write or maintain.
Every data team supporting an operation that runs on Confluence hits the same wall: the documentation, decisions, and action items that move the work forward stay trapped in the wiki, the rest of the analytical stack lives in the warehouse, and joining the two becomes an engineering project nobody planned. What starts as a Python script on a cron job turns, three months later, into a pipeline nobody understands, that fails silently, and that no engineer wants to inherit.
The problem with hand-built Confluence pipelines
The Confluence Cloud REST API authenticates with an Atlassian API token over Basic authentication, and list endpoints return results paginated by cursor, exposed in the response links. The pattern is simple enough to convince any engineer to build an integration in one afternoon. The real cost shows up later, and it is predictable.
Expansion that multiplies call volume. Every child resource lives behind its own call: a single page record requires chasing versions, labels, footer and inline comments, likes, properties, and hierarchy endpoints one by one. A mistake in the iteration logic does not raise an exception: the pipeline simply stops fetching older results and keeps reporting success.
Nested resources that never get synced. To materialize one page completely you need the page, its versions and labels, its footer comments with replies, likes, and versions of their own, plus attachments with labels and versions of their own. A hand-built pipeline tends to sync the top-level tables and quietly skip the rest, and the warehouse ends up with pages without version history and comments without replies.
Atlassian rate limits. Confluence Cloud applies rate limits per method, and a historical backfill across many spaces easily runs into throttled responses. If the retry logic does not respect the limits with backoff, entire windows of data are lost with no error surfaced anywhere.
Content that never stops changing. Pages are edited constantly, comments are added and resolved, attachments are replaced. A sync keyed on the page record alone does not revisit any of that, so stale rows quietly accumulate next to fresh ones, and the dashboard never knows which is which.
Bodies as nested JSON. Page and blog post bodies arrive in Atlassian Document Format, a nested JSON whose shape varies with the content: macros, tables, mentions, and panels each produce a different structure. Flattening it into columns is a project of its own, and when Atlassian ships a new content type the dbt model that ran clean starts failing in production, or worse, keeps running with a null where a column used to be.
Permissions tied to one account. The API token inherits the permissions of the Atlassian account that generated it. When that account loses access to a space, the space simply stops appearing in the next sync without any error: the pipeline stays green and the warehouse gets smaller.
Observability that does not exist. A 200 response does not mean the data arrived complete. Without record counts per endpoint per run, comparison against the previous window, and volume drop alerts, you are flying blind. The pipeline that ran successfully may have brought half the site because a space fell out of the token's reach.
The worst scenario is not the pipeline that breaks and sends an alert. It is the pipeline that executes successfully and delivers wrong data, and the stale-content metric that goes up to the team's dashboard is already calculated over an incomplete base.
What you can do once Confluence data reaches your warehouse
Before detailing the connector, it is worth documenting what you actually gain when the spaces, pages, and tasks your team writes every day leave the SaaS and get modeled next to the rest of your stack.
A content inventory that answers which documentation is stale
With spaces, pages, blogposts, and pages_versions in the warehouse, you measure pages per space, pages untouched for months, authorship concentration, and depth of the page tree. Confluence's native search and space reports answer where a page is; only the raw tables in the warehouse answer which spaces stopped being maintained last quarter and who owns the pages nobody updated.
Contribution by team, crossed with the org
pages_versions carries authorId and createdAt for every edit. Joined in the warehouse with an HR employee base or with your product analytics, documentation becomes measurable: which teams write, who reviews, whether new authors start contributing after an onboarding change, and whether a rewrite of a documentation space coincided with a product launch. That join only exists when Confluence and the org data share the same database.
Documentation crossed with support and product data
Runbooks and release notes live in Confluence; incidents and tickets live in your support tools; feature usage lives in product analytics. With all of them in the same warehouse you can test, for example, whether pages about a feature get updated after its launch, or whether incident runbooks were touched in the weeks before a response-time drop. Confluence's own analytics never crosses these systems; the warehouse does.
Governance and access history for audits
space_permissions, space_roles, space_permission_assignments, space_role_assignments, and classification_levels materialized on every run accumulate a historical trail of who could access which space and under which classification level. Security reviews and compliance audits ask for point-in-time history that spot queries against the API do not preserve; a full refresh landing in the warehouse every day does.
Follow-through on action items
Tasks are the one Confluence object in this connector with incremental sync, cursor keyed on createdAt. With tasks, comments, and likes in the warehouse you measure completion rate per assignee, overdue items by team, and whether decisions recorded in pages actually turn into completed follow-ups. Native Confluence shows each task in context; the warehouse aggregates them across every space and crosses them with CRM or project data.
A knowledge taxonomy you can actually query
labels, label_pages, label_blogposts, and label_attachments let you audit the label taxonomy as data: which labels concentrate content, which duplicated labels fragment it, and how much content carries no label at all. Teams running enterprise search or building internal knowledge assistants need exactly this table in their own warehouse to power retrieval on their own terms, instead of depending on the vendor's built-in graph.
What's available in the connector
The Confluence connector delivers sixty-three endpoints ready to be materialized in your destination warehouse:
Endpoint | What it contains |
|---|---|
spaces | Every space in the site, with key, name, type, status, author, homepage, description, and icon |
space_detail | The full record of each space, adding labels, properties, operations, permissions, and role assignments to the base fields |
space_labels | Labels applied to each space, with prefix and name |
space_operations | Operations allowed on each space and their target types |
space_permission_assignments | Permission assignments on each space, with the principal that holds each one |
space_properties | Custom properties stored on each space, with version and links |
space_role_assignments | Role assignments on each space, with the principal that holds each role |
spaces_blogposts | Blog posts published in each space, with title, author, owner, creation date, and version |
spaces_content_labels | Labels applied to the content of each space |
spaces_pages | Pages that belong to each space, with parent, position, author, owner, and creation date |
pages | Every page in the site, with space, parent, position, author, owner, creation date, and version |
pages_attachments | Files attached to each page, with media type, file size, author, and comment |
pages_footer_comments | Footer comments left on each page, with body, author, and creation date |
pages_inline_comments | Inline comments made on page content, with resolved properties |
pages_direct_children | First-level child pages of each page |
pages_children | Child pages of each page |
pages_ancestors | The ancestor chain of each page, up to the space root |
pages_descendants | All descendant pages below each page |
page_labels | Labels applied to each page |
pages_versions | Version history of each page, with version number, edit message, and author of each version |
page_likes_count | Like counts for each page |
page_likes_users | Users who liked each page, with account, email, and profile data |
page_properties | Custom properties stored on each page, with version |
page_operations | Operations allowed on each page and their target types |
page_detail | The full record of each page, adding body, labels, properties, operations, likes, and versions to the base fields |
blogposts | Every blog post in the site, with space, author, owner, creation date, and version |
blogposts_attachments | Files attached to each blog post, with media type, file size, author, and comment |
blogposts_footer_comments | Footer comments left on each blog post, with body, author, and creation date |
blogposts_inline_comments | Inline comments made on blog post content, with resolved properties |
blogpost_labels | Labels applied to each blog post |
blogpost_versions | Version history of each blog post, with version number, edit message, and author of each version |
blogpost_likes_count | Like counts for each blog post |
blogpost_likes_users | Users who liked each blog post, with account, email, and profile data |
blogpost_properties | Custom properties stored on each blog post, with version |
blogpost_operations | Operations allowed on each blog post and their target types |
blogpost_detail | The full record of each blog post, adding body, labels, properties, operations, likes, and versions to the base fields |
attachments | Every file attached to pages and blog posts, with media type, file size, author, and comment |
attachments_labels | Labels applied to each attachment |
attachment_versions | Version history of each attachment, with version number and author |
attachments_footer_comments | Footer comments left on each attachment, with body and author |
attachment_properties | Custom properties stored on each attachment, with version |
attachment_operations | Operations allowed on each attachment and their target types |
attachment_detail | The full record of each attachment, adding labels, properties, operations, versions, and collaborators to the base fields |
labels | Every label in the site, with prefix and name |
label_pages | Pages that carry each label |
label_blogposts | Blog posts that carry each label |
label_attachments | Attachments that carry each label |
footer_comments | Every footer comment on pages and blog posts, with body, author, and creation date |
footer_comment_children | Replies nested under each footer comment |
footer_comment_likes_users | Users who liked each footer comment, with account and profile data |
footer_comment_likes_count | Like counts for each footer comment |
footer_comment_versions | Version history of each footer comment, with version number and author |
footer_comment_operations | Operations allowed on each footer comment and their target types |
inline_comments | Every inline comment on page and blog post content, with body, author, and resolved properties |
inline_comment_children | Replies nested under each inline comment |
inline_comment_likes_users | Users who liked each inline comment, with account and profile data |
inline_comment_likes_count | Like counts for each inline comment |
inline_comment_versions | Version history of each inline comment, with version number and author |
inline_comment_operations | Operations allowed on each inline comment and their target types |
classification_levels | Data classification levels defined for the site, with name and description |
space_permissions | Space permissions across the site, with principal and permission type |
space_roles | Space roles defined in the site, with keys and bundled permissions |
tasks | Tasks created in pages, blog posts, and inline comments, with status, body, assignee, due date, and completion date; the only endpoint with incremental sync, cursor keyed on createdAt |
Sixty-two endpoints operate in full refresh, which fits content and configuration tables that are always re-read in full. Tasks is the only endpoint with incremental sync, cursor keyed on createdAt. The relationships between entities come preserved in the destination: spaces parent of their details, labels, permissions, pages, and blog posts; pages parent of attachments, comments, versions, likes, and hierarchy; blog posts parent of their own attachments, comments, and engagement; labels parent of the content that carries them; footer and inline comments parent of replies, likes, and versions.
How to authenticate
- Site: your Confluence site subdomain, for example mycompany if you access Confluence at mycompany.atlassian.net
- Token: your Atlassian API token, used to authenticate all requests to the Confluence API together with your Atlassian account email
- Where to find the token: go to id.atlassian.com/manage-profile/security/api-tokens, click Create API token, give it a label such as Erathos, and copy the value right away, since it is displayed only once
- Access requirement: the Atlassian account used to generate the token must have access to the Confluence site you want to sync, because the token inherits that account's permissions
In the Erathos platform, fill in the Site field with your subdomain and paste the token into the Token field, then click Save and close to complete the setup. The full process is documented at https://docs.erathos.com/connectors/apis/confluence.
Why outsource ingestion to Erathos
The premise of the connector is direct: ingestion maintenance engineering should not be your data team's responsibility. Cursor pagination, nested-resource chasing, rate limit handling, schema evolution, failure alerts, volume drop alerts, backfill. All of it belongs to whoever operates the ingestion platform. With the connector configured, the platform delivers:
End-to-end visibility into every run
Extraction time per endpoint, record count per window, which windows were processed, where a retry happened. When a metric changes on a dashboard and a team opens a ticket with the data team, you have the complete trail to find the root cause.
Alerts configured out of the box
Run failure, volume drop per endpoint, and window delay are detected and routed through the alerting integrations your team already uses, such as Slack and email. You do not write this code.
Reprocessing as a supported operation
When you need to reprocess a window, for example to reload a space after fixing the model that flattens page bodies, or to recover a period affected by a permission change on the token, that is a platform operation, not an improvised sequence of DELETE and INSERT in the warehouse.
Correct pagination, nested-resource handling, rate limit management, schema evolution, and backfill are the platform's responsibility. The data team focuses on the model, not the plumbing.
Available pipelines
- Confluence → Azure SQL Server: https://www.erathos.com/pipelines/confluence-azure-sql-server
- Confluence → Azure Synapse: https://www.erathos.com/pipelines/confluence-azure-synapse
- Confluence → BigQuery: https://www.erathos.com/pipelines/confluence-bigquery
- Confluence → ClickHouse: https://www.erathos.com/pipelines/confluence-clickhouse
- Confluence → Databricks: https://www.erathos.com/pipelines/confluence-databricks
- Confluence → PostgreSQL: https://www.erathos.com/pipelines/confluence-postgresql
- Confluence → Redshift: https://www.erathos.com/pipelines/confluence-redshift
- Confluence → S3 Iceberg: https://www.erathos.com/pipelines/confluence-amazon-s3
- Confluence → Snowflake: https://www.erathos.com/pipelines/confluence-snowflake
- Confluence → Supabase: https://www.erathos.com/pipelines/confluence-supabase
Get started
Create your Erathos account at https://app.erathos.com/signup?slug=blog&button=cta&utm_campaign=confluence_release and connect Confluence to your warehouse in minutes. With your Atlassian API token, your first data lands at the destination with zero pipeline code to write, maintain, or monitor.
Documentation, decisions, and action items written into Confluence every day shouldn't stay locked inside a SaaS, disconnected from the rest of your analytics model. Or worse: stuck in a homegrown pipeline that will keep costing your team's attention every month, forever.
See the connector's full documentation at https://docs.erathos.com/connectors/apis/confluence.