Migrando do BigQuery para o Databricks: Guia de Ferramentas

Moving a warehouse from BigQuery to Databricks means converting SQL, copying history, keeping both sides in sync during cutover, and repointing the pipelines and dashboards around the warehouse. Each of those jobs has different tools. This guide names them, along with the limits that decide between them: export file sizes, type precision, and how sync works when the source is BigQuery.

Which tools help with each part of the migration?

Databricks Lakebridge converts the SQL, and a native export or a managed connector from Fivetran or Airbyte moves the historical tables and keeps the sync running during cutover. Lakehouse Federation covers coexistence, and Erathos repoints ingestion pipelines at Databricks.

BigQuery source support in a tool means it can copy table data. SQL translation, validation, and BI cutover are separate work, and picking a connector settles only one row of this table.

Can Databricks query BigQuery while we migrate?

Yes. Lakehouse Federation runs federated queries on BigQuery data through a Unity Catalog connection and a foreign catalog, so Databricks can read tables that have not moved yet.

Setup needs a Google service account with the BigQuery User and BigQuery Data Viewer roles. On the Databricks side, clusters need Databricks Runtime 16.1 or above with standard or dedicated access mode, and SQL warehouses must be pro or serverless.

Two limits matter for a migration. Views and external tables are always materialized: BigQuery runs the view and writes the result before Databricks reads it, which adds BigQuery compute charges on every query. And BigQuery INTERVAL columns are unsupported, so tables with them need another path.

Federation queries BigQuery where it is, so the tables still need a copy job to become Delta tables. Its role in a migration is coexistence: dashboards already on Databricks can join against tables that have not moved, and you can compare a migrated table with its BigQuery source in a single query.

Which tool moves the historical data?

A BigQuery extract job to Google Cloud Storage in Parquet, followed by a load into Delta tables, moves historical data with no export charge, because batch exports use the free shared slot pool. Fivetran and Airbyte do the same transfer as a managed service.

An extract job writes up to 1 GB per file, so any real table needs a wildcard URI that splits the output across files. Parquet and Avro keep nested and repeated fields, while CSV cannot hold them. The default extract limit is 50 TiB per project per day, and the target bucket must be in the same location as the dataset.

Two details to plan for. Rows loaded while an extract job runs are left out of that export, so the snapshot time marks where cutover sync needs to start. And in Parquet exports, a BIGNUMERIC column comes out as DECIMAL(76, 38) while JSON columns come out as strings, so the load into Delta tables has to handle both.

Fivetran's BigQuery connector reads standard tables through the Storage Read API. A table import that runs longer than 6 hours fails, and Fivetran recommends deselecting tables over 1 TB, so the biggest tables favor the native export. BigQuery tables have no primary keys, so Fivetran adds a generated column named _fivetran_id. Airbyte's source runs full refresh and incremental syncs of tables and views.

How do we keep Databricks in sync with BigQuery during cutover?

Fivetran, Airbyte, and Estuary keep Databricks in sync by querying BigQuery on a schedule, using cursor columns or full-table scans. None of them do log-based change data capture (reading the database's change log instead of querying it) from BigQuery, and each sync cycle runs on BigQuery compute until the day you cut over.

Airbyte's incremental sync follows a cursor field that must always increase and never be null, and its docs recommend a cursor that matches the table's partitioning. Frequent syncs increase BigQuery slot usage. Fivetran detects changes in standard tables with Time Travel, which scans the full table; views sync through Teleport, which also scans the full table.

Estuary's BigQuery connector runs periodic queries against tables with cursor columns. Its Databricks side applies changes through a Unity Catalog Volume with a default sync delay of 30 minutes.

The cursor choice is the main design decision here. It's a timestamp or increasing ID that exists on every table you sync. Tables without one fall back to full refreshes, and every one of those is a full scan billed by BigQuery.

How do BigQuery data types map to Databricks?

Most BigQuery types have a direct Databricks match; the ones to test first are BIGNUMERIC, DATETIME, JSON, and nested fields. BIGNUMERIC holds up to 76 digits of precision while Spark's DecimalType stops at 38, so high-precision values cannot survive the default mapping.

This is how Lakehouse Federation reads BigQuery types:

Connectors map differently. Fivetran turns ARRAY, STRUCT, and JSON columns into STRING. Airbyte keeps arrays and structs as arrays and objects but maps DATE, DATETIME, TIMESTAMP, and TIME to String, and its Databricks destination stores numbers as DECIMAL(38, 10).

The same table lands with different types depending on the path. A good first test is moving one worst-case table, the one with BIGNUMERIC, JSON, and nested fields, through the chosen path and comparing schemas on both sides before scheduling the rest.

Can Lakebridge convert our BigQuery SQL?

Databricks says Lakebridge automates up to 80% of migration tasks, with an Analyzer for assessment, a Converter that turns legacy SQL scripts, stored procedures, and ETL (extract, transform, load) workflows into Databricks SQL or Spark SQL, and a Validator for reconciliation. It is free for Databricks customers and partners.

The published source list names Teradata, Snowflake, Oracle, and SQL Server among more than 10 supported warehouses, with more connectors in private preview. BigQuery is absent from that list, so I'd confirm dialect coverage with Databricks before planning the SQL conversion around it. The conversion engine came from the BladeBridge acquisition in February 2025, which covered code analysis and conversion across more than 20 warehouses and ETL tools.

Whatever the converter covers, the rest still needs a manual pass: scheduled queries, user-defined functions, and scripts that call BigQuery-only functions.

What changes in cost after the move?

On-demand BigQuery bills $6.25 per TiB scanned, with the first TiB free each month; capacity pricing bills per slot-hour instead. Databricks bills compute in DBUs (Databricks Units), priced by SKU (the specific compute product) and cloud. The units have no direct conversion, so the real cost estimate is a proof of concept with your own heaviest queries.

Databricks publishes its SKU price history in the system.billing.list_prices table, so after a trial you can read what each workload cost instead of modeling it.

The migration window itself has BigQuery costs. An extract job processes for free, while EXPORT DATA runs as a query and bills the scanned bytes. Cursor syncs and Time Travel scans also run against BigQuery on every cycle, so a long parallel-run window keeps both bills open.

What happens to the pipelines that load BigQuery today?

The pipelines loading BigQuery from your databases and SaaS tools need a new destination too, and repointing them is its own workstream. Erathos connects over 100 data sources to Databricks, and per source you choose batch loads, cursor-based incremental loads, or change data capture.

Pipelines into the Databricks destination run incrementally by default. The connection takes a server hostname, an HTTP path, a catalog name, and a personal access token or OAuth, on any Databricks instance on AWS, Azure, or Google Cloud. During each job, Erathos stages the extracted data in a temporary cloud bucket, loads it with a COPY command or equivalent, and deletes the temporary files.

During the parallel-run window, the old pipeline keeps loading BigQuery while the new one loads Databricks, so both warehouses stay current and dashboards can switch one at a time. The Databricks integration has a 14-day free test. For teams that keep pipelines in Python, dlt's Databricks destination loads Parquet through a staging bucket on S3, Azure Blob Storage, or Google Cloud Storage.

How do we validate and repoint BI tools without a big-bang cutover?

The cutover order that avoids surprises: copy history, start sync, prove parity, repoint BI tools one at a time, then turn BigQuery ingestion off last.

1. Reconcile every migrated table. Lakebridge includes a Validator for reconciliation between source and target; row counts and column aggregates per table are the baseline check.
2. Run your scheduled queries on both warehouses for a full cycle and compare outputs. Converted SQL shows its problems here, on real data.
3. Pin the freshness boundary. Each table's export snapshot time is the point the sync has to cover, and a table is done when Databricks matches BigQuery past that point.
4. Repoint BI connections one dashboard at a time. Federation lets Databricks read the tables that have not moved, so a half-migrated state still works.
5. Turn off BigQuery ingestion and the sync connectors last, after the dashboards have run on Databricks without falling back.

Ready to repoint your ingestion pipelines? Try Erathos free for 14 days and load Databricks from over 100 sources.