Professional Data Engineer · Domain 1: Designing data processing systems · 22% of the exam
Task 1.4: Designing data migrations.
Assessing stakeholders, users, processes and technologies, then planning and validating a move to Google Cloud with the BigQuery Data Transfer Service, Database Migration Service, Transfer Appliance, Google Cloud networking and Datastream.
Study it
Migration planning: assessment, transfer options and validation
Try each one before opening the answer. Every option is explained, with the Google Cloud documentation page that proves it.
Question 1 · choose 1
A research institute must move 400 TB of archived instrument data from its data center to Cloud Storage within six weeks. Its internet link is 1 Gbps and is shared with production traffic, and more bandwidth cannot be bought before the deadline. What should the data engineer plan?
AStorage Transfer Service agents with a bandwidth cap, running for six weeks
BTransfer Appliance, copying the archive locally and shipping it to Google
CParallel gcloud storage uploads from several hosts over the existing link
DAn event-driven Storage Transfer Service transfer from the source share
Show the answer and why
AStorage Transfer Service agents with a bandwidth cap, running for six weeks
Incorrect
At 1 Gbps, 100 TB takes about 12 days, so 400 TB needs roughly seven weeks at full line rate, and a cap to protect production traffic makes it longer.
BTransfer Appliance, copying the archive locally and shipping it to Google
Correct
Google recommends Transfer Appliance when the available bandwidth cannot move the data before the deadline and more bandwidth cannot be acquired in time.
CParallel gcloud storage uploads from several hosts over the existing link
Incorrect
More parallel uploads do not add bandwidth; the shared 1 Gbps link is still the limit.
DAn event-driven Storage Transfer Service transfer from the source share
Incorrect
Event-driven transfers react to change notifications from cloud sources such as Amazon S3 or Cloud Storage; they do not speed up a bulk copy over a slow link.
The choice between online and offline transfer is a bandwidth calculation. Google's guidance puts 100 TB at about 12 days over 1 Gbps, so 400 TB over a shared 1 Gbps link cannot finish in six weeks. When bandwidth cannot be bought in time, Transfer Appliance moves the data offline.
A company is moving the 3 TB on-premises MySQL 8.0 database behind its order system to Cloud SQL for MySQL. The order system can be down for only a few minutes at cutover. The team wants one managed service that runs the initial load, keeps the Cloud SQL instance in sync with ongoing changes, and then lets them promote it to a standalone primary. What should the data engineer use?
ADatabase Migration Service with a continuous migration job
BDatastream streaming the changes from MySQL into a BigQuery dataset
CA mysqldump export to Cloud Storage and then a Cloud SQL import
DStorage Transfer Service copying the MySQL data files to Cloud Storage
Show the answer and why
ADatabase Migration Service with a continuous migration job
Correct
A continuous migration copies the full database first and then replicates changes until the team promotes the destination, which keeps downtime to the cutover itself.
BDatastream streaming the changes from MySQL into a BigQuery dataset
Incorrect
Datastream replicates into BigQuery, Cloud Storage or Apache Iceberg tables for analytics and events; reaching Cloud SQL needs extra Dataflow templates, and it has no promotion step.
CA mysqldump export to Cloud Storage and then a Cloud SQL import
Incorrect
An export and import is a one-time, point-in-time copy; writes made after the export are missed unless the source stops taking writes for the whole copy.
DStorage Transfer Service copying the MySQL data files to Cloud Storage
Incorrect
Copying raw database files to a bucket does not create a running Cloud SQL database or keep one in sync.
Database Migration Service migrates MySQL, PostgreSQL, SQL Server and Oracle databases to Cloud SQL or AlloyDB. A continuous migration starts with a full initial copy of the database and keeps replicating changes, so the cutover happens when source and destination are in sync and the destination is then promoted to a standalone primary.
A Teradata-to-BigQuery migration includes about 6,000 BTEQ and SQL scripts kept in a Git repository. Before any data moves, the team wants all of them translated to GoogleSQL in bulk, with a report of warnings and errors to plan the manual fixes. What should the data engineer use?
AThe interactive SQL translator in BigQuery Studio, one script at a time
BScheduled queries in BigQuery that run the scripts unchanged
CThe batch SQL translator, with the scripts uploaded to Cloud Storage
DGemini in BigQuery, rewriting each script from a natural-language prompt
Show the answer and why
AThe interactive SQL translator in BigQuery Studio, one script at a time
Incorrect
The interactive translator is for reviewing and debugging a single query; 6,000 scripts one at a time is the case the batch tool exists for.
BScheduled queries in BigQuery that run the scripts unchanged
Incorrect
A scheduled query must be valid GoogleSQL, so Teradata BTEQ and SQL have to be translated before BigQuery can run them, and nothing reports what needs fixing.
CThe batch SQL translator, with the scripts uploaded to Cloud Storage
Correct
The batch translator converts many scripts at once from files in Cloud Storage and writes one output file per input plus a summary report of warnings and errors.
DGemini in BigQuery, rewriting each script from a natural-language prompt
Incorrect
Gemini assistance helps write individual queries in the editor; it is not a bulk dialect translator with a migration report.
BigQuery's migration tooling has a batch SQL translator for bulk work and an interactive translator for single queries. The batch job reads the source files from Cloud Storage, writes the GoogleSQL output alongside, and produces a CSV summary of warnings and errors that the team uses to plan the fixes.
A media company keeps 50 TB of project files on an on-premises NFS filer and has a 10 Gbps Cloud Interconnect connection to Google Cloud. It wants to copy everything to Cloud Storage now and then keep copying new and changed files every night for three months, with bandwidth limits so production traffic is not affected. What should the data engineer use?
ATransfer Appliance, shipped once for the initial copy of the filer
BA cron job on one host that runs gcloud storage rsync each night
CThe BigQuery Data Transfer Service, with a scheduled file transfer
DStorage Transfer Service with agents installed near the filer
Show the answer and why
ATransfer Appliance, shipped once for the initial copy of the filer
Incorrect
The link can move 50 TB in well under a day, and an appliance does not provide the nightly incremental copies.
BA cron job on one host that runs gcloud storage rsync each night
Incorrect
A script on one host has no management interface, agent fault tolerance or transfer logs, and is limited to that host's throughput.
CThe BigQuery Data Transfer Service, with a scheduled file transfer
Incorrect
The BigQuery Data Transfer Service loads data into BigQuery tables; it does not copy files from an NFS filer into Cloud Storage.
DStorage Transfer Service with agents installed near the filer
Correct
Storage Transfer Service for file systems uses agents in the data center for full and incremental copies on a schedule, with bandwidth caps and a record of every file moved.
When bandwidth is sufficient, an online transfer is the right tool. For on-premises file systems, Storage Transfer Service runs fault-tolerant agents near the data, supports recurring incremental transfers and bandwidth limits, and logs every file it moves.
A retailer is considering a move from Amazon Redshift to BigQuery. Stakeholders first want a report of what the current warehouse holds and how it is used, and an estimate of what storing the data and running the existing workload would cost in BigQuery, before any data is moved. What should the data engineer run?
AA BigQuery migration assessment on extracted metadata and query logs
BA trial Data Transfer Service transfer of the largest Redshift tables
CThe batch SQL translator on the warehouse's stored procedures
DThe BigQuery pricing calculator with storage estimated by hand
Show the answer and why
AA BigQuery migration assessment on extracted metadata and query logs
Correct
The migration assessment analyzes the source warehouse's metadata and query logs and produces a report with BigQuery storage and workload cost estimates.
BA trial Data Transfer Service transfer of the largest Redshift tables
Incorrect
A trial transfer moves data, which stakeholders wanted to avoid, and says little about how the whole warehouse is used.
CThe batch SQL translator on the warehouse's stored procedures
Incorrect
Translation converts SQL; it does not inventory the warehouse or estimate BigQuery costs.
DThe BigQuery pricing calculator with storage estimated by hand
Incorrect
A hand estimate leaves out how the workload actually uses the warehouse, which the assessment reads from the query logs.
The BigQuery migration assessment is the first step of a warehouse migration. Metadata and query logs extracted from the source, such as Amazon Redshift, Teradata or Snowflake, are analyzed to describe the existing warehouse and estimate BigQuery costs, with results in a Data Studio report.
An operational PostgreSQL database stays on-premises, where it keeps serving the application. Analysts want its tables replicated into BigQuery with inserts, updates and deletes applied within minutes, without writing or running pipeline code. What should the data engineer use?
AA nightly Cloud Run job that exports the tables to CSV and loads them
BDatastream, with a stream from PostgreSQL to a BigQuery destination
CA nightly BigQuery Data Transfer Service transfer from PostgreSQL
DFederated queries from BigQuery to the on-premises database
Show the answer and why
AA nightly Cloud Run job that exports the tables to CSV and loads them
Incorrect
A nightly export and load is batch code to maintain, and it cannot apply changes within minutes.
BDatastream, with a stream from PostgreSQL to a BigQuery destination
Correct
Datastream is a serverless change data capture service that replicates CDC and backfill data into BigQuery for near real-time analytics.
CA nightly BigQuery Data Transfer Service transfer from PostgreSQL
Incorrect
A nightly batch transfer cannot apply changes within minutes.
DFederated queries from BigQuery to the on-premises database
Incorrect
Federated queries reach Cloud SQL, AlloyDB and Spanner, not an on-premises database, and they replicate nothing.
Datastream reads changes from MySQL, PostgreSQL, Oracle, SQL Server and other sources and applies them to BigQuery, together with a backfill of existing data, with no servers or pipelines to manage.
A warehouse migration to BigQuery has loaded all tables. Before the business switches its reports over, the migration lead wants evidence that each BigQuery table matches its source table, for example in row counts and column aggregates. Which step does Google's migration guidance include for this?
ARerun the migration assessment against the BigQuery tables
BTranslate the reports again with the interactive SQL translator
CRun the data validation tool to compare source and destination
DCheck each table's row count on its BigQuery Details tab
Show the answer and why
ARerun the migration assessment against the BigQuery tables
Incorrect
The assessment analyzes the source warehouse before the move; it does not compare source and destination tables.
BTranslate the reports again with the interactive SQL translator
Incorrect
Translation converts SQL; it says nothing about whether the data matches.
CRun the data validation tool to compare source and destination
Correct
After migrating, Google's guidance is to run the data validation tool to confirm that the source and destination tables match.
DCheck each table's row count on its BigQuery Details tab
Incorrect
Looking at BigQuery alone gives no comparison with the source, and row counts by hand do not scale or cover column values.
A migration is not finished when the data is loaded. Google's migration overview ends with validation: the data validation tool compares source and destination tables, so that consumers move to BigQuery with evidence that the data is complete.