Skip to content

Associate Data Practitioner · Domain 3: Data Pipeline Orchestration · 18% of the exam

Task 3.2: Schedule, automate, and monitor basic data processing tasks.

Scheduled queries, monitoring Dataflow jobs in the job UI, Cloud Logging and Cloud Monitoring, choosing an orchestration option, Pub/Sub to BigQuery ingestion and Eventarc triggers.

Study it

Sample questions

Try each one before opening the answer. Every option is explained, with the Google Cloud documentation page that proves it.

Question 1 · choose 1

A streaming Dataflow job reads events from Pub/Sub and writes them to BigQuery. It was launched without Streaming Engine. At peak hours its backlog keeps growing, and the on-call team wants the Dataflow monitoring interface to show which step is holding the pipeline back and the likely cause, so that they fix the right thing. What should the team do?

  1. AOpen the Job graph, find the step that shows a bottleneck alert, and read its likely cause in the Step info panel
  2. BClick each step in the Job graph and take the step with the largest system watermark lag in the Step info panel as the cause
  3. CRelaunch the job with Streaming Engine, then open the step marked with a bottleneck alert and read its likely cause
  4. DRelaunch the job with Streaming Engine, then compare the CPU utilization charts of the workers on the Job metrics tab
Show the answer and why
  • AOpen the Job graph, find the step that shows a bottleneck alert, and read its likely cause in the Step info panel

    Incorrect

    Bottleneck detection is not supported for jobs that don't run on Streaming Engine, so this job's graph shows no bottleneck alerts to read.

  • BClick each step in the Job graph and take the step with the largest system watermark lag in the Step info panel as the cause

    Incorrect

    A backlog builds up in the queues upstream of the step that causes it, and later steps wait on its output, so many steps show a large lag at once; the lag points to no single step and names no cause.

  • CRelaunch the job with Streaming Engine, then open the step marked with a bottleneck alert and read its likely cause

    Correct

    On Streaming Engine jobs, the bottleneck detector marks steps with an active bottleneck in the Job graph, and the Step info panel gives the kind of bottleneck and its likely cause.

  • DRelaunch the job with Streaming Engine, then compare the CPU utilization charts of the workers on the Job metrics tab

    Incorrect

    CPU utilization is a per-worker metric for the whole job; it names neither the step that holds the pipeline back nor a cause such as a throttled sink.

In a streaming pipeline, one slow step can back up the queues upstream of it, so that a large part of the pipeline looks unhealthy. The Dataflow bottleneck detector finds the step: a step whose delay passes five minutes shows an alert in the Job graph, and its Step info panel lists the kind of bottleneck, such as falling behind or stuck, and the likely cause, if known. Bottleneck detection is supported only for jobs that run on Streaming Engine, and it also exports its findings to Cloud Monitoring.

Question 2 · choose 1

A streaming Dataflow job on Streaming Engine computes running totals per site from sensor readings, grouping the readings by site_id; the company has 12 sites. Since traffic tripled, the backlog grows for more than 30 minutes at every peak and the workers' CPU stays high, yet the job never runs more than 12 workers, although max_num_workers is 100. What change lets the job keep up?

  1. ARaise max_num_workers to 200, so that the autoscaler has more room to add workers at peak times
  2. BSet the job's autoscaling latency tier to low_latency, so that it scales up as soon as a short backlog builds
  3. CGroup first by site and sensor, then combine the results per site, so that the work spreads over more keys
  4. DSet a lower worker utilization hint, so that the autoscaler adds workers at a lower CPU utilization
Show the answer and why
  • ARaise max_num_workers to 200, so that the autoscaler has more room to add workers at peak times

    Incorrect

    The job is far below its maximum of 100; Dataflow can't scale a stage beyond the number of distinct keys it processes, so a higher maximum adds nothing.

  • BSet the job's autoscaling latency tier to low_latency, so that it scales up as soon as a short backlog builds

    Incorrect

    low_latency is already the default tier, which aims for a backlog of less than 15 seconds, and no tier lets a stage use more workers than it has keys to spread the work over.

  • CGroup first by site and sensor, then combine the results per site, so that the work spreads over more keys

    Correct

    Dataflow can't scale beyond the number of distinct processing keys at a stage, so with 12 site keys extra workers would sit idle; more keys let the autoscaler use more workers.

  • DSet a lower worker utilization hint, so that the autoscaler adds workers at a lower CPU utilization

    Incorrect

    The utilization hint is only one factor in autoscaling, and available keys can override it; with 12 keys the work can't be distributed across more workers, so adding them doesn't help.

Streaming autoscaling adds workers when there is a backlog or high worker utilization, within the configured range and settings such as the utilization hint and latency tier. It can't go beyond the parallelism of the pipeline: a stage can't use more workers than it has distinct keys, so a stream with few unique keys stays capped even with a high maximum. The fix is in the pipeline, for example a finer key followed by a second, combining step.

Question 3 · choose 1

A data team has about 30 Apache Airflow DAGs written in Python that chain tasks across an on-premises database, BigQuery and another cloud, with dependencies between tasks, schedules and some runs triggered by new files. It is moving them to Google Cloud with as little rework as possible and wants no servers of its own to run, patch or scale. Which orchestration option should it choose?

  1. AWorkflows, deploying each DAG file as a workflow definition
  2. BBigQuery scheduled queries, one for each task in the DAGs
  3. CCloud Scheduler, with one cron job for each task in the DAGs
  4. DManaged Service for Apache Airflow (Cloud Composer in the guide)
Show the answer and why
  • AWorkflows, deploying each DAG file as a workflow definition

    Incorrect

    Workflows is serverless, but a workflow definition is written in Workflows syntax as YAML or JSON steps, so each Python DAG would have to be rewritten.

  • BBigQuery scheduled queries, one for each task in the DAGs

    Incorrect

    A scheduled query runs SQL in BigQuery on a schedule; it cannot run tasks against the on-premises database or the other cloud, and nothing makes one task wait for another.

  • CCloud Scheduler, with one cron job for each task in the DAGs

    Incorrect

    Cloud Scheduler sends cron jobs to targets on a schedule; it does not model dependencies between tasks or run DAG code.

  • DManaged Service for Apache Airflow (Cloud Composer in the guide)

    Correct

    Managed Airflow runs Apache Airflow DAGs written in Python in managed environments, with no installation or management overhead, and DAGs can run on schedules or be triggered by events, across clouds and on-premises systems.

Pick orchestration by what the work needs. Managed Service for Apache Airflow (formerly Cloud Composer) runs existing Airflow DAGs, with task dependencies, schedules and event triggers across clouds and on-premises systems, without managing Airflow. Workflows suits serverless sequences of API calls written in YAML or JSON, scheduled queries run SQL in BigQuery, and Cloud Scheduler only triggers single jobs on a schedule.

Question 4 · choose 1

A batch Dataflow job with about 60 steps now takes twice as long as it did last month. In the Dataflow job monitoring interface, the analyst wants to find quickly which steps take the most time. What should the analyst do?

  1. AOpen the Job graph tab, switch the job steps view to Table view and sort the steps by wall time
  2. BOpen the job's Job metrics tab and compare the resource utilization charts over the run
  3. CCreate an uptime check in Cloud Monitoring against the job's workers
  4. DOpen the job's Autoscaling tab and compare the worker counts over time
Show the answer and why
  • AOpen the Job graph tab, switch the job steps view to Table view and sort the steps by wall time

    Correct

    The table view shows the same step information as the graph and can be sorted by a property such as wall time, which is the documented way to identify slow steps in a job with many stages.

  • BOpen the job's Job metrics tab and compare the resource utilization charts over the run

    Incorrect

    Job metrics charts show metrics for the whole job over its duration, such as resource use; they do not rank the 60 steps by the time each one takes.

  • CCreate an uptime check in Cloud Monitoring against the job's workers

    Incorrect

    Uptime checks test whether an endpoint responds; they say nothing about the time spent in each pipeline step.

  • DOpen the job's Autoscaling tab and compare the worker counts over time

    Incorrect

    Autoscaling information shows how the number of workers changed, not which steps consumed the time.

The Dataflow job graph shows each step with its status, data lag, wall time and maximum operation latency. For jobs with many stages, the table view of the job steps lets you sort by wall time to find the slowest steps.

Question 5 · choose 1

A team starts about 20 streaming Dataflow jobs from deployment scripts with the gcloud CLI. Finance wants an email whenever a job's vCPU usage jumps far above that job's normal level, and the team does not want to design alert policies itself. The project has no Cloud Monitoring notification channels yet. What should the team do?

  1. ACreate an email notification channel, then add the goog-turnkey-alerts label to each running job
  2. BCreate an email notification channel, then add --enable-turnkey-alerts when the jobs are next submitted or updated
  3. CAdd --enable-turnkey-alerts to the job submissions, so that Dataflow emails the project's owners when a policy fires
  4. DCreate an email notification channel, then turn on turnkey alerts for each running job from its Autoscaling tab
Show the answer and why
  • ACreate an email notification channel, then add the goog-turnkey-alerts label to each running job

    Incorrect

    The goog-turnkey-alerts label adds a Cloud Monitoring alert policy to the turnkey suite; a job itself is enrolled with the gcloud flag when it is submitted.

  • BCreate an email notification channel, then add --enable-turnkey-alerts when the jobs are next submitted or updated

    Correct

    Turnkey alerts are a preconfigured suite of alert policies, including a vCPU growth policy, that a job joins through the gcloud flag at submission; alerts reach the project's notification channels, so one must exist.

  • CAdd --enable-turnkey-alerts to the job submissions, so that Dataflow emails the project's owners when a policy fires

    Incorrect

    Turnkey alerts notify through Cloud Monitoring notification channels, and at least one channel must be configured; without one, the policies fire but nobody is emailed.

  • DCreate an email notification channel, then turn on turnkey alerts for each running job from its Autoscaling tab

    Incorrect

    The Autoscaling tab only charts how workers changed; a job is enrolled in turnkey alerts with the gcloud flag when it is submitted, and the setting carries over to job updates.

Dataflow turnkey alerts give streaming jobs a recommended set of Cloud Monitoring alert policies without manual setup, such as a policy that fires when the 1-hour average vCPU count rises far above the 7-day average. A job is enrolled with --enable-turnkey-alerts when it is submitted, the setting is kept across updates, and alerts are delivered to the project's notification channels, at least one of which must exist.

Question 6 · choose 1

A new Dataflow job never makes progress: the job graph shows no step running, and the job messages say that workers are not becoming ready. The analyst suspects a problem while the workers download the job's files and start up. Which logs should the analyst read first?

  1. AThe job's worker-startup logs
  2. BThe job's shuffler logs
  3. CThe project's Admin Activity audit logs
  4. DThe job's harness logs
Show the answer and why
  • AThe job's worker-startup logs

    Correct

    Worker-startup logs capture the startup process, including downloading the job's files from Cloud Storage and starting the workers, which is where a problem starting workers shows up.

  • BThe job's shuffler logs

    Incorrect

    Shuffler logs come from workers that consolidate the results of parallel operations, which only happens once the pipeline is processing data.

  • CThe project's Admin Activity audit logs

    Incorrect

    Audit logs record administrative changes to resources; they do not hold the startup output of Dataflow workers.

  • DThe job's harness logs

    Incorrect

    Harness logs hold messages from the Portable Runner harness that runs pipeline code, not the startup steps that come before it.

Dataflow writes several log types, among them job-message logs for job-level events, worker-startup logs for the startup process, and harness and shuffler logs for specific components. When workers fail to start, the worker-startup logs are the place to look.

Question 7 · choose 1

An analyst wants a BigQuery scheduled query to refresh a small summary table as often as the service allows. What is the minimum time between runs of a scheduled query?

  1. A1 minute
  2. B15 minutes
  3. C5 minutes
  4. D24 hours
Show the answer and why
  • A1 minute

    Incorrect

    Scheduled queries cannot run this often; the minimum duration between runs is 5 minutes.

  • B15 minutes

    Incorrect

    Fifteen minutes is allowed, but it is not the minimum; runs can be as close as 5 minutes apart.

  • C5 minutes

    Correct

    The scheduled queries page states that the minimum duration between scheduled queries is 5 minutes.

  • D24 hours

    Incorrect

    Every 24 hours is the default schedule when none is given, not the shortest one.

Scheduled queries accept custom, cron-like schedules such as every 6 hours or every monday 09:00; the default is every 24 hours and the minimum duration between runs is 5 minutes.

Google Cloud documentation

Question 8 · choose 1

A nightly Dataflow batch job finished, but it took longer and cost more than the team expected. A data practitioner opens the job in the Dataflow monitoring interface and wants Dataflow's own insights into what could be changed to make this job cheaper and faster. Which tab should she open?

  1. AThe Recommendations tab of the job
  2. BThe Estimated cost tab of the job
  3. CThe Job metrics tab of the job
  4. DThe Execution details tab of the job
Show the answer and why
  • AThe Recommendations tab of the job

    Correct

    This tab shows insights from Dataflow about the pipeline, aimed at situations where cost and performance could be improved, as well as help with errors.

  • BThe Estimated cost tab of the job

    Incorrect

    It shows the estimated cost of the job based on resource usage metrics; it reports what the job cost, not what to change.

  • CThe Job metrics tab of the job

    Incorrect

    It charts metrics over the duration of the job, such as resource utilization; reading the charts gives no suggested changes.

  • DThe Execution details tab of the job

    Incorrect

    It shows the execution stages of the job and how they progressed; it describes how the run went, not what to change.

For each job, the Dataflow monitoring interface shows the job graph, execution details, job metrics, estimated cost, recommendations and logs, plus autoscaling charts for streaming jobs. The Recommendations tab lists Dataflow's insights for improving job performance, reducing cost and troubleshooting errors; each is kept for 30 days from its update date.

Question 9 · choose 1

A developer added logging statements inside a DoFn of a running Dataflow pipeline to trace odd records. Which Dataflow log type holds the messages that this pipeline code writes?

  1. AJob-message logs
  2. BWorker logs
  3. CShuffler logs
  4. DCloud Audit Logs
Show the answer and why
  • AJob-message logs

    Incorrect

    Job-message logs contain job-level messages generated by Dataflow components, such as the autoscaling configuration and job progress, not the messages your DoFn logs.

  • BWorker logs

    Correct

    Worker logs are produced by the Dataflow workers that do the pipeline work, such as applying ParDo transforms, and contain messages logged by your code.

  • CShuffler logs

    Incorrect

    Shuffler logs come from workers that consolidate the results of parallel operations, not from user code in a DoFn.

  • DCloud Audit Logs

    Incorrect

    Audit logs record administrative activity and data access on Google Cloud resources, not messages from pipeline code.

In the Dataflow monitoring interface, selecting a step switches the logs panel to step logs from your code, and Logs Explorer shows the worker logs for the job.

Question 10 · choose 1

A data team only needs to deliver one kind of Google Cloud event to a single destination that starts its pipeline, filtering the events by their source and type. It has no need for a central event bus shared across teams or for transforming events. Which Eventarc option does Google recommend for this?

  1. AEventarc Advanced, with a bus and an enrollment
  2. BA Cloud Scheduler job that polls for the events
  3. CA Workflows workflow with a step that lists new events
  4. DEventarc Standard, with a trigger that filters the events
Show the answer and why
  • AEventarc Advanced, with a bus and an enrollment

    Incorrect

    Eventarc Advanced, with its central bus, enrollments and optional event transformation, targets complex eventing needs across teams.

  • BA Cloud Scheduler job that polls for the events

    Incorrect

    Cloud Scheduler sends jobs on a time schedule; it does not receive or filter events as they happen.

  • CA Workflows workflow with a step that lists new events

    Incorrect

    Workflows runs steps when it is called; it is a common destination of events, not the service that routes them.

  • DEventarc Standard, with a trigger that filters the events

    Correct

    Eventarc Standard is recommended when the focus is simply delivering events from a provider to a destination, using triggers that filter events by source, type and other attributes.

Eventarc comes in two editions with the same data model. Eventarc Standard routes events through triggers that filter by source, type and attributes to a destination; Eventarc Advanced adds a central bus, enrollments, transformation and cross-project delivery for complex architectures.

Question 11 · choose 1

A load job failed overnight, and an analyst wants to read the individual log entries around the failure and troubleshoot what happened. Which Cloud Logging page does Google recommend for this?

  1. AObservability Analytics
  2. BLogs Explorer in Cloud Logging
  3. CMetrics Explorer in Cloud Monitoring
  4. DThe Cloud Logging dashboard
Show the answer and why
  • AObservability Analytics

    Incorrect

    Observability Analytics lets you query log data with SQL to find trends and patterns, such as average latency over time, rather than read individual entries.

  • BLogs Explorer in Cloud Logging

    Correct

    Logs Explorer lets you view individual log entries, and Google recommends it for troubleshooting services and applications.

  • CMetrics Explorer in Cloud Monitoring

    Incorrect

    Metrics Explorer charts metric data over time; it does not show the text of individual log entries.

  • DThe Cloud Logging dashboard

    Incorrect

    The preconfigured dashboard shows error rates of Google Cloud services, not the individual entries of one failed job.

Cloud Logging offers Logs Explorer for viewing and troubleshooting individual log entries and Observability Analytics for SQL queries over log data that reveal trends. The Logging dashboard summarizes error rates.

Question 12 · choose 1

A team set up event-driven ingestion: an app publishes order events to a Pub/Sub topic, and a BigQuery subscription with the default service account settings should write each message to an existing BigQuery table. No rows arrive, and the errors show that the writes to the table are denied. Which grant fixes this?

  1. ABigQuery Data Editor on the table for the app's publishing service account
  2. BBigQuery Data Editor on the table for the Pub/Sub service agent
  3. CBigQuery Data Editor on the table for the user who created the subscription
  4. DPub/Sub Editor on the project for the Pub/Sub service agent
Show the answer and why
  • ABigQuery Data Editor on the table for the app's publishing service account

    Incorrect

    The app only publishes to the topic; the rows are written to BigQuery by the subscription's service account, not by the publisher.

  • BBigQuery Data Editor on the table for the Pub/Sub service agent

    Correct

    By default a BigQuery subscription writes as the Pub/Sub service agent, and to write to a BigQuery table that service account needs the BigQuery Data Editor role on the project or the table.

  • CBigQuery Data Editor on the table for the user who created the subscription

    Incorrect

    The creator needs Pub/Sub permissions to create the subscription, but the writes themselves are made by the service account the subscription uses.

  • DPub/Sub Editor on the project for the Pub/Sub service agent

    Incorrect

    Pub/Sub Editor contains the permissions to create subscriptions; it grants no permission to write data into a BigQuery table.

A BigQuery subscription writes messages to an existing BigQuery table as they arrive, with no subscriber code. Pub/Sub makes those writes with a service account, by default the Pub/Sub service agent, which needs BigQuery Data Editor on the project or the table; a user-managed service account can be configured instead for finer control.

Practice domain 3 →Practice all domains →