Task 2.3: Deploying and operationalizing the pipelines.

• AA classic template with each runtime parameter wrapped in a ValueProvider

  Incorrect

  A classic template stores one serialized, static job graph, so it cannot choose a different source at launch.

• BA Flex Template whose container image includes the custom packages

  Correct

  A Flex Template is packaged as a container image and builds the job graph at launch from the runtime parameters, so it can pick its I/O, and the image carries the packages.

• CA Google-provided template with a JavaScript UDF that selects the source

  Incorrect

  Google-provided templates have fixed sources and sinks; a UDF transforms elements, it does not change what the pipeline reads from.

• DA Managed Airflow DAG that runs the pipeline code in a Python operator

  Incorrect

  Running the code from Airflow needs an environment and still has to package the dependencies; it is not a template that operations can launch in Dataflow.

Classic templates contain a serialized job graph and need ValueProvider for runtime parameters. Flex Templates package the pipeline as a Docker image in Artifact Registry with a template spec in Cloud Storage, and the graph is built when the job is launched, so parameters can change the graph. Google recommends Flex Templates for new templates.

• ADrain the job, then start the new code as a new job on the same subscription

  Incorrect

  Draining closes open windows early, fires their triggers and stops the job; the new job does not inherit that state, so the aggregations restart from nothing.

• BCancel the job, then launch the new version under the same job name

  Incorrect

  Cancelling stops processing at once and drops buffered data and state.

• CUpdate the job in place and let Dataflow match the renamed step itself

  Incorrect

  Dataflow matches steps by name; without a mapping for the renamed transform, the compatibility check fails and the update is rejected.

• DLaunch a replacement job with --update and a transform name mapping

  Correct

  An update replaces the job while preserving its intermediate state and in-flight data, and the name mapping tells Dataflow that ParseV2 is the old Parse step.

Updating a streaming job launches a replacement with the same name; if the compatibility check passes, the old job stops and its state and buffered records carry over. Renamed transforms need --transformNameMapping (Java) or --transform_name_mapping (Python and Go). Drain and cancel both end the job, so the next job starts without its state.

• ARun the vendor's image as a task with the KubernetesPodOperator

  Correct

  KubernetesPodOperator runs the task in a pod from any image, which Google recommends for binary dependencies that the worker image does not have.

• BAdd the tool to the environment's custom PyPI packages list

  Incorrect

  PyPI packages install Python libraries into the environment; they do not bring the system libraries that a compiled binary needs.

• CUse a BashOperator that downloads and runs the binary each time

  Incorrect

  The binary would still run on the worker image, which lacks its system libraries, and every run would download it again.

• DUse a PythonVirtualenvOperator that installs the tool's wrapper

  Incorrect

  A virtualenv isolates Python packages only; the missing system libraries are still missing.

Python dependencies belong in the environment's PyPI packages, but custom binaries and system libraries that the stock worker image lacks are a case for the KubernetesPodOperator, which runs the task in its own pod from a container image that already holds them.

• AA Managed Airflow DAG triggered once for each order

  Incorrect

  Managed Airflow is built for data pipelines as DAGs, starts tasks in seconds and bills for provisioned capacity, a poor fit for many short service calls.

• BA Cloud Scheduler job that runs the three calls each minute

  Incorrect

  A schedule runs on the clock, not when an order is placed, and has no steps or retries across calls.

• CA Dataflow streaming job that calls the services per element

  Incorrect

  Dataflow processes data streams; using it to sequence API calls with retries means writing and running a pipeline for an orchestration task.

• DA workflow in Workflows that calls each service in turn

  Correct

  Workflows orchestrates HTTP-based services with retries and error handling, starts in milliseconds, scales to zero and bills per step executed.

Google's guidance separates the two orchestrators: Workflows for service orchestration of HTTP and API calls (serverless, millisecond latency, usage-based billing), and Managed Airflow for data-driven ETL and ELT workflows written as Python DAGs, with backfills and reruns.

• AA Managed Airflow environment with a DAG that runs the query

  Incorrect

  An environment adds provisioned cost and upkeep for a single statement with no dependencies.

• BA BigQuery scheduled query with a destination table

  Correct

  Scheduled queries run a query on a recurring schedule in BigQuery itself, with nothing else to deploy.

• CA Dataflow batch job launched from a template each night

  Incorrect

  Dataflow would read the data out of BigQuery and write it back, which is more moving parts than one SQL statement needs.

• DA Cloud Run job that calls the BigQuery API on a schedule

  Incorrect

  A custom job works but is code to build and maintain for what scheduled queries do natively.

Orchestration should fit the job. A single recurring statement is what BigQuery scheduled queries are for; Dataform or Managed Airflow become worth it when there are many dependent steps, tests or cross-system dependencies.

• ATest each release by running it on production input data in production

  Incorrect

  Testing in production risks production data; end-to-end tests belong in a preproduction environment.

• BWrite unit tests with TestPipeline and PAssert run on the Direct Runner

  Correct

  Unit tests check DoFns and composite transforms with TestPipeline and PAssert on the Direct Runner, using small test data in local memory.

• CSkip unit tests and rely on Dataflow job metrics after each deploy

  Incorrect

  Metrics after deployment find problems late, after bad output has been written.

• DRun each unit test as a separate Dataflow job on the Dataflow Runner

  Incorrect

  Unit tests are meant to be fast and local; launching jobs for them is slow and costly.

• ERun end-to-end tests on the Dataflow Runner in a preproduction project

  Correct

  End-to-end tests run the full pipeline on the Dataflow Runner in a preproduction environment that closely resembles production.

Google's testing guidance for Dataflow layers the tests: unit tests of transforms with TestPipeline and PAssert on the Direct Runner, integration tests of the whole pipeline, and end-to-end tests on the Dataflow Runner in preproduction, with artifacts promoted to production only when those pass.

• ATerraform configurations for the environments, applied per stage

  Correct

  Managed Airflow environments can be created and changed with Terraform, so the same reviewed code produces each stage's environment.

• BEnvironment snapshots copied from production into the other stages

  Incorrect

  Snapshots save an environment's state for recovery or migration; they are not reviewed definitions of configuration.

• CA shared document listing each setting to apply by hand in the console

  Incorrect

  Manual steps are how the environments drifted in the first place.

• DOne shared environment for the three stages with separate DAG folders per stage

  Incorrect

  Sharing one environment removes the isolation between test and production rather than making the stages consistent.

Infrastructure as code turns environment configuration into reviewed, versioned text. Google documents creating Managed Airflow environments with Terraform, which also covers related resources such as service accounts and buckets.