Skip to content

Professional Cloud DevOps Engineer · Domain 4: Implementing observability practices and troubleshooting issues · 22% of the exam

Task 4.4: Troubleshooting issues.

Diagnosing infrastructure, CI/CD pipeline, application, observability, and performance and latency problems.

Study it

  • Troubleshooting infrastructure, pipeline, application and latency issues

    Partly covered by: Tracing a Request

Sample questions

Try each one before opening the answer. Every option is explained, with the Google Cloud documentation page that proves it.

Question 1 · choose 1

Builds in project app-ci run as a user-managed service account, builder@app-ci.iam.gserviceaccount.com. A new step stores the built image through the images field in an Artifact Registry Docker repository in another project, shared-artifacts, and the build now fails with a permission denied error on uploading artifacts. Pulling base images from that repository works. What should the DevOps engineer grant?

  1. AArtifact Registry Writer on the shared-artifacts repository to the legacy Cloud Build service account of app-ci
  2. BArtifact Registry Reader on the shared-artifacts project to builder@app-ci.iam.gserviceaccount.com
  3. CArtifact Registry Writer on the shared-artifacts repository to builder@app-ci.iam.gserviceaccount.com
  4. DArtifact Registry Repository Administrator on the app-ci project to builder@app-ci.iam.gserviceaccount.com
Show the answer and why
  • AArtifact Registry Writer on the shared-artifacts repository to the legacy Cloud Build service account of app-ci

    Incorrect

    The build runs as the user-managed service account, so roles on the legacy Cloud Build service account are not used by this build.

  • BArtifact Registry Reader on the shared-artifacts project to builder@app-ci.iam.gserviceaccount.com

    Incorrect

    Reader only views and gets artifacts, which is why pulls already work; an upload needs write access.

  • CArtifact Registry Writer on the shared-artifacts repository to builder@app-ci.iam.gserviceaccount.com

    Correct

    Writer reads and writes artifacts; across projects, or with a user-provided service account, the permission must be granted to the identity the build actually uses.

  • DArtifact Registry Repository Administrator on the app-ci project to builder@app-ci.iam.gserviceaccount.com

    Incorrect

    A role in app-ci grants nothing on a repository in shared-artifacts, and repository administrator would also allow deleting artifacts.

Default service accounts get automatic access only within their own project. When Artifact Registry is in another project, or the build uses a user-provided service account, the engineer must grant the role to that account; Artifact Registry Writer on the target repository is the least privilege that allows pushes.

Question 2 · choose 1

A Java service on Cloud Run keeps 3 minimum instances for its baseline traffic. During unpredictable bursts, its 99th percentile latency jumps; traces show the slow requests waiting for new instances, whose startup takes about 25 seconds, most of it CPU-bound class loading. Finance will not pay for many more idle instances. Which change best reduces the latency of the bursts?

  1. ARaise the minimum number of instances to the size of the largest burst seen last month
  2. BLower the maximum concurrent requests per instance to 1 so requests never share an instance
  3. CRaise the service's request timeout so that requests waiting on instance startup are not cut off
  4. DTurn on startup CPU boost for the service's revisions
Show the answer and why
  • ARaise the minimum number of instances to the size of the largest burst seen last month

    Incorrect

    Idle minimum instances keep costing money, which finance rejects, and a burst larger than last month's still waits for cold starts.

  • BLower the maximum concurrent requests per instance to 1 so requests never share an instance

    Incorrect

    One request per instance makes Cloud Run start even more instances during a burst, so more requests wait for slow startups.

  • CRaise the service's request timeout so that requests waiting on instance startup are not cut off

    Incorrect

    A longer timeout avoids errors but leaves the requests waiting just as long, so latency does not improve.

  • DTurn on startup CPU boost for the service's revisions

    Correct

    Startup CPU boost temporarily gives instances more CPU while they start, which shortens CPU-bound startup, and is charged only for the startup time.

Cold starts that are CPU-bound shrink when an instance gets more CPU during startup. Startup CPU boost provides that for each new instance, during bursts as well as at the baseline, without paying for idle instances the way a higher minimum does.

Question 3 · choose 1

A nightly integration build now takes about 80 minutes. Its cloudbuild.yaml contains: steps: - name: 'gcr.io/cloud-builders/docker' args: ['build', '-t', '${_IMAGE}', '.'] - name: '${_IMAGE}' entrypoint: './run-integration-tests.sh' timeout: 5400s The build is stopped with status TIMEOUT after exactly 60 minutes, while the test step is still running. What should the team change?

  1. AAdd a top-level timeout of at least 5400s, because the build's own timeout defaults to 60 minutes otherwise
  2. BRaise the test step's timeout to 7200s, so that it has more room than the 80 minutes the tests need
  3. CSplit the tests into a second build triggered when the first one ends, because builds cannot run for more than an hour
  4. DMove the build to a larger machine type, because TIMEOUT means the worker ran out of CPU for the test step
Show the answer and why
  • AAdd a top-level timeout of at least 5400s, because the build's own timeout defaults to 60 minutes otherwise

    Correct

    Without a build-level timeout the build uses the 60-minute default, and a step's timeout cannot exceed the build's timeout, so the build stops first.

  • BRaise the test step's timeout to 7200s, so that it has more room than the 80 minutes the tests need

    Incorrect

    The step already allows 90 minutes; the build-level default of 60 minutes is what ends the build.

  • CSplit the tests into a second build triggered when the first one ends, because builds cannot run for more than an hour

    Incorrect

    A build's timeout can be set up to 24 hours, so one build can run the whole suite.

  • DMove the build to a larger machine type, because TIMEOUT means the worker ran out of CPU for the test step

    Incorrect

    TIMEOUT reports that a time limit was reached; it does not indicate a shortage of CPU on the worker.

Cloud Build has two timeout fields. A step's timeout limits that step and must not exceed the build's timeout. The build's timeout limits the whole build; if it isn't set, the build uses a default of 60 minutes, and it can be raised up to 24 hours.

Question 4 · choose 1

After a team added a relabeling rule to its Managed Service for Prometheus collection that renames http_server_requests_total to api_requests_total, plain PromQL queries of the new name show gaps, quantile functions on its histograms return odd results, and samples ingested have roughly doubled. Metric types with an "unknown:counter" suffix also appear in Cloud Monitoring. What is the cause?

  1. AThe renamed metric needs a new PodMonitoring resource, because each metric name must be listed in its own resource
  2. BThe scrape interval is too short for the new metric name, so Monarch drops samples and duplicates others
  3. CRenaming the metric separated it from its Prometheus TYPE, so it is ingested untyped, as both a gauge and a counter
  4. DThe collector's service account lacks Monitoring Metric Writer for the new metric, so only part of the data is written
Show the answer and why
  • AThe renamed metric needs a new PodMonitoring resource, because each metric name must be listed in its own resource

    Incorrect

    PodMonitoring selects targets to scrape; it does not list metric names, and the symptoms point to missing type information.

  • BThe scrape interval is too short for the new metric name, so Monarch drops samples and duplicates others

    Incorrect

    A scrape interval changes how often samples arrive; it does not create "unknown"-suffixed metric types.

  • CRenaming the metric separated it from its Prometheus TYPE, so it is ingested untyped, as both a gauge and a counter

    Correct

    A relabeling rule that modifies a metric name disassociates it from its TYPE; untyped metrics get "unknown" suffixes, are ingested twice, and break histogram functions.

  • DThe collector's service account lacks Monitoring Metric Writer for the new metric, so only part of the data is written

    Incorrect

    A permission problem stops writes; it does not explain doubled samples and "unknown"-suffixed metric types.

Monarch is strongly typed, so Managed Service for Prometheus ingests metrics without a Prometheus TYPE twice, as a gauge and as a counter, with "unknown" suffixes. Common causes are federation, Prometheus Remote Write, relabeling rules that change metric names, and exporters that omit TYPE. Remove the renaming rule, check that each metric has a # TYPE line, and delete the conflicting "unknown" metric descriptors.

Question 5 · choose 1

Cloud Deploy rollouts to a GKE target fail in the deploy job a few minutes after they start. The job logs show only short warning-level messages from kubectl, which do not explain the failure. The team wants debug-level output from Skaffold, kubectl and the gcloud CLI for this target's deploy jobs. What should it change?

  1. AEnable Data Access audit logs for the Cloud Deploy API, so that every call made during the deploy job is logged
  2. BSet verbose: true in the target definition's executionConfigs entry whose usages include DEPLOY
  3. CRaise the executionTimeout in the target's execution environment above the default of one hour for each job
  4. DAdd a verify job to the deploy phase, so that Cloud Deploy runs extra checks and logs them
Show the answer and why
  • AEnable Data Access audit logs for the Cloud Deploy API, so that every call made during the deploy job is logged

    Incorrect

    Audit logs record API calls to Cloud Deploy, not the detailed output of the tools that run inside the deploy job.

  • BSet verbose: true in the target definition's executionConfigs entry whose usages include DEPLOY

    Correct

    verbose: true raises the log level of Skaffold, the gcloud CLI and kubectl from their defaults, typically warn, to debug for that usage.

  • CRaise the executionTimeout in the target's execution environment above the default of one hour for each job

    Incorrect

    The jobs fail after a few minutes, long before the one-hour default timeout, so a longer timeout changes nothing.

  • DAdd a verify job to the deploy phase, so that Cloud Deploy runs extra checks and logs them

    Incorrect

    A verify job runs after the deployment and does not change the log level of the tools in the failing deploy job.

A Cloud Deploy execution environment sets the worker pool, service account, artifact storage, timeout (one hour by default) and log verbosity for render, predeploy, deploy, verify and postdeploy jobs. Setting verbose to true in an executionConfigs entry switches Skaffold, gcloud CLI and kubectl to debug logging for those usages.

Question 6 · choose 1

A Cloud Run service generates large PDF reports. Since a new report type was added, some requests fail with HTTP 504 after exactly five minutes, while CPU and memory stay well below their limits and the container keeps running. The longest reports take about twelve minutes. What should the team do?

  1. ARaise the service's maximum instances, because the 504 errors mean no instance was free to take the request
  2. BGive each instance more memory and CPU, so that the twelve-minute reports finish within five minutes
  3. CSet minimum instances to 1, so that a warm instance is always ready when a long report request arrives
  4. DRaise the service's request timeout above twelve minutes; the default 300 seconds can go up to 3,600
Show the answer and why
  • ARaise the service's maximum instances, because the 504 errors mean no instance was free to take the request

    Incorrect

    A shortage of instances does not make requests fail at one fixed age; a 504 after exactly five minutes points to a time limit.

  • BGive each instance more memory and CPU, so that the twelve-minute reports finish within five minutes

    Incorrect

    Resources are not the bottleneck here; the request hits a time limit, not a resource limit.

  • CSet minimum instances to 1, so that a warm instance is always ready when a long report request arrives

    Incorrect

    Minimum instances keep instances ready to serve; they do not change how long a single request may run.

  • DRaise the service's request timeout above twelve minutes; the default 300 seconds can go up to 3,600

    Correct

    Cloud Run returns HTTP 504 when a response takes longer than the request timeout, which defaults to 5 minutes and can be set up to 60 minutes.

Cloud Run closes the connection and returns HTTP 504 when a service doesn't respond within its request timeout, 5 minutes by default and up to 60 minutes. The container may keep processing the aborted request. For timeouts beyond 15 minutes, Google recommends retries and handlers that are idempotent or can resume.

Practice domain 4 →Practice all domains →