Skip to content

Professional Cloud DevOps Engineer · Domain 3: Applying site reliability engineering practices · 18% of the exam

Task 3.3: Mitigating incident impact on users.

Draining or redirecting traffic, adding capacity and rolling back during an incident.

Study it

  • Mitigating incidents: draining traffic, adding capacity and rolling back

    Partly covered by: Load Balancing

Sample questions

Try each one before opening the answer. Every option is explained, with the Google Cloud documentation page that proves it.

Question 1 · choose 1

A file-upload service runs on a managed instance group behind an external Application Load Balancer. Uploads can take up to three minutes. Whenever the autoscaler removes VMs or a rolling update replaces them, users whose uploads were in progress on those VMs see their connections reset. What should the team configure?

  1. ASession affinity on the backend service, so each user's requests keep going to the same VM
  2. BA connection draining timeout of about 300 seconds on the backend service
  3. CA maximum unavailable value of 0 in the group's rolling update settings
  4. DA higher unhealthy threshold on the load balancer's health check
Show the answer and why
  • ASession affinity on the backend service, so each user's requests keep going to the same VM

    Incorrect

    Affinity decides where new requests go; it does not give in-progress requests time to finish when their VM is removed.

  • BA connection draining timeout of about 300 seconds on the backend service

    Correct

    Connection draining gives existing, in-progress requests time to complete when a VM is removed from the instance group, including removals by the autoscaler and the MIG updater.

  • CA maximum unavailable value of 0 in the group's rolling update settings

    Incorrect

    This keeps serving capacity during updates, but old VMs are still removed when replaced, and it does nothing for scale-in by the autoscaler.

  • DA higher unhealthy threshold on the load balancer's health check

    Incorrect

    The health check decides when a backend is considered unhealthy; it does not delay removal of VMs that the autoscaler or updater deletes.

When a VM leaves a backend instance group, through resize, deletion, autoscaling or the MIG updater, connection draining lets its in-flight requests finish for up to the configured timeout (0 to 3,600 seconds).

Question 2 · choose 1

A global external Application Load Balancer sends traffic to two managed instance groups, one in us-east1 and one in europe-west1. A bad configuration push has made the europe-west1 VMs return errors. The on-call engineer wants new requests to stop reaching that group at once, wants its VMs left running for debugging, and wants to bring the group back later in steps of 10% of its capacity. What should the engineer do?

  1. ARemove the europe-west1 instance group from the backend service, and add it back once it is fixed
  2. BResize the europe-west1 managed instance group to zero VMs, then grow it back in steps
  3. CLower the target capacity (maximum rate per instance) of the europe-west1 backend
  4. DSet the capacity scaler of the europe-west1 backend to 0, then raise it in steps of 0.1
Show the answer and why
  • ARemove the europe-west1 instance group from the backend service, and add it back once it is fixed

    Incorrect

    Removing the group stops traffic, but bringing it back is all or nothing, and connection draining does not apply when a backend group is removed.

  • BResize the europe-west1 managed instance group to zero VMs, then grow it back in steps

    Incorrect

    Traffic stops, but resizing the group down removes the VMs the engineer wants to debug.

  • CLower the target capacity (maximum rate per instance) of the europe-west1 backend

    Incorrect

    Target capacity shapes how traffic spreads once backends fill, so the group still receives requests and keeps failing them.

  • DSet the capacity scaler of the europe-west1 backend to 0, then raise it in steps of 0.1

    Correct

    A capacity scaler of 0 drains the backend completely while it stays attached, and values from 0.1 to 1.0 bring its capacity back gradually.

The capacity scaler on a backend of a backend service scales its target capacity by hand: 0 drains the backend (allowed when the service has more than one backend), and 0.1 to 1.0 serve a share of its capacity. It is the incident tool for redirecting traffic away from one region without deleting or detaching anything.

Question 3 · choose 2

During a flash sale, a checkout Deployment on GKE Standard is slow. Its horizontal Pod autoscaler targets 60% CPU with 5 to 20 replicas and already runs 20 Pods at about 95% CPU. The node pool's cluster autoscaler is at its maximum of 10 nodes, and those nodes have no room for more checkout Pods. Which TWO changes together let the service add capacity now? (Choose TWO.)

  1. ARaise the autoscaler's CPU target to 90% so each Pod is allowed to work harder before scaling
  2. BRaise maxReplicas on the horizontal Pod autoscaler above 20
  3. CAdd a vertical Pod autoscaler in Auto mode on CPU and memory next to the horizontal one
  4. DAdd a PodDisruptionBudget for the checkout Pods with minAvailable set to 20
  5. ERaise the maximum size of the node pool in its cluster autoscaler settings
Show the answer and why
  • ARaise the autoscaler's CPU target to 90% so each Pod is allowed to work harder before scaling

    Incorrect

    A higher target makes the autoscaler want fewer replicas for the same load, the opposite of adding capacity.

  • BRaise maxReplicas on the horizontal Pod autoscaler above 20

    Correct

    The autoscaler cannot scale the Deployment beyond its maximum number of replicas, which it has reached.

  • CAdd a vertical Pod autoscaler in Auto mode on CPU and memory next to the horizontal one

    Incorrect

    Google advises against combining the standard vertical autoscaler with the horizontal one on CPU or memory; they would act on the same signal.

  • DAdd a PodDisruptionBudget for the checkout Pods with minAvailable set to 20

    Incorrect

    A disruption budget limits voluntary evictions such as node upgrades; it adds neither Pods nor nodes.

  • ERaise the maximum size of the node pool in its cluster autoscaler settings

    Correct

    The cluster autoscaler adds nodes for Pods that cannot be scheduled only up to the node pool's maximum size, which it has reached.

Two ceilings are reached at once: the horizontal Pod autoscaler's maximum replicas and the node pool's maximum size. New replicas need both raised: the HPA can then create Pods, and the cluster autoscaler can add nodes for the Pods that would otherwise stay pending.

Question 4 · choose 1

One VM in an autoscaled managed instance group behind an external Application Load Balancer has started returning corrupted responses, and users who land on it see errors. Engineers want that VM out of the load balancer at once, but kept running with its memory and disks intact so that they can investigate it. What should they do?

  1. AAbandon the VM from the managed instance group, which takes it out of the group's load balancers and leaves it running
  2. BDelete the VM from the group, so that the MIG replaces it with a fresh VM built from the instance template
  3. CRecreate the VM in the group, so that it restarts from the template and comes back into service in a healthy state
  4. DResize the group down by one VM, so that the load balancer has one fewer backend serving the faulty responses
Show the answer and why
  • AAbandon the VM from the managed instance group, which takes it out of the group's load balancers and leaves it running

    Correct

    Abandoning separates a VM from its MIG for debugging and removes it from the load balancers assigned to the group; the target size drops and the autoscaler keeps managing capacity.

  • BDelete the VM from the group, so that the MIG replaces it with a fresh VM built from the instance template

    Incorrect

    Deleting the VM ends the problem for users but destroys the running state the engineers want to investigate.

  • CRecreate the VM in the group, so that it restarts from the template and comes back into service in a healthy state

    Incorrect

    Recreation rebuilds the VM from the group's configuration, which also discards the memory state and possibly the disks under investigation.

  • DResize the group down by one VM, so that the load balancer has one fewer backend serving the faulty responses

    Incorrect

    When a MIG scales in, it chooses which VMs to delete, so a healthy VM may go while the faulty one keeps serving users.

Abandoning a managed instance separates it from the group so that you can debug it without affecting the group as a whole, and also removes it from the load balancers assigned to the group. The group's target size decreases, autoscaling continues, and an abandoned VM cannot be added back to the group.

Question 5 · choose 1

A product catalog API sits behind Cloud CDN with responses cached for one hour. When the origin failed for a whole day last month, users got errors once the default stale window ran out. For future origin outages, the team wants cached responses served for up to two days after they expire, while Cloud CDN keeps trying to revalidate. What should it configure?

  1. AHave the origin add Cache-Control: stale-if-error=172800 to its responses, so that errors trigger stale content
  2. BSet the serve-while-stale option on the backend's Cloud CDN policy to 172,800 seconds, which is two days
  3. CSwitch to the FORCE_CACHE_ALL cache mode and add stale-while-revalidate to the origin's responses
  4. DRaise the responses' max-age to two days, so that cached objects never expire during an origin outage
Show the answer and why
  • AHave the origin add Cache-Control: stale-if-error=172800 to its responses, so that errors trigger stale content

    Incorrect

    Cloud CDN does not support the stale-if-error directive.

  • BSet the serve-while-stale option on the backend's Cloud CDN policy to 172,800 seconds, which is two days

    Correct

    serve-while-stale sets how long past expiry Cloud CDN may keep serving a stale object while it revalidates; it can be raised from the one-day default up to one week.

  • CSwitch to the FORCE_CACHE_ALL cache mode and add stale-while-revalidate to the origin's responses

    Incorrect

    In FORCE_CACHE_ALL mode, Cloud CDN ignores the stale-while-revalidate directive from the origin.

  • DRaise the responses' max-age to two days, so that cached objects never expire during an origin outage

    Incorrect

    A longer max-age also delays fresh catalog data for two days in normal operation, which is not what the team asked for.

Cloud CDN can serve stale, expired content when the origin is unreachable or returns errors. The origin can send stale-while-revalidate, or the serve-while-stale policy setting caps how long stale content is served: default one day, maximum one week, 0 to disable. Cloud CDN does not support stale-if-error, and FORCE_CACHE_ALL ignores stale-while-revalidate.

Question 6 · choose 1

An internal API name in a Cloud DNS private zone uses a weighted round robin policy that sends traffic to internal load balancers in two regions, with weights of 50 and 50. Region B is returning slow, partial results, yet its backends still pass health checks. The team wants all new lookups sent to region A now, then to bring region B back step by step after the fix. What should it change?

  1. ANothing in DNS, because Cloud DNS health checks will soon stop returning region B's address on their own
  2. BDelete region B's entry from the policy, and add it back with weight 50 once the fix is deployed
  3. CChange the record to a geolocation policy, so that clients are routed to the region that is closest to them
  4. DSet region B's weight to 0 and region A's to 50, then raise B's weight in small steps after the fix
Show the answer and why
  • ANothing in DNS, because Cloud DNS health checks will soon stop returning region B's address on their own

    Incorrect

    Health checks remove only unhealthy targets; region B still passes its checks, so it keeps receiving its share.

  • BDelete region B's entry from the policy, and add it back with weight 50 once the fix is deployed

    Incorrect

    Adding the entry back at 50 sends half the traffic at once, which is not the gradual return the team wants.

  • CChange the record to a geolocation policy, so that clients are routed to the region that is closest to them

    Incorrect

    Geolocation routes by the client's location, so clients near region B would still be sent there.

  • DSet region B's weight to 0 and region A's to 50, then raise B's weight in small steps after the fix

    Correct

    Weights from 0.0 to 1000.0 set each target's share as its weight over the total, so a weight of 0 sends region B nothing and small increases bring it back gradually.

A Cloud DNS weighted round robin policy routes to each target in proportion to its weight divided by the total of all weights, with weights from 0.0 to 1000.0. Changing weights lets operators drain a target and return traffic in steps; health-checked targets are removed only when they fail their health checks.

Question 7 · choose 1

During a marketing launch, a Cloud Run service starts returning HTTP 429 with "The request was aborted because there was no available instance". The Container instance count metric sits flat at the service's configured limit, and the database behind the service has plenty of spare connection capacity. What should the on-call engineer do first to restore service?

  1. ALower the service's maximum concurrent requests per instance, so that each instance handles fewer requests
  2. BAdd exponential backoff and retries in the clients, and wait for the launch traffic to fall off
  3. CRaise the service's maximum number of instances, requesting more quota if the new limit needs it
  4. DRoll back to the previous revision, because a code change must be making the instances refuse requests
Show the answer and why
  • ALower the service's maximum concurrent requests per instance, so that each instance handles fewer requests

    Incorrect

    With the instance count already at its cap, fewer requests per instance means less total capacity and more rejected requests.

  • BAdd exponential backoff and retries in the clients, and wait for the launch traffic to fall off

    Incorrect

    Retries help with short transient errors, but here the service is held at a fixed limit, so customers would keep failing during the launch.

  • CRaise the service's maximum number of instances, requesting more quota if the new limit needs it

    Correct

    HTTP 429 with no available instance means the service reached its maximum instance limit; raising that limit, with a quota increase if needed, adds capacity.

  • DRoll back to the previous revision, because a code change must be making the instances refuse requests

    Incorrect

    The symptoms point to the instance limit, not to a faulty revision; the previous revision would hit the same cap.

Cloud Run returns HTTP 429 "no available instance" when a service has reached its maximum container instance limit, for example after a sudden traffic increase. Check the Container instance count metric and raise the maximum instances setting, requesting a quota increase if more instances are needed. HTTP 500 with the same message points to scaling that could not keep up instead.

Practice domain 3 →Practice all domains →