Skip to content

Professional Cloud DevOps Engineer · Domain 3: Applying site reliability engineering practices · 18% of the exam

Task 3.1: Balancing change, velocity, and reliability of the service.

Defining SLIs, SLOs and SLAs, error budgets in Cloud Service Mesh, Cloud Run and Cloud Monitoring, and the cost of each extra nine.

Study it

  • SLIs, SLOs, SLAs, error budgets and the cost of nines

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Google Cloud documentation page that proves it.

Question 1 · choose 1

A legacy service exports only one latency metric to Cloud Monitoring: its 95th percentile response time, computed every minute. There is no count of requests and no latency distribution. The team wants an SLO that says the service is fast enough 99% of the time over 28 days. What kind of SLO can it build on this metric?

  1. AA windows-based SLO that counts the minutes in which the 95th percentile stays under the threshold
  2. BA request-based SLO that divides the requests faster than the threshold by all requests
  3. CA request-based SLO over a calendar month, which estimates the month's total traffic from the rate seen so far
  4. DAn availability SLO built from an uptime check instead of the percentile metric
Show the answer and why
  • AA windows-based SLO that counts the minutes in which the 95th percentile stays under the threshold

    Correct

    When the raw metric is a latency percentile, the SLO must be windows-based: each measurement interval is good or bad, and compliance is the share of good intervals.

  • BA request-based SLO that divides the requests faster than the threshold by all requests

    Incorrect

    A request-based SLI needs counts of good and total requests, which this service does not export.

  • CA request-based SLO over a calendar month, which estimates the month's total traffic from the rate seen so far

    Incorrect

    The calendar period changes how the budget is projected, but the SLO is still request-based and needs request counts the metric lacks.

  • DAn availability SLO built from an uptime check instead of the percentile metric

    Incorrect

    That measures whether the endpoint answers, not whether responses are fast enough, so it is a different SLI.

Cloud Monitoring supports request-based SLOs (good requests divided by total requests) and windows-based SLOs (good measurement intervals divided by all intervals). A percentile metric holds no request counts, so only a windows-based SLO can use it, for example "p95 latency under 300 ms in at least 99% of one-minute windows".

Question 2 · choose 2

An SRE team is choosing service level indicators for a checkout API that customers call from a mobile app. It wants indicators that move when customers are affected, stay flat when they are not, and read as a percentage so that an SLO target such as 99.9% is easy to state. Which TWO indicators fit best? (Choose TWO.)

  1. AThe share of checkout requests that receive a response without a server error
  2. BThe average CPU utilization of the Pods that serve the checkout API
  3. CThe share of checkout requests answered in under 400 ms
  4. DThe number of ERROR lines that the checkout service writes to its logs per minute
  5. EThe number of checkout Pods that report Ready in the GKE cluster at each minute
Show the answer and why
  • AThe share of checkout requests that receive a response without a server error

    Correct

    Counting kept promises (a successful response) over all requests is a proxy for user happiness that ranges from 0% to 100%.

  • BThe average CPU utilization of the Pods that serve the checkout API

    Incorrect

    CPU load can change while customers are unaffected and stay normal while they fail, so it is a poor proxy for user happiness.

  • CThe share of checkout requests answered in under 400 ms

    Correct

    A latency promise counted per request is a 0% to 100% measure that changes when customers wait too long.

  • DThe number of ERROR lines that the checkout service writes to its logs per minute

    Incorrect

    A raw count is not a 0% to 100% measure, and log volume does not track how many customers were affected.

  • EThe number of checkout Pods that report Ready in the GKE cluster at each minute

    Incorrect

    Ready Pods describe the platform's state, not what customers experience, so the indicator can look healthy during a customer-facing failure.

Good SLIs are proxies for user happiness, scale monotonically with it, and range from 0% to 100%. Counting promises kept, such as successful responses or responses under a latency threshold, over all requests gives SLIs with these properties; resource and platform metrics do not.

Google Cloud documentation

Question 3 · choose 1

A service has a windows-based availability SLO over a rolling 30 days, with one-minute windows and a goal of 99.9%. Product managers ask to raise the goal to 99.95%, and the SRE lead wants to show what that does to the time the team can spend on risky changes and failures. How many bad minutes does each goal allow in 30 days?

  1. AAbout 4.3 minutes at 99.9%, and about 2.2 minutes at 99.95%
  2. BAbout 43.2 minutes at 99.9%, and about 4.3 minutes at 99.95%
  3. CAbout 432 minutes at 99.9%, and about 216 minutes at 99.95%
  4. DAbout 43.2 minutes at 99.9%, and about 21.6 minutes at 99.95%
Show the answer and why
  • AAbout 4.3 minutes at 99.9%, and about 2.2 minutes at 99.95%

    Incorrect

    These values are a tenth of the real budgets: 30 days hold 43,200 one-minute windows, and 0.1% of them is 43.2.

  • BAbout 43.2 minutes at 99.9%, and about 4.3 minutes at 99.95%

    Incorrect

    4.3 minutes is the budget of a 99.99% goal; 99.95% leaves 0.05% of 43,200 minutes, about 21.6.

  • CAbout 432 minutes at 99.9%, and about 216 minutes at 99.95%

    Incorrect

    432 minutes is 1% of the windows, the budget of a 99% goal.

  • DAbout 43.2 minutes at 99.9%, and about 21.6 minutes at 99.95%

    Correct

    The error budget is 1 minus the goal: 0.1% and 0.05% of the 43,200 windows in 30 days.

An error budget is the difference between 100% good service and the SLO. With one-minute windows over 30 days there are 43,200 windows, so 99.9% allows about 43 bad minutes and 99.95% about 22. Moving from 99.9% to 99.95% halves the room for releases and incidents, which is the opportunity cost of a stricter target.

Question 4 · choose 1

A platform team offers GKE clusters to internal teams and wants an SLO on the share of Kubernetes API server requests that return errors, so that it can track an error budget for the control plane. The clusters use the default monitoring settings. What must the team do before it can build this SLI in Cloud Monitoring?

  1. ANothing, because the clusters' default system metrics in Cloud Monitoring already include API server request counts
  2. BEnable the control plane metrics package on the clusters, so that API server metrics reach Cloud Monitoring
  3. CEnable the kube state metrics package, which reports the API server's request errors along with object states
  4. DInstall the Ops Agent on the cluster nodes, so that it scrapes the control plane's API server metrics
Show the answer and why
  • ANothing, because the clusters' default system metrics in Cloud Monitoring already include API server request counts

    Incorrect

    Default system metrics describe low-level resources such as CPU, memory and storage; API server metrics come from the separate control plane metrics package.

  • BEnable the control plane metrics package on the clusters, so that API server metrics reach Cloud Monitoring

    Correct

    Control plane metrics, such as those of the API server and scheduler, must be enabled before they can be used for custom SLIs; Managed Service for Prometheus collects them.

  • CEnable the kube state metrics package, which reports the API server's request errors along with object states

    Incorrect

    Kube state metrics describe Kubernetes objects such as Pods and Deployments, not the requests the API server handles.

  • DInstall the Ops Agent on the cluster nodes, so that it scrapes the control plane's API server metrics

    Incorrect

    The API server is part of the GKE control plane, not a process on the nodes; its metrics come through the control plane metrics package.

GKE sends system metrics by default. Packages of observability metrics, such as control plane metrics (API server, scheduler, controller manager), kube state metrics and cAdvisor/Kubelet metrics, are configured separately. API server metrics make useful custom SLIs for request errors and latency once their collection is enabled.

Question 5 · choose 1

A GKE-based service is defined as a service in Cloud Monitoring. Managed Service for Prometheus collects its counter http_requests_total, which has a code label holding the HTTP status. The team wants a request-based availability SLI equal to the share of requests that did not return a 5xx status. How should it build the SLI in the console?

  1. AChoose Other, then a time-series ratio indicator: numerator filter that excludes 5xx codes, denominator of all requests
  2. BChoose the predefined Availability metric, which reads the request counts of every GKE service without extra configuration
  3. CChoose Other, then a distribution-cut indicator on the counter with a range below 500, so that 5xx responses fall outside it
  4. DChoose a windows-based SLI on the Pods' CPU usage, with each minute below 80% counted as a good window
Show the answer and why
  • AChoose Other, then a time-series ratio indicator: numerator filter that excludes 5xx codes, denominator of all requests

    Correct

    For GKE-based services the only metric choice is Other; a time-series ratio indicator is built from numerator and denominator filters on metric labels, which fits a counter split by status code.

  • BChoose the predefined Availability metric, which reads the request counts of every GKE service without extra configuration

    Incorrect

    Predefined availability and latency metrics exist for Cloud Service Mesh, Istio on GKE and App Engine services; Prometheus metrics are not part of the default SLOs.

  • CChoose Other, then a distribution-cut indicator on the counter with a range below 500, so that 5xx responses fall outside it

    Incorrect

    Distribution-cut indicators take a distribution metric and a value range; a request counter labeled by status needs label filters in a ratio.

  • DChoose a windows-based SLI on the Pods' CPU usage, with each minute below 80% counted as a good window

    Incorrect

    CPU usage says nothing about whether requests succeeded, and the team asked for a request-based SLI on status codes.

When you create an SLO for GKE-based or custom services, the only SLI metric choice is Other. A collected Prometheus metric can be the custom SLI: use a time-series ratio indicator with numerator and denominator filters on labels, or a distribution-cut indicator with a range for distribution metrics such as latency histograms.

Google Cloud documentation

Question 6 · choose 1

A team wants to track the error budget of its Cloud Run service checkout-api in Cloud Monitoring. A week after the service was deployed, it still does not appear on the Services overview page, while the company's App Engine services appear there automatically with their dashboards. What should the team do?

  1. AWait for automatic detection, which adds Cloud Run services as soon as they have served traffic for a full compliance period
  2. BEnable Cloud Service Mesh for the Cloud Run service, because only mesh services can have SLOs in Cloud Monitoring
  3. CCreate an uptime check for the service URL, because Monitoring builds services and SLOs only from uptime check results
  4. DSelect checkout-api from the candidate services list to define it as a Monitoring service, then create an SLO on it
Show the answer and why
  • AWait for automatic detection, which adds Cloud Run services as soon as they have served traffic for a full compliance period

    Incorrect

    Cloud Monitoring auto-detects App Engine and Cloud Service Mesh services; Cloud Run services are not added on their own.

  • BEnable Cloud Service Mesh for the Cloud Run service, because only mesh services can have SLOs in Cloud Monitoring

    Incorrect

    Cloud Run services can be Monitoring services without a mesh; SLOs are not limited to mesh services.

  • CCreate an uptime check for the service URL, because Monitoring builds services and SLOs only from uptime check results

    Incorrect

    Uptime checks probe availability from outside; they are not how a Monitoring service is defined for SLOs.

  • DSelect checkout-api from the candidate services list to define it as a Monitoring service, then create an SLO on it

    Correct

    Monitoring lists Cloud Run services as candidates; once the team selects one, Monitoring creates the service so that SLOs and their error budgets can be attached.

Cloud Monitoring automatically detects App Engine services and Cloud Service Mesh services. For GKE namespaces, services and workloads, and for Cloud Run services, it builds a list of candidate services; you pick the ones you want, Monitoring creates the service, and you then define SLOs and alerts on it.

Question 7 · choose 1

An availability SLI for a web shop counts successful queries in the order database. During last week's outage, a misconfigured frontend returned errors to most shoppers for an hour, yet the SLI stayed at 100% because few requests reached the database. Collecting data from shoppers' browsers is not feasible yet. Where should the team measure the SLI instead?

  1. AIn the order database, with a stricter target so that smaller drops in query success are noticed
  2. BIn the backend order service, counting the requests that it completes without an exception
  3. CAt the external load balancer in front of the shop, as the success ratio of shoppers' requests
  4. DIn the GKE cluster, as the share of order service Pods that pass their readiness checks
Show the answer and why
  • AIn the order database, with a stricter target so that smaller drops in query success are noticed

    Incorrect

    The database never saw the failed requests, so a stricter target there would still miss the outage.

  • BIn the backend order service, counting the requests that it completes without an exception

    Incorrect

    The backend sits behind the frontend that failed, so its success ratio would also look healthy during the outage.

  • CAt the external load balancer in front of the shop, as the success ratio of shoppers' requests

    Correct

    When client-side data is not feasible, move the measurement point as close to users as possible, such as the load balancer or frontend.

  • DIn the GKE cluster, as the share of order service Pods that pass their readiness checks

    Incorrect

    Pod readiness describes internal system state, not whether shoppers' requests succeeded.

Reliability should be measured as users experience it, for example as the success ratio of user requests. Ideally the data comes from users' devices or browsers; when that is not feasible, measure progressively further from the user, such as at the load balancer or frontend service, rather than deep in the backend.

Practice domain 3 →Practice all domains →