Skip to content

Professional Cloud DevOps Engineer · Domain 4: Implementing observability practices and troubleshooting issues · 22% of the exam

Task 4.1: Instrumenting and collecting telemetry.

Collecting logs and metrics with the Ops Agent, OpenTelemetry and Managed Service for Prometheus, optimizing log volume, synthetic monitors, log-based and custom metrics, and telemetry from services and agents.

Study it

  • Instrumenting: Ops Agent, OpenTelemetry, Managed Service for Prometheus and log-based metrics

    Partly covered by: Observability Basics

Sample questions

Try each one before opening the answer. Every option is explained, with the Google Cloud documentation page that proves it.

Question 1 · choose 1

Managed collection for Managed Service for Prometheus is enabled on a GKE cluster. The cart Pods run in the shop namespace and expose Prometheus metrics on a port named metrics. The platform team applied this resource, but no cart metrics reach Cloud Monitoring: apiVersion: monitoring.googleapis.com/v1 kind: PodMonitoring metadata: name: cart-metrics namespace: monitoring spec: selector: matchLabels: app: cart endpoints: - port: metrics interval: 30s What is the fix?

  1. ALower the scrape interval to 15s so that the collector picks up the Pods sooner
  2. BInstall the Ops Agent on the cluster's nodes and add a Prometheus receiver for the port
  3. CCreate the PodMonitoring in the shop namespace, or use a ClusterPodMonitoring instead
  4. DAdd a Kubernetes Service that exposes the metrics port of the cart Pods to the monitoring namespace
Show the answer and why
  • ALower the scrape interval to 15s so that the collector picks up the Pods sooner

    Incorrect

    The interval controls how often targets are scraped; it does not change which Pods the resource can discover.

  • BInstall the Ops Agent on the cluster's nodes and add a Prometheus receiver for the port

    Incorrect

    The Ops Agent is the agent for Compute Engine VMs; on GKE, managed collection with PodMonitoring resources does the scraping.

  • CCreate the PodMonitoring in the shop namespace, or use a ClusterPodMonitoring instead

    Correct

    A PodMonitoring scrapes targets only in the namespace it is deployed in; a ClusterPodMonitoring does not limit discovery to one namespace.

  • DAdd a Kubernetes Service that exposes the metrics port of the cart Pods to the monitoring namespace

    Incorrect

    PodMonitoring selects Pods by label in its own namespace; a Service in front of the Pods does not extend that namespace boundary.

With managed collection, PodMonitoring and ClusterPodMonitoring resources tell the collectors what to scrape. A PodMonitoring is namespaced and only finds Pods in its own namespace, so it must live next to the workload, or a ClusterPodMonitoring must be used to select Pods across namespaces.

Question 2 · choose 1

An aggregated sink on a folder routes the application logs of 25 projects into one log bucket, central-apps, in the project obs-hub. The SRE team created a counter log-based metric in obs-hub for entries containing PAYMENT_FAILED so it can alert on all 25 projects with one policy, but the metric counts almost nothing. What should the team do?

  1. ACreate the same project-level log-based metric in each of the 25 projects and add them to the metrics scope of obs-hub
  2. BGrant the SRE team Logs Viewer on the 25 source projects so the metric can read their entries
  3. CChange the metric from a counter to a distribution metric that extracts a value from each central-apps entry
  4. DCreate a bucket-scoped log-based metric on central-apps with the same filter
Show the answer and why
  • ACreate the same project-level log-based metric in each of the 25 projects and add them to the metrics scope of obs-hub

    Incorrect

    That works only as 25 separate metrics that each project must maintain, not the single metric over the central bucket the team wants.

  • BGrant the SRE team Logs Viewer on the 25 source projects so the metric can read their entries

    Incorrect

    Log-based metrics are not limited by the viewer's roles; a project-scoped metric counts only entries that pass through its own project's sinks, and the routed entries never do.

  • CChange the metric from a counter to a distribution metric that extracts a value from each central-apps entry

    Incorrect

    The metric type changes what is measured, not which entries are evaluated, so the routed entries would still be missed.

  • DCreate a bucket-scoped log-based metric on central-apps with the same filter

    Correct

    Bucket-scoped log-based metrics are calculated from all logs destined for the bucket, wherever they originated, which suits logs routed by aggregated sinks.

A project-scoped log-based metric counts entries that pass through the log sinks of its own project. Entries routed from other projects into a bucket are counted by a bucket-scoped log-based metric defined on that bucket, which sees everything stored there regardless of origin.

Question 3 · choose 1

A Java service on Compute Engine is instrumented with the OpenTelemetry SDK, which exports traces, metrics and logs over OTLP gRPC to the Ops Agent's OTLP receiver on the same VM. The traces and the metrics reach Cloud Trace and Cloud Monitoring, but none of the service's log records reach Cloud Logging. What should the team do?

  1. ASwitch the SDK's OTLP exporter from gRPC to HTTP, which the receiver uses for log records
  2. BHave the service write its logs to a file and collect that file with an Ops Agent logging receiver
  3. CGrant the Logs Writer role to the VM's service account so that the agent can write the OTLP log records
  4. DAdd the log records to a logging pipeline that names the OTLP receiver under the agent's service section
Show the answer and why
  • ASwitch the SDK's OTLP exporter from gRPC to HTTP, which the receiver uses for log records

    Incorrect

    The Ops Agent's OTLP listener supports the gRPC transport, not HTTP, and it does not collect logs over either.

  • BHave the service write its logs to a file and collect that file with an Ops Agent logging receiver

    Correct

    The OTLP receiver collects metrics and traces only; logs on a VM are collected with the Ops Agent's other logging receivers, such as a files receiver.

  • CGrant the Logs Writer role to the VM's service account so that the agent can write the OTLP log records

    Incorrect

    Permissions are not the cause: the OTLP receiver never accepts log records, whatever the service account can write.

  • DAdd the log records to a logging pipeline that names the OTLP receiver under the agent's service section

    Incorrect

    The OTLP receiver belongs to the metrics and traces side of the agent; it cannot feed a logging pipeline because it does not collect logs.

The Ops Agent OTLP receiver collects user-defined metrics and traces over gRPC. It doesn't support direct collection of logs; logs are collected with the agent's logging receivers, and log information can be attached to spans if needed.

Question 4 · choose 1

An application has written "payment declined" entries to Cloud Logging for months. On Monday a team creates a user-defined counter log-based metric for these entries and builds a chart to compare this week with the last quarter. The chart shows data only from Monday onward. Why?

  1. AThe chart's alignment period hides older points until a full quarter of data has been aligned
  2. BThe _Default bucket keeps logs for 30 days, so the metric can only reach back 30 days from today
  3. CUser-defined log-based metrics count only entries that arrive after the metric is created
  4. DUser-defined counter log-based metrics reset when the application restarts, so earlier entries were dropped
Show the answer and why
  • AThe chart's alignment period hides older points until a full quarter of data has been aligned

    Incorrect

    Alignment changes how points are grouped in time; it does not hide data that exists.

  • BThe _Default bucket keeps logs for 30 days, so the metric can only reach back 30 days from today

    Incorrect

    Retention is not the cause: the metric would show nothing before Monday even for logs still stored in the bucket.

  • CUser-defined log-based metrics count only entries that arrive after the metric is created

    Correct

    Data for a user-defined log-based metric comes only from log entries received after its creation; it is not filled in retroactively.

  • DUser-defined counter log-based metrics reset when the application restarts, so earlier entries were dropped

    Incorrect

    A log-based metric counts matching log entries in Cloud Logging; it does not depend on counters held by the application.

A user-defined log-based metric starts collecting when it is created: only log entries received after that point are counted, and the metric is not backfilled from logs already stored. Create log-based metrics before you need their history, and use log queries or Observability Analytics for questions about older logs.

Question 5 · choose 1

A company runs Prometheus with prometheus-operator on Amazon EKS and wants those metrics in Managed Service for Prometheus next to its GKE metrics. It wants to keep its existing scrape configurations and operator setup with as few changes as possible. What should the team do on the EKS cluster?

  1. AReplace the Prometheus image with the Managed Service for Prometheus drop-in binary and pass credentials explicitly
  2. BReplace the Prometheus image with the drop-in binary and rely on Workload Identity Federation for GKE to supply its credentials
  3. CInstall the Ops Agent on each EKS node with its Prometheus receiver, so that it scrapes the same targets
  4. DKeep upstream Prometheus and add a remote-write receiver on GKE that exports what it receives to Managed Service for Prometheus
Show the answer and why
  • AReplace the Prometheus image with the Managed Service for Prometheus drop-in binary and pass credentials explicitly

    Correct

    Self-deployed collection keeps the existing Prometheus setup and swaps only the binary; outside GKE the collector needs credentials provided through flags or GOOGLE_APPLICATION_CREDENTIALS.

  • BReplace the Prometheus image with the drop-in binary and rely on Workload Identity Federation for GKE to supply its credentials

    Incorrect

    Credentials are picked up automatically only on GKE; on a non-GKE cluster they must be provided explicitly.

  • CInstall the Ops Agent on each EKS node with its Prometheus receiver, so that it scrapes the same targets

    Incorrect

    The Ops Agent's Prometheus receiver is the recommended option for Compute Engine VMs, and it would replace the operator setup the team wants to keep.

  • DKeep upstream Prometheus and add a remote-write receiver on GKE that exports what it receives to Managed Service for Prometheus

    Incorrect

    Managed Service for Prometheus does not support exporting metrics from a server used as a remote-write receiver.

With self-deployed collection you manage your Prometheus installation as before, but run the Managed Service for Prometheus drop-in replacement binary, which also works with prometheus-operator setups. On GKE it picks up credentials from the environment; on other Kubernetes clusters you provide them explicitly. For Compute Engine VMs, the Ops Agent's Prometheus receiver is the recommended path.

Question 6 · choose 1

A team adds this to the Ops Agent configuration on Compute Engine VMs that run nginx: metrics: receivers: nginx: type: nginx stub_status_url: http://127.0.0.1:80/nginx_status service: pipelines: nginx: receivers: [nginx] The nginx access and error logs arrive, but no nginx connection metrics appear in Cloud Monitoring. What is the most likely cause?

  1. AThe metrics pipeline must also list the nginx_access receiver, because connection metrics are derived from the access log
  2. BThe receiver must listen on a public IP address, because the agent cannot reach services bound to 127.0.0.1
  3. CThe nginx metrics receiver supports only Container-Optimized OS, so the VMs need a different image
  4. DThe stub_status module is not enabled in the nginx configuration, so the status page the receiver reads does not exist
Show the answer and why
  • AThe metrics pipeline must also list the nginx_access receiver, because connection metrics are derived from the access log

    Incorrect

    nginx_access is a logging receiver; connection metrics come from the status page scraped by the metrics receiver.

  • BThe receiver must listen on a public IP address, because the agent cannot reach services bound to 127.0.0.1

    Incorrect

    The documented example uses http://127.0.0.1:80/nginx_status; the agent runs on the same VM as nginx.

  • CThe nginx metrics receiver supports only Container-Optimized OS, so the VMs need a different image

    Incorrect

    The nginx integration runs with the Ops Agent on the VM; the guest image is not why the metrics are missing.

  • DThe stub_status module is not enabled in the nginx configuration, so the status page the receiver reads does not exist

    Correct

    The integration reads connection metrics from nginx's status page, which exists only after stub_status is enabled at a location that matches stub_status_url.

The Ops Agent nginx integration collects connection metrics and access logs. For metrics you must enable the stub_status module in nginx and point the receiver's stub_status_url at it; for logs you configure nginx_access and nginx_error receivers in a logging pipeline.

Google Cloud documentation

Question 7 · choose 1

A GKE Standard cluster uses the default logging settings. A team also runs its own fluentbit DaemonSet that parses application logs in a custom format and sends them to Cloud Logging. Every application log entry now appears twice, and ingestion costs have doubled. The team wants to keep its own agent for application logs. What should it change?

  1. ASet the cluster's logging to NONE, so that only the custom DaemonSet sends logs from the cluster
  2. BSet the cluster's logging to SYSTEM only, so that the managed GKE agent stops collecting application logs
  3. CEdit the managed GKE logging agent's configuration so that it skips the namespaces the custom agent reads
  4. DTurn on the high-throughput logging agent, which removes duplicate entries before they reach Cloud Logging
Show the answer and why
  • ASet the cluster's logging to NONE, so that only the custom DaemonSet sends logs from the cluster

    Incorrect

    If logs go to Cloud Logging, system logs must be sent; turning off GKE logging entirely also drops the system logs and events.

  • BSet the cluster's logging to SYSTEM only, so that the managed GKE agent stops collecting application logs

    Correct

    A Standard cluster can capture only system logs; the custom agent then becomes the single source of application logs, ending the duplicates.

  • CEdit the managed GKE logging agent's configuration so that it skips the namespaces the custom agent reads

    Incorrect

    You cannot directly customize the configuration of the managed logging agent.

  • DTurn on the high-throughput logging agent, which removes duplicate entries before they reach Cloud Logging

    Incorrect

    The high-throughput configuration raises log throughput per node; it does not stop two agents from sending the same logs.

GKE deploys a managed fluentbit-based logging agent whose configuration you cannot change. A custom agent that collects the same workload logs creates duplicates. Standard clusters can be set to collect only system logs (the SYSTEM value), which are required whenever logs go to Cloud Logging, while exclusion filters can further reduce volume.

Google Cloud documentation

Practice domain 4 →Practice all domains →