Professional Cloud DevOps Engineer · Domain 4: Implementing observability practices and troubleshooting issues · 22% of the exam
Task 4.1: Instrumenting and collecting telemetry.
Collecting logs and metrics with the Ops Agent, OpenTelemetry and Managed Service for Prometheus, optimizing log volume, synthetic monitors, log-based and custom metrics, and telemetry from services and agents.
Study it
Instrumenting: Ops Agent, OpenTelemetry, Managed Service for Prometheus and log-based metrics
Try each one before opening the answer. Every option is explained, with the Google Cloud documentation page that proves it.
Question 1 · choose 1
Managed collection for Managed Service for Prometheus is enabled on a GKE
cluster. The cart Pods run in the shop namespace and expose Prometheus
metrics on a port named metrics. The platform team applied this resource,
but no cart metrics reach Cloud Monitoring:
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
name: cart-metrics
namespace: monitoring
spec:
selector:
matchLabels:
app: cart
endpoints:
- port: metrics
interval: 30s
What is the fix?
ALower the scrape interval to 15s so that the collector picks up the Pods sooner
BInstall the Ops Agent on the cluster's nodes and add a Prometheus receiver for the port
CCreate the PodMonitoring in the shop namespace, or use a ClusterPodMonitoring instead
DAdd a Kubernetes Service that exposes the metrics port of the cart Pods to the monitoring namespace
Show the answer and why
ALower the scrape interval to 15s so that the collector picks up the Pods sooner
Incorrect
The interval controls how often targets are scraped; it does not change which Pods the resource can discover.
BInstall the Ops Agent on the cluster's nodes and add a Prometheus receiver for the port
Incorrect
The Ops Agent is the agent for Compute Engine VMs; on GKE, managed collection with PodMonitoring resources does the scraping.
CCreate the PodMonitoring in the shop namespace, or use a ClusterPodMonitoring instead
Correct
A PodMonitoring scrapes targets only in the namespace it is deployed in; a ClusterPodMonitoring does not limit discovery to one namespace.
DAdd a Kubernetes Service that exposes the metrics port of the cart Pods to the monitoring namespace
Incorrect
PodMonitoring selects Pods by label in its own namespace; a Service in front of the Pods does not extend that namespace boundary.
With managed collection, PodMonitoring and ClusterPodMonitoring resources tell the collectors what to scrape. A PodMonitoring is namespaced and only finds Pods in its own namespace, so it must live next to the workload, or a ClusterPodMonitoring must be used to select Pods across namespaces.
An aggregated sink on a folder routes the application logs of 25 projects into one log bucket, central-apps, in the project obs-hub. The SRE team created a counter log-based metric in obs-hub for entries containing PAYMENT_FAILED so it can alert on all 25 projects with one policy, but the metric counts almost nothing. What should the team do?
ACreate the same project-level log-based metric in each of the 25 projects and add them to the metrics scope of obs-hub
BGrant the SRE team Logs Viewer on the 25 source projects so the metric can read their entries
CChange the metric from a counter to a distribution metric that extracts a value from each central-apps entry
DCreate a bucket-scoped log-based metric on central-apps with the same filter
Show the answer and why
ACreate the same project-level log-based metric in each of the 25 projects and add them to the metrics scope of obs-hub
Incorrect
That works only as 25 separate metrics that each project must maintain, not the single metric over the central bucket the team wants.
BGrant the SRE team Logs Viewer on the 25 source projects so the metric can read their entries
Incorrect
Log-based metrics are not limited by the viewer's roles; a project-scoped metric counts only entries that pass through its own project's sinks, and the routed entries never do.
CChange the metric from a counter to a distribution metric that extracts a value from each central-apps entry
Incorrect
The metric type changes what is measured, not which entries are evaluated, so the routed entries would still be missed.
DCreate a bucket-scoped log-based metric on central-apps with the same filter
Correct
Bucket-scoped log-based metrics are calculated from all logs destined for the bucket, wherever they originated, which suits logs routed by aggregated sinks.
A project-scoped log-based metric counts entries that pass through the log sinks of its own project. Entries routed from other projects into a bucket are counted by a bucket-scoped log-based metric defined on that bucket, which sees everything stored there regardless of origin.
A Java service on Compute Engine is instrumented with the OpenTelemetry SDK, which exports traces, metrics and logs over OTLP gRPC to the Ops Agent's OTLP receiver on the same VM. The traces and the metrics reach Cloud Trace and Cloud Monitoring, but none of the service's log records reach Cloud Logging. What should the team do?
ASwitch the SDK's OTLP exporter from gRPC to HTTP, which the receiver uses for log records
BHave the service write its logs to a file and collect that file with an Ops Agent logging receiver
CGrant the Logs Writer role to the VM's service account so that the agent can write the OTLP log records
DAdd the log records to a logging pipeline that names the OTLP receiver under the agent's service section
Show the answer and why
ASwitch the SDK's OTLP exporter from gRPC to HTTP, which the receiver uses for log records
Incorrect
The Ops Agent's OTLP listener supports the gRPC transport, not HTTP, and it does not collect logs over either.
BHave the service write its logs to a file and collect that file with an Ops Agent logging receiver
Correct
The OTLP receiver collects metrics and traces only; logs on a VM are collected with the Ops Agent's other logging receivers, such as a files receiver.
CGrant the Logs Writer role to the VM's service account so that the agent can write the OTLP log records
Incorrect
Permissions are not the cause: the OTLP receiver never accepts log records, whatever the service account can write.
DAdd the log records to a logging pipeline that names the OTLP receiver under the agent's service section
Incorrect
The OTLP receiver belongs to the metrics and traces side of the agent; it cannot feed a logging pipeline because it does not collect logs.
The Ops Agent OTLP receiver collects user-defined metrics and traces over gRPC. It doesn't support direct collection of logs; logs are collected with the agent's logging receivers, and log information can be attached to spans if needed.
An application has written "payment declined" entries to Cloud Logging for months. On Monday a team creates a user-defined counter log-based metric for these entries and builds a chart to compare this week with the last quarter. The chart shows data only from Monday onward. Why?
AThe chart's alignment period hides older points until a full quarter of data has been aligned
BThe _Default bucket keeps logs for 30 days, so the metric can only reach back 30 days from today
CUser-defined log-based metrics count only entries that arrive after the metric is created
DUser-defined counter log-based metrics reset when the application restarts, so earlier entries were dropped
Show the answer and why
AThe chart's alignment period hides older points until a full quarter of data has been aligned
Incorrect
Alignment changes how points are grouped in time; it does not hide data that exists.
BThe _Default bucket keeps logs for 30 days, so the metric can only reach back 30 days from today
Incorrect
Retention is not the cause: the metric would show nothing before Monday even for logs still stored in the bucket.
CUser-defined log-based metrics count only entries that arrive after the metric is created
Correct
Data for a user-defined log-based metric comes only from log entries received after its creation; it is not filled in retroactively.
DUser-defined counter log-based metrics reset when the application restarts, so earlier entries were dropped
Incorrect
A log-based metric counts matching log entries in Cloud Logging; it does not depend on counters held by the application.
A user-defined log-based metric starts collecting when it is created: only log entries received after that point are counted, and the metric is not backfilled from logs already stored. Create log-based metrics before you need their history, and use log queries or Observability Analytics for questions about older logs.
A company runs Prometheus with prometheus-operator on Amazon EKS and wants those metrics in Managed Service for Prometheus next to its GKE metrics. It wants to keep its existing scrape configurations and operator setup with as few changes as possible. What should the team do on the EKS cluster?
AReplace the Prometheus image with the Managed Service for Prometheus drop-in binary and pass credentials explicitly
BReplace the Prometheus image with the drop-in binary and rely on Workload Identity Federation for GKE to supply its credentials
CInstall the Ops Agent on each EKS node with its Prometheus receiver, so that it scrapes the same targets
DKeep upstream Prometheus and add a remote-write receiver on GKE that exports what it receives to Managed Service for Prometheus
Show the answer and why
AReplace the Prometheus image with the Managed Service for Prometheus drop-in binary and pass credentials explicitly
Correct
Self-deployed collection keeps the existing Prometheus setup and swaps only the binary; outside GKE the collector needs credentials provided through flags or GOOGLE_APPLICATION_CREDENTIALS.
BReplace the Prometheus image with the drop-in binary and rely on Workload Identity Federation for GKE to supply its credentials
Incorrect
Credentials are picked up automatically only on GKE; on a non-GKE cluster they must be provided explicitly.
CInstall the Ops Agent on each EKS node with its Prometheus receiver, so that it scrapes the same targets
Incorrect
The Ops Agent's Prometheus receiver is the recommended option for Compute Engine VMs, and it would replace the operator setup the team wants to keep.
DKeep upstream Prometheus and add a remote-write receiver on GKE that exports what it receives to Managed Service for Prometheus
Incorrect
Managed Service for Prometheus does not support exporting metrics from a server used as a remote-write receiver.
With self-deployed collection you manage your Prometheus installation as before, but run the Managed Service for Prometheus drop-in replacement binary, which also works with prometheus-operator setups. On GKE it picks up credentials from the environment; on other Kubernetes clusters you provide them explicitly. For Compute Engine VMs, the Ops Agent's Prometheus receiver is the recommended path.
A team adds this to the Ops Agent configuration on Compute Engine VMs that
run nginx:
metrics:
receivers:
nginx:
type: nginx
stub_status_url: http://127.0.0.1:80/nginx_status
service:
pipelines:
nginx:
receivers: [nginx]
The nginx access and error logs arrive, but no nginx connection metrics
appear in Cloud Monitoring. What is the most likely cause?
AThe metrics pipeline must also list the nginx_access receiver, because connection metrics are derived from the access log
BThe receiver must listen on a public IP address, because the agent cannot reach services bound to 127.0.0.1
CThe nginx metrics receiver supports only Container-Optimized OS, so the VMs need a different image
DThe stub_status module is not enabled in the nginx configuration, so the status page the receiver reads does not exist
Show the answer and why
AThe metrics pipeline must also list the nginx_access receiver, because connection metrics are derived from the access log
Incorrect
nginx_access is a logging receiver; connection metrics come from the status page scraped by the metrics receiver.
BThe receiver must listen on a public IP address, because the agent cannot reach services bound to 127.0.0.1
Incorrect
The documented example uses http://127.0.0.1:80/nginx_status; the agent runs on the same VM as nginx.
CThe nginx metrics receiver supports only Container-Optimized OS, so the VMs need a different image
Incorrect
The nginx integration runs with the Ops Agent on the VM; the guest image is not why the metrics are missing.
DThe stub_status module is not enabled in the nginx configuration, so the status page the receiver reads does not exist
Correct
The integration reads connection metrics from nginx's status page, which exists only after stub_status is enabled at a location that matches stub_status_url.
The Ops Agent nginx integration collects connection metrics and access logs. For metrics you must enable the stub_status module in nginx and point the receiver's stub_status_url at it; for logs you configure nginx_access and nginx_error receivers in a logging pipeline.
A GKE Standard cluster uses the default logging settings. A team also runs its own fluentbit DaemonSet that parses application logs in a custom format and sends them to Cloud Logging. Every application log entry now appears twice, and ingestion costs have doubled. The team wants to keep its own agent for application logs. What should it change?
ASet the cluster's logging to NONE, so that only the custom DaemonSet sends logs from the cluster
BSet the cluster's logging to SYSTEM only, so that the managed GKE agent stops collecting application logs
CEdit the managed GKE logging agent's configuration so that it skips the namespaces the custom agent reads
DTurn on the high-throughput logging agent, which removes duplicate entries before they reach Cloud Logging
Show the answer and why
ASet the cluster's logging to NONE, so that only the custom DaemonSet sends logs from the cluster
Incorrect
If logs go to Cloud Logging, system logs must be sent; turning off GKE logging entirely also drops the system logs and events.
BSet the cluster's logging to SYSTEM only, so that the managed GKE agent stops collecting application logs
Correct
A Standard cluster can capture only system logs; the custom agent then becomes the single source of application logs, ending the duplicates.
CEdit the managed GKE logging agent's configuration so that it skips the namespaces the custom agent reads
Incorrect
You cannot directly customize the configuration of the managed logging agent.
DTurn on the high-throughput logging agent, which removes duplicate entries before they reach Cloud Logging
Incorrect
The high-throughput configuration raises log throughput per node; it does not stop two agents from sending the same logs.
GKE deploys a managed fluentbit-based logging agent whose configuration you cannot change. A custom agent that collects the same workload logs creates duplicates. Standard clusters can be set to collect only system logs (the SYSTEM value), which are required whenever logs go to Cloud Logging, while exclusion filters can further reduce volume.