Professional Machine Learning Engineer · Domain 4: Serving and scaling models · 20% of the exam
Task 4.2: Scaling online model serving.
Serving features from Feature Store, public and private endpoints, serving hardware, autoscaling the serving backend for throughput, and tuning models for production training and serving.
Try each one before opening the answer. Every option is explained, with the Google Cloud documentation page that proves it.
Question 1 · choose 1
A bank's fraud model must be reachable only over private networking from its VPC and from on-premises, inside a VPC Service Controls perimeter. The team also needs traffic splits for canary releases and request-response logging to BigQuery. Which endpoint type should the ML engineer use?
AA private endpoint that uses private services access over VPC peering
BA dedicated private endpoint that uses Private Service Connect
CA dedicated public endpoint with an IAM condition on the calling service accounts
DA shared public endpoint restricted with IAM roles
Show the answer and why
AA private endpoint that uses private services access over VPC peering
Incorrect
This endpoint is private, but it supports neither traffic split nor request-response logging.
BA dedicated private endpoint that uses Private Service Connect
Correct
It routes privately, supports VPC Service Controls, and supports traffic split and request-response logging.
CA dedicated public endpoint with an IAM condition on the calling service accounts
Incorrect
A dedicated public endpoint is reached over the public internet and is not supported inside a VPC Service Controls perimeter.
DA shared public endpoint restricted with IAM roles
Incorrect
IAM limits who may call it, but the requests still travel over the public internet.
Agent Platform offers dedicated public, shared public, dedicated private (Private Service Connect) and private (private services access) endpoints. The Private Service Connect endpoint is recommended for production enterprise use: it is private, supports VPC Service Controls, traffic split and request-response logging. The private services access endpoint supports neither traffic split nor logging.
An image model runs on GPU replicas behind an endpoint. Its autoscaling was configured with CPU utilization as the only metric. At peak, GPU utilization stays near 100%, CPU stays at 20%, latency triples, and no replicas are added. What should the ML engineer change?
ARaise maxReplicaCount and keep CPU utilization as the only autoscaling metric
BLower the CPU utilization target from 60% to 15%
CSwitch the model to a machine type with more vCPUs
DAdd the GPU duty cycle metric to the deployment's scaling metrics
Show the answer and why
ARaise maxReplicaCount and keep CPU utilization as the only autoscaling metric
Incorrect
A higher maximum is never reached, because CPU never crosses its target.
BLower the CPU utilization target from 60% to 15%
Incorrect
That would scale on CPU noise rather than the real GPU bottleneck, adding replicas even when the GPUs are idle.
CSwitch the model to a machine type with more vCPUs
Incorrect
More vCPUs lower CPU utilization further and do nothing for the saturated GPU.
DAdd the GPU duty cycle metric to the deployment's scaling metrics
Correct
A deployment that scales only on CPU usage will not scale up even if GPU usage is high; adding the accelerator metric lets it scale on the GPU.
By default, a GPU deployment scales so that CPU or GPU usage, whichever is higher, matches the 60% target. If autoscalingMetricSpecs is set to scale only on CPU, the deployment does not scale up when GPU usage is high. The GPU utilization metric is aiplatform.googleapis.com/prediction/online/accelerator/duty_cycle.
A vLLM server for an open model runs on GKE with GPUs. The Horizontal Pod Autoscaler scales on GPU utilization, and replicas are added late or not at all while latency climbs. The goal is the highest throughput at the lowest cost while staying within a latency target that the server meets at its maximum batch size. Which autoscaling signal should the ML engineer use?
AGPU utilization with a lower target value, to add replicas earlier in a spike
BThe model server's batch size, to keep each request's latency as low as possible
CThe model server's queue size, exported through Managed Service for Prometheus to the HPA
DCPU utilization of the serving Pods with the default HPA target
Show the answer and why
AGPU utilization with a lower target value, to add replicas earlier in a spike
Incorrect
GPU utilization measures duty cycle, not how much work is done, so it maps poorly to latency and throughput at any target.
BThe model server's batch size, to keep each request's latency as low as possible
Incorrect
Batch size suits latency targets below what the maximum batch allows; this team optimizes throughput and cost.
CThe model server's queue size, exported through Managed Service for Prometheus to the HPA
Correct
Queue size tracks waiting requests, rises quickly with load, and is the recommended metric for throughput and cost.
DCPU utilization of the serving Pods with the default HPA target
Incorrect
CPU use of a GPU-bound inference server does not reflect the load on the model.
For LLM inference on GKE, server metrics such as queue size and batch size are better autoscaling signals than GPU utilization, which measures only the duty cycle. Google recommends queue size to maximize throughput and minimize cost when the latency target is achievable at the server's maximum batch size, and batch size for lower latency targets.
A document-analysis model on a shared public endpoint fails on large requests: some payloads are 6 MB, and some inferences take about 3 minutes. Callers use the public internet and there is no private networking requirement. What should the ML engineer do?
AKeep the shared public endpoint and add replicas to run more requests at once
BMove the model to a private endpoint with private services access
CCompress the payloads and keep the 60-second timeout
DMove the model to a dedicated public endpoint with a longer timeout
Show the answer and why
AKeep the shared public endpoint and add replicas to run more requests at once
Incorrect
More replicas do not change the shared endpoint's 1.5 MB payload limit or its 60-second timeout.
BMove the model to a private endpoint with private services access
Incorrect
That endpoint also times out at 60 seconds and requires private networking the callers do not use.
CCompress the payloads and keep the 60-second timeout
Incorrect
Smaller payloads do not help the inferences that run for three minutes.
DMove the model to a dedicated public endpoint with a longer timeout
Correct
Dedicated public endpoints accept payloads up to 10 MB and have a timeout configurable up to one hour.
A shared public endpoint limits requests to 1.5 MB and times out after 60 seconds. A dedicated public endpoint provides production isolation, payloads up to 10 MB and an inference timeout configurable up to one hour, and it is the recommended public option.
A scikit-learn gradient boosting model with 300 trees answers 2,000 requests per second, each with one row of 50 features, within a 30 ms target. The team deployed it on GPU machines and costs are high while GPU utilization is near zero. Which serving hardware should the ML engineer choose?
ACPU-only machine types with autoscaled replicas
BLarger GPU machine types with more GPU memory
CCloud TPU VMs for the endpoint
DOne very large CPU machine with no autoscaling, sized for the peak
Show the answer and why
ACPU-only machine types with autoscaled replicas
Correct
A tree ensemble scoring single rows runs on CPUs, and the GPUs bring cost but no speedup for this model.
BLarger GPU machine types with more GPU memory
Incorrect
The GPUs are already idle; bigger ones cost more and stay idle.
CCloud TPU VMs for the endpoint
Incorrect
TPUs suit matrix-heavy deep models; they would not speed up a scikit-learn tree model.
DOne very large CPU machine with no autoscaling, sized for the peak
Incorrect
One machine is a single point of failure and cannot follow the load the way autoscaled replicas do.
Accelerators help models dominated by matrix computations. A scikit-learn tree ensemble that scores one row at a time runs well on CPU machine types, and Agent Platform scales CPU replicas on their utilization.
An internal document-tagging service calls an online endpoint. Requests are retried automatically, a few seconds of extra latency during a replica restart is acceptable, and traffic varies from near zero at night to a peak each afternoon. The endpoint runs a fixed 12 on-demand replicas. Which TWO changes cut cost within these limits? (Choose TWO.)
ADeploy the model on Spot VMs
BTurn on access logging for the endpoint to track usage
CSet minReplicaCount low and maxReplicaCount at the daily peak
DKeep 12 replicas and switch to a larger machine type
ECreate a reservation for the 12 replicas
Show the answer and why
ADeploy the model on Spot VMs
Correct
Spot VMs cost much less and can be reclaimed, which retried requests and tolerated latency absorb.
BTurn on access logging for the endpoint to track usage
Incorrect
Access logs record each request's timing; they add visibility but do not lower what the replicas cost.
CSet minReplicaCount low and maxReplicaCount at the daily peak
Correct
With max above min, the deployment adds replicas for the peak and removes them when traffic falls.
DKeep 12 replicas and switch to a larger machine type
Incorrect
Fixed, larger replicas raise the cost instead of lowering it.
ECreate a reservation for the 12 replicas
Incorrect
A reservation guarantees capacity for the replicas; it does not lower their price.
Online inference can run on Spot VMs, which are much cheaper but can be preempted, so they suit workloads that tolerate interruptions. Setting maxReplicaCount greater than minReplicaCount turns on autoscaling, so the deployment follows traffic instead of running the peak capacity all day.
A team wants to serve an open model for online inference on Cloud TPUs in Agent Platform and is allowed to use only generally available features. The model fits on one TPU host. Which deployment should it plan?
AA multi-host TPU deployment spread across several hosts
BA single-host Cloud TPU v5e, v6e or TPU7x machine
CA Cloud TPU v4 Pod slice behind the endpoint
DA GPU machine type, because endpoints do not support TPUs
Show the answer and why
AA multi-host TPU deployment spread across several hosts
Incorrect
Multi-host TPU deployment for online inference is in Public Preview, which the team cannot use.
BA single-host Cloud TPU v5e, v6e or TPU7x machine
Correct
Agent Platform supports deploying models to single-host TPU v5e, v6e or TPU7x for online inference.
CA Cloud TPU v4 Pod slice behind the endpoint
Incorrect
Other Cloud TPU generations are not supported for this deployment.
DA GPU machine type, because endpoints do not support TPUs
Incorrect
Endpoints do support TPU deployment, so GPUs are not required.
You can deploy models to a single-host Cloud TPU v5e, v6e or TPU7x for online inference on Agent Platform; other TPU generations are not supported, and multi-host TPU deployment is in Public Preview.
A GKE cluster serves 12 small models, each on its own NVIDIA L4 GPU, and each GPU is busy less than 10% of the time. The team wants to cut GPU cost while keeping each model in its own Pod. What should the ML engineer configure?
AGPU time-sharing, with several Pods sharing one physical GPU
BA larger GPU per node so each model gets more memory
CA node pool of Spot VMs with one GPU per Pod
DAn HPA that scales each model's Pods down to zero replicas at night
Show the answer and why
AGPU time-sharing, with several Pods sharing one physical GPU
Correct
Time-sharing gives multiple workloads access to one GPU, which improves utilization and lowers cost.
BA larger GPU per node so each model gets more memory
Incorrect
A bigger GPU per model raises cost when the GPUs are already idle.
CA node pool of Spot VMs with one GPU per Pod
Incorrect
Spot lowers the price but keeps 12 mostly idle GPUs.
DAn HPA that scales each model's Pods down to zero replicas at night
Incorrect
Scaling down helps at night but leaves each GPU underused during the day.
GKE GPU time-sharing lets multiple workloads get time-shared access to a single NVIDIA GPU on a node, so lightly used inference workloads can share GPUs and reduce running costs.
An Agent Platform endpoint with autoscaling went from 2 to 8 replicas within seconds of a short traffic spike. Traffic dropped back to normal a minute later, but the endpoint kept 8 replicas for several more minutes. The team asks whether autoscaling is broken. What should the ML engineer explain?
AAutoscaling removes replicas only when the model is redeployed, so the team must redeploy after each spike
BThe spike raised minReplicaCount to 8, which now has to be lowered by hand
CAutoscaling averages utilization over the past 24 hours, so a spike keeps the replica count high for the rest of the day
DAutoscaling keeps the highest replica target from the last 5 minutes, so it scales up fast and scales down only after the spike leaves that window
Show the answer and why
AAutoscaling removes replicas only when the model is redeployed, so the team must redeploy after each spike
Incorrect
Autoscaling scales down on its own; no redeployment is needed.
BThe spike raised minReplicaCount to 8, which now has to be lowered by hand
Incorrect
Autoscaling changes the replica count between the minimum and maximum; it does not change the configured minimum.
CAutoscaling averages utilization over the past 24 hours, so a spike keeps the replica count high for the rest of the day
Incorrect
The autoscaler looks at a 5-minute window, not a daily average.
DAutoscaling keeps the highest replica target from the last 5 minutes, so it scales up fast and scales down only after the spike leaves that window
Correct
Every 15 seconds the system applies the highest target from the previous 5-minute window, which favors fast scale-up over fast scale-down.
Agent Platform adjusts the number of replicas every 15 seconds using data from the previous 5-minute window. Each cycle computes a target replica count from current and target utilization, and the endpoint is set to the highest target seen in that window. It therefore scales up within about 15 seconds but does not scale down while a recent spike is still inside the window. Starting and stopping replicas adds further delay.
Several endpoints in one project each allow up to 20 GPU replicas. During a busy day some deployments stop adding replicas even though their utilization is above target, and errors mention quota. What is the most likely cause?
AThe 60% utilization target is set too high for the replicas to scale out when needed
BThe deployments use a prebuilt container instead of a custom one
CThe sum of the deployments' replicas needs more CPU or GPU quota than the project has
DThe endpoints are dedicated ones, which have a lower project quota than shared ones
Show the answer and why
AThe 60% utilization target is set too high for the replicas to scale out when needed
Incorrect
Utilization above target is already triggering scale-up; the requests fail on quota.
BThe deployments use a prebuilt container instead of a custom one
Incorrect
The container type does not affect replica quota.
CThe sum of the deployments' replicas needs more CPU or GPU quota than the project has
Correct
Quota is counted per CPU and GPU for all active replicas, so busy deployments can run out of quota before reaching their maximum.
DThe endpoints are dedicated ones, which have a lower project quota than shared ones
Incorrect
Endpoint type does not change how serving quota is counted for the replicas.
Custom model serving quota is based on the deployed models' real-time use of compute. Endpoints scale per machine, but quota is counted per CPU or GPU, so if the sum of maxReplicaCount across deployments exceeds the project's quota, some deployments may fail to autoscale.