Task 6.2: Monitoring, testing, and troubleshooting AI solutions.

• APrediction drift detection with no training data

  Incorrect

  Drift detection compares production data over time; it is the choice when training data is not available.

• BFeature attribution monitoring on the endpoint with no baseline dataset

  Incorrect

  Attribution monitoring tracks feature contributions; it does not compare input values with the training data.

• CRequest-response logging to BigQuery

  Incorrect

  Logging stores requests but sends no alerts on differences from the training data.

• DTraining-serving skew detection with the original training data as baseline

  Correct

  Skew detection compares production inputs with the training data, which must be provided as the baseline.

Model Monitoring (v1, generally available, configured on endpoints) detects training-serving skew when the original training data is provided, and drift over time when it is not. You can enable both.

• ALower the drift detection alert thresholds on every input feature until alerts appear

  Incorrect

  The inputs have not changed, so tighter thresholds still raise no alerts.

• BContinuous evaluation that joins the delayed labels with logged predictions

  Correct

  When the relation between inputs and labels changes, only comparing predictions with ground truth shows the drop in quality.

• CSkew detection against a newer copy of the training data and its labels

  Incorrect

  Skew detection compares input distributions too, and they match.

• DDaily retraining on the inputs of the last 24 hours

  Incorrect

  Retraining without fresh labels cannot learn the new fraud patterns.

Concept drift changes the relationship between features and labels without changing the feature distributions, so skew and drift detection do not see it. When ground truth becomes available, join it with the predicted labels of the serving instances to evaluate the model continuously and alert on its metrics.

• AA weekly rerun of the pre-launch test set through the evaluation service with the same launch rubric

  Incorrect

  The fixed test set does not contain the new questions users are asking.

• BAlerts on the endpoint's latency and error rate in Cloud Monitoring

  Incorrect

  Operational metrics show health, not whether the answers are good.

• CModel Monitoring skew detection on the assistant's input features

  Incorrect

  Skew detection is for tabular input features; it does not judge the quality of generated answers.

• DA weekly evaluation run on a sample of production logs, scored with the same rubric

  Correct

  Evaluation datasets can be sampled from production logs, so quality is measured on real usage with the same rubric.

The Gen AI evaluation service can build evaluation datasets by sampling directly from production logs, which evaluates real-world usage, and runs rubric-based or computation-based metrics on them. Repeating that on a schedule tracks quality as usage changes.

• ARemove the alert thresholds for all features

  Incorrect

  Thresholds decide when to alert; removing them loses the alerts without reducing the analysis.

• BAdd more email addresses to the monitoring notifications

  Incorrect

  Notification recipients have no effect on cost.

• CSwitch from skew detection to both skew and drift detection

  Incorrect

  Adding drift detection adds analysis instead of reducing it.

• DLower the prediction sampling rate

  Correct

  The sampling rate sets what share of requests is logged and analyzed, which drives cost.

• ELower the monitoring frequency to once a day

  Correct

  The monitoring frequency sets how often the logged inputs are analyzed for skew and drift.

When you create a model deployment monitoring job, you set the prediction sampling rate (the share of requests logged to BigQuery and analyzed) and the monitoring frequency, along with alert thresholds and notifications. Sampling fewer requests and analyzing less often reduce the cost of monitoring.

• ALower the safety filter thresholds for every category

  Incorrect

  The response was not blocked for safety; a SAFETY finish reason would show that.

• BMove the requests from the regional endpoint to the global endpoint

  Incorrect

  The endpoint affects capacity and availability, not output length.

• CRetry the request with exponential backoff

  Incorrect

  A retry produces the same cut-off answer; nothing transient failed.

• DRaise maxOutputTokens or ask for a shorter summary in the prompt

  Correct

  MAX_TOKENS means generation stopped at the maximum number of tokens set in the request.

The finish reason explains why generation stopped: STOP for a natural end, MAX_TOKENS when the response reached the maximum number of tokens set in the request, SAFETY when a configurable content filter blocked it, and other codes for citation and non-configurable filters.

• ATurn on request-response logging to BigQuery for the deployed model

  Incorrect

  Request-response logging samples inputs and outputs; it does not capture the container's own messages.

• BEnable container logging by updating the deployed model

  Correct

  Container logging sends the inference nodes' stdout and stderr to Cloud Logging and can be enabled when the deployed model is mutated.

• CAdd Model Monitoring with drift detection

  Incorrect

  Model Monitoring analyzes inputs for skew and drift; it is not a debugging log.

• DTurn on access logging for the endpoint and its container

  Incorrect

  Access logs record request timing and latency, not stdout from the container.

Agent Platform online inference offers container logging (stdout and stderr of the inference nodes, needed for debugging), access logging (timestamp and latency per request) and request-response logging (a sample of requests and responses in BigQuery). Container logging can be enabled or disabled at deployment or by mutating the deployed model.

• ASet a higher alert threshold for days_since_signup only, and keep the default threshold for the other features

  Correct

  Thresholds can be set per feature, and by default every feature is monitored with a threshold of 0.3.

• BRaise the threshold for every feature to the level that silences days_since_signup

  Incorrect

  This also silences the useful alerts on the other features.

• CRemove the notification emails so that the alerts stop reaching the on-call team

  Incorrect

  Without notifications, real anomalies on the other features would go unseen too.

• DDelete the monitoring job and check the feature distributions by hand each month

  Incorrect

  Manual monthly checks lose the automatic alerts the team relies on.

When you create a Model Monitoring job, you can set an alert threshold for each feature, for example Age=0.4, and Model Monitoring alerts when the statistical distance between the input and baseline distributions for that feature exceeds it. By default every categorical and numerical feature is monitored with a threshold of 0.3, so a feature that drifts harmlessly can get its own, higher threshold.

• ARequest-response logging on the endpoint, with the logs written to a BigQuery table

  Incorrect

  This stores sampled requests and responses; it does not record why the monitoring pipeline runs fail.

• BData Access audit logs for Agent Platform in the project

  Incorrect

  Audit logs record who accessed what; they do not explain why monitoring runs fail.

• CThe enableMonitoringPipelineLogs field in the monitoring job, which writes the model_monitoring log to Cloud Logging

  Correct

  With this field set to true, debugging logs are written when the job is set up and at each monitoring interval.

• DMore notification email addresses on the monitoring job

  Incorrect

  More recipients get the same failure emails, with no extra detail.

Model Monitoring emails each recipient when a scheduled monitoring pipeline run fails. To debug those runs, set enableMonitoringPipelineLogs to true in the model deployment monitoring job. Debugging logs are then written to Cloud Logging, under the model_monitoring log name, when the job is set up and at each monitoring interval. Anomaly alerts can separately be written to the model_monitoring_anomaly log.