Task 4.1: Monitor ML and AI model inference and performance.

• ACloudWatch alarms on the endpoint Invocations and ModelLatency metrics

  Incorrect

  These metrics show traffic and speed. A feature distribution can shift completely while invocation counts and latency stay normal.

• BData capture plus a scheduled Evidently drift job with SNS alerts

  Correct

  This is the pattern of the open-source SageMaker AI monitoring solutions that AWS recommends: data capture from the endpoint, Evidently drift presets that compare current data with the training reference, results logged to a SageMaker AI MLflow App, and SNS alerts above a drift threshold.

• CA weekly retraining schedule that runs whether or not the data has changed

  Incorrect

  Retraining on a timer may help, but it does not detect or report drift, store drift results, or alert anyone.

• DMore instances behind the endpoint, chosen with auto scaling

  Incorrect

  Scaling changes capacity, not the data. It has no effect on whether the inputs still look like the training data.

Drift detection compares the statistics of live inputs with a training baseline. AWS documents an open-source path built on endpoint data capture, Evidently AI, MLflow, EventBridge, Lambda and SNS, with CloudWatch for system-level metrics.

• ARun the second model as a shadow variant and compare its responses offline

  Incorrect

  Only the production variant's responses go back to callers in a shadow test, so customers never see the shadow model's recommendations and its effect on purchases cannot be measured.

• BTwo production variants with 90/10 traffic weights

  Correct

  Production variants on a multi-variant endpoint split live invocations by the weights you set, so each model serves real customers and their purchases can be attributed to the variant that served them.

• CScore last month's sessions with both models in batch transform jobs

  Incorrect

  Offline scoring shows what each model would have recommended, not whether customers would have bought anything.

• DCompare the models' validation AUC from training

  Incorrect

  Offline metrics are useful for selection, but they do not measure the business outcome the retailer cares about.

A/B testing exposes real users to each model and measures business outcomes. Shadow testing is safer but can only compare technical behavior, because users never see the shadow responses.

• AEnable CloudTrail data events for Bedrock to record the prompts

  Incorrect

  CloudTrail records API activity, such as who called which operation and when. The full prompt and response text is what model invocation logging collects.

• BRun Amazon Inspector on the application

  Incorrect

  Inspector scans workloads for software vulnerabilities. It neither keeps prompts nor watches throttling.

• CTurn on Bedrock model invocation logging to CloudWatch Logs or S3

  Correct

  Model invocation logging collects the request data, response data and metadata of model invocations and delivers them to CloudWatch Logs, Amazon S3, or both. It is off by default.

• DTurn on AWS Config recording for Bedrock resources

  Incorrect

  AWS Config records resource configurations and evaluates them against rules. It does not capture inference content or request-level throttling.

• ECreate a CloudWatch alarm on the InvocationThrottles metric

  Correct

  Bedrock publishes runtime metrics such as InvocationThrottles to CloudWatch, so an alarm on that metric can notify the team soon after throttling begins.

For foundation model applications, keep three signals apart: runtime metrics in CloudWatch for health and throttling, invocation logs for content, and CloudTrail for who called what.

• AAWS CloudTrail management events for the account

  Incorrect

  CloudTrail records API calls made in the account. It does not show the internal steps, tool calls and latencies of an agent session.

• BAn Amazon Bedrock model evaluation job on sample prompts

  Incorrect

  An evaluation job scores model responses offline. It does not trace live production sessions step by step.

• CAgentCore Memory with long-term memory turned on

  Incorrect

  Memory stores conversation context and extracted insights. It is not a tracing or monitoring tool.

• DAgentCore Observability traces in Amazon CloudWatch

  Correct

  AgentCore Observability visualizes each step of the agent workflow, so you can inspect the execution path, audit intermediate outputs and find bottlenecks and failures. Its metrics, spans and logs are stored in CloudWatch.

Agents fail in the middle: a slow tool, a failed call, a bad intermediate output. Step-level traces from AgentCore Observability, viewed in CloudWatch, are how those problems are found.

• AStep caching turned on for every step of the nightly pipeline

  Incorrect

  Caching reuses earlier successful outputs. It does not notify anyone about a failure.

• BAn EventBridge rule on failed pipeline executions, sending to SNS

  Correct

  SageMaker AI sends pipeline execution state change events to EventBridge, and a rule can send matching events to an SNS topic that emails the team.

• CAn AWS Config rule that evaluates the pipeline resource

  Incorrect

  Config evaluates resource configurations against rules. It does not react to an individual pipeline execution failing.

• DA CloudTrail trail that records the StartPipelineExecution call

  Incorrect

  The trail would show that a run started, not that a step inside it failed later.

Pipeline and job state changes arrive in EventBridge in near real time. Matching the failure states and routing them to SNS is the simplest failure alert.

• AThe endpoint's log group in CloudWatch Logs

  Correct

  Anything a model container sends to stdout or stderr is also sent to CloudWatch Logs, in the /aws/sagemaker/Endpoints/[EndpointName] log group with a stream per variant and instance.

• BThe AWS CloudTrail event history for the endpoint

  Incorrect

  CloudTrail records API activity in SageMaker AI, not what a container writes to stderr.

• CThe endpoint's Invocations metric in CloudWatch

  Incorrect

  Metrics count requests and errors over time, but they do not contain the container's output.

• DThe S3 bucket that stores the model artifacts

  Incorrect

  The artifact bucket holds model files; container output goes to CloudWatch Logs, not to that bucket.

Container stdout and stderr land in CloudWatch Logs per endpoint, variant and instance, which is the first place to debug inference errors.

• AModelLatency

  Incorrect

  ModelLatency is the time the model takes to respond as seen from SageMaker AI, including the inference in the container, not the time to launch compute.

• BModelSetupTime

  Correct

  ModelSetupTime is the time it takes to launch new compute resources for a serverless endpoint, and it varies with model size, model download time and container start-up time.

• COverheadLatency

  Incorrect

  OverheadLatency is the time SageMaker AI adds to a request beyond ModelLatency, not the time to launch compute resources.

• DInvocationsPerInstance

  Incorrect

  This metric counts invocations per instance; it does not measure launch time.

For serverless endpoints, ModelSetupTime shows the cold-start cost of new compute, separately from model and overhead latency.

• AThe InvocationLatency p50 statistic alone

  Incorrect

  InvocationLatency alone cannot tell slower generation apart from more tokens per request, which is exactly what changed here.

• BThe OutputTokenCount metric alone

  Incorrect

  Output token count reflects how long answers are, not how fast the model generates them.

• COutput tokens per second from metric math

  Correct

  Output tokens per second (OTPS) is OutputTokenCount divided by InvocationLatency minus TimeToFirstToken, times 1,000; it isolates throughput, so longer prompts or answers do not cause false alarms.

• DThe InvocationThrottles metric

  Incorrect

  Throttles count requests the system rejected; they do not measure generation speed.

Separate workload changes from service changes: OTPS stays stable when only prompt or answer length grows. TimeToFirstToken is published for the streaming operations.

• ASubscribe a Lambda function that counts lines

  Incorrect

  Subscriptions can deliver log events to Lambda, but counting and alerting would then be custom code.

• BCreate a CloudTrail trail for the endpoint

  Incorrect

  CloudTrail records API calls, not lines that a container writes to its log.

• CCreate a metric filter on the endpoint's log group

  Correct

  Metric filters define terms and patterns to look for in log data and turn matching log events into CloudWatch metrics that you can graph or alarm on.

• DAdd an AWS Config rule for the endpoint

  Incorrect

  Config evaluates resource configurations, not log contents.

• ECreate an alarm on that metric with an SNS action

  Correct

  A CloudWatch alarm watches a metric against a threshold over a period and can notify an Amazon SNS topic, which can send email.

Turn log patterns into metrics with metric filters, then alarm on them, so model behavior signals need no custom code.

• AInvocationThrottles and Invocations

  Incorrect

  Throttled requests are not counted in Invocations or in the error metrics, so this pair cannot separate client from server errors.

• BInputTokenCount and OutputTokenCount per model ID

  Incorrect

  Token counts measure request and response volume, not where errors come from.

• CInvocationLatency and TimeToFirstToken by model

  Incorrect

  Latency metrics measure speed, not the source of failures.

• DInvocationClientErrors and InvocationServerErrors

  Correct

  InvocationClientErrors counts invocations that end in client-side errors and InvocationServerErrors counts those that end in AWS server-side errors, so comparing them shows where failures start.

Split error metrics by origin first: a jump in client errors after a client release points at the requests, not the service.