Task 3.4: Describe methods to evaluate FM performance.

• ABERTScore

  Incorrect

  BERTScore compares embeddings of the generated and reference text, not overlapping word sequences.

• BROUGE

  Correct

  ROUGE-N computes the n-gram overlap between the reference summary and the model's summary, and it is used to score summarization accuracy.

• CToxicity score

  Incorrect

  A toxicity score comes from a toxicity detector model and measures harmful content, not agreement with a reference.

• DClassification accuracy

  Incorrect

  Accuracy is the ratio of correctly classified items to all items. A summary is generated text, not a class label.

ROUGE counts shared n-grams with a reference; BERTScore compares meaning through embeddings. Both are used to judge summaries against a gold standard.

• AA model evaluation job that uses human workers

  Incorrect

  This job type brings in employees or subject-matter experts to rate responses, which is exactly what the team wants to avoid.

• BA programmatic evaluation that computes BERTScore

  Incorrect

  BERTScore is a computed similarity metric against reference text. It does not judge helpfulness or explain each score.

• CA model evaluation job that uses an LLM as a judge

  Correct

  In a judge-model evaluation, a second LLM scores the generator model's responses on the metrics you select, such as helpfulness, and explains each score.

• DThe contextual grounding check in Amazon Bedrock Guardrails

  Incorrect

  Guardrails filter content at run time; the grounding check flags responses not supported by the source. It is not an evaluation job that rates helpfulness.

LLM-as-a-judge scales subjective evaluation that once needed people, and it explains its scores. Spot-check it with humans for high-stakes uses.

• AStereotyping

  Incorrect

  Stereotyping measures generalized statements about individuals or groups in responses, not retrieval quality or grounding.

• BRefusal

  Incorrect

  Refusal measures how evasive the responses are in answering questions.

• CLogical coherence

  Incorrect

  Logical coherence checks responses for logical gaps, inconsistencies or contradictions, not whether they stick to the retrieved texts.

• DContext relevance

  Correct

  Context relevance measures how contextually relevant the retrieved texts are to the questions, the first problem.

• EFaithfulness

  Correct

  Faithfulness measures how well responses avoid hallucination with respect to the retrieved texts, the second problem.

Evaluate RAG in two halves: retrieval (context relevance, context coverage) and generation (faithfulness, correctness, completeness).

• AThe average number of words in the agent's replies

  Incorrect

  Reply length says nothing about whether a reset was completed; a short or long reply can both fail the task.

• BThe BERTScore of the agent's replies measured against a set of sample replies

  Incorrect

  BERTScore compares generated text with reference text through embeddings. It measures wording similarity, not task success.

• CThe number of tokens the agent processes per day

  Incorrect

  Token volume drives cost on token-based pricing; it measures usage, not whether users got their passwords reset.

• DThe task completion rate, the share of requests resolved without a person

  Correct

  Task completion rate is one of the business objective alignment metrics the exam guide lists for AI applications, alongside user satisfaction and cost per interaction.

For agents, measure outcomes: did the task get done, how satisfied was the user, and what did each interaction cost.

• AA human-based evaluation job staffed by the brand's copywriters

  Correct

  Human-based evaluations bring human input to the process; the workers can be employees of your company or subject-matter experts.

• BAn automatic evaluation that computes toxicity

  Incorrect

  Toxicity scores measure harmful content, not whether text matches a brand voice.

• CA retrieve-only RAG evaluation job

  Incorrect

  Retrieve-only RAG evaluations measure retrieval from a knowledge base, not writing style.

• DAn automatic evaluation job that runs on a built-in open-source prompt dataset

  Incorrect

  Built-in datasets are generic prompt sets; they cannot capture the brand's own voice.

When the quality bar is subjective and only experts can judge it, human-in-the-loop evaluation is the right tool.

• AA human-based evaluation job with a newly recruited team of reviewers

  Incorrect

  Human evaluation is slower to set up and suits subjective criteria; the team wants a quick automated first pass.

• BA Guardrails contextual grounding check

  Incorrect

  The grounding check filters ungrounded responses at run time; it is not a model comparison.

• CAn automatic evaluation with built-in prompt datasets

  Correct

  Amazon Bedrock provides built-in prompt datasets, each based on an open-source dataset, that can be used in automatic model evaluation jobs.

• DA Provisioned Throughput purchase for each model

  Incorrect

  Provisioned Throughput buys capacity for a model; it does not evaluate or compare models.

Benchmark-style built-in datasets give a fast, repeatable first comparison; follow up with your own data for the final choice.

• AROUGE-N

  Incorrect

  ROUGE-N counts n-gram overlaps between the reference and the model's text, so different wording lowers it.

• BClassification accuracy

  Incorrect

  Accuracy is the ratio of correctly classified items, used for class labels rather than generated text.

• CWord error rate

  Incorrect

  Word error rate is a robustness metric in Bedrock evaluations; it does not compare the meaning of two texts.

• DBERTScore

  Correct

  BERTScore compares embeddings of the summary and the reference, so it reflects similarity of meaning.

ROUGE rewards matching words; BERTScore rewards matching meaning. Using both gives a fuller picture of summary quality.

• AAmazon Bedrock AgentCore Memory

  Incorrect

  Memory lets agents remember past interactions; it does not score them.

• BAmazon Bedrock AgentCore Evaluations

  Correct

  AgentCore Evaluations measures how well agents and tools perform tasks, handle edge cases and stay consistent, using built-in and custom evaluators, before and after deployment.

• CAmazon Bedrock AgentCore Identity

  Incorrect

  Identity manages agent identities and credentials for accessing resources; it does not assess quality.

• DAmazon Bedrock AgentCore Observability dashboards

  Incorrect

  Observability traces, debugs and monitors agent workflows step by step; scoring task success with evaluators is what Evaluations adds.

Evaluating an agent means scoring task outcomes over whole sessions, which AgentCore Evaluations does with LLM-as-a-judge evaluators.

• ARun a retrieve-only RAG evaluation job

  Correct

  A retrieve-only job bases its report on the data retrieved from the RAG source, so it isolates the search step from response generation.

• BRun an automatic model evaluation job on a built-in prompt dataset

  Incorrect

  A model evaluation job scores a model's own outputs for a task type such as text generation or summarization; it does not look at what a knowledge base retrieves.

• CScore the job with the citation precision metric

  Incorrect

  Citation precision is a retrieve-and-generate metric about the passages a generated response cites, so it does not apply to a retrieve-only job.

• DScore the job with the context coverage metric

  Correct

  Context coverage is a built-in retrieve-only metric that measures how much of the information in the ground-truth texts the retrieved texts cover, and it needs exactly the ground truth the team has prepared.

• ETurn on the contextual grounding check in Amazon Bedrock Guardrails

  Incorrect

  The contextual grounding check detects and filters hallucinations in model responses; it is a safeguard on responses, not an evaluation of what the retrieval step finds.

Amazon Bedrock RAG evaluations come in two job types. Retrieve only scores what the knowledge base returns (context relevance, context coverage); retrieve and generate also scores the generated answers (correctness, faithfulness, citation metrics and more). To find out whether retrieval is the weak link, evaluate retrieval on its own first.