AI-300 · Domain 4: Implement generative AI quality assurance and observability · 14% of the exam
Task 4.1: Configure evaluation and validation for generative AI applications and agents
Test datasets and data mapping, quality metrics such as groundedness, relevance, coherence and fluency, risk and safety evaluations, and automated evaluation workflows with built-in and custom metrics.
Study it
Test datasets, data mapping and AI quality evaluators
Try each one before opening the answer. Every option is explained, with the Microsoft documentation page that proves it.
Question 1 · choose 1
An evaluation run combines coherence, fluency, violence and self-harm evaluators. The project is in a region that isn't on the risk and safety evaluator region list, and the team passed no deployment_name to any evaluator. Which change makes the whole run work?
AGive all four evaluators the same judge deployment_name, and keep the project in its current region
BRemove deployment_name everywhere, since every built-in evaluator uses the hosted evaluation service
CGive coherence and fluency a judge deployment_name, and run the evaluation from a project in a supported safety region
DMove only the coherence and fluency judges to a supported region, and leave the safety evaluators as they are
Show the answer and why
AGive all four evaluators the same judge deployment_name, and keep the project in its current region
Incorrect
Safety evaluators don't take a deployment_name; they need a project in a supported region.
BRemove deployment_name everywhere, since every built-in evaluator uses the hosted evaluation service
Incorrect
Only risk and safety evaluators use the hosted service; coherence and fluency need a judge.
CGive coherence and fluency a judge deployment_name, and run the evaluation from a project in a supported safety region
Correct
Judge evaluators need a model deployment, and safety evaluators need a project in a supported region.
DMove only the coherence and fluency judges to a supported region, and leave the safety evaluators as they are
Incorrect
The region requirement applies to the safety evaluators' project.
LLM-as-judge evaluators such as coherence and fluency take a deployment_name initialization parameter for the judge model. Risk and safety evaluators use hosted evaluation language models in the Foundry evaluation service, don't require a deployment_name, and need your Foundry project information, so the project must be in a region that supports them.
An evaluation sends each row's query to an agent target and scores the agent's answers with relevance. The query comes from the dataset; the answers are generated during the run. How should the relevance evaluator's data mapping look?
Aquery mapped to {{sample.query}}, response mapped to {{item.response}}
Bquery mapped to {{item.query}}, response mapped to {{sample.output_items}}
Cquery mapped to {{item.query}}, response mapped to {{item.response}}
DNo mapping; evaluators find columns with matching names by themselves
Show the answer and why
Aquery mapped to {{sample.query}}, response mapped to {{item.response}}
Incorrect
Dataset fields use item, and generated answers use sample.
Bquery mapped to {{item.query}}, response mapped to {{sample.output_items}}
Correct
item fields come from the dataset, and sample.output_items holds the target's responses.
Cquery mapped to {{item.query}}, response mapped to {{item.response}}
Incorrect
There's no response column; the agent generates it during the run.
DNo mapping; evaluators find columns with matching names by themselves
Incorrect
Mapping can be skipped only for standard dataset columns; answers generated at run time need a sample mapping.
Data mapping connects evaluator inputs to data. {{item.field_name}} references fields from the test dataset, such as {{item.query}}, while {{sample.output_items}} references agent responses generated or retrieved during the evaluation, for example when evaluating with an agent target.
A GitHub Actions pipeline runs cloud evaluations through a service principal that has Foundry User on the project. Runs that only score stored responses succeed, but runs that send queries to a model deployment fail with authorization errors. What should you add?
ACognitive Services OpenAI User for the service principal on the project
BFoundry User for the service principal at the Foundry account scope
CAn API key for the deployment stored as a pipeline secret
DFoundry Owner for the service principal on the project
Show the answer and why
ACognitive Services OpenAI User for the service principal on the project
Incorrect
Foundry guidance uses Foundry User at account scope for evaluation inference.
BFoundry User for the service principal at the Foundry account scope
Correct
A project-only assignment doesn't authorize model inference for evaluation jobs.
CAn API key for the deployment stored as a pipeline secret
Incorrect
Evaluation jobs use Microsoft Entra ID, not keys.
DFoundry Owner for the service principal on the project
Incorrect
Owner on the project still doesn't add account-scope inference rights, and it's broader.
For portal, SDK and CI/CD evaluations, the identity running the evaluation needs Foundry User (or equivalent) on the resource or project that contains it. Evaluation jobs use Microsoft Entra ID, and if an evaluation invokes a model deployment you must assign Foundry User at the Foundry account scope, because a project-only role assignment doesn't authorize model inference.
A violence evaluator returns a severity score of 4 for a response, using the default threshold. What label does the row get?
APass, because scores of 3 and above count as passing on these evaluators
BPass, because only scores of 7 cross the safety evaluators' default threshold
CFail, because the score is above the default threshold of 3 on the 0-7 scale
DNo label, because safety evaluators return only the reason field
Show the answer and why
APass, because scores of 3 and above count as passing on these evaluators
Incorrect
That's the quality evaluators' convention; safety evaluators pass at or below the threshold.
BPass, because only scores of 7 cross the safety evaluators' default threshold
Incorrect
The default threshold is 3, not 7.
CFail, because the score is above the default threshold of 3 on the 0-7 scale
Correct
Safety evaluators pass at or below the threshold and fail above it.
DNo label, because safety evaluators return only the reason field
Incorrect
They return a score, a label and a reason.
Content safety evaluators (violence, sexual, self-harm, hate and unfairness) use a 0-7 severity scale. Given a numerical threshold, by default 3, the evaluator outputs pass if the score is less than or equal to the threshold and fail otherwise, along with a reason.
What file layout do Foundry evaluation datasets typically use?
AJSONL, one test case per line
BA single JSON array in one object
CA Parquet table per evaluator
DA ZIP of text files
Show the answer and why
AJSONL, one test case per line
Correct
Each line of a JSONL file is one test case.
BA single JSON array in one object
Incorrect
The typical format is JSONL.
CA Parquet table per evaluator
Incorrect
Datasets aren't split per evaluator.
DA ZIP of text files
Incorrect
The typical format is JSONL.
Evaluation datasets typically use JSONL, where each line contains one evaluation test case; the scenario and selected evaluators determine the required columns.
Which column is required to evaluate simulated conversations?
Aground_truth
Bresponse
Ctest_case_description
Ddesired_num_turns
Show the answer and why
Aground_truth
Incorrect
Simulation doesn't need a reference answer.
Bresponse
Incorrect
The simulation generates the responses.
Ctest_case_description
Correct
It describes what the simulator should act out.
Ddesired_num_turns
Incorrect
It's optional guidance.
For conversation simulation, test_case_description is required: it holds the user's situation, goal, constraints and behavior for the simulator. desired_num_turns is optional.
AOne that scores query and response pairs from a file
BOne that scores stored agent responses by response ID
COne that compares responses with ground truth
DOne that sends dataset queries to an agent
Show the answer and why
AOne that scores query and response pairs from a file
Incorrect
Those pairs come from a dataset.
BOne that scores stored agent responses by response ID
Correct
Response-ID evaluations retrieve the stored responses directly.
COne that compares responses with ground truth
Incorrect
Ground truth comes from a dataset.
DOne that sends dataset queries to an agent
Incorrect
The queries come from a dataset.
Evaluations based on existing traces, response IDs or generated synthetic queries don't require an input dataset; with response IDs, Foundry retrieves the stored responses by ID. Other scenarios use a dataset with the columns the evaluators need.
A team chats with its agent in the agents playground of a new Foundry project. What's true of the playground's evaluations by default?
AThey're off until an administrator enables them per project
BThey're free, since the playground isn't billed separately
CThey can be turned off only by deleting the project
DThey're on, and billed by consumption, for every project
Show the answer and why
AThey're off until an administrator enables them per project
Incorrect
They're enabled by default for all Foundry projects.
BThey're free, since the playground isn't billed separately
Incorrect
Playground evaluations are included in consumption-based billing.
CThey can be turned off only by deleting the project
Incorrect
You turn them off by unselecting all evaluators under metrics in the playground.
DThey're on, and billed by consumption, for every project
Correct
They're enabled for all projects and billed; unselect all evaluators to stop them.
Evaluations in the agents playground are enabled by default for all Foundry projects and are included in consumption-based billing. To turn off playground evaluations, select metrics in the upper right of the agents playground and unselect all evaluators.