Generative AI Engineer Associate · Domain 3: Application Development · 30% of the exam
Task 3.3: Qualitatively assess responses to identify common issues such as quality and safety; Implement LLM guardrails to prevent negative outcomes; Compare the evaluation and monitoring phases of the Gen AI application life cycle
Reading traces for hallucination, irrelevance and unsafe output, safety and guideline judges, guardrails at the gateway and in code, and what changes between development evaluation and production monitoring.
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
In support chats, users give their order number early, yet several turns later the assistant asks for it again. Each turn is traced with inputs that hold the full messages array of the conversation so far. The team wants a judge in mlflow.genai.evaluate that flags these answers, without writing expected answers. Which judge fits?
AA Guidelines judge whose rule forbids asking for details the user already gave
BRelevanceToQuery, which compares each answer with the user's latest question
CRetrievalGroundedness, which checks each answer against the retrieved context
DCorrectness, with each turn's expected answer written into the dataset
Show the answer and why
AA Guidelines judge whose rule forbids asking for details the user already gave
Correct
When the inputs hold more than one message, the Guidelines judge receives the whole messages array as the request, so a rule can check the answer against what the user said earlier, with no ground truth.
BRelevanceToQuery, which compares each answer with the user's latest question
Incorrect
Relevance asks whether the response addresses the request; an answer that asks again for the order number can still be on topic.
CRetrievalGroundedness, which checks each answer against the retrieved context
Incorrect
The order number came from the user, not from a retriever, so groundedness in retrieved context does not test it.
DCorrectness, with each turn's expected answer written into the dataset
Incorrect
Correctness needs expected answers or facts, which the team does not want to write.
Reading the traces shows a memory failure in the answers, not a retrieval or relevance failure. The Guidelines judge builds its request from the inputs: a single message becomes its content, and several messages become the whole array serialized as JSON, so a pass or fail rule about earlier turns can be checked on every trace.
A bank's production policy allows only generally available platform features. Its agent must never return account numbers, which sometimes appear in retrieved documents. Which guardrail meets the policy?
APII detection in AI Guardrails on the legacy model serving endpoint
BA built-in PII service policy attached to the Unity Gateway model service that the agent calls
CA post-processing step in the agent that redacts account-number patterns before replying
DThe Safety judge in mlflow.genai.evaluate run before each release
Show the answer and why
APII detection in AI Guardrails on the legacy model serving endpoint
Incorrect
AI Guardrails on serving endpoints are in Public Preview, which the bank's policy excludes.
BA built-in PII service policy attached to the Unity Gateway model service that the agent calls
Incorrect
Unity Gateway is generally available, but service policies are in Beta and need a preview to be turned on.
CA post-processing step in the agent that redacts account-number patterns before replying
Correct
Post-processing guardrails in the chain check output before it reaches the user and depend on no preview feature.
DThe Safety judge in mlflow.genai.evaluate run before each release
Incorrect
An evaluation judge scores responses offline; it does not block anything a user receives in production, and it targets harmful content rather than PII.
Guardrails can sit in the chain itself (post-processing that enforces format and content rules) or at the gateway. On Databricks today the gateway-level PII guardrails are AI Guardrails (Public Preview) and service policies (Beta).
After launch, reviewers find logged production traces in which the agent quotes a retired refund policy. The team changes the prompt and needs evidence before the next release that the fix works on those cases and breaks nothing else. Which plan uses the evaluation side of the lifecycle well?
ARelease the change and watch whether new production traces still quote the retired policy
BRe-score the original traces with the same judges, since they show the problem
CAdd those traces to the evaluation dataset and evaluate the changed agent on all of it
DReplace the evaluation dataset with the failing traces so the run targets the bug
Show the answer and why
ARelease the change and watch whether new production traces still quote the retired policy
Incorrect
That only learns about the fix after users meet it, which is what the team wants to avoid.
BRe-score the original traces with the same judges, since they show the problem
Incorrect
Those traces hold the old outputs, so scoring them again says nothing about the changed prompt.
CAdd those traces to the evaluation dataset and evaluate the changed agent on all of it
Correct
Problem cases from production become test cases, and running the whole dataset checks the fix and guards against regressions.
DReplace the evaluation dataset with the failing traces so the run targets the bug
Incorrect
Dropping the other examples removes the regression check the team asked for.
Evaluation happens during development on a curated set; monitoring happens on production traffic. Evaluation datasets can be built from existing traces, so problems found in production feed back into the set used to test fixes and prevent regressions before a release.
A user reports a wrong answer about overtime rules. The trace shows the retriever returned, as its first result, the chunk that states the correct rule, and the final answer contradicts that chunk. Which conclusion and next step follow from the trace?
ATreat it as a generation failure and revise how the prompt presents retrieved context
BTreat it as a retrieval failure and re-chunk the source documents into smaller, more precise chunks
CTreat it as an embedding failure and re-embed the index with a larger embedding model
DTreat it as a source-data gap and add more overtime documents to the corpus
Show the answer and why
ATreat it as a generation failure and revise how the prompt presents retrieved context
Correct
The trace shows retrieval found the right content, so the root cause lies after retrieval, in prompt augmentation or the model.
BTreat it as a retrieval failure and re-chunk the source documents into smaller, more precise chunks
Incorrect
The correct chunk was already ranked first, so smaller chunks do not address the error.
CTreat it as an embedding failure and re-embed the index with a larger embedding model
Incorrect
Better embeddings change what is retrieved; the right chunk was already retrieved.
DTreat it as a source-data gap and add more overtime documents to the corpus
Incorrect
The corpus already holds the correct rule; more documents add noise without fixing the generation step.
Production traces record inputs, outputs and intermediate steps such as retrieval, so a reviewer can tell whether a low-quality answer comes from the retrieval step or from the model ignoring or misreading good context.
Each row of an evaluation dataset carries its own rule, such as "must mention the refund deadline" for one question and "must not quote prices" for another. Which built-in judge applies these per-example criteria?
AExpectationsGuidelines
BCorrectness
CRetrievalSufficiency
DToolCallEfficiency
Show the answer and why
AExpectationsGuidelines
Correct
ExpectationsGuidelines checks whether the response meets per-example natural-language criteria supplied as guidelines in expectations.
BCorrectness
Incorrect
Correctness compares answers with ground truth rather than checking row-specific rules.
CRetrievalSufficiency
Incorrect
This checks whether retrieved context holds the needed facts.
DToolCallEfficiency
Incorrect
This checks for redundant tool calls.
Built-in judges include Guidelines for criteria shared by every row and ExpectationsGuidelines for criteria that vary by example and are stored in each row's expectations.
A developer is building a RAG assistant for tax rules but is not a tax expert, so the developer cannot tell whether many answers are right. Which component of evaluation does Databricks describe for this gap?
AA larger judge model that replaces expert review
BMore test questions written by the developer
CA review interface where experts give feedback
DProduction latency and error-rate dashboards
Show the answer and why
AA larger judge model that replaces expert review
Incorrect
LLM judges scale assessment, but they do not remove the need for expert feedback on domain quality.
BMore test questions written by the developer
Incorrect
A bigger set written without expertise still leaves the developer unable to judge the answers.
CA review interface where experts give feedback
Correct
A stakeholder-facing UI lets domain experts interact with the app and give detailed feedback the developer cannot provide.
DProduction latency and error-rate dashboards
Incorrect
Operational metrics say nothing about whether tax answers are correct.
Databricks lists the components of evaluation and monitoring: an evaluation set, metric definitions, LLM judges, an evaluation harness, a stakeholder-facing UI for expert feedback, and production trace logging. The UI exists because developers are often not experts in the content.
Why does Databricks say evaluating a RAG application needs both human feedback and LLM judges, unlike classical ML?
ALLM outputs are deterministic, so one reference answer works
BClassical metrics cost too much to compute for every answer
CHuman feedback replaces automated metrics for GenAI apps
DAnswers can be right in many ways, so fixed metrics fall short
Show the answer and why
ALLM outputs are deterministic, so one reference answer works
Incorrect
The opposite is the case for open-ended generation.
BClassical metrics cost too much to compute for every answer
Incorrect
The issue is fit, not cost.
CHuman feedback replaces automated metrics for GenAI apps
Incorrect
Humans do not scale to every response; judges are needed too.
DAnswers can be right in many ways, so fixed metrics fall short
Correct
GenAI answers are non-deterministic and some are more right than others, so human feedback builds confidence and LLM judges scale evaluation.
In classical ML an answer is deterministically right or wrong. In GenAI there are many right answers, which calls for human feedback plus LLM-judged metrics.
A post-processing step cross-references each answer with the retrieved documents and finds an answer that fails the check. What fallback does Databricks describe?
AReturn the answer unchanged and log a warning for review
BGenerate an alternative response or add a disclaimer
CDelete the retrieved documents that the answer cited
DRaise the temperature and resend the same prompt
Show the answer and why
AReturn the answer unchanged and log a warning for review
Incorrect
Logging alone still delivers an unsupported answer.
BGenerate an alternative response or add a disclaimer
Correct
Fallback mechanisms for answers that fail factual checks include generating alternative responses or giving the user a disclaimer.
CDelete the retrieved documents that the answer cited
Incorrect
The documents are the evidence; deleting them does not fix the answer.
DRaise the temperature and resend the same prompt
Incorrect
More randomness does not make answers more factual.
Hallucination handling can live in post-processing by cross-referencing output with retrieved documents or using another LLM to validate it, with fallbacks when checks fail.