Task 6.2: Evaluate agent performance using MLflow scoring and tracing; Identify evaluation judges that require ground truth; Use Databricks custom Scorers for evaluating agents and LLMs; Incorporate SME feedback to improve agent performance

• ACorrectness

  Correct

  Correctness compares the response with the provided ground truth, so it needs expectations.

• BRetrievalGroundedness

  Incorrect

  Groundedness compares the response with the retrieved context and needs no ground truth.

• CRelevanceToQuery

  Incorrect

  Relevance compares the response with the request and needs no ground truth.

• DRetrievalSufficiency

  Correct

  RetrievalSufficiency checks whether the context contains the ground-truth facts, so it needs expectations.

• ESafety

  Incorrect

  Safety inspects the content itself and needs no ground truth.

Among built-in judges, Correctness, RetrievalSufficiency and ToolCallCorrectness require ground truth in expectations. RelevanceToQuery, RetrievalRelevance, Safety, RetrievalGroundedness and Guidelines need only inputs and outputs.

• AA Guidelines judge stating that each answer must be JSON citing chunks the retriever returned

  Incorrect

  Guidelines judges use an LLM to grade natural-language rules, so results are not deterministic and ids are not checked against the trace exactly.

• BA custom judge built with make_judge that reads the full trace of each request

  Incorrect

  A custom judge is still an LLM-based scorer; it adds nuance the check does not need and loses determinism.

• CA code-based @scorer that parses the output and checks its citations against the retriever span

  Correct

  Code-based scorers receive outputs and the full trace, so they can check format and cited ids exactly, with no LLM involved.

• DThe Correctness judge with the expected citations stored as expectations per row

  Incorrect

  Correctness is an LLM judge of answer quality, which is neither a format check nor deterministic.

Use code-based scorers for programmatic, deterministic checks such as format validation and exact matching. They can return booleans, numbers or Feedback objects and can inspect intermediate steps through the trace.

• AWorkspace access and Databricks SQL access entitlements, and CAN_EDIT on the experiment

  Correct

  The Review App loads traces through workspace routes and queries a SQL warehouse, and reviewers need CAN_EDIT on the experiment.

• BCAN QUERY on the agent's serving endpoint and nothing else

  Incorrect

  Querying the endpoint is not what labeling existing traces requires.

• CWorkspace admin rights so they can see every experiment

  Incorrect

  Admin rights grant far more than labeling needs; the specific entitlements and experiment permission suffice.

• DSELECT on the Unity Catalog trace tables only, with no workspace entitlement assigned to them

  Incorrect

  Account provisioning and table access are not enough; the Review App needs the Workspace access entitlement.

Domain experts labeling existing traces must be provisioned in the account, have the Workspace access and Databricks SQL access entitlements, and have CAN_EDIT permission on the MLflow experiment.

• AReturn a pandas DataFrame for each call it makes

  Incorrect

  It must return a JSON-serializable dictionary.

• BRun without MLflow tracing, so that evaluate() adds it

  Incorrect

  predict_fn must be instrumented with tracing.

• CAccept the inputs keys as kwargs; emit one trace per call

  Correct

  predict_fn must accept the inputs keys as kwargs, return a JSON-serializable dict, be traced and emit exactly one trace per call.

• DTake a Spark DataFrame holding all the rows at once

  Incorrect

  evaluate() calls predict_fn per row with keyword arguments.

In direct evaluation, mlflow.genai.evaluate() calls predict_fn for each row and applies scorers to the resulting traces.

• AAnswer sheet evaluation

  Correct

  Answer sheet evaluation applies scorers to pre-computed outputs or existing traces.

• BDirect evaluation

  Incorrect

  Direct evaluation calls the app through predict_fn.

• CEndpoint load testing

  Incorrect

  Load tests measure throughput, not quality.

• DBacktesting

  Incorrect

  This is not an MLflow evaluation mode.

mlflow.genai.evaluate() supports direct evaluation (recommended, calls your app) and answer sheet evaluation (scores outputs you provide).

• AWith an environment variable only, never in the judge definition

  Incorrect

  The documented way is the judge's model argument.

• BIt cannot be changed

  Incorrect

  Each judge's model can be changed.

• CBy editing the judge prompt

  Incorrect

  Prompt text does not select the model.

• DWith model="databricks:/<model-name>"

  Correct

  Judges accept a model argument in the format <provider>:/<model-name>.

By default judges use a Databricks-hosted LLM tuned for quality assessment; the model argument overrides it.

• ABuilt-in judges cannot be aligned

  Incorrect

  Alignment applies to built-in and custom judges.

• BThe feedback name must match the judge name, relevance_to_query

  Correct

  The human feedback assessment name must exactly match the judge's name attribute.

• CAlignment needs a million traces

  Incorrect

  At least 10 traces work; 50 to 100 are better.

• DFeedback must be stored as expectations on an evaluation dataset, not on traces

  Incorrect

  Alignment uses human feedback assessments on traces.

Judge alignment generates assessments, collects human corrections and calls align() to build a judge closer to human standards.