Generative AI Engineer Associate · Domain 3: Application Development · 30% of the exam
Task 3.4: Select the best LLM based on the attributes of the application to be developed; Select a model from a model hub or marketplace for a task based on model metadata/model cards; Select the best model for a given task based on common metrics generated in experiments
Context window, latency, cost, tool calling and license as selection criteria, Marketplace and model cards, and comparing candidate models on MLflow evaluation runs.
Study it
Choosing an LLM: context, latency, cost, model cards and experiment metrics
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
Twelve candidate versions of an agent, each logged as an MLflow LoggedModel with a different model behind it, were evaluated on the same dataset. In code, the team wants the versions whose correctness score on that dataset is at least 0.85, best first. Which call fits?
Amlflow.search_runs filtered on models.model_id
Bmlflow.search_logged_models with filter and order_by
Cmlflow.get_logged_model with the metric threshold
Dmlflow.genai.evaluate run again on all twelve versions
Show the answer and why
Amlflow.search_runs filtered on models.model_id
Incorrect
This returns the runs that used one given model as input or output; it does not rank models by a metric.
Bmlflow.search_logged_models with filter and order_by
Correct
search_logged_models filters Logged Models on metrics, parameters or tags and can order them by a metric computed on a named dataset.
Cmlflow.get_logged_model with the metric threshold
Incorrect
get_logged_model fetches a single Logged Model by its model_id.
Dmlflow.genai.evaluate run again on all twelve versions
Incorrect
Evaluating again produces new results; the metrics are already logged and only need to be queried.
A LoggedModel ties together the metrics, parameters and traces of one application version, so versions can be compared and the most performant one found. search_logged_models queries them programmatically.
For a commercial Spanish contract-summary product, the team compares two open models found in Databricks Marketplace. Model 1's card restricts it to non-commercial research. Model 2's card shows an Apache 2.0 license and a base architecture listed for provisioned throughput. The product needs dedicated capacity. Which choice fits?
AModel 2, registered in Unity Catalog and served on a provisioned throughput endpoint
BModel 1, because Marketplace listings are cleared by Databricks for any use
CModel 1 behind a pay-per-token endpoint, since pay-per-token pricing replaces the model license
DModel 2 on a CPU custom model serving endpoint with scale to zero
Show the answer and why
AModel 2, registered in Unity Catalog and served on a provisioned throughput endpoint
Correct
Its license permits commercial use, and provisioned throughput serves supported base architectures, including models obtained from Marketplace, with dedicated capacity.
BModel 1, because Marketplace listings are cleared by Databricks for any use
Incorrect
Customers are responsible for complying with model terms; a listing does not override a non-commercial license.
CModel 1 behind a pay-per-token endpoint, since pay-per-token pricing replaces the model license
Incorrect
The serving mode does not change a model's license, and pay-per-token serves Databricks-hosted models rather than arbitrary uploads.
DModel 2 on a CPU custom model serving endpoint with scale to zero
Incorrect
Large language models belong on Foundation Model APIs rather than CPU custom serving, and scale to zero conflicts with dedicated capacity.
Model cards carry the license, architecture and intended use. Provisioned throughput supports base models from Marketplace or Hugging Face registered in Unity Catalog, and you remain responsible for applicable model terms.
Three candidate models were evaluated on the same MLflow evaluation dataset. Requirements: Correctness pass rate of at least 85 percent and p95 latency under 2 seconds, then lowest cost. Model X: 91 percent, 3.1 s, high cost. Model Y: 87 percent, 1.6 s, medium cost. Model Z: 79 percent, 0.7 s, low cost. Which model should be chosen?
AModel X
BModel Z
CModel X, since Correctness outweighs latency
DModel Y
Show the answer and why
AModel X
Incorrect
X has the best pass rate but misses the latency limit and costs the most.
BModel Z
Incorrect
Z is fastest and cheapest but falls short of the Correctness threshold.
CModel X, since Correctness outweighs latency
Incorrect
The requirements make both thresholds mandatory, and X misses the latency one.
DModel Y
Correct
Y is the only model that meets both the 85 percent pass rate and the 2-second latency limit.
Running every candidate through mlflow.genai.evaluate() on the same dataset produces comparable judge results; pick the cheapest model that clears the quality and latency thresholds.
Two candidate chat models score the same on an evaluation dataset, and the one that is cheaper per request should win. The evaluation produced an MLflow trace for every row. Where can the team read each request's input and output token counts?
AIn the judge assessments attached to each trace
BIn the parameters of the evaluation run
CIn trace.info.token_usage on each trace
DIn the model signature of the logged agent
Show the answer and why
AIn the judge assessments attached to each trace
Incorrect
Assessments hold judge scores and rationales, not the token counts of the model calls.
BIn the parameters of the evaluation run
Incorrect
Run parameters hold configuration values; token counts are recorded on each trace.
CIn trace.info.token_usage on each trace
Correct
MLflow Tracing tracks token usage of LLM calls from the counts that provider APIs return, readable as trace.info.token_usage.
DIn the model signature of the logged agent
Incorrect
A signature describes input and output schemas, not per-request usage.
A trace captures each step with its inputs, outputs, latency and token usage, so candidates with equal quality scores can be compared on cost and latency from the same evaluation.
A team wants a quick proof of concept with a general-purpose LLM before deciding whether to invest in anything custom. Which Foundation Model APIs mode is recommended for starting?
APay-per-token
BReserved provisioned throughput
COn-demand provisioned throughput
DA custom GPU serving endpoint
Show the answer and why
APay-per-token
Correct
Pay-per-token is the easiest way to start and suits proofs of concept before investing in a custom model.
BReserved provisioned throughput
Incorrect
Reserved capacity is a fixed-term commitment for predictable production use.
COn-demand provisioned throughput
Incorrect
On-demand provisioned throughput allocates dedicated capacity, which is more than a first experiment needs.
DA custom GPU serving endpoint
Incorrect
Self-hosting an LLM is the custom investment the team wants to postpone.
Foundation Model APIs pay-per-token offers preconfigured endpoints that need no deployment of your own, which suits getting started and quick prototypes.
A partner provider announces that a model a team calls through Foundation Model APIs pay-per-token will retire in two weeks, which is shorter than the transition Databricks publishes. What happens to the team's requests?
AThey start failing on the date the partner set, two weeks out
BThey keep working on the retired model indefinitely
CThey move to the newest model, whatever its price
DRedirected to a compatible version for the full transition
Show the answer and why
AThey start failing on the date the partner set, two weeks out
Incorrect
Databricks redirects requests so customers keep the full one-month transition.
BThey keep working on the retired model indefinitely
Incorrect
At the end of the full transition period, queries fail.
CThey move to the newest model, whatever its price
Incorrect
Redirection happens only to a replacement with the same price that is backwards compatible.
DRedirected to a compatible version for the full transition
Correct
Databricks bridges short partner notice by temporarily redirecting to a similar version at the same price that is backwards compatible.
Partner models follow the Databricks deprecation policy, with retirement dates announced at least one month ahead. When a partner gives less notice, Databricks redirects temporarily; teams should still migrate before the period ends.
A team installs an AI model listing from Databricks Marketplace and fine-tunes the model. Registering the tuned version into the catalog that the listing created fails. Why, and where should the version go?
AReinstall the listing so its catalog grants write access
BThe shared catalog is read-only; use a catalog the team owns
CGet CAN MANAGE on a model serving endpoint before registering
DEach catalog may hold only one version of a model
Show the answer and why
AReinstall the listing so its catalog grants write access
Incorrect
Marketplace shares reach consumers as read-only catalogs; installing again does not change that.
BThe shared catalog is read-only; use a catalog the team owns
Correct
Marketplace datasets and models are shared in a read-only catalog, so derived model versions belong in the team's own catalog.
CGet CAN MANAGE on a model serving endpoint before registering
Incorrect
Serving endpoint permissions do not govern registering a model in a catalog.
DEach catalog may hold only one version of a model
Incorrect
Version counts are not the issue; consumers cannot write to a shared catalog at all.
Marketplace assets arrive as read-only catalogs governed by Unity Catalog. A derived asset, such as a fine-tuned model, is registered in a catalog and schema where the team has the right privileges.
A team shortlists an Anthropic model on Foundation Model APIs for long contract reviews and checks its context window against the provider's own model documentation. What does Databricks state about the context window and maximum output tokens of OpenAI, Google Gemini and Anthropic models?
ADatabricks caps them at 128K tokens for every model
BThey depend on the workspace's pricing tier
CThey match the values the model provider publishes
DThey are halved when the model is served on Databricks
Show the answer and why
ADatabricks caps them at 128K tokens for every model
Incorrect
There is no common cap; the supported models page lists windows well above 128K for several models.
BThey depend on the workspace's pricing tier
Incorrect
Pricing tier does not change a model's context window or output limit.
CThey match the values the model provider publishes
Correct
For these partner models, the supported models page states that the context window and maximum output tokens match the provider's values.
DThey are halved when the model is served on Databricks
Incorrect
Serving through Databricks does not shrink the provider's limits.
The supported models page describes each model's attributes and links to model cards and terms. For OpenAI, Google Gemini and Anthropic models, the context window and maximum output tokens are the ones the provider publishes, so the provider documentation can be used to check fit.