Generative AI Engineer Associate · Domain 6: Evaluation and Monitoring · 12% of the exam
Task 6.1: Select an LLM choice (size and architecture) based on a set of quantitative evaluation metrics; Select key metrics to monitor for a specific LLM deployment scenario; Use Databricks features to control LLM costs
Reading evaluation metrics to pick model size, latency and token metrics to watch, and cost levers: pay-per-token versus provisioned throughput, scale to zero, batch inference and rate limits.
Study it
Choosing model size from evaluation metrics; metrics to monitor
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
Evaluation of a two-step chain gives these results. Intent step: 8B model 97 percent accuracy, 70B model 98 percent. Answer step: 8B model 71 percent Correctness, 70B model 89 percent. The target is at least 85 percent Correctness at the lowest cost, and latency allows one large-model call per request. Which configuration should the team deploy?
A8B for both steps to minimize cost
B70B for both steps to maximize quality
C70B for the intent step and 8B for the answer step
D8B for intent, 70B for answers
Show the answer and why
A8B for both steps to minimize cost
Incorrect
The 8B answer step reaches 71 percent, well short of the 85 percent target.
B70B for both steps to maximize quality
Incorrect
This meets quality but spends two large-model calls per request, beyond the latency allowance and at higher cost.
C70B for the intent step and 8B for the answer step
Incorrect
This puts the large model where it adds one point and the small model where it misses the target.
D8B for intent, 70B for answers
Correct
The 8B model matches the 70B model on intent, while only the 70B model meets the Correctness target, and this uses one large call per request.
Quantitative evaluation per component shows where model size matters. Start small, scale up where metrics demand it, and use different models for different steps.
Token spend on a Unity Gateway model service spiked after one team's batch script, running as a service principal, started calling it in a tight loop. Interactive users must not be affected. What should the engineer configure?
AA lower service-level rate limit for all traffic
BA token-based rate limit on the MCP services that the batch team's agents call
CScale to zero on the model service during business hours
DA custom tokens-per-minute limit for that service principal
Show the answer and why
AA lower service-level rate limit for all traffic
Incorrect
The service limit is a global maximum; lowering it throttles every user, including interactive ones.
BA token-based rate limit on the MCP services that the batch team's agents call
Incorrect
MCP services support only requests-per-minute limits, and the spend is on the model service.
CScale to zero on the model service during business hours
Incorrect
Scale to zero is a serving-endpoint idle setting; it does not limit one consumer's token use.
DA custom tokens-per-minute limit for that service principal
Correct
Custom rate limits can target individual users or service principals, and token-based limits cap consumption on model services.
Rate limits on Unity Gateway services can be set for the whole service, as a per-user default, or as custom limits for users, service principals and groups, using QPM or TPM for model services.
A custom model endpoint must handle 40 queries per second, and each request takes 0.5 seconds of model time. What provisioned concurrency does the Databricks formula give?
A80
B40
C20
D0.5
Show the answer and why
A80
Incorrect
This divides by the execution time instead of multiplying.
B40
Incorrect
This ignores execution time.
C20
Correct
Provisioned concurrency equals QPS times model execution time: 40 x 0.5 = 20.
D0.5
Incorrect
This is the execution time alone.
Estimate concurrency with QPS x execution time and confirm it with a load test.