Data Engineer Professional · Domain 1: Developing Code for Data Processing using Python and SQL · 23% of the exam
Task 1.10: Choose appropriate compute and configuration for environments and dependencies — including serverless compute (serverless environments, dependency management, performance mode), high-memory notebook tasks, and auto-optimization settings (e.g., disallowing retries).
Serverless environment versions, standard and performance-optimized modes, memory settings for serverless notebooks, serverless auto-optimization and retries, and when classic compute is needed.
Study it
Serverless environments, performance mode, memory and retries
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A serverless job task posts invoices to a partner API that has no idempotency key, so a second attempt would bill customers twice. The task has no retries configured, yet the team saw it attempted twice after a transient failure. The task must run at most once per job run. What should the data engineer do?
ASet the job's maximum concurrent runs to 1 and turn queueing off
BClear Enable serverless auto-optimization in the Retry Policy
CTurn on performance optimized mode for the serverless job
DSet a task timeout of a few minutes
Show the answer and why
ASet the job's maximum concurrent runs to 1 and turn queueing off
Incorrect
This limits runs of the job, not attempts of a task within one run.
BClear Enable serverless auto-optimization in the Retry Policy
Correct
Serverless auto-optimization can retry failed tasks; turning it off is the documented setting for workloads that must run at most once.
CTurn on performance optimized mode for the serverless job
Incorrect
Performance mode affects start-up and run speed; it does not stop a failed task from being retried.
DSet a task timeout of a few minutes
Incorrect
A timeout ends a long attempt; the platform can still retry the task after a failure.
Serverless compute for jobs has auto-optimization on by default; it adjusts the compute and retries failed tasks. For jobs that are not idempotent, Databricks documents turning auto-optimization off in the task's retry policy so that the task is attempted only once.
Notebooks that ran on Databricks Runtime for Machine Learning are being moved to serverless compute. They rely on the ML libraries that runtime preinstalls. Which serverless setting fits?
AThe Standard base environment with no dependencies
BAn init script that installs the ML runtime packages
CA compute policy that selects an ML runtime version
DThe ML base environment in the Environment side pane
Show the answer and why
AThe Standard base environment with no dependencies
Incorrect
Standard has Databricks-provided libraries, not the ML runtime set.
BAn init script that installs the ML runtime packages
Incorrect
Serverless compute does not run init scripts.
CA compute policy that selects an ML runtime version
Incorrect
Serverless does not support compute policies.
DThe ML base environment in the Environment side pane
Correct
The ML base environment preinstalls the Python and system packages of Databricks Runtime for Machine Learning.
For serverless notebooks, the base environment sets preinstalled libraries. The ML base environment is meant for migrating Databricks Runtime for Machine Learning workloads to serverless.
A classic jobs cluster uses spot instances to cut cost. The data engineer wants to avoid losing the whole run if AWS reclaims capacity. Which configuration follows Databricks guidance?
AA driver taken from a pool of spot instances
BSpot instances for the driver and all workers
CSpot workers with an on-demand driver
DA single-node cluster on a spot instance
Show the answer and why
AA driver taken from a pool of spot instances
Incorrect
A reclaimed driver ends the run; use an on-demand driver.
BSpot instances for the driver and all workers
Incorrect
Losing the driver loses the run, so the driver should be on-demand.
CSpot workers with an on-demand driver
Correct
With spot enabled, the first instance (the driver) is on-demand, and Databricks says not to use a spot pool for the driver.
DA single-node cluster on a spot instance
Incorrect
Its only node is the driver, so a reclaim ends the run.
When spot instances are used, the first instance is the on-demand driver and later instances are spot. Do not use a pool with spot instances as the driver type, so the driver is not reclaimed.
A scheduled nightly notebook runs on a shared all-purpose cluster that is kept up all day for it. It must stay on classic compute because of a custom cluster setting. How can the team lower its cost?
ARun it on classic jobs compute defined in the job
BKeep the all-purpose cluster and add more workers to finish sooner
CTurn off autoscaling on the shared all-purpose cluster
DMove the notebook into a Git folder
Show the answer and why
ARun it on classic jobs compute defined in the job
Correct
Classic jobs compute is part of the job definition and runs for the job; Databricks does not recommend all-purpose compute for production jobs.
BKeep the all-purpose cluster and add more workers to finish sooner
Incorrect
More workers raise cost on a cluster that already sits idle.
CTurn off autoscaling on the shared all-purpose cluster
Incorrect
That does not stop the cluster from running all day.
DMove the notebook into a Git folder
Incorrect
Source location does not change compute cost.
For production jobs that must stay on classic compute, use classic jobs compute configured in the job. All-purpose compute is meant for interactive work, and Databricks does not recommend it for production jobs.
A serverless Python script task needs a newer Python version and newer preinstalled libraries than it has today. Where is this chosen?
AThe Databricks Runtime version of the job cluster
BThe environment version of the task's environment
CThe Performance optimized setting of the job
DThe job's maximum concurrent runs
Show the answer and why
AThe Databricks Runtime version of the job cluster
Incorrect
Serverless tasks have no job cluster or runtime version to pick.
BThe environment version of the task's environment
Correct
The environment version sets the Python version and the preinstalled libraries for serverless tasks.
CThe Performance optimized setting of the job
Incorrect
Performance mode affects speed and cost, not versions.
DThe job's maximum concurrent runs
Incorrect
Concurrency has nothing to do with library versions.
Each serverless job task runs in an environment whose version sets the Python and Scala versions and the preinstalled libraries, plus any dependencies the task adds.