Reading compute event logs and termination codes, init script and library installation failures, driver versus executor out of memory, and the fixes for each.
Study it
Cluster start failures, library conflicts and out-of-memory errors
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A classic jobs cluster fails to start, and the run shows the termination code INIT_SCRIPT_FAILURE for a cluster-scoped init script stored in a volume. Where should the data engineer look for the cause?
AThe Executors tab of the Spark UI for the failed cluster launch
BThe init script logs in the init_scripts folder of the cluster log path
CThe task values that the failed run published for the job
DThe system.lakeflow.job_run_timeline rows recorded for this failed run
Show the answer and why
AThe Executors tab of the Spark UI for the failed cluster launch
Incorrect
The cluster never started, so there is no Spark application and no Spark UI for this launch.
BThe init script logs in the init_scripts folder of the cluster log path
Correct
Init script stdout and stderr logs are written to <log-path>/<cluster-id>/init_scripts when cluster log delivery is configured.
CThe task values that the failed run published for the job
Incorrect
Task values hold data a task chose to publish; a task that did not start publishes no values.
DThe system.lakeflow.job_run_timeline rows recorded for this failed run
Incorrect
Jobs system tables hold records about runs, not the stdout and stderr that the init script wrote on the cluster's nodes.
INIT_SCRIPT_FAILURE means a cluster-scoped script could not be loaded or exited non-zero. Enable log delivery if needed, rerun, and read the .stderr.log files to find the failing command.
One notebook on a shared cluster needs a newer version of a Python library than the Databricks Runtime includes. Other notebooks on the same cluster must keep the built-in version. How should the library be installed?
AAs a compute-scoped library on the cluster
BWith a cluster-scoped init script
CWith %pip install in that notebook
DBy switching the cluster to the ML runtime
Show the answer and why
AAs a compute-scoped library on the cluster
Incorrect
Compute-scoped libraries are available to every notebook and job on the compute, so the others would change too.
BWith a cluster-scoped init script
Incorrect
Init scripts change the whole cluster environment and are a common source of library conflicts.
CWith %pip install in that notebook
Correct
Notebook-scoped libraries create an environment for that notebook session only and do not affect other notebooks on the compute.
DBy switching the cluster to the ML runtime
Incorrect
The ML runtime adds a large set of libraries that can conflict with your own and changes the environment for everyone.
Use notebook-scoped %pip installs or environments to avoid conflicts; Databricks advises against init scripts and compute-scoped libraries for this.
A job fails with ExecutorLostFailure messages saying containers may have exceeded thresholds. The compute event log shows no spot losses or autoscaling. Workers have 4 cores and 16 GB each. How can the data engineer confirm that memory is the cause?
ARerun on workers with 4 cores and 32 GB
BRerun with twice as many workers of the same type
CRerun with a larger driver node
DRerun with adaptive query execution disabled
Show the answer and why
ARerun on workers with 4 cores and 32 GB
Correct
Doubling memory per core is the suggested check; if the job fails later or not at all, memory is likely the cause.
BRerun with twice as many workers of the same type
Incorrect
More workers of the same type leave memory per core unchanged.
CRerun with a larger driver node
Incorrect
The lost executors are on the workers; the driver is not involved here.
DRerun with adaptive query execution disabled
Incorrect
Disabling AQE removes optimizations without testing memory.
The ratio of memory to cores matters, not total memory. Then look for causes such as too few shuffle partitions, large broadcasts, UDFs or skew.
A classic jobs cluster fails to start with AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE for its instance type in one availability zone. What should the data engineer try?
AGive the driver node more memory in the cluster configuration
BFix the cluster-scoped init script and start the cluster again
CLower spark.sql.shuffle.partitions in the cluster's Spark config
DUse the auto availability zone setting or another instance type
Show the answer and why
AGive the driver node more memory in the cluster configuration
Incorrect
The cluster could not get instances; driver memory does not affect that.
BFix the cluster-scoped init script and start the cluster again
Incorrect
An init script failure has its own termination code; this error is about cloud capacity.
CLower spark.sql.shuffle.partitions in the cluster's Spark config
Incorrect
Spark settings matter only after the cluster has started.
DUse the auto availability zone setting or another instance type
Correct
The recommended fix is another or automatic availability zone, a different instance type, or on-demand instead of spot.
Capacity errors come from the cloud provider. Workspace admins can also enable flexible node types so Databricks falls back to compatible instance types.
In a notebook, a data engineer imports a library, then runs %pip install to upgrade it. Code in the next cell still uses the old version. What should the engineer do?
ARun the same %pip install command a second time in a new cell
BCall dbutils.library.restartPython() and import the library again
CInstall the library as a compute-scoped library on the cluster instead
DRestart the cluster and run the import cell again
Show the answer and why
ARun the same %pip install command a second time in a new cell
Incorrect
Installing again does not reload a library that the Python process has already imported.
BCall dbutils.library.restartPython() and import the library again
Correct
%pip does not restart the Python process, so restart it to load the newly installed version.
CInstall the library as a compute-scoped library on the cluster instead
Incorrect
This changes the environment for every notebook on the compute and still needs a restart to take effect.
DRestart the cluster and run the import cell again
Incorrect
Restarting the cluster discards notebook-scoped installs, so the old built-in version is imported again.
After %pip changes packages that are already loaded, restart the Python process with dbutils.library.restartPython().