Data Engineer Professional · Domain 1: Developing Code for Data Processing using Python and SQL · 23% of the exam
Task 1.2: Apply troubleshooting techniques to dependency conflicts and installation failures for external libraries (PyPI, local wheels, source archives) across serverless, pipeline, and bundle-deployed environments.
Serverless environment versions and dependencies, notebook-scoped installs, pipeline environments, wheels and requirements files from volumes and workspace files, private repositories, and reading installation errors.
Study it
Library failures: serverless environments, pipeline dependencies, wheels and requirements files
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A serverless job task installs an in-house wheel from a Unity Catalog volume. The wheel contains C extensions and was built on an x86_64 workstation. Some runs install it and succeed, while others fail during dependency installation with an error that the wheel is not supported on this platform. The code did not change between runs. What fixes this?
APin the task to an older serverless environment version for its x86_64 native wheel
BPublish aarch64 and x86_64 builds of the wheel, each selected by a platform_machine marker
CMove the wheel from the Unity Catalog volume into workspace files and install it from there
DInstall the wheel from an init script before the task's Python process starts
Show the answer and why
APin the task to an older serverless environment version for its x86_64 native wheel
Incorrect
An older environment version does not change the CPU architecture that a run lands on.
BPublish aarch64 and x86_64 builds of the wheel, each selected by a platform_machine marker
Correct
Serverless compute can run on either CPU architecture and may change between runs, so a native wheel needs a variant per architecture (or a pure-Python py3-none-any build).
CMove the wheel from the Unity Catalog volume into workspace files and install it from there
Incorrect
Both locations are supported for serverless dependencies; the failure comes from the wheel's platform tag, not from where it is stored.
DInstall the wheel from an init script before the task's Python process starts
Incorrect
Serverless compute does not support init scripts; dependencies are declared in the environment.
Serverless compute does not guarantee a CPU architecture, so a run can land on aarch64 or x86_64. A wheel with native extensions is built for one of them and fails to install on the other. Use pure-Python wheels where you can, or ship both builds and select each with a platform_machine marker.
After adding a package to the Dependencies of a serverless notebook, the session stops with an error. The package lists pyspark among its own requirements. What should the data engineer do?
APin the package's pyspark requirement to the newest release
BInstall the package with %pip in the first cell instead
CRemove the package and reset the notebook environment
DChoose a larger memory setting for the notebook
Show the answer and why
APin the package's pyspark requirement to the newest release
Incorrect
Any PySpark install conflicts with the serverless Spark environment, whatever the version.
BInstall the package with %pip in the first cell instead
Incorrect
The package still installs PySpark, which stops the session.
CRemove the package and reset the notebook environment
Correct
Installing PySpark, or a library that installs it, is not supported on serverless; remove it and reset the environment.
DChoose a larger memory setting for the notebook
Incorrect
Memory does not change a conflict with the installed Spark libraries.
Do not install PySpark, or any library that installs it, on serverless notebooks; doing so stops the session with an error. Remove the library and reset the environment.
A serverless job task must install a package from the company's private Python repository. A notebook cell that sets HTTPS_PROXY before the code runs did not help. Which approach works?
APut --index-url or --extra-index-url in the dependency specification
BSet HTTPS_PROXY as a compute-scoped environment variable in the task settings
CDownload the package inside the notebook with requests and add it to sys.path
DAdd a compute-scoped init script that writes a pip.conf for the repository
Show the answer and why
APut --index-url or --extra-index-url in the dependency specification
Correct
Dependencies are resolved before task code runs, and an index URL in the specification tells the installer where to fetch from.
BSet HTTPS_PROXY as a compute-scoped environment variable in the task settings
Incorrect
Serverless compute does not support compute-scoped environment variables, so proxy settings do not affect installation.
CDownload the package inside the notebook with requests and add it to sys.path
Incorrect
That bypasses dependency management and is not how serverless dependencies are installed.
DAdd a compute-scoped init script that writes a pip.conf for the repository
Incorrect
Serverless compute does not run init scripts.
Serverless resolves and installs dependencies before notebook or task code runs, so code cannot change how they are fetched. Specify index-url or extra-index-url in the dependency, or have a workspace admin configure the repository as a default package repository.
A job moved from Databricks Runtime 14.3 LTS to a newer LTS release fails to install a wheel that is stored in the DBFS root. Where does Databricks recommend storing the wheel?
AIn a DBFS mount point backed by a new S3 bucket
BIn the driver node's local /tmp folder
CIn the job's notebook, embedded as a base64 string
DIn workspace files or a Unity Catalog volume
Show the answer and why
AIn a DBFS mount point backed by a new S3 bucket
Incorrect
DBFS-based library storage is the pattern being retired; volumes and workspace files are recommended.
BIn the driver node's local /tmp folder
Incorrect
Local disk does not persist across clusters and is not a library source.
CIn the job's notebook, embedded as a base64 string
Incorrect
That is not a supported way to install a library.
DIn workspace files or a Unity Catalog volume
Correct
Databricks recommends uploading libraries to workspace files or Unity Catalog volumes, or using package repositories.
Libraries in the DBFS root can be installed on Databricks Runtime 14.3 LTS and below, but storing them there is deprecated and disabled by default from Databricks Runtime 15.1, because any workspace user can modify those files.
A serverless notebook keeps failing after a package upgrade broke its environment. The data engineer removed the package and reconnected, but the problem remains. How do they force a fresh install of the dependencies?
ADetach the notebook and start a new session on serverless
BUse Reset to defaults in the Environment side pane
CClear all cell outputs and run the notebook again
DMove the notebook into a new folder and reattach it
Show the answer and why
ADetach the notebook and start a new session on serverless
Incorrect
Starting a new session does not clear the entire environment cache.
BUse Reset to defaults in the Environment side pane
Correct
Reset to defaults clears the cached virtual environment and reinstalls the listed dependencies.
CClear all cell outputs and run the notebook again
Incorrect
Outputs have nothing to do with the cached Python environment.
DMove the notebook into a new folder and reattach it
Incorrect
The location of the notebook does not reset its environment.
Serverless caches a notebook's virtual environment. If an installed package breaks the environment, remove it and use Reset to defaults to clear the cache and reinstall; a new session alone does not clear it.