Machine Learning Associate · Domain 1: Databricks Machine Learning · 38% of the exam
Task 1.2: Identify how AutoML facilitates model/feature selection; Identify the advantages AutoML brings to the model development process
How AutoML prepares data, splits it, tunes several algorithms and picks the best trial, the generated notebooks you can review and rerun, and where AutoML runs today.
Study it
AutoML: data preparation, trials, generated notebooks and where it runs
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
On Databricks Runtime 17.3 LTS ML, a data scientist starts an AutoML classification run with the Python API. The integer column store_id holds store identifiers, and she passes imputers={"store_id": {"strategy": "constant", "fill_value": -1}} to fill its missing values. The trial notebooks treat store_id as a numeric feature. How can she have AutoML treat store_id as categorical while keeping her imputation?
AChange the strategy for store_id to "most_frequent", which keeps semantic type detection on
BSet its semantic type annotation to categorical before calling AutoML
CAdd store_id to exclude_cols so that AutoML encodes it separately
DSet the semantic type annotation of store_id to native in the column metadata
Show the answer and why
AChange the strategy for store_id to "most_frequent", which keeps semantic type detection on
Incorrect
Any non-default imputation method turns off semantic type detection for that column, so the column would still be treated as numeric.
BSet its semantic type annotation to categorical before calling AutoML
Correct
A semantic type annotation sets the type manually, so it applies even though the custom imputer turned off automatic detection for the column.
CAdd store_id to exclude_cols so that AutoML encodes it separately
Incorrect
exclude_cols tells AutoML to ignore the column during its calculations, so it would not be a feature at all.
DSet the semantic type annotation of store_id to native in the column metadata
Incorrect
The native annotation disables semantic type detection on the column; it does not mark the column as categorical.
AutoML normally detects that a numeric column holding categorical IDs is a categorical feature, but it skips semantic type detection for any column with a custom imputation method. A manual annotation (spark.contentAnnotation.semanticType set to categorical) assigns the type regardless; the keyword native disables detection.
After an AutoML regression experiment on Databricks Runtime 17.3 LTS ML, a data scientist wants to modify and rerun the code of the trial that ranked third. Her experiment folder holds only the best trial notebook and the data exploration notebook. How can she get the third trial's notebook into her workspace?
ARead notebook_path from that trial's TrialInfo in the AutoMLSummary and open that path
BRerun the experiment with experiment_dir set to her folder so that every trial notebook is imported
COpen the data exploration notebook, which contains the code of every trial
DCall databricks.automl.import_notebook with that trial's artifact_uri and a workspace path
Show the answer and why
ARead notebook_path from that trial's TrialInfo in the AutoMLSummary and open that path
Incorrect
For classification and regression, notebook_path is set only for the best trial; it is None for the other trials.
BRerun the experiment with experiment_dir set to her folder so that every trial notebook is imported
Incorrect
For classification and regression, only the best trial and data exploration notebooks are imported, whatever folder is used.
COpen the data exploration notebook, which contains the code of every trial
Incorrect
The data exploration notebook holds summary statistics about the dataset, not the code of the trials.
DCall databricks.automl.import_notebook with that trial's artifact_uri and a workspace path
Correct
Notebooks of the other trials are saved as MLflow artifacts, and import_notebook imports one into the workspace from its artifact URI.
For classification and regression experiments, AutoML imports the data exploration notebook and the best trial notebook into the workspace and saves the notebooks of the other trials as MLflow artifacts. You can import one with databricks.automl.import_notebook (or from the experiment UI); TrialInfo sets notebook_path and notebook_url only for the best trial.
A data scientist calls databricks.automl.classify on Databricks Runtime 17.3 LTS ML without passing primary_metric. Which metric does AutoML use to evaluate and rank the trials?
Aaccuracy
Broc_auc
Clog_loss
Df1
Show the answer and why
Aaccuracy
Incorrect
accuracy is a supported option, but it is not the default.
Broc_auc
Incorrect
roc_auc is a supported option, but it is not the default.
Clog_loss
Incorrect
log_loss is a supported option, but it is not the default.
Df1
Correct
primary_metric defaults to f1 for classification.
primary_metric is the metric used to evaluate and rank model performance. For classification it supports f1 (the default), log_loss, precision, accuracy and roc_auc.
An AutoML classification run on Databricks Runtime 17.3 LTS ML reports that it sampled the large training dataset. The data scientist wants AutoML to use a larger sample. What does the AutoML documentation suggest for this runtime?
ASet max_trials to a lower value
BPass the dataset as a pandas DataFrame
CChoose a worker instance type with more total memory
DSet pos_label to the minority class
Show the answer and why
ASet max_trials to a lower value
Incorrect
max_trials is not supported in Databricks Runtime 11.0 ML and above.
BPass the dataset as a pandas DataFrame
Incorrect
AutoML estimates memory needs and samples whatever form the dataset takes.
CChoose a worker instance type with more total memory
Correct
From Databricks Runtime 12.2 LTS ML, AutoML allocates more CPU cores per task, and you increase the sample size with more total memory.
DSet pos_label to the minority class
Incorrect
pos_label names the positive class for metrics such as precision and recall; it does not change sampling.
AutoML trains each model on a single worker node, so it estimates the memory needed to load and train the dataset and samples it if necessary. In Databricks Runtime 12.2 LTS ML and above, AutoML handles larger datasets by allocating more CPU cores per training task, and you increase the sample size by selecting an instance with more total memory.
A data scientist wants to run AutoML with its built-in library. Which Databricks Runtime should her classic compute use?
ADatabricks Runtime 18.0 ML, the newest version with AutoML
BAny standard Databricks Runtime, since AutoML is part of every runtime
CDatabricks Runtime 19 ML, with AutoML installed by %pip
DDatabricks Runtime 17.3 LTS ML
Show the answer and why
ADatabricks Runtime 18.0 ML, the newest version with AutoML
Incorrect
AutoML is not included as a built-in library in Databricks Runtime 18.0 ML or above.
BAny standard Databricks Runtime, since AutoML is part of every runtime
Incorrect
AutoML depends on the libraries of Databricks Runtime for Machine Learning.
CDatabricks Runtime 19 ML, with AutoML installed by %pip
Incorrect
AutoML is not built into Databricks Runtime 18.0 ML or above.
DDatabricks Runtime 17.3 LTS ML
Correct
17.3 LTS ML still includes AutoML, which is not built into 18.0 ML or above.
AutoML is not included as a built-in library in Databricks Runtime 18.0 ML or above (the 18.0 ML release notes say it has been removed), so AutoML work uses a runtime that includes it, such as 17.3 LTS ML.
A data scientist opens the best trial notebook of an AutoML regression run and wants Shapley value (SHAP) plots of feature importance, which the notebook does not show. What should she do?
ARerun AutoML with the explainability option of the AutoML UI turned on
BSet shap_enabled = True in the notebook's feature importance section and rerun the notebook
CInstall the shap package on the cluster and restart it
DOpen the data exploration notebook, which holds the SHAP plots
Show the answer and why
ARerun AutoML with the explainability option of the AutoML UI turned on
Incorrect
The SHAP code is already in the generated notebook; it is enabled there, not in the experiment setup.
BSet shap_enabled = True in the notebook's feature importance section and rerun the notebook
Correct
SHAP values are not calculated by default because they are memory intensive; setting shap_enabled = True turns them on.
CInstall the shap package on the cluster and restart it
Incorrect
The notebooks already include the SHAP code; it is skipped by default, not missing.
DOpen the data exploration notebook, which holds the SHAP plots
Incorrect
Feature importance code is in the trial notebooks, not in the data exploration notebook.
AutoML regression and classification notebooks include code to calculate Shapley values, which estimate each feature's importance. The calculation is memory intensive and is not run by default: go to the Feature importance section, set shap_enabled = True and rerun the notebook.
Before an AutoML run on Databricks Runtime 17.3 LTS ML, an engineer wants to upgrade scikit-learn on the cluster to a newer release. What does the AutoML documentation say about this?
AAutoML picks up the newer release and uses it in every trial
BUpgrades are allowed if the engineer also installs databricks-automl-runtime
COnly downgrades cause failures; upgrades are safe
DAny change to the pre-installed library versions makes AutoML runs fail
Show the answer and why
AAutoML picks up the newer release and uses it in every trial
Incorrect
Changing library versions breaks AutoML runs rather than improving them.
BUpgrades are allowed if the engineer also installs databricks-automl-runtime
Incorrect
databricks-automl-runtime simplifies the generated notebooks; it does not make other version changes safe.
COnly downgrades cause failures; upgrades are safe
Incorrect
Removals, upgrades and downgrades all cause failures.
DAny change to the pre-installed library versions makes AutoML runs fail
Correct
No libraries beyond those pre-installed in Databricks Runtime ML should be installed, and any modification to existing versions results in run failures.
AutoML requires that no libraries other than those pre-installed in Databricks Runtime for Machine Learning be installed on the cluster. Any modification of existing library versions, whether removal, upgrade or downgrade, results in run failures due to incompatibility.
When calling the AutoML Python API, a data scientist wonders whether to set data_dir to a DBFS folder. What does Databricks recommend?
ALeave it empty so the dataset is saved as an MLflow artifact
BSet it to a folder that every user can write to
CSet it to the workspace folder that holds the generated notebooks
DSet it to a Unity Catalog table name
Show the answer and why
ALeave it empty so the dataset is saved as an MLflow artifact
Correct
Databricks recommends leaving data_dir empty; a custom path does not inherit the experiment's access permissions.
BSet it to a folder that every user can write to
Incorrect
A custom path does not inherit the AutoML experiment's permissions, which is why Databricks recommends leaving it empty.
CSet it to the workspace folder that holds the generated notebooks
Incorrect
Notebooks go to experiment_dir; data_dir is a DBFS path for the training dataset.
DSet it to a Unity Catalog table name
Incorrect
data_dir takes a DBFS path in the form dbfs:/<folder-name>, not a table name.
data_dir is an optional DBFS path for storing the training dataset. Databricks recommends leaving it empty so that AutoML saves the dataset as an MLflow artifact; with a custom path, the dataset does not inherit the AutoML experiment's access permissions.