Skip to content

Machine Learning Associate · Domain 1: Databricks Machine Learning · 38% of the exam

Task 1.2: Identify how AutoML facilitates model/feature selection; Identify the advantages AutoML brings to the model development process

How AutoML prepares data, splits it, tunes several algorithms and picks the best trial, the generated notebooks you can review and rerun, and where AutoML runs today.

Study it

  • AutoML: data preparation, trials, generated notebooks and where it runs

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

On Databricks Runtime 17.3 LTS ML, a data scientist starts an AutoML classification run with the Python API. The integer column store_id holds store identifiers, and she passes imputers={"store_id": {"strategy": "constant", "fill_value": -1}} to fill its missing values. The trial notebooks treat store_id as a numeric feature. How can she have AutoML treat store_id as categorical while keeping her imputation?

  1. AChange the strategy for store_id to "most_frequent", which keeps semantic type detection on
  2. BSet its semantic type annotation to categorical before calling AutoML
  3. CAdd store_id to exclude_cols so that AutoML encodes it separately
  4. DSet the semantic type annotation of store_id to native in the column metadata
Show the answer and why
  • AChange the strategy for store_id to "most_frequent", which keeps semantic type detection on

    Incorrect

    Any non-default imputation method turns off semantic type detection for that column, so the column would still be treated as numeric.

  • BSet its semantic type annotation to categorical before calling AutoML

    Correct

    A semantic type annotation sets the type manually, so it applies even though the custom imputer turned off automatic detection for the column.

  • CAdd store_id to exclude_cols so that AutoML encodes it separately

    Incorrect

    exclude_cols tells AutoML to ignore the column during its calculations, so it would not be a feature at all.

  • DSet the semantic type annotation of store_id to native in the column metadata

    Incorrect

    The native annotation disables semantic type detection on the column; it does not mark the column as categorical.

AutoML normally detects that a numeric column holding categorical IDs is a categorical feature, but it skips semantic type detection for any column with a custom imputation method. A manual annotation (spark.contentAnnotation.semanticType set to categorical) assigns the type regardless; the keyword native disables detection.

Question 2 · choose 1

After an AutoML regression experiment on Databricks Runtime 17.3 LTS ML, a data scientist wants to modify and rerun the code of the trial that ranked third. Her experiment folder holds only the best trial notebook and the data exploration notebook. How can she get the third trial's notebook into her workspace?

  1. ARead notebook_path from that trial's TrialInfo in the AutoMLSummary and open that path
  2. BRerun the experiment with experiment_dir set to her folder so that every trial notebook is imported
  3. COpen the data exploration notebook, which contains the code of every trial
  4. DCall databricks.automl.import_notebook with that trial's artifact_uri and a workspace path
Show the answer and why
  • ARead notebook_path from that trial's TrialInfo in the AutoMLSummary and open that path

    Incorrect

    For classification and regression, notebook_path is set only for the best trial; it is None for the other trials.

  • BRerun the experiment with experiment_dir set to her folder so that every trial notebook is imported

    Incorrect

    For classification and regression, only the best trial and data exploration notebooks are imported, whatever folder is used.

  • COpen the data exploration notebook, which contains the code of every trial

    Incorrect

    The data exploration notebook holds summary statistics about the dataset, not the code of the trials.

  • DCall databricks.automl.import_notebook with that trial's artifact_uri and a workspace path

    Correct

    Notebooks of the other trials are saved as MLflow artifacts, and import_notebook imports one into the workspace from its artifact URI.

For classification and regression experiments, AutoML imports the data exploration notebook and the best trial notebook into the workspace and saves the notebooks of the other trials as MLflow artifacts. You can import one with databricks.automl.import_notebook (or from the experiment UI); TrialInfo sets notebook_path and notebook_url only for the best trial.

Question 3 · choose 1

A data scientist calls databricks.automl.classify on Databricks Runtime 17.3 LTS ML without passing primary_metric. Which metric does AutoML use to evaluate and rank the trials?

  1. Aaccuracy
  2. Broc_auc
  3. Clog_loss
  4. Df1
Show the answer and why
  • Aaccuracy

    Incorrect

    accuracy is a supported option, but it is not the default.

  • Broc_auc

    Incorrect

    roc_auc is a supported option, but it is not the default.

  • Clog_loss

    Incorrect

    log_loss is a supported option, but it is not the default.

  • Df1

    Correct

    primary_metric defaults to f1 for classification.

primary_metric is the metric used to evaluate and rank model performance. For classification it supports f1 (the default), log_loss, precision, accuracy and roc_auc.

Question 4 · choose 1

An AutoML classification run on Databricks Runtime 17.3 LTS ML reports that it sampled the large training dataset. The data scientist wants AutoML to use a larger sample. What does the AutoML documentation suggest for this runtime?

  1. ASet max_trials to a lower value
  2. BPass the dataset as a pandas DataFrame
  3. CChoose a worker instance type with more total memory
  4. DSet pos_label to the minority class
Show the answer and why
  • ASet max_trials to a lower value

    Incorrect

    max_trials is not supported in Databricks Runtime 11.0 ML and above.

  • BPass the dataset as a pandas DataFrame

    Incorrect

    AutoML estimates memory needs and samples whatever form the dataset takes.

  • CChoose a worker instance type with more total memory

    Correct

    From Databricks Runtime 12.2 LTS ML, AutoML allocates more CPU cores per task, and you increase the sample size with more total memory.

  • DSet pos_label to the minority class

    Incorrect

    pos_label names the positive class for metrics such as precision and recall; it does not change sampling.

AutoML trains each model on a single worker node, so it estimates the memory needed to load and train the dataset and samples it if necessary. In Databricks Runtime 12.2 LTS ML and above, AutoML handles larger datasets by allocating more CPU cores per training task, and you increase the sample size by selecting an instance with more total memory.

Question 5 · choose 1

A data scientist wants to run AutoML with its built-in library. Which Databricks Runtime should her classic compute use?

  1. ADatabricks Runtime 18.0 ML, the newest version with AutoML
  2. BAny standard Databricks Runtime, since AutoML is part of every runtime
  3. CDatabricks Runtime 19 ML, with AutoML installed by %pip
  4. DDatabricks Runtime 17.3 LTS ML
Show the answer and why
  • ADatabricks Runtime 18.0 ML, the newest version with AutoML

    Incorrect

    AutoML is not included as a built-in library in Databricks Runtime 18.0 ML or above.

  • BAny standard Databricks Runtime, since AutoML is part of every runtime

    Incorrect

    AutoML depends on the libraries of Databricks Runtime for Machine Learning.

  • CDatabricks Runtime 19 ML, with AutoML installed by %pip

    Incorrect

    AutoML is not built into Databricks Runtime 18.0 ML or above.

  • DDatabricks Runtime 17.3 LTS ML

    Correct

    17.3 LTS ML still includes AutoML, which is not built into 18.0 ML or above.

AutoML is not included as a built-in library in Databricks Runtime 18.0 ML or above (the 18.0 ML release notes say it has been removed), so AutoML work uses a runtime that includes it, such as 17.3 LTS ML.

Question 6 · choose 1

A data scientist opens the best trial notebook of an AutoML regression run and wants Shapley value (SHAP) plots of feature importance, which the notebook does not show. What should she do?

  1. ARerun AutoML with the explainability option of the AutoML UI turned on
  2. BSet shap_enabled = True in the notebook's feature importance section and rerun the notebook
  3. CInstall the shap package on the cluster and restart it
  4. DOpen the data exploration notebook, which holds the SHAP plots
Show the answer and why
  • ARerun AutoML with the explainability option of the AutoML UI turned on

    Incorrect

    The SHAP code is already in the generated notebook; it is enabled there, not in the experiment setup.

  • BSet shap_enabled = True in the notebook's feature importance section and rerun the notebook

    Correct

    SHAP values are not calculated by default because they are memory intensive; setting shap_enabled = True turns them on.

  • CInstall the shap package on the cluster and restart it

    Incorrect

    The notebooks already include the SHAP code; it is skipped by default, not missing.

  • DOpen the data exploration notebook, which holds the SHAP plots

    Incorrect

    Feature importance code is in the trial notebooks, not in the data exploration notebook.

AutoML regression and classification notebooks include code to calculate Shapley values, which estimate each feature's importance. The calculation is memory intensive and is not run by default: go to the Feature importance section, set shap_enabled = True and rerun the notebook.

Databricks documentation

Question 7 · choose 1

Before an AutoML run on Databricks Runtime 17.3 LTS ML, an engineer wants to upgrade scikit-learn on the cluster to a newer release. What does the AutoML documentation say about this?

  1. AAutoML picks up the newer release and uses it in every trial
  2. BUpgrades are allowed if the engineer also installs databricks-automl-runtime
  3. COnly downgrades cause failures; upgrades are safe
  4. DAny change to the pre-installed library versions makes AutoML runs fail
Show the answer and why
  • AAutoML picks up the newer release and uses it in every trial

    Incorrect

    Changing library versions breaks AutoML runs rather than improving them.

  • BUpgrades are allowed if the engineer also installs databricks-automl-runtime

    Incorrect

    databricks-automl-runtime simplifies the generated notebooks; it does not make other version changes safe.

  • COnly downgrades cause failures; upgrades are safe

    Incorrect

    Removals, upgrades and downgrades all cause failures.

  • DAny change to the pre-installed library versions makes AutoML runs fail

    Correct

    No libraries beyond those pre-installed in Databricks Runtime ML should be installed, and any modification to existing versions results in run failures.

AutoML requires that no libraries other than those pre-installed in Databricks Runtime for Machine Learning be installed on the cluster. Any modification of existing library versions, whether removal, upgrade or downgrade, results in run failures due to incompatibility.

Databricks documentation

Question 8 · choose 1

When calling the AutoML Python API, a data scientist wonders whether to set data_dir to a DBFS folder. What does Databricks recommend?

  1. ALeave it empty so the dataset is saved as an MLflow artifact
  2. BSet it to a folder that every user can write to
  3. CSet it to the workspace folder that holds the generated notebooks
  4. DSet it to a Unity Catalog table name
Show the answer and why
  • ALeave it empty so the dataset is saved as an MLflow artifact

    Correct

    Databricks recommends leaving data_dir empty; a custom path does not inherit the experiment's access permissions.

  • BSet it to a folder that every user can write to

    Incorrect

    A custom path does not inherit the AutoML experiment's permissions, which is why Databricks recommends leaving it empty.

  • CSet it to the workspace folder that holds the generated notebooks

    Incorrect

    Notebooks go to experiment_dir; data_dir is a DBFS path for the training dataset.

  • DSet it to a Unity Catalog table name

    Incorrect

    data_dir takes a DBFS path in the form dbfs:/<folder-name>, not a table name.

data_dir is an optional DBFS path for storing the training dataset. Databricks recommends leaving it empty so that AutoML saves the dataset as an MLflow artifact; with a custom path, the dataset does not inherit the AutoML experiment's access permissions.

Practice domain 1 →Practice all domains →