Machine Learning Associate · Domain 3: Model Development · 31% of the exam
Task 3.2: Use Hyperopt's fmin operation to tune a model's hyperparameters; Perform random or grid search or Bayesian search as a method for tuning hyperparameters; Parallelize single node models for hyperparameter tuning
Objective functions and search spaces, grid, random and adaptive search, parallel trials with Optuna on Spark and the trade-off with adaptivity, and what replaced Hyperopt on Databricks.
Study it
Hyperparameter tuning: grid, random and adaptive search, parallel trials
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A team still on Databricks Runtime 16.4 LTS ML tunes a Spark MLlib GBTClassifier with Hyperopt's fmin and tpe.suggest on a cluster with 200 worker cores, and it wants the tuning to finish as soon as possible. Which trials setting should it pass to fmin?
ATrials(), because MLlib already parallelizes each model's training on the cluster
BSparkTrials(parallelism=8), so that eight GBT models are trained at once on the workers
CTrials() with max_queue_len=8, so that eight GBT models are trained at the same time
DSparkTrials(parallelism=200), one trial for each of the cluster's 200 cores
Show the answer and why
ATrials(), because MLlib already parallelizes each model's training on the cluster
Correct
For distributed training algorithms such as MLlib, Databricks says to use the default Hyperopt Trials class rather than SparkTrials.
BSparkTrials(parallelism=8), so that eight GBT models are trained at once on the workers
Incorrect
SparkTrials is designed to distribute single-machine models such as scikit-learn; it should not be used with MLlib.
CTrials() with max_queue_len=8, so that eight GBT models are trained at the same time
Incorrect
max_queue_len is how many hyperparameter settings Hyperopt generates ahead of time; it does not train models concurrently.
DSparkTrials(parallelism=200), one trial for each of the cluster's 200 cores
Incorrect
SparkTrials parallelism is capped at 128, and SparkTrials is not meant for MLlib models in any case.
fmin takes the objective (fn), the search space, the search algorithm (rand.suggest for random search or tpe.suggest for TPE), max_evals and a trials object. SparkTrials distributes single-machine models such as scikit-learn; for MLlib or Horovod, use Trials. Hyperopt is no longer maintained and is not included in Databricks Runtime ML after 16.4 LTS ML; Databricks recommends Optuna, which can run parallel studies on Spark, for a similar experience.
A data scientist tunes a gradient boosting classifier with an Optuna study. Her objective trains a model with values from trial.suggest_int and trial.suggest_float and ends with return roc_auc. After 32 trials, the best parameters the study reports belong to one of the weakest models. What should she change?
ARaise n_trials, so that the sampler has enough trials to reach the highest AUC
BReturn -roc_auc from the objective, because the study minimizes the objective value by default
CLog roc_auc with mlflow.log_metric inside the objective, because the study ranks trials by their logged metrics
DUse Hyperopt's fmin with tpe.suggest instead, because it maximizes the value that the objective returns
Show the answer and why
ARaise n_trials, so that the sampler has enough trials to reach the highest AUC
Incorrect
More trials do not change the direction of the search; the study would keep moving toward lower AUC.
BReturn -roc_auc from the objective, because the study minimizes the objective value by default
Correct
Databricks' tutorial negates the AUC for exactly this reason: Optuna minimizes the objective by default.
CLog roc_auc with mlflow.log_metric inside the objective, because the study ranks trials by their logged metrics
Incorrect
The study ranks trials by the objective value that the function returns, not by metrics logged to MLflow.
DUse Hyperopt's fmin with tpe.suggest instead, because it maximizes the value that the objective returns
Incorrect
fmin also minimizes, treating the returned value as a loss, and Hyperopt is not included in Databricks Runtime ML after 16.4 LTS ML.
An Optuna objective defines the search space with trial.suggest_* calls and returns the objective value linked with the trial. The study minimizes that value by default, so to maximize a metric such as AUC the objective returns its negative, as Databricks' getting-started tutorial does. Hyperopt's fmin also minimizes the loss returned by its objective.
On Databricks Runtime 17.3 LTS ML, a data scientist tunes a scikit-learn model over 32 trials and wants four trials at a time to run on the cluster's workers, with every trial stored in an MLflow experiment. Which approach fits?
AMlflowSparkStudy with MlflowStorage and optimize(objective, n_trials=4)
Boptuna.create_study(storage=MlflowStorage(...)) and study.optimize(objective, n_trials=32, n_jobs=4)
CA ParamGridBuilder grid built with parallelism=4
DMlflowSparkStudy with MlflowStorage and optimize(objective, n_trials=32, n_jobs=4)
Show the answer and why
AMlflowSparkStudy with MlflowStorage and optimize(objective, n_trials=4)
Incorrect
n_trials is the number of calls of the objective in total, so this runs only four trials; n_jobs sets how many run at once.
Boptuna.create_study(storage=MlflowStorage(...)) and study.optimize(objective, n_trials=32, n_jobs=4)
Incorrect
MlflowStorage only keeps the study in the MLflow tracking server; launching the trials on PySpark executors is what MlflowSparkStudy adds.
CA ParamGridBuilder grid built with parallelism=4
Incorrect
ParamGridBuilder only builds the grid; parallelism is a setting of CrossValidator and TrainValidationSplit.
DMlflowSparkStudy with MlflowStorage and optimize(objective, n_trials=32, n_jobs=4)
Correct
MlflowSparkStudy launches parallel Optuna trials on PySpark executors, and MlflowStorage keeps the study in the MLflow tracking server.
For single-node libraries such as scikit-learn, MLflow's Optuna integration distributes trials: MlflowSparkStudy runs them on Spark executors, n_trials sets how many trials run in total and n_jobs how many run at once, and MlflowStorage uses the MLflow tracking server as the storage backend. Hyperopt and SparkTrials are gone after 16.4 LTS ML, and CrossValidator parallelism applies to Spark ML estimators.
A data scientist wants to use MLflow's Optuna integration for tuning on a cluster running Databricks Runtime 16.4 LTS ML. What does the Databricks Optuna page say to do first on a runtime this old?
ANothing; MLflow 3.0 and Optuna are pre-installed on every ML runtime
BInstall Hyperopt, which the integration uses underneath
CInstall the latest MLflow and Optuna with %pip in the notebook
DEnable Photon on the cluster
Show the answer and why
ANothing; MLflow 3.0 and Optuna are pre-installed on every ML runtime
Incorrect
MLflow 3.0 is pre-installed only in Databricks Runtime 17.1 ML and above.
BInstall Hyperopt, which the integration uses underneath
Incorrect
The integration is built on Optuna, not Hyperopt.
CInstall the latest MLflow and Optuna with %pip in the notebook
Correct
On runtimes older than 17.1 ML, the page says to install the latest Optuna and MLflow.
DEnable Photon on the cluster
Incorrect
Photon is a query engine setting; it does not provide MLflow 3.0 or Optuna.
MLflow 3.0 adds Optuna integration classes for hyperparameter tuning, and it is pre-installed in Databricks Runtime 17.1 ML and above. On older runtimes, the Optuna page says to run %pip install mlflow --upgrade and %pip install optuna first.
A team already runs Ray on Databricks for distributed Python work and wants scalable hyperparameter search on the same Ray cluster. Which Ray library does the Ray on Databricks page name for hyperparameter tuning?
ARay Core
BLakeflow Jobs
CDelta Lake
DRay Tune
Show the answer and why
ARay Core
Incorrect
The page names Ray Core for parallel mathematical computations, not for hyperparameter search.
BLakeflow Jobs
Incorrect
Lakeflow Jobs automates workflows; it is not a Ray library.
CDelta Lake
Incorrect
Delta Lake is the data layer that Ray applications can read from, not a tuning library.
DRay Tune
Correct
The page lists Ray Tune for performant and scalable hyperparameter search.
Ray on Databricks lets you use Ray's machine learning libraries: Ray Tune for performant and scalable hyperparameter search, plus distributed deep learning training and distributed training, evaluation and batch inference for traditional ML models.
A CrossValidator over a parameter grid evaluates one parameter map at a time. The data scientist wants several maps evaluated in parallel on the cluster. What does Spark's ML Tuning guide advise?
ASet parallelism to 2 or more; a value up to 10 is generally sufficient for most clusters
BSet parallelism to 1, which evaluates every map at once
CSet parallelism to the total number of grid points, however large
DSet parallelism equal to numFolds, since each fold needs its own task
Show the answer and why
ASet parallelism to 2 or more; a value up to 10 is generally sufficient for most clusters
Correct
Parameter maps are evaluated serially by default, and parallelism of 2 or more runs them in parallel; up to 10 suits most clusters.
BSet parallelism to 1, which evaluates every map at once
Incorrect
A parallelism of 1 is serial evaluation, the default.
CSet parallelism to the total number of grid points, however large
Incorrect
Larger values may not improve performance; parallelism should not exceed cluster resources.
DSet parallelism equal to numFolds, since each fold needs its own task
Incorrect
parallelism is about how many parameter maps are evaluated at once and is sized to the cluster, not to the folds.
By default, CrossValidator and TrainValidationSplit evaluate the parameter maps in serial. Setting parallelism to 2 or more evaluates them in parallel; choose it so as not to exceed cluster resources, since larger values may not help, and a value up to 10 should be sufficient for most clusters.
A team tunes a Spark ML pipeline with CrossValidator on Databricks Runtime ML and wants the tuning runs tracked in MLflow. What does the Databricks hyperparameter tuning page say to use?
AMLlib automated MLflow tracking, which is on by default
Bmlflow.pyspark.ml.autolog()
CHyperopt's SparkTrials, which logs nested runs
DWriting each fold's metric to a Delta table by hand
Show the answer and why
AMLlib automated MLflow tracking, which is on by default
Incorrect
MLlib automated MLflow tracking is deprecated and disabled by default on 10.4 LTS ML and above.
Bmlflow.pyspark.ml.autolog()
Correct
Databricks says to use mlflow.pyspark.ml.autolog(), which is enabled by default with Databricks Autologging.
CHyperopt's SparkTrials, which logs nested runs
Incorrect
SparkTrials belongs to Hyperopt, which is no longer included after 16.4 LTS ML, and it does not track CrossValidator.
DWriting each fold's metric to a Delta table by hand
Incorrect
MLflow autologging already tracks the tuning runs.
MLlib automated MLflow tracking is deprecated and disabled by default on Databricks Runtime 10.4 LTS ML and above. Instead, use MLflow PySpark ML autologging by calling mlflow.pyspark.ml.autolog(), which is enabled by default with Databricks Autologging.