Skip to content

Machine Learning Associate · Domain 3: Model Development · 31% of the exam

Task 3.2: Use Hyperopt's fmin operation to tune a model's hyperparameters; Perform random or grid search or Bayesian search as a method for tuning hyperparameters; Parallelize single node models for hyperparameter tuning

Objective functions and search spaces, grid, random and adaptive search, parallel trials with Optuna on Spark and the trade-off with adaptivity, and what replaced Hyperopt on Databricks.

Study it

  • Hyperparameter tuning: grid, random and adaptive search, parallel trials

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A team still on Databricks Runtime 16.4 LTS ML tunes a Spark MLlib GBTClassifier with Hyperopt's fmin and tpe.suggest on a cluster with 200 worker cores, and it wants the tuning to finish as soon as possible. Which trials setting should it pass to fmin?

  1. ATrials(), because MLlib already parallelizes each model's training on the cluster
  2. BSparkTrials(parallelism=8), so that eight GBT models are trained at once on the workers
  3. CTrials() with max_queue_len=8, so that eight GBT models are trained at the same time
  4. DSparkTrials(parallelism=200), one trial for each of the cluster's 200 cores
Show the answer and why
  • ATrials(), because MLlib already parallelizes each model's training on the cluster

    Correct

    For distributed training algorithms such as MLlib, Databricks says to use the default Hyperopt Trials class rather than SparkTrials.

  • BSparkTrials(parallelism=8), so that eight GBT models are trained at once on the workers

    Incorrect

    SparkTrials is designed to distribute single-machine models such as scikit-learn; it should not be used with MLlib.

  • CTrials() with max_queue_len=8, so that eight GBT models are trained at the same time

    Incorrect

    max_queue_len is how many hyperparameter settings Hyperopt generates ahead of time; it does not train models concurrently.

  • DSparkTrials(parallelism=200), one trial for each of the cluster's 200 cores

    Incorrect

    SparkTrials parallelism is capped at 128, and SparkTrials is not meant for MLlib models in any case.

fmin takes the objective (fn), the search space, the search algorithm (rand.suggest for random search or tpe.suggest for TPE), max_evals and a trials object. SparkTrials distributes single-machine models such as scikit-learn; for MLlib or Horovod, use Trials. Hyperopt is no longer maintained and is not included in Databricks Runtime ML after 16.4 LTS ML; Databricks recommends Optuna, which can run parallel studies on Spark, for a similar experience.

Question 2 · choose 1

A data scientist tunes a gradient boosting classifier with an Optuna study. Her objective trains a model with values from trial.suggest_int and trial.suggest_float and ends with return roc_auc. After 32 trials, the best parameters the study reports belong to one of the weakest models. What should she change?

  1. ARaise n_trials, so that the sampler has enough trials to reach the highest AUC
  2. BReturn -roc_auc from the objective, because the study minimizes the objective value by default
  3. CLog roc_auc with mlflow.log_metric inside the objective, because the study ranks trials by their logged metrics
  4. DUse Hyperopt's fmin with tpe.suggest instead, because it maximizes the value that the objective returns
Show the answer and why
  • ARaise n_trials, so that the sampler has enough trials to reach the highest AUC

    Incorrect

    More trials do not change the direction of the search; the study would keep moving toward lower AUC.

  • BReturn -roc_auc from the objective, because the study minimizes the objective value by default

    Correct

    Databricks' tutorial negates the AUC for exactly this reason: Optuna minimizes the objective by default.

  • CLog roc_auc with mlflow.log_metric inside the objective, because the study ranks trials by their logged metrics

    Incorrect

    The study ranks trials by the objective value that the function returns, not by metrics logged to MLflow.

  • DUse Hyperopt's fmin with tpe.suggest instead, because it maximizes the value that the objective returns

    Incorrect

    fmin also minimizes, treating the returned value as a loss, and Hyperopt is not included in Databricks Runtime ML after 16.4 LTS ML.

An Optuna objective defines the search space with trial.suggest_* calls and returns the objective value linked with the trial. The study minimizes that value by default, so to maximize a metric such as AUC the objective returns its negative, as Databricks' getting-started tutorial does. Hyperopt's fmin also minimizes the loss returned by its objective.

Question 3 · choose 1

On Databricks Runtime 17.3 LTS ML, a data scientist tunes a scikit-learn model over 32 trials and wants four trials at a time to run on the cluster's workers, with every trial stored in an MLflow experiment. Which approach fits?

  1. AMlflowSparkStudy with MlflowStorage and optimize(objective, n_trials=4)
  2. Boptuna.create_study(storage=MlflowStorage(...)) and study.optimize(objective, n_trials=32, n_jobs=4)
  3. CA ParamGridBuilder grid built with parallelism=4
  4. DMlflowSparkStudy with MlflowStorage and optimize(objective, n_trials=32, n_jobs=4)
Show the answer and why
  • AMlflowSparkStudy with MlflowStorage and optimize(objective, n_trials=4)

    Incorrect

    n_trials is the number of calls of the objective in total, so this runs only four trials; n_jobs sets how many run at once.

  • Boptuna.create_study(storage=MlflowStorage(...)) and study.optimize(objective, n_trials=32, n_jobs=4)

    Incorrect

    MlflowStorage only keeps the study in the MLflow tracking server; launching the trials on PySpark executors is what MlflowSparkStudy adds.

  • CA ParamGridBuilder grid built with parallelism=4

    Incorrect

    ParamGridBuilder only builds the grid; parallelism is a setting of CrossValidator and TrainValidationSplit.

  • DMlflowSparkStudy with MlflowStorage and optimize(objective, n_trials=32, n_jobs=4)

    Correct

    MlflowSparkStudy launches parallel Optuna trials on PySpark executors, and MlflowStorage keeps the study in the MLflow tracking server.

For single-node libraries such as scikit-learn, MLflow's Optuna integration distributes trials: MlflowSparkStudy runs them on Spark executors, n_trials sets how many trials run in total and n_jobs how many run at once, and MlflowStorage uses the MLflow tracking server as the storage backend. Hyperopt and SparkTrials are gone after 16.4 LTS ML, and CrossValidator parallelism applies to Spark ML estimators.

Question 4 · choose 1

A data scientist wants to use MLflow's Optuna integration for tuning on a cluster running Databricks Runtime 16.4 LTS ML. What does the Databricks Optuna page say to do first on a runtime this old?

  1. ANothing; MLflow 3.0 and Optuna are pre-installed on every ML runtime
  2. BInstall Hyperopt, which the integration uses underneath
  3. CInstall the latest MLflow and Optuna with %pip in the notebook
  4. DEnable Photon on the cluster
Show the answer and why
  • ANothing; MLflow 3.0 and Optuna are pre-installed on every ML runtime

    Incorrect

    MLflow 3.0 is pre-installed only in Databricks Runtime 17.1 ML and above.

  • BInstall Hyperopt, which the integration uses underneath

    Incorrect

    The integration is built on Optuna, not Hyperopt.

  • CInstall the latest MLflow and Optuna with %pip in the notebook

    Correct

    On runtimes older than 17.1 ML, the page says to install the latest Optuna and MLflow.

  • DEnable Photon on the cluster

    Incorrect

    Photon is a query engine setting; it does not provide MLflow 3.0 or Optuna.

MLflow 3.0 adds Optuna integration classes for hyperparameter tuning, and it is pre-installed in Databricks Runtime 17.1 ML and above. On older runtimes, the Optuna page says to run %pip install mlflow --upgrade and %pip install optuna first.

Question 5 · choose 1

A team already runs Ray on Databricks for distributed Python work and wants scalable hyperparameter search on the same Ray cluster. Which Ray library does the Ray on Databricks page name for hyperparameter tuning?

  1. ARay Core
  2. BLakeflow Jobs
  3. CDelta Lake
  4. DRay Tune
Show the answer and why
  • ARay Core

    Incorrect

    The page names Ray Core for parallel mathematical computations, not for hyperparameter search.

  • BLakeflow Jobs

    Incorrect

    Lakeflow Jobs automates workflows; it is not a Ray library.

  • CDelta Lake

    Incorrect

    Delta Lake is the data layer that Ray applications can read from, not a tuning library.

  • DRay Tune

    Correct

    The page lists Ray Tune for performant and scalable hyperparameter search.

Ray on Databricks lets you use Ray's machine learning libraries: Ray Tune for performant and scalable hyperparameter search, plus distributed deep learning training and distributed training, evaluation and batch inference for traditional ML models.

Question 6 · choose 1

A CrossValidator over a parameter grid evaluates one parameter map at a time. The data scientist wants several maps evaluated in parallel on the cluster. What does Spark's ML Tuning guide advise?

  1. ASet parallelism to 2 or more; a value up to 10 is generally sufficient for most clusters
  2. BSet parallelism to 1, which evaluates every map at once
  3. CSet parallelism to the total number of grid points, however large
  4. DSet parallelism equal to numFolds, since each fold needs its own task
Show the answer and why
  • ASet parallelism to 2 or more; a value up to 10 is generally sufficient for most clusters

    Correct

    Parameter maps are evaluated serially by default, and parallelism of 2 or more runs them in parallel; up to 10 suits most clusters.

  • BSet parallelism to 1, which evaluates every map at once

    Incorrect

    A parallelism of 1 is serial evaluation, the default.

  • CSet parallelism to the total number of grid points, however large

    Incorrect

    Larger values may not improve performance; parallelism should not exceed cluster resources.

  • DSet parallelism equal to numFolds, since each fold needs its own task

    Incorrect

    parallelism is about how many parameter maps are evaluated at once and is sized to the cluster, not to the folds.

By default, CrossValidator and TrainValidationSplit evaluate the parameter maps in serial. Setting parallelism to 2 or more evaluates them in parallel; choose it so as not to exceed cluster resources, since larger values may not help, and a value up to 10 should be sufficient for most clusters.

Databricks documentation

Question 7 · choose 1

A team tunes a Spark ML pipeline with CrossValidator on Databricks Runtime ML and wants the tuning runs tracked in MLflow. What does the Databricks hyperparameter tuning page say to use?

  1. AMLlib automated MLflow tracking, which is on by default
  2. Bmlflow.pyspark.ml.autolog()
  3. CHyperopt's SparkTrials, which logs nested runs
  4. DWriting each fold's metric to a Delta table by hand
Show the answer and why
  • AMLlib automated MLflow tracking, which is on by default

    Incorrect

    MLlib automated MLflow tracking is deprecated and disabled by default on 10.4 LTS ML and above.

  • Bmlflow.pyspark.ml.autolog()

    Correct

    Databricks says to use mlflow.pyspark.ml.autolog(), which is enabled by default with Databricks Autologging.

  • CHyperopt's SparkTrials, which logs nested runs

    Incorrect

    SparkTrials belongs to Hyperopt, which is no longer included after 16.4 LTS ML, and it does not track CrossValidator.

  • DWriting each fold's metric to a Delta table by hand

    Incorrect

    MLflow autologging already tracks the tuning runs.

MLlib automated MLflow tracking is deprecated and disabled by default on Databricks Runtime 10.4 LTS ML and above. Instead, use MLflow PySpark ML autologging by calling mlflow.pyspark.ml.autolog(), which is enabled by default with Databricks Autologging.

Practice domain 3 →Practice all domains →