Machine Learning Associate · Domain 3: Model Development · 31% of the exam
Task 3.3: Describe the benefits and downsides of using cross-validation over a train-validation split; Perform cross-validation as a part of model fitting; Identify the number of models being trained in conjunction with a grid-search and cross-validation process
CrossValidator and TrainValidationSplit, folds, evaluators and ParamGridBuilder, the refit on the full data, and counting the models a grid and k folds train.
Study it
Cross-validation versus a train-validation split; counting models
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 2
A team compares CrossValidator(numFolds=5) with TrainValidationSplit(trainRatio=0.8) for tuning a Spark ML pipeline over a grid of 12 parameter combinations on a modest dataset. Which two statements are correct? (Choose TWO.)
ACrossValidator fits each combination on five training folds and averages the metric, so the search costs about five times as many fits
BTrainValidationSplit gives more reliable results than CrossValidator when the dataset is small
CCrossValidator returns the model trained in its best fold, without fitting again
DTrainValidationSplit evaluates each combination on several random splits of the data
ETrainValidationSplit evaluates each combination once, which is cheaper but less reliable when the training data is not large enough
Show the answer and why
ACrossValidator fits each combination on five training folds and averages the metric, so the search costs about five times as many fits
Correct
Each parameter map is evaluated on k (training, test) pairs, so k folds multiply the number of fits by k.
BTrainValidationSplit gives more reliable results than CrossValidator when the dataset is small
Incorrect
TrainValidationSplit is cheaper but less reliable when the training dataset is not sufficiently large.
CCrossValidator returns the model trained in its best fold, without fitting again
Incorrect
After choosing the best parameter map, CrossValidator re-fits the estimator on the entire dataset.
DTrainValidationSplit evaluates each combination on several random splits of the data
Incorrect
TrainValidationSplit creates a single (training, validation) pair.
ETrainValidationSplit evaluates each combination once, which is cheaper but less reliable when the training data is not large enough
Correct
It evaluates each combination once instead of k times, at the cost of reliability on smaller data.
CrossValidator splits the data into k folds and averages the metric over the k models fitted for each parameter map, then re-fits the best one on the whole dataset; it is reliable but expensive. TrainValidationSplit uses one split set by trainRatio and evaluates each combination once, which is less expensive but not as reliable when the training data is not sufficiently large.
A data scientist runs the following code, where pipeline ends in a LogisticRegression and grid holds six parameter maps.
cv = CrossValidator(estimator=pipeline, estimatorParamMaps=grid, evaluator=BinaryClassificationEvaluator(), numFolds=3); cv_model = cv.fit(train_df); preds = cv_model.transform(test_df)
Which model produces preds?
AThe model fitted in the fold where the best parameter map scored highest
BAn average of the three fold models for the best parameter map
CA pipeline re-fitted on all of train_df with the best parameter map
DThe pipeline fitted with the first parameter map in the grid
Show the answer and why
AThe model fitted in the fold where the best parameter map scored highest
Incorrect
Fold models are used only to score each parameter map; none of them is the final model.
BAn average of the three fold models for the best parameter map
Incorrect
The metric is averaged across the folds, not the models.
CA pipeline re-fitted on all of train_df with the best parameter map
Correct
After choosing the best parameter map, CrossValidator re-fits the estimator on the entire dataset it was given.
DThe pipeline fitted with the first parameter map in the grid
Incorrect
CrossValidator selects the parameter map with the best average metric, not the first one.
CrossValidator computes the average evaluation metric of the k fold models for each parameter map, picks the best map and re-fits the estimator, here the whole pipeline, on the entire input dataset. The fitted CrossValidatorModel transforms new data with that best model.
A grid is built with ParamGridBuilder().addGrid(lr.regParam, [0.01, 0.1, 1.0]).addGrid(lr.elasticNetParam, [0.0, 0.5]).addGrid(lr.maxIter, [50, 100]).build() and passed to CrossValidator(estimator=lr, numFolds=4). Counting the final refit, how many times is the LogisticRegression fitted when cv.fit runs?
A49, including the final refit
B48, which is 12 combinations times 4 folds
C13, one fit per combination plus the refit
D52, which is 13 fits in each of the 4 folds
Show the answer and why
A49, including the final refit
Correct
The grid has 3 x 2 x 2 = 12 combinations, each fitted on 4 folds (48), and the best one is re-fitted once on the entire dataset.
B48, which is 12 combinations times 4 folds
Incorrect
This leaves out the final refit of the best parameter map on the entire dataset.
C13, one fit per combination plus the refit
Incorrect
CrossValidator fits every combination once per fold, not once in total.
D52, which is 13 fits in each of the 4 folds
Incorrect
The refit happens once, on the entire dataset, not once in each fold.
ParamGridBuilder builds every combination of the listed values (3 x 2 x 2 = 12). CrossValidator fits the estimator for each combination on each of the k folds (12 x 4 = 48) and then re-fits the best combination on the entire dataset, for 49 fits in all. Spark's guide works through the same arithmetic: (3 x 2) x 2 = 12 models for its example.
After cv_model = cv.fit(train_df), where cv is a CrossValidator, a data scientist wants to log the fitted model that cross-validation selected as an MLflow model. Which expression gives her that fitted model?
Acv_model.getEstimator()
Bcv_model.bestModel
Ccv_model.getEvaluator()
Dcv_model.getEstimatorParamMaps()[0]
Show the answer and why
Acv_model.getEstimator()
Incorrect
This returns the estimator param that was passed to CrossValidator, not a fitted model.
Bcv_model.bestModel
Correct
bestModel holds the best model from cross-validation, the one the CrossValidatorModel uses to transform data.
Ccv_model.getEvaluator()
Incorrect
This returns the evaluator that scored the parameter maps.
Dcv_model.getEstimatorParamMaps()[0]
Incorrect
This is the first parameter map of the grid, a set of parameter values rather than a fitted model.
A CrossValidatorModel contains the model with the highest average cross-validation metric across folds and uses it to transform input data; the API exposes that model as bestModel. getEstimator, getEvaluator and getEstimatorParamMaps return the estimator, evaluator and parameter maps that were set on the CrossValidator.
A TrainValidationSplit is created with trainRatio=0.75. How does it divide the data?
AInto four folds, each used once for validation
B75% for validation and 25% for training
C75% for training and 25% for validation
DInto 75 random samples
Show the answer and why
AInto four folds, each used once for validation
Incorrect
TrainValidationSplit makes a single split, not folds.
B75% for validation and 25% for training
Incorrect
trainRatio is the share used for training.
C75% for training and 25% for validation
Correct
With trainRatio=0.75, 75% of the data trains the models and 25% validates them.
DInto 75 random samples
Incorrect
trainRatio is a proportion of the data, not a count of samples.
Unlike CrossValidator, TrainValidationSplit creates a single (training, validation) dataset pair, split by trainRatio: with 0.75, 75% of the data is used for training and 25% for validation.
A data scientist wants all rows of the same customer to land in the same fold, so she adds an integer column fold_id with values 0 to 2 and sets numFolds=3. Which CrossValidator parameter makes it use her folds?
Aseed="fold_id"
BfoldCol="fold_id"
CestimatorParamMaps="fold_id"
Dparallelism=3
Show the answer and why
Aseed="fold_id"
Incorrect
seed controls the random split; it does not name a fold column.
BfoldCol="fold_id"
Correct
foldCol names a column of user-specified fold numbers in [0, numFolds), and CrossValidator then skips the random split.
CestimatorParamMaps="fold_id"
Incorrect
estimatorParamMaps takes the parameter grid, not a fold column.
Dparallelism=3
Incorrect
parallelism sets how many parameter maps are evaluated at once.
CrossValidator's foldCol is the column of user-specified fold numbers. Once it is set, CrossValidator does not do a random k-fold split; the column must be an integer type with values in [0, numFolds).
After cv_model = cv.fit(train_df), a data scientist wants the average validation metric of every parameter map in the grid. Which attribute of the CrossValidatorModel holds it?
Acv_model.getEstimatorParamMaps()
Bcv_model.getNumFolds()
Ccv_model.transform(train_df)
Dcv_model.avgMetrics, a list
Show the answer and why
Acv_model.getEstimatorParamMaps()
Incorrect
This returns the parameter maps themselves, not their metrics.
Bcv_model.getNumFolds()
Incorrect
This returns the number of folds.
Ccv_model.transform(train_df)
Incorrect
transform returns predictions, not the validation metrics.
Dcv_model.avgMetrics, a list
Correct
avgMetrics holds the average evaluation metric of each parameter map, in grid order.
For each parameter map, CrossValidator computes the average evaluation metric over the k fold models. The fitted CrossValidatorModel exposes these averages in avgMetrics, one per parameter map, as in the API reference example.
Before cross-validating, a data scientist holds out a test set with df.randomSplit([8.0, 2.0], seed=42). What happens to the weights?
ASpark raises an error because the weights do not sum to 1.0
BThe first DataFrame gets 8 rows and the second gets 2
CThey are normalized, giving about an 80/20 split that the seed makes repeatable
DThe seed is ignored when the weights are not fractions
Show the answer and why
ASpark raises an error because the weights do not sum to 1.0
Incorrect
Weights that do not sum to 1.0 are normalized, not rejected.
BThe first DataFrame gets 8 rows and the second gets 2
Incorrect
Weights are relative proportions, not row counts.
CThey are normalized, giving about an 80/20 split that the seed makes repeatable
Correct
Weights are normalized if they don't sum to 1.0, and seed sets the seed for sampling.
DThe seed is ignored when the weights are not fractions
Incorrect
The seed applies to the sampling whatever the weights are.
DataFrame.randomSplit splits a DataFrame randomly with the provided weights, which are normalized if they don't sum to 1.0; the optional seed is the seed for sampling, so the same seed gives the same split.