Machine Learning Associate · Domain 3: Model Development · 31% of the exam
Task 3.4: Use common classification metrics: F1, Log Loss, ROC/AUC, etc; Use common regression metrics: RMSE, MAE, R-squared, etc; Choose the most appropriate metric for a given scenario objective; Identify the need to exponentiate log-transformed variables before calculating evaluation metrics or interpreting predictions; Assess the impact of model complexity and the bias variance tradeoff on model performance
Evaluator metric names and defaults, which metric fits an objective, back-transforming log-scale predictions before scoring them, and how model complexity moves bias and variance.
Study it
Classification and regression metrics and choosing one
Lesson coming
Log-scale targets and the bias-variance trade-off
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A binary churn model's predictions DataFrame has the columns label, prediction and churn_probs, the vector of class probabilities; it has no column named probability. The team wants the log loss of these predictions. Which evaluator computes it?
MulticlassClassificationEvaluator supports logLoss and reads the class probabilities from probabilityCol, which is used only for logLoss.
Log loss is one of MulticlassClassificationEvaluator's metrics and can be used for a binary label as well. It needs the class conditional probabilities, which the evaluator reads from probabilityCol (default "probability"); probabilities of 0 or 1 are clipped by eps. BinaryClassificationEvaluator offers areaUnderROC and areaUnderPR only.
A data scientist evaluates a house price model with RegressionEvaluator(labelCol="price", predictionCol="pred").evaluate(preds) and reports the returned value as the model's R-squared. What did the call compute, and how does she get R-squared?
AR-squared already, because r2 is the default metric
BMean absolute error; set metricName="r2"
CRMSE, the default; set metricName="r2"
DExplained variance; set metricName="rsquared"
Show the answer and why
AR-squared already, because r2 is the default metric
Incorrect
The default metricName of RegressionEvaluator is rmse, not r2.
BMean absolute error; set metricName="r2"
Incorrect
mae is available but is not the default; the default metric is rmse.
CRMSE, the default; set metricName="r2"
Correct
RegressionEvaluator returns rmse unless metricName is set, and r2 is the name of the R-squared metric.
DExplained variance; set metricName="rsquared"
Incorrect
Explained variance is the var metric, and the R-squared metric is named r2.
RegressionEvaluator supports rmse (the default), mse, r2, mae and var (explained variance). R-squared, the coefficient of determination, is requested with metricName="r2".
A ticket-routing model predicts four classes indexed 0 to 3. The team cares only about class 2, outage, and wants one number that balances how often predicted outages are right and how many real outages are caught, for that class alone. Which evaluation gives it?
Without metricLabel the by-label metric is computed for the default class 0, not class 2.
CMulticlassClassificationEvaluator(metricName="accuracy") evaluated on preds.filter("label = 2.0")
Incorrect
On the real outages alone, accuracy is the share of them that were caught; wrong outage predictions sit in rows with other labels, so precision is left out.
fMeasureByLabel computes the F-measure of the class named by metricLabel, here class 2.
The F-measure combines precision and recall. MulticlassClassificationEvaluator offers label-based metrics (precisionByLabel, recallByLabel, fMeasureByLabel and others) for the class given by metricLabel, whose default is 0, as well as weighted metrics over all classes and accuracy across all labels.
After fitting a binary pyspark.ml LogisticRegression, a data scientist wants to choose the probability threshold that gives the highest F-measure on the training data and apply it to the model. Which approach does that?
ARead model.summary.roc and use the threshold with the lowest false positive rate
BRead model.summary.fMeasureByThreshold and set the threshold with the highest F-Measure
CEvaluate the predictions once with MulticlassClassificationEvaluator using f1, which applies the best threshold
DRead model.summary.precisionByThreshold and set the threshold with the highest precision
Show the answer and why
ARead model.summary.roc and use the threshold with the lowest false positive rate
Incorrect
The lowest false positive rate does not maximize the F-measure, and the roc DataFrame holds rates rather than F-measures.
BRead model.summary.fMeasureByThreshold and set the threshold with the highest F-Measure
Correct
The training summary reports the F-measure at each threshold, so the best one can be found and passed to setThreshold.
CEvaluate the predictions once with MulticlassClassificationEvaluator using f1, which applies the best threshold
Incorrect
The evaluator scores the predictions made at the current threshold; it does not search for a threshold.
DRead model.summary.precisionByThreshold and set the threshold with the highest precision
Incorrect
Maximizing precision alone is not the same as maximizing the F-measure, which also depends on recall.
The training summary of a binary logistic regression model reports metrics by threshold, including fMeasureByThreshold, precisionByThreshold and recallByThreshold, plus the ROC curve. Spark's guide finds the threshold with the maximum F-Measure there and sets it on the model with setThreshold.
A Spark ML regressor was trained with label_log = log1p(price) as its label column. Its predictions DataFrame has the columns price, label_log and prediction. Stakeholders want the RMSE in dollars. Which evaluation is correct? (F is pyspark.sql.functions.)
AAdd pred_price = F.exp("prediction"), then evaluate with RegressionEvaluator(labelCol="price", predictionCol="pred_price")
BCompute the RMSE on the log scale, then report exp(rmse) - 1 as the dollar RMSE
CAdd pred_price = F.exp("prediction") - 1, then evaluate with RegressionEvaluator(labelCol="price", predictionCol="pred_price")
The model predicts on the scale of its training label, log1p of price, so this compares dollars with log-scale values.
Transformed variables lose their original meaning, and a model trained on log1p(price) predicts on that scale. When a transformed variable is the target of a regression, applying the inverse transformation to the predicted values makes them interpretable again: log1p is the natural logarithm of the value plus one, so exp(prediction) - 1 undoes it. Only then does an RMSE against the original label come out in the original units.
A data scientist calls MulticlassClassificationEvaluator().evaluate(preds) without setting metricName. Which metric is returned?
Af1
Baccuracy
ClogLoss
DweightedPrecision
Show the answer and why
Af1
Correct
MulticlassClassificationEvaluator's metricName defaults to f1.
Baccuracy
Incorrect
accuracy is available but must be set with metricName.
ClogLoss
Incorrect
logLoss is available but must be set with metricName.
DweightedPrecision
Incorrect
weightedPrecision is available but must be set with metricName.
MulticlassClassificationEvaluator's default metricName is f1. Other metrics, such as accuracy, weightedPrecision, weightedRecall, logLoss and the by-label metrics, are chosen with metricName.
A data scientist calls BinaryClassificationEvaluator().evaluate(preds) without setting metricName. Which metric is returned?
Af1
BareaUnderPR, the area under the precision-recall curve
CareaUnderROC
Daccuracy
Show the answer and why
Af1
Incorrect
f1 belongs to MulticlassClassificationEvaluator.
BareaUnderPR, the area under the precision-recall curve
Incorrect
areaUnderPR is available but is not the default.
CareaUnderROC
Correct
BinaryClassificationEvaluator's metricName defaults to areaUnderROC.
Daccuracy
Incorrect
accuracy belongs to MulticlassClassificationEvaluator.
BinaryClassificationEvaluator expects a rawPrediction column, a label column and an optional weight column. Its metricName defaults to areaUnderROC, and areaUnderPR is the other option.
A data scientist fits lr_model = LinearRegression().fit(train_df) on a dataset with many features and wants the adjusted R-squared of the model on its training data. Which expression returns it?
r2adj is not one of RegressionEvaluator's metric names.
Dlr_model.summary.numInstances
Incorrect
numInstances is the number of rows in the predictions DataFrame.
A fitted LinearRegressionModel exposes summary, the summary of the model on its training set. That LinearRegressionSummary reports r2, the coefficient of determination, and r2adj, the adjusted coefficient of determination, among other values. Adjusted R-squared is not among RegressionEvaluator's metric names.
A model outputs, for each row, only a double column holding the probability of label 1. Can BinaryClassificationEvaluator use that column as rawPredictionCol?
ANo; rawPrediction must be a vector of raw scores
BNo; only MulticlassClassificationEvaluator accepts probabilities
CYes; rawPrediction can be a double holding a 0/1 prediction or the probability of label 1
DYes, but only after converting it to a string column
Show the answer and why
ANo; rawPrediction must be a vector of raw scores
Incorrect
A double column is also accepted, as a 0/1 prediction or the probability of label 1.
BNo; only MulticlassClassificationEvaluator accepts probabilities
CYes; rawPrediction can be a double holding a 0/1 prediction or the probability of label 1
Correct
The rawPrediction column can be a double (0/1 prediction or probability of label 1) or a length-2 vector.
DYes, but only after converting it to a string column
Incorrect
The column is used as a double or a vector, not as a string.
BinaryClassificationEvaluator expects rawPrediction, label and an optional weight column. The rawPrediction column can be a double (a binary 0/1 prediction or the probability of label 1) or a vector (a length-2 vector of raw predictions, scores or label probabilities).