Skip to content

Machine Learning Associate · Domain 3: Model Development · 31% of the exam

Task 3.4: Use common classification metrics: F1, Log Loss, ROC/AUC, etc; Use common regression metrics: RMSE, MAE, R-squared, etc; Choose the most appropriate metric for a given scenario objective; Identify the need to exponentiate log-transformed variables before calculating evaluation metrics or interpreting predictions; Assess the impact of model complexity and the bias variance tradeoff on model performance

Evaluator metric names and defaults, which metric fits an objective, back-transforming log-scale predictions before scoring them, and how model complexity moves bias and variance.

Study it

  • Classification and regression metrics and choosing one

    Lesson coming

  • Log-scale targets and the bias-variance trade-off

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A binary churn model's predictions DataFrame has the columns label, prediction and churn_probs, the vector of class probabilities; it has no column named probability. The team wants the log loss of these predictions. Which evaluator computes it?

  1. AMulticlassClassificationEvaluator(metricName="logLoss")
  2. BBinaryClassificationEvaluator(rawPredictionCol="churn_probs", metricName="logLoss")
  3. CMulticlassClassificationEvaluator(metricName="logLoss", predictionCol="churn_probs")
  4. DMulticlassClassificationEvaluator(metricName="logLoss", probabilityCol="churn_probs")
Show the answer and why
  • AMulticlassClassificationEvaluator(metricName="logLoss")

    Incorrect

    logLoss reads probabilityCol, whose default is the column probability, which this DataFrame does not have.

  • BBinaryClassificationEvaluator(rawPredictionCol="churn_probs", metricName="logLoss")

    Incorrect

    BinaryClassificationEvaluator supports only areaUnderROC and areaUnderPR.

  • CMulticlassClassificationEvaluator(metricName="logLoss", predictionCol="churn_probs")

    Incorrect

    predictionCol holds the predicted labels; the probabilities that log loss needs are read from probabilityCol.

  • DMulticlassClassificationEvaluator(metricName="logLoss", probabilityCol="churn_probs")

    Correct

    MulticlassClassificationEvaluator supports logLoss and reads the class probabilities from probabilityCol, which is used only for logLoss.

Log loss is one of MulticlassClassificationEvaluator's metrics and can be used for a binary label as well. It needs the class conditional probabilities, which the evaluator reads from probabilityCol (default "probability"); probabilities of 0 or 1 are clipped by eps. BinaryClassificationEvaluator offers areaUnderROC and areaUnderPR only.

Question 2 · choose 1

A data scientist evaluates a house price model with RegressionEvaluator(labelCol="price", predictionCol="pred").evaluate(preds) and reports the returned value as the model's R-squared. What did the call compute, and how does she get R-squared?

  1. AR-squared already, because r2 is the default metric
  2. BMean absolute error; set metricName="r2"
  3. CRMSE, the default; set metricName="r2"
  4. DExplained variance; set metricName="rsquared"
Show the answer and why
  • AR-squared already, because r2 is the default metric

    Incorrect

    The default metricName of RegressionEvaluator is rmse, not r2.

  • BMean absolute error; set metricName="r2"

    Incorrect

    mae is available but is not the default; the default metric is rmse.

  • CRMSE, the default; set metricName="r2"

    Correct

    RegressionEvaluator returns rmse unless metricName is set, and r2 is the name of the R-squared metric.

  • DExplained variance; set metricName="rsquared"

    Incorrect

    Explained variance is the var metric, and the R-squared metric is named r2.

RegressionEvaluator supports rmse (the default), mse, r2, mae and var (explained variance). R-squared, the coefficient of determination, is requested with metricName="r2".

Question 3 · choose 1

A ticket-routing model predicts four classes indexed 0 to 3. The team cares only about class 2, outage, and wants one number that balances how often predicted outages are right and how many real outages are caught, for that class alone. Which evaluation gives it?

  1. AMulticlassClassificationEvaluator(metricName="weightedFMeasure", metricLabel=2.0)
  2. BMulticlassClassificationEvaluator(metricName="fMeasureByLabel")
  3. CMulticlassClassificationEvaluator(metricName="accuracy") evaluated on preds.filter("label = 2.0")
  4. DMulticlassClassificationEvaluator(metricName="fMeasureByLabel", metricLabel=2.0)
Show the answer and why
  • AMulticlassClassificationEvaluator(metricName="weightedFMeasure", metricLabel=2.0)

    Incorrect

    metricLabel is read only by the by-label metrics; weightedFMeasure combines the F-measures of all classes, weighted by class.

  • BMulticlassClassificationEvaluator(metricName="fMeasureByLabel")

    Incorrect

    Without metricLabel the by-label metric is computed for the default class 0, not class 2.

  • CMulticlassClassificationEvaluator(metricName="accuracy") evaluated on preds.filter("label = 2.0")

    Incorrect

    On the real outages alone, accuracy is the share of them that were caught; wrong outage predictions sit in rows with other labels, so precision is left out.

  • DMulticlassClassificationEvaluator(metricName="fMeasureByLabel", metricLabel=2.0)

    Correct

    fMeasureByLabel computes the F-measure of the class named by metricLabel, here class 2.

The F-measure combines precision and recall. MulticlassClassificationEvaluator offers label-based metrics (precisionByLabel, recallByLabel, fMeasureByLabel and others) for the class given by metricLabel, whose default is 0, as well as weighted metrics over all classes and accuracy across all labels.

Question 4 · choose 1

After fitting a binary pyspark.ml LogisticRegression, a data scientist wants to choose the probability threshold that gives the highest F-measure on the training data and apply it to the model. Which approach does that?

  1. ARead model.summary.roc and use the threshold with the lowest false positive rate
  2. BRead model.summary.fMeasureByThreshold and set the threshold with the highest F-Measure
  3. CEvaluate the predictions once with MulticlassClassificationEvaluator using f1, which applies the best threshold
  4. DRead model.summary.precisionByThreshold and set the threshold with the highest precision
Show the answer and why
  • ARead model.summary.roc and use the threshold with the lowest false positive rate

    Incorrect

    The lowest false positive rate does not maximize the F-measure, and the roc DataFrame holds rates rather than F-measures.

  • BRead model.summary.fMeasureByThreshold and set the threshold with the highest F-Measure

    Correct

    The training summary reports the F-measure at each threshold, so the best one can be found and passed to setThreshold.

  • CEvaluate the predictions once with MulticlassClassificationEvaluator using f1, which applies the best threshold

    Incorrect

    The evaluator scores the predictions made at the current threshold; it does not search for a threshold.

  • DRead model.summary.precisionByThreshold and set the threshold with the highest precision

    Incorrect

    Maximizing precision alone is not the same as maximizing the F-measure, which also depends on recall.

The training summary of a binary logistic regression model reports metrics by threshold, including fMeasureByThreshold, precisionByThreshold and recallByThreshold, plus the ROC curve. Spark's guide finds the threshold with the maximum F-Measure there and sets it on the model with setThreshold.

Question 5 · choose 1

A Spark ML regressor was trained with label_log = log1p(price) as its label column. Its predictions DataFrame has the columns price, label_log and prediction. Stakeholders want the RMSE in dollars. Which evaluation is correct? (F is pyspark.sql.functions.)

  1. AAdd pred_price = F.exp("prediction"), then evaluate with RegressionEvaluator(labelCol="price", predictionCol="pred_price")
  2. BCompute the RMSE on the log scale, then report exp(rmse) - 1 as the dollar RMSE
  3. CAdd pred_price = F.exp("prediction") - 1, then evaluate with RegressionEvaluator(labelCol="price", predictionCol="pred_price")
  4. DRegressionEvaluator(labelCol="price", predictionCol="prediction")
Show the answer and why
  • AAdd pred_price = F.exp("prediction"), then evaluate with RegressionEvaluator(labelCol="price", predictionCol="pred_price")

    Incorrect

    log1p is the natural logarithm of the value plus one, so exp alone gives back price plus one; the one must also be subtracted.

  • BCompute the RMSE on the log scale, then report exp(rmse) - 1 as the dollar RMSE

    Incorrect

    The inverse transformation is applied to the predictions, not to an error metric; exponentiating a log-scale RMSE does not give the RMSE in dollars.

  • CAdd pred_price = F.exp("prediction") - 1, then evaluate with RegressionEvaluator(labelCol="price", predictionCol="pred_price")

    Correct

    Applying the inverse of log1p to the predictions puts them back in dollars, so the RMSE against price is in dollars.

  • DRegressionEvaluator(labelCol="price", predictionCol="prediction")

    Incorrect

    The model predicts on the scale of its training label, log1p of price, so this compares dollars with log-scale values.

Transformed variables lose their original meaning, and a model trained on log1p(price) predicts on that scale. When a transformed variable is the target of a regression, applying the inverse transformation to the predicted values makes them interpretable again: log1p is the natural logarithm of the value plus one, so exp(prediction) - 1 undoes it. Only then does an RMSE against the original label come out in the original units.

Question 6 · choose 1

A data scientist calls MulticlassClassificationEvaluator().evaluate(preds) without setting metricName. Which metric is returned?

  1. Af1
  2. Baccuracy
  3. ClogLoss
  4. DweightedPrecision
Show the answer and why
  • Af1

    Correct

    MulticlassClassificationEvaluator's metricName defaults to f1.

  • Baccuracy

    Incorrect

    accuracy is available but must be set with metricName.

  • ClogLoss

    Incorrect

    logLoss is available but must be set with metricName.

  • DweightedPrecision

    Incorrect

    weightedPrecision is available but must be set with metricName.

MulticlassClassificationEvaluator's default metricName is f1. Other metrics, such as accuracy, weightedPrecision, weightedRecall, logLoss and the by-label metrics, are chosen with metricName.

Question 7 · choose 1

A data scientist calls BinaryClassificationEvaluator().evaluate(preds) without setting metricName. Which metric is returned?

  1. Af1
  2. BareaUnderPR, the area under the precision-recall curve
  3. CareaUnderROC
  4. Daccuracy
Show the answer and why
  • Af1

    Incorrect

    f1 belongs to MulticlassClassificationEvaluator.

  • BareaUnderPR, the area under the precision-recall curve

    Incorrect

    areaUnderPR is available but is not the default.

  • CareaUnderROC

    Correct

    BinaryClassificationEvaluator's metricName defaults to areaUnderROC.

  • Daccuracy

    Incorrect

    accuracy belongs to MulticlassClassificationEvaluator.

BinaryClassificationEvaluator expects a rawPrediction column, a label column and an optional weight column. Its metricName defaults to areaUnderROC, and areaUnderPR is the other option.

Question 8 · choose 1

A data scientist fits lr_model = LinearRegression().fit(train_df) on a dataset with many features and wants the adjusted R-squared of the model on its training data. Which expression returns it?

  1. Alr_model.summary.r2
  2. Blr_model.summary.r2adj
  3. CRegressionEvaluator(metricName="r2adj").evaluate(lr_model.transform(train_df))
  4. Dlr_model.summary.numInstances
Show the answer and why
  • Alr_model.summary.r2

    Incorrect

    r2 is the plain coefficient of determination, not the adjusted one.

  • Blr_model.summary.r2adj

    Correct

    The model's training summary returns the adjusted coefficient of determination as r2adj.

  • CRegressionEvaluator(metricName="r2adj").evaluate(lr_model.transform(train_df))

    Incorrect

    r2adj is not one of RegressionEvaluator's metric names.

  • Dlr_model.summary.numInstances

    Incorrect

    numInstances is the number of rows in the predictions DataFrame.

A fitted LinearRegressionModel exposes summary, the summary of the model on its training set. That LinearRegressionSummary reports r2, the coefficient of determination, and r2adj, the adjusted coefficient of determination, among other values. Adjusted R-squared is not among RegressionEvaluator's metric names.

Question 9 · choose 1

A model outputs, for each row, only a double column holding the probability of label 1. Can BinaryClassificationEvaluator use that column as rawPredictionCol?

  1. ANo; rawPrediction must be a vector of raw scores
  2. BNo; only MulticlassClassificationEvaluator accepts probabilities
  3. CYes; rawPrediction can be a double holding a 0/1 prediction or the probability of label 1
  4. DYes, but only after converting it to a string column
Show the answer and why
  • ANo; rawPrediction must be a vector of raw scores

    Incorrect

    A double column is also accepted, as a 0/1 prediction or the probability of label 1.

  • BNo; only MulticlassClassificationEvaluator accepts probabilities

    Incorrect

    BinaryClassificationEvaluator accepts probabilities too.

  • CYes; rawPrediction can be a double holding a 0/1 prediction or the probability of label 1

    Correct

    The rawPrediction column can be a double (0/1 prediction or probability of label 1) or a length-2 vector.

  • DYes, but only after converting it to a string column

    Incorrect

    The column is used as a double or a vector, not as a string.

BinaryClassificationEvaluator expects rawPrediction, label and an optional weight column. The rawPrediction column can be a double (a binary 0/1 prediction or the probability of label 1) or a vector (a length-2 vector of raw predictions, scores or label probabilities).

Practice domain 3 →Practice all domains →