Skip to content

Machine Learning Associate · Domain 2: Data Processing · 19% of the exam

Task 2.2: Compare and contrast imputing missing values with the mean or median or mode value; Impute missing values with the mode, mean, or median value; Use one-hot encoding for categorical features; Identify and explain the model types or data sets for which one-hot encoding is or is not appropriate; Identify scenarios where log scale transformation is appropriate

Imputer strategies and when each fits, StringIndexer and OneHotEncoder in a pipeline, when trees or many categories make one-hot encoding a poor choice, and log transforms for skewed values.

Study it

  • Imputing missing values: mean, median or mode

    Lesson coming

  • One-hot encoding and when not to use it; log transforms

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A data scientist adds Imputer(inputCols=["income", "age"], outputCols=["income_imp", "age_imp"]) to a pipeline, assuming that it fills missing values with each column's median. Both columns have nulls in the training data. What does the fitted ImputerModel fill them with?

  1. AEach column's median, computed exactly over all its training values
  2. BEach column's mean over its non-missing training values
  3. CEach column's most frequent value
  4. DZero, because missingValue was not set
Show the answer and why
  • AEach column's median, computed exactly over all its training values

    Incorrect

    Median is not the default strategy, and when it is chosen Spark computes it approximately with approxQuantile.

  • BEach column's mean over its non-missing training values

    Correct

    mean is the default strategy, and the value is computed after missing values are filtered out; nulls count as missing.

  • CEach column's most frequent value

    Incorrect

    The mode is used only when the strategy is set to it; the default is mean.

  • DZero, because missingValue was not set

    Incorrect

    missingValue tells Imputer which value marks a missing entry; it is not the replacement value.

pyspark.ml.feature.Imputer completes missing values with the mean, median or mode of each column; its strategy defaults to mean. The statistic is computed after filtering out missing values, nulls are treated as missing, and the median is computed with approxQuantile at a relative error of 0.001.

Databricks documentation

Question 2 · choose 1

In a Spark DataFrame of sensor readings, the column temp_c uses -999.0 for a missing reading and contains no nulls. A data scientist fits Imputer(strategy="median", inputCols=["temp_c"], outputCols=["temp_imp"]) and finds -999.0 values still present in temp_imp. Which change makes the Imputer replace them?

  1. ASwitch the strategy to mean
  2. BRun df.fillna() on temp_c before fitting the Imputer
  3. CSet missingValue=-999.0 on the Imputer
  4. DSet relativeError to 0 so that the median is exact
Show the answer and why
  • ASwitch the strategy to mean

    Incorrect

    The strategy changes the statistic used, but -999.0 is still treated as a real reading rather than as missing.

  • BRun df.fillna() on temp_c before fitting the Imputer

    Incorrect

    fillna replaces nulls, and the column has no nulls; the sentinel stays.

  • CSet missingValue=-999.0 on the Imputer

    Correct

    missingValue sets the placeholder that marks a missing entry; its default is NaN, so -999.0 is otherwise treated as data.

  • DSet relativeError to 0 so that the median is exact

    Incorrect

    relativeError controls the precision of the approximate median, not which values count as missing.

Imputer treats NaN (the default missingValue) and nulls as missing. A sentinel such as -999.0 is imputed only when missingValue is set to it; the median or mean is then computed after filtering out the missing entries.

Question 3 · choose 1

A fitted PipelineModel that applies StringIndexer(inputCol="city", outputCol="city_idx") and then OneHotEncoder scores new data every night. A city that was not in the training data makes the job fail, and every row must still receive a prediction. Which change to the StringIndexer fixes this?

  1. ASet handleInvalid="keep"
  2. BSet handleInvalid="skip"
  3. CSet stringOrderType="alphabetAsc"
  4. DSet dropLast=False on it
Show the answer and why
  • ASet handleInvalid="keep"

    Correct

    With keep, unseen labels are put in an extra bucket at index numLabels, so the rows are scored.

  • BSet handleInvalid="skip"

    Incorrect

    skip removes the rows that contain an unseen label, so they get no prediction.

  • CSet stringOrderType="alphabetAsc"

    Incorrect

    stringOrderType only sets how labels are ordered into indices; unseen labels still raise an error.

  • DSet dropLast=False on it

    Incorrect

    dropLast is a parameter of OneHotEncoder, not of StringIndexer, so it cannot stop the StringIndexer from failing on the unseen city.

StringIndexer's handleInvalid parameter decides what happens to labels that were not seen when it was fitted: error (the default) throws, skip drops the row, and keep puts the label in an additional bucket at index numLabels.

Question 4 · choose 1

A Spark ML pipeline indexes the string column merchant_category (300 distinct values) with StringIndexer and assembles the index with numeric columns, without one-hot encoding, for a RandomForestClassifier with default settings. Fitting fails with an error about the categorical feature. Keeping the random forest and the indexed column, what fixes it?

  1. AAdd a StandardScaler stage before the classifier
  2. BRaise maxDepth so that the trees can split on all 300 categories
  3. CSet maxBins to at least 300 on the classifier
  4. DSet handleInvalid="keep" on the StringIndexer
Show the answer and why
  • AAdd a StandardScaler stage before the classifier

    Incorrect

    Tree methods do not require feature scaling, and scaling does not change the number of categories.

  • BRaise maxDepth so that the trees can split on all 300 categories

    Incorrect

    The limit that applies to categorical features is maxBins, not the depth of the trees.

  • CSet maxBins to at least 300 on the classifier

    Correct

    maxBins must be at least the number of categories of any categorical feature, and its default is 32.

  • DSet handleInvalid="keep" on the StringIndexer

    Incorrect

    handleInvalid deals with labels unseen at fit time; it does not change the bin limit of the trees.

Spark's tree methods handle categorical features directly and do not need one-hot encoding or feature scaling; they read DataFrame metadata, such as the one StringIndexer writes, to tell categorical from continuous features. The maxBins parameter (default 32) must be at least the number of categories of any categorical feature, so a 300-value feature needs maxBins of 300 or more.

Question 5 · choose 1

The feature items_purchased is a count with many zeros and a long right tail. A data scientist wants a log-scale version of it for a linear model. Which Spark expression gives a finite value for every row? (F is pyspark.sql.functions.)

  1. AF.log("items_purchased")
  2. BF.log(2.0, "items_purchased")
  3. CF.log(F.col("items_purchased") - 1)
  4. DF.log1p("items_purchased")
Show the answer and why
  • AF.log("items_purchased")

    Incorrect

    The logarithm is not defined at 0, so the zero counts get no usable value.

  • BF.log(2.0, "items_purchased")

    Incorrect

    A base-2 logarithm has the same problem at 0 as the natural logarithm.

  • CF.log(F.col("items_purchased") - 1)

    Incorrect

    Subtracting 1 makes the zero counts negative, and the logarithm is undefined for negative values.

  • DF.log1p("items_purchased")

    Correct

    log1p computes the natural logarithm of the value plus one, so a count of 0 maps to 0.

A log transform suits heavily skewed data, but the logarithm is not defined for 0 or for negative values, so a constant is added first. Spark's log1p computes the natural logarithm of the value plus one in a single function.

Question 6 · choose 1

A StringIndexer with default settings is fitted on a color column whose values are red (500 rows), blue (300 rows) and green (200 rows). Which index does red receive?

  1. A2.0, because red comes last alphabetically
  2. B1.0, because indices start at 1
  3. CA random index that changes on every fit
  4. D0.0, as the most frequent label
Show the answer and why
  • A2.0, because red comes last alphabetically

    Incorrect

    The default ordering is by frequency, not alphabetical.

  • B1.0, because indices start at 1

    Incorrect

    Indices are in [0, numLabels), so they start at 0.

  • CA random index that changes on every fit

    Incorrect

    The ordering is deterministic: by label frequency by default.

  • D0.0, as the most frequent label

    Correct

    stringOrderType defaults to frequencyDesc, so the most frequent label gets index 0.

StringIndexer maps a string column of labels to indices in [0, numLabels). By default the ordering is by label frequency (stringOrderType frequencyDesc), so the most frequent label gets index 0.

Databricks documentation

Question 7 · choose 1

A Spark OneHotEncoder with default settings encodes a category index column with five categories (0.0 to 4.0). What does the input value 4.0 become?

  1. AAn all-zeros vector of length 4
  2. BA vector of length 5 with a 1.0 in the last position
  3. CThe scalar 4.0, unchanged
  4. DAn error, because 4.0 is out of range
Show the answer and why
  • AAn all-zeros vector of length 4

    Correct

    dropLast defaults to true, so the last category has no slot and maps to all zeros.

  • BA vector of length 5 with a 1.0 in the last position

    Incorrect

    That is the result only with dropLast set to false.

  • CThe scalar 4.0, unchanged

    Incorrect

    OneHotEncoder outputs binary vectors, not the index.

  • DAn error, because 4.0 is out of range

    Incorrect

    4.0 is a valid index for five categories.

Spark's OneHotEncoder maps a category index to a binary vector with at most a single one-value. The last category is not included by default (dropLast), because including it would make the entries linearly dependent; with five categories, 4.0 maps to [0.0, 0.0, 0.0, 0.0]. Unlike scikit-learn's encoder, it does not keep all categories.

Databricks documentation

Question 8 · choose 1

A regression model needs a zip_code feature with 30,000 distinct values, and one-hot encoding would add tens of thousands of columns. Which Spark feature transformer does the MLlib guide describe as usually performing better than one-hot encoding with lower dimensionality?

  1. ABinarizer
  2. BTargetEncoder
  3. CVectorAssembler
  4. DStandardScaler
Show the answer and why
  • ABinarizer

    Incorrect

    Binarizer thresholds numeric values into 0 and 1; it does not encode categories.

  • BTargetEncoder

    Correct

    Target encoding maps high-cardinality categories to quasi-continuous values without a binary vector, reducing dimensionality.

  • CVectorAssembler

    Incorrect

    VectorAssembler merges columns into one vector; it does not encode a categorical column.

  • DStandardScaler

    Incorrect

    StandardScaler rescales numeric features; it does not encode categories.

TargetEncoder transforms high-cardinality categorical features into quasi-continuous scalar values based on their relationship with the target. The MLlib guide says it usually performs better than one-hot encoding and does not need a final binary vector, decreasing the dataset's dimensionality; string categories are usually indexed with StringIndexer first.

Question 9 · choose 1

A data scientist plans to add the string column segment to the inputCols of pyspark.ml.feature.Imputer so that its missing values are filled. What does the Imputer documentation say about this?

  1. AImputer fills string columns with the most common label automatically
  2. BImputer converts the strings to indices first and then imputes them
  3. CImputer supports string columns only with strategy="median"
  4. DImputer expects numeric columns and does not support categorical features
Show the answer and why
  • AImputer fills string columns with the most common label automatically

    Incorrect

    Imputer works on numeric input columns only.

  • BImputer converts the strings to indices first and then imputes them

    Incorrect

    Imputer does no indexing; its input columns must already be numeric.

  • CImputer supports string columns only with strategy="median"

    Incorrect

    No strategy makes Imputer support categorical features.

  • DImputer expects numeric columns and does not support categorical features

    Correct

    The input columns should be numeric, and Imputer does not support categorical features, which it could fill with incorrect values.

pyspark.ml.feature.Imputer completes missing values with the mean, median or mode of numeric columns. Its documentation states that it does not currently support categorical features and may create incorrect values for them.

Databricks documentation

Question 10 · choose 1

A data scientist prepares a small training set with the pandas API on Spark (import pyspark.pandas as ps). The DataFrame psdf has the string column payment_method with five values, and she wants one indicator column per value before training. Which call does that?

  1. Apsdf["payment_method"].factorize(sort=True)
  2. Bps.get_dummies(psdf, columns=["payment_method"])
  3. Cpsdf["payment_method"].value_counts(normalize=True)
  4. Dpsdf.describe()
Show the answer and why
  • Apsdf["payment_method"].factorize(sort=True)

    Incorrect

    factorize encodes the values as a single column of integer codes, not one indicator column per value.

  • Bps.get_dummies(psdf, columns=["payment_method"])

    Correct

    get_dummies converts a categorical variable into dummy (indicator) columns, which is one-hot encoding.

  • Cpsdf["payment_method"].value_counts(normalize=True)

    Incorrect

    value_counts returns the relative frequency of each unique value; it adds no columns to psdf.

  • Dpsdf.describe()

    Incorrect

    describe returns summary statistics, not encoded columns.

The pandas API on Spark is used on Databricks with import pyspark.pandas as ps. Its get_dummies function converts a categorical variable into dummy/indicator variables, also known as one-hot encoding; the columns argument names the DataFrame columns to encode. factorize returns integer codes and the unique values, value_counts returns counts of unique values, and describe returns summary statistics.

Question 11 · choose 1

A data scientist sets up an AutoML classification experiment in the AutoML UI on Databricks Runtime 17.3 LTS ML. The numeric column income has nulls, and she wants them filled with the column's median instead of the method AutoML would pick on its own. Where does she set that?

  1. ABy setting pos_label to income
  2. BNowhere; AutoML always fills numeric nulls with the mean
  3. CIn the Impute with column of the table schema
  4. DBy adding income to exclude_cols
Show the answer and why
  • ABy setting pos_label to income

    Incorrect

    pos_label names the positive class for metrics such as precision and recall; it does not impute features.

  • BNowhere; AutoML always fills numeric nulls with the mean

    Incorrect

    AutoML picks a default method from the column type and content, and a different method can be chosen per column.

  • CIn the Impute with column of the table schema

    Correct

    In the UI, the Impute with drop-down of the table schema sets how a column's nulls are imputed.

  • DBy adding income to exclude_cols

    Incorrect

    exclude_cols makes AutoML ignore the column, so income would no longer be a feature.

On Databricks Runtime 10.4 LTS ML and above, you can choose how AutoML imputes null values: in the UI with the drop-down in the Impute with column of the table schema, and in the API with the imputers parameter, whose string values are mean, median or most_frequent. Without a choice, AutoML selects an imputation method based on the column type and content.

Practice domain 2 →Practice all domains →