Machine Learning Associate · Domain 2: Data Processing · 19% of the exam
Task 2.2: Compare and contrast imputing missing values with the mean or median or mode value; Impute missing values with the mode, mean, or median value; Use one-hot encoding for categorical features; Identify and explain the model types or data sets for which one-hot encoding is or is not appropriate; Identify scenarios where log scale transformation is appropriate
Imputer strategies and when each fits, StringIndexer and OneHotEncoder in a pipeline, when trees or many categories make one-hot encoding a poor choice, and log transforms for skewed values.
Study it
Imputing missing values: mean, median or mode
Lesson coming
One-hot encoding and when not to use it; log transforms
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A data scientist adds Imputer(inputCols=["income", "age"], outputCols=["income_imp", "age_imp"]) to a pipeline, assuming that it fills missing values with each column's median. Both columns have nulls in the training data. What does the fitted ImputerModel fill them with?
AEach column's median, computed exactly over all its training values
BEach column's mean over its non-missing training values
CEach column's most frequent value
DZero, because missingValue was not set
Show the answer and why
AEach column's median, computed exactly over all its training values
Incorrect
Median is not the default strategy, and when it is chosen Spark computes it approximately with approxQuantile.
BEach column's mean over its non-missing training values
Correct
mean is the default strategy, and the value is computed after missing values are filtered out; nulls count as missing.
CEach column's most frequent value
Incorrect
The mode is used only when the strategy is set to it; the default is mean.
DZero, because missingValue was not set
Incorrect
missingValue tells Imputer which value marks a missing entry; it is not the replacement value.
pyspark.ml.feature.Imputer completes missing values with the mean, median or mode of each column; its strategy defaults to mean. The statistic is computed after filtering out missing values, nulls are treated as missing, and the median is computed with approxQuantile at a relative error of 0.001.
In a Spark DataFrame of sensor readings, the column temp_c uses -999.0 for a missing reading and contains no nulls. A data scientist fits Imputer(strategy="median", inputCols=["temp_c"], outputCols=["temp_imp"]) and finds -999.0 values still present in temp_imp. Which change makes the Imputer replace them?
ASwitch the strategy to mean
BRun df.fillna() on temp_c before fitting the Imputer
CSet missingValue=-999.0 on the Imputer
DSet relativeError to 0 so that the median is exact
Show the answer and why
ASwitch the strategy to mean
Incorrect
The strategy changes the statistic used, but -999.0 is still treated as a real reading rather than as missing.
BRun df.fillna() on temp_c before fitting the Imputer
Incorrect
fillna replaces nulls, and the column has no nulls; the sentinel stays.
CSet missingValue=-999.0 on the Imputer
Correct
missingValue sets the placeholder that marks a missing entry; its default is NaN, so -999.0 is otherwise treated as data.
DSet relativeError to 0 so that the median is exact
Incorrect
relativeError controls the precision of the approximate median, not which values count as missing.
Imputer treats NaN (the default missingValue) and nulls as missing. A sentinel such as -999.0 is imputed only when missingValue is set to it; the median or mean is then computed after filtering out the missing entries.
A fitted PipelineModel that applies StringIndexer(inputCol="city", outputCol="city_idx") and then OneHotEncoder scores new data every night. A city that was not in the training data makes the job fail, and every row must still receive a prediction. Which change to the StringIndexer fixes this?
ASet handleInvalid="keep"
BSet handleInvalid="skip"
CSet stringOrderType="alphabetAsc"
DSet dropLast=False on it
Show the answer and why
ASet handleInvalid="keep"
Correct
With keep, unseen labels are put in an extra bucket at index numLabels, so the rows are scored.
BSet handleInvalid="skip"
Incorrect
skip removes the rows that contain an unseen label, so they get no prediction.
CSet stringOrderType="alphabetAsc"
Incorrect
stringOrderType only sets how labels are ordered into indices; unseen labels still raise an error.
DSet dropLast=False on it
Incorrect
dropLast is a parameter of OneHotEncoder, not of StringIndexer, so it cannot stop the StringIndexer from failing on the unseen city.
StringIndexer's handleInvalid parameter decides what happens to labels that were not seen when it was fitted: error (the default) throws, skip drops the row, and keep puts the label in an additional bucket at index numLabels.
A Spark ML pipeline indexes the string column merchant_category (300 distinct values) with StringIndexer and assembles the index with numeric columns, without one-hot encoding, for a RandomForestClassifier with default settings. Fitting fails with an error about the categorical feature. Keeping the random forest and the indexed column, what fixes it?
AAdd a StandardScaler stage before the classifier
BRaise maxDepth so that the trees can split on all 300 categories
CSet maxBins to at least 300 on the classifier
DSet handleInvalid="keep" on the StringIndexer
Show the answer and why
AAdd a StandardScaler stage before the classifier
Incorrect
Tree methods do not require feature scaling, and scaling does not change the number of categories.
BRaise maxDepth so that the trees can split on all 300 categories
Incorrect
The limit that applies to categorical features is maxBins, not the depth of the trees.
CSet maxBins to at least 300 on the classifier
Correct
maxBins must be at least the number of categories of any categorical feature, and its default is 32.
DSet handleInvalid="keep" on the StringIndexer
Incorrect
handleInvalid deals with labels unseen at fit time; it does not change the bin limit of the trees.
Spark's tree methods handle categorical features directly and do not need one-hot encoding or feature scaling; they read DataFrame metadata, such as the one StringIndexer writes, to tell categorical from continuous features. The maxBins parameter (default 32) must be at least the number of categories of any categorical feature, so a 300-value feature needs maxBins of 300 or more.
The feature items_purchased is a count with many zeros and a long right tail. A data scientist wants a log-scale version of it for a linear model. Which Spark expression gives a finite value for every row? (F is pyspark.sql.functions.)
AF.log("items_purchased")
BF.log(2.0, "items_purchased")
CF.log(F.col("items_purchased") - 1)
DF.log1p("items_purchased")
Show the answer and why
AF.log("items_purchased")
Incorrect
The logarithm is not defined at 0, so the zero counts get no usable value.
BF.log(2.0, "items_purchased")
Incorrect
A base-2 logarithm has the same problem at 0 as the natural logarithm.
CF.log(F.col("items_purchased") - 1)
Incorrect
Subtracting 1 makes the zero counts negative, and the logarithm is undefined for negative values.
DF.log1p("items_purchased")
Correct
log1p computes the natural logarithm of the value plus one, so a count of 0 maps to 0.
A log transform suits heavily skewed data, but the logarithm is not defined for 0 or for negative values, so a constant is added first. Spark's log1p computes the natural logarithm of the value plus one in a single function.
A StringIndexer with default settings is fitted on a color column whose values are red (500 rows), blue (300 rows) and green (200 rows). Which index does red receive?
A2.0, because red comes last alphabetically
B1.0, because indices start at 1
CA random index that changes on every fit
D0.0, as the most frequent label
Show the answer and why
A2.0, because red comes last alphabetically
Incorrect
The default ordering is by frequency, not alphabetical.
B1.0, because indices start at 1
Incorrect
Indices are in [0, numLabels), so they start at 0.
CA random index that changes on every fit
Incorrect
The ordering is deterministic: by label frequency by default.
D0.0, as the most frequent label
Correct
stringOrderType defaults to frequencyDesc, so the most frequent label gets index 0.
StringIndexer maps a string column of labels to indices in [0, numLabels). By default the ordering is by label frequency (stringOrderType frequencyDesc), so the most frequent label gets index 0.
A Spark OneHotEncoder with default settings encodes a category index column with five categories (0.0 to 4.0). What does the input value 4.0 become?
AAn all-zeros vector of length 4
BA vector of length 5 with a 1.0 in the last position
CThe scalar 4.0, unchanged
DAn error, because 4.0 is out of range
Show the answer and why
AAn all-zeros vector of length 4
Correct
dropLast defaults to true, so the last category has no slot and maps to all zeros.
BA vector of length 5 with a 1.0 in the last position
Incorrect
That is the result only with dropLast set to false.
CThe scalar 4.0, unchanged
Incorrect
OneHotEncoder outputs binary vectors, not the index.
DAn error, because 4.0 is out of range
Incorrect
4.0 is a valid index for five categories.
Spark's OneHotEncoder maps a category index to a binary vector with at most a single one-value. The last category is not included by default (dropLast), because including it would make the entries linearly dependent; with five categories, 4.0 maps to [0.0, 0.0, 0.0, 0.0]. Unlike scikit-learn's encoder, it does not keep all categories.
A regression model needs a zip_code feature with 30,000 distinct values, and one-hot encoding would add tens of thousands of columns. Which Spark feature transformer does the MLlib guide describe as usually performing better than one-hot encoding with lower dimensionality?
ABinarizer
BTargetEncoder
CVectorAssembler
DStandardScaler
Show the answer and why
ABinarizer
Incorrect
Binarizer thresholds numeric values into 0 and 1; it does not encode categories.
BTargetEncoder
Correct
Target encoding maps high-cardinality categories to quasi-continuous values without a binary vector, reducing dimensionality.
CVectorAssembler
Incorrect
VectorAssembler merges columns into one vector; it does not encode a categorical column.
DStandardScaler
Incorrect
StandardScaler rescales numeric features; it does not encode categories.
TargetEncoder transforms high-cardinality categorical features into quasi-continuous scalar values based on their relationship with the target. The MLlib guide says it usually performs better than one-hot encoding and does not need a final binary vector, decreasing the dataset's dimensionality; string categories are usually indexed with StringIndexer first.
A data scientist plans to add the string column segment to the inputCols of pyspark.ml.feature.Imputer so that its missing values are filled. What does the Imputer documentation say about this?
AImputer fills string columns with the most common label automatically
BImputer converts the strings to indices first and then imputes them
CImputer supports string columns only with strategy="median"
DImputer expects numeric columns and does not support categorical features
Show the answer and why
AImputer fills string columns with the most common label automatically
Incorrect
Imputer works on numeric input columns only.
BImputer converts the strings to indices first and then imputes them
Incorrect
Imputer does no indexing; its input columns must already be numeric.
CImputer supports string columns only with strategy="median"
Incorrect
No strategy makes Imputer support categorical features.
DImputer expects numeric columns and does not support categorical features
Correct
The input columns should be numeric, and Imputer does not support categorical features, which it could fill with incorrect values.
pyspark.ml.feature.Imputer completes missing values with the mean, median or mode of numeric columns. Its documentation states that it does not currently support categorical features and may create incorrect values for them.
A data scientist prepares a small training set with the pandas API on Spark (import pyspark.pandas as ps). The DataFrame psdf has the string column payment_method with five values, and she wants one indicator column per value before training. Which call does that?
value_counts returns the relative frequency of each unique value; it adds no columns to psdf.
Dpsdf.describe()
Incorrect
describe returns summary statistics, not encoded columns.
The pandas API on Spark is used on Databricks with import pyspark.pandas as ps. Its get_dummies function converts a categorical variable into dummy/indicator variables, also known as one-hot encoding; the columns argument names the DataFrame columns to encode. factorize returns integer codes and the unique values, value_counts returns counts of unique values, and describe returns summary statistics.
A data scientist sets up an AutoML classification experiment in the AutoML UI on Databricks Runtime 17.3 LTS ML. The numeric column income has nulls, and she wants them filled with the column's median instead of the method AutoML would pick on its own. Where does she set that?
ABy setting pos_label to income
BNowhere; AutoML always fills numeric nulls with the mean
CIn the Impute with column of the table schema
DBy adding income to exclude_cols
Show the answer and why
ABy setting pos_label to income
Incorrect
pos_label names the positive class for metrics such as precision and recall; it does not impute features.
BNowhere; AutoML always fills numeric nulls with the mean
Incorrect
AutoML picks a default method from the column type and content, and a different method can be chosen per column.
CIn the Impute with column of the table schema
Correct
In the UI, the Impute with drop-down of the table schema sets how a column's nulls are imputed.
DBy adding income to exclude_cols
Incorrect
exclude_cols makes AutoML ignore the column, so income would no longer be a feature.
On Databricks Runtime 10.4 LTS ML and above, you can choose how AutoML imputes null values: in the UI with the drop-down in the Impute with column of the table schema, and in the API with the imputers parameter, whose string values are mean, median or most_frequent. Without a choice, AutoML selects an imputation method based on the column type and content.