Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
On Spark 4.3 with default settings, a developer adds an order total in cents:
from pyspark.sql import functions as F
orders = spark.createDataFrame([(1, 50000, 60000)], "id INT, qty INT, unit_cents INT")
out = orders.withColumn("total", (F.col("qty") * F.col("unit_cents")).cast("bigint"))
out.show()
What happens?
Ashow() raises an ARITHMETIC_OVERFLOW error from the multiplication
Bshow() prints 3000000000 in total, because the cast makes it a BIGINT
Cshow() prints -1294967296 in total, the INT result wrapped around
Dshow() prints NULL in total, because the product does not fit
Show the answer and why
Ashow() raises an ARITHMETIC_OVERFLOW error from the multiplication
Correct
Both operands are INT, so the product is computed as an INT before the cast; 3,000,000,000 is beyond the INT range, and an overflow raises an exception at run time by default.
Bshow() prints 3000000000 in total, because the cast makes it a BIGINT
Incorrect
The cast applies to the result of the multiplication; the multiplication itself runs on two INT operands, so its INT result overflows first.
Cshow() prints -1294967296 in total, the INT result wrapped around
Incorrect
A wrapped negative result is the behaviour with spark.sql.ansi.enabled set to false; by default an integer overflow throws an exception.
Dshow() prints NULL in total, because the product does not fit
Incorrect
By default an integer overflow throws an exception instead of returning a NULL result.
The least common type of the operands decides the operand type of an arithmetic operation; for two INT columns that is INT, whose range ends at 2147483647, so 50000 * 60000 = 3,000,000,000 overflows before cast("bigint") is applied to the result. With spark.sql.ansi.enabled true, the default, Spark throws an arithmetic exception at run time for numeric overflows (ARITHMETIC_OVERFLOW), and the run time is the show() action. Casting one operand to BIGINT before multiplying would make the product BIGINT.
A developer runs:
from pyspark.sql import functions as F
df = spark.createDataFrame([("a,b,c,d",)], "path STRING")
df.select(F.element_at(F.split("path", ",", 3), -1).alias("last")).show()
Which value does the column last hold?
Ad
Bc,d
Cc
DAn error, because element_at indexes start at 1 and cannot be negative
Show the answer and why
Ad
Incorrect
With a limit of 3, the array has at most three entries and the last one keeps all input beyond the last match, so d is not an entry of its own.
Bc,d
Correct
split with limit 3 returns a, b and c,d, and element_at with index -1 accesses the array from the last element.
Cc
Incorrect
The limit does not drop the rest of the input: the last entry contains everything beyond the last matched pattern.
DAn error, because element_at indexes start at 1 and cannot be negative
Incorrect
Negative indexes are allowed: an index below 0 accesses elements from the last to the first, and -1 is within this three-entry array.
split(str, pattern, limit) with limit > 0 returns an array whose length is not more than limit, and its last entry contains all input beyond the last matched pattern, so a,b,c,d split on commas with limit 3 gives [a, b, c,d]. element_at(array, index) uses 1-based positions, and an index below 0 accesses elements from the last to the first, so -1 returns c,d.
A developer runs:
from pyspark.sql import functions as F
orders = spark.createDataFrame([(1, "EUR", 20.0), (2, "USD", 35.0)], "order_id INT, currency STRING, amount DOUBLE")
rates = spark.createDataFrame([("EUR", 1.08), ("USD", 1.0)], "currency STRING, rate DOUBLE")
priced = orders.withColumn("rate", rates["rate"])
The last line raises an error. Which change adds each order's rate as a
column?
This is still a column of another DataFrame, which withColumn rejects.
Corders.withColumn("rate", F.col("rate"))
Incorrect
orders has no column named rate, so the expression cannot be resolved.
Dorders.join(rates, on="currency", how="left")
Correct
A join on currency brings each order's rate from rates into the result as a column.
The column expression given to withColumn must be an expression over this DataFrame; adding a column from some other DataFrame raises an error. Values from another DataFrame are brought in by joining on a key, here currency, with a left join keeping orders whose currency has no rate.
A developer runs:
from pyspark.sql import functions as F
df = spark.createDataFrame([(1, 10.0)], "id INT, price DOUBLE")
out = df.withColumn("price", F.col("price") * 2)
Which columns does out have?
Aid, price and price, with the doubled value in the second price column
Bid and price, with price holding 20.0
Cid and price_1, with the doubled value in price_1
DNone: withColumn raises an error because price already exists
Show the answer and why
Aid, price and price, with the doubled value in the second price column
Incorrect
withColumn replaces a column that has the same name instead of adding a second one.
Bid and price, with price holding 20.0
Correct
withColumn adds a column or replaces the existing column that has the same name, so price is replaced by the doubled value.
Cid and price_1, with the doubled value in price_1
Incorrect
Spark does not rename the new column; it replaces the existing price column.
DNone: withColumn raises an error because price already exists
Incorrect
An existing name is allowed: the column of that name is replaced.
DataFrame.withColumn(colName, col) returns a new DataFrame by adding a column or replacing the existing column that has the same name. Here price is replaced by price * 2, so out keeps two columns, id and price, with price equal to 20.0.
A DataFrame df has the column amount, and a developer wants to keep all columns and add gross = amount * 1.2 by writing the expression as a SQL string. Which call does that?
Adf.selectExpr("*", "amount * 1.2 AS gross")
Bdf.select("*", "amount * 1.2 AS gross")
Cdf.withColumn("gross", "amount * 1.2")
Ddf.filter("amount * 1.2 AS gross")
Show the answer and why
Adf.selectExpr("*", "amount * 1.2 AS gross")
Correct
selectExpr is a variant of select that accepts SQL expressions, so the string is parsed as an expression with an alias.
Bdf.select("*", "amount * 1.2 AS gross")
Incorrect
select reads a string as a column name, and there is no column with that name.
Cdf.withColumn("gross", "amount * 1.2")
Incorrect
withColumn needs a Column expression for the new column, not a SQL string.
Ddf.filter("amount * 1.2 AS gross")
Incorrect
filter keeps rows that meet a condition; it does not add a column.
DataFrame.selectExpr(*expr) projects a set of SQL expressions and returns a new DataFrame; it is a variant of select() that accepts SQL expressions, so "amount * 1.2 AS gross" defines a new column. select takes column names as strings or Column expressions, and withColumn takes a Column.
A developer runs:
from pyspark.sql import functions as F
df = spark.createDataFrame([(1, {"color": "red", "size": "M"})], "id INT, attrs MAP<STRING, STRING>")
out = df.select("id", F.explode("attrs"))
Which columns does out have?
Aid and col
Bid, key and value
Cid, color and size
Did and attrs
Show the answer and why
Aid and col
Incorrect
col is the default name for array elements; map entries get two columns.
Bid, key and value
Correct
For a map, explode uses the default column names key and value, with one row per entry.
Cid, color and size
Incorrect
explode turns map entries into rows; it does not turn map keys into columns.
Did and attrs
Incorrect
explode replaces the map with its entries; the map column itself is not selected.
pyspark.sql.functions.explode returns one row per array item or map key value pair. It uses the default column name col for elements of an array, and key and value for entries of a map, unless specified otherwise. Here out has two rows, (1, color, red) and (1, size, M), in the columns id, key and value.