Associate Developer for Apache Spark · Domain 3: Developing Apache Spark DataFrame/DataSet API Applications · 30% of the exam
Task 3.6: Create and invoke user-defined functions with or without stateful operators, including StateStores; Describe different types of variables in Spark, including broadcast variables and accumulators
Python and Arrow UDFs, return types and nulls, registering a UDF for SQL, stateful streaming operators and the state store, and broadcast variables and accumulators.
Study it
User-defined functions and stateful streaming operators
Lesson coming
Broadcast variables and accumulators
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A developer runs:
from pyspark.sql import functions as F
@F.udf
def word_count(text):
return len(text.split())
df = spark.createDataFrame([("a b c",)], "text STRING")
print(df.select(word_count("text").alias("n")).dtypes)
What does it print?
A[('n', 'string')]
B[('n', 'int')]
C[('n', 'bigint')]
DAn error, because the decorator does not name a return type
Show the answer and why
A[('n', 'string')]
Correct
A UDF created without a returnType returns StringType, whatever the Python function returns.
B[('n', 'int')]
Incorrect
Spark does not infer the UDF's type from the Python code: without a returnType it is StringType.
C[('n', 'bigint')]
Incorrect
The returnType is not taken from the Python int; it defaults to StringType.
DAn error, because the decorator does not name a return type
Incorrect
udf can be used as a bare decorator; the returnType then takes its default, StringType.
pyspark.sql.functions.udf creates a user-defined function whose result type is returnType, a DataType or a DDL type string, defaulting to StringType. Used as @udf without arguments, the column the UDF produces is therefore a string column; dtypes reads that from the schema. Declaring @F.udf("int") would give an integer column.
A developer runs:
from pyspark.sql import functions as F
@F.udf("double")
def ratio(a, b):
return a / b
df = spark.createDataFrame([(10, 2), (5, 0)], "a INT, b INT")
df.withColumn("r", F.when(F.col("b") != 0, ratio("a", "b"))).show()
show() fails with a ZeroDivisionError raised inside ratio. Which change
returns NULL for the second row and keeps both rows?
AAdd .otherwise(None) after the when(...) call
BFilter out the rows with b = 0 before calling withColumn
CRegister ratio with spark.udf.register and call it inside a SQL CASE WHEN
DHandle b = 0 inside ratio, returning None in that case
Show the answer and why
AAdd .otherwise(None) after the when(...) call
Incorrect
UDFs do not support conditional expressions or short circuiting, so ratio still runs on the row with b = 0.
BFilter out the rows with b = 0 before calling withColumn
Incorrect
filter keeps only the rows that meet the condition, so the second row would be removed, while both rows must stay.
CRegister ratio with spark.udf.register and call it inside a SQL CASE WHEN
Incorrect
Inside SQL the UDF still does not support conditional expressions, so it can run on every row.
DHandle b = 0 inside ratio, returning None in that case
Correct
UDFs are evaluated without short circuiting, so the workaround is to incorporate the condition into the function.
Python UDFs do not support conditional expressions or short circuiting in boolean expressions; they end up being executed on all rows internally. When a function can fail on special rows, the documented workaround is to incorporate the condition into the function itself, for example return None when b is 0, which Spark stores as NULL.
A Python UDF assigns each row a random test group with random.randint, and the developer notices that the optimizer may call it a different number of times than it appears in the query. What does the udf documentation say to do for such a function?
ADeclare its returnType as StringType
BCache the DataFrame before the UDF is applied
CCall asNondeterministic() on the UDF
DWrap the UDF call in F.when(...)
Show the answer and why
ADeclare its returnType as StringType
Incorrect
The return type does not tell Spark whether the function is deterministic.
BCache the DataFrame before the UDF is applied
Incorrect
Caching stores data for reuse; it does not tell Spark that the function is not deterministic.
CCall asNondeterministic() on the UDF
Correct
UDFs are considered deterministic by default, so a nondeterministic function should be marked with asNondeterministic.
DWrap the UDF call in F.when(...)
Incorrect
Conditional expressions do not change how Spark treats a UDF's determinism.
User-defined functions are considered deterministic by default. Because of optimization, duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query. If a function is not deterministic, call asNondeterministic on the user-defined function, as the udf documentation shows for a random-number UDF.
On Spark 4.3, where Python UDFs use Arrow for (de)serialization by default, a team wants one particular UDF to run without Arrow while the others keep the default. Which definition does that?
This setting applies to every UDF whose useArrow is None, and true keeps Arrow on.
pyspark.sql.functions.udf has a useArrow parameter that decides whether Arrow optimizes the (de)serialization of that UDF; when it is None, the configuration spark.sql.execution.pythonUDF.arrow.enabled takes effect, and since Spark 4.2 Python UDFs use Arrow by default. Passing useArrow=False to one UDF opts it out without changing the others.
On a classic cluster, a job counts invalid rows with an accumulator created by spark.sparkContext.accumulator(0) and incremented inside a function that runs on the executors. Where can the accumulator's value be read?
AIn any task, which sees the running total of all tasks
BIn each executor's logs, written at the end of each task
COnly in the driver program, through its value
DIn each task, once the stage that updates it has finished
Show the answer and why
AIn any task, which sees the running total of all tasks
Incorrect
Tasks can add to an accumulator but cannot read its value.
BIn each executor's logs, written at the end of each task
Incorrect
The value is not published to executor logs; it is read by the driver.
COnly in the driver program, through its value
Correct
Tasks running on the cluster can only add to an accumulator; only the driver program can read its value, using its value method.
DIn each task, once the stage that updates it has finished
Incorrect
Tasks can only add to an accumulator, whether or not the stage has finished; reading the value is reserved to the driver.
Accumulators are variables that tasks only add to, such as counters and sums. Tasks running on a cluster can add to an accumulator with add or +=, but they cannot read its value; only the driver program can read the accumulator's value, using its value method.