Skip to content

Associate Developer for Apache Spark · Domain 3: Developing Apache Spark DataFrame/DataSet API Applications · 30% of the exam

Task 3.6: Create and invoke user-defined functions with or without stateful operators, including StateStores; Describe different types of variables in Spark, including broadcast variables and accumulators

Python and Arrow UDFs, return types and nulls, registering a UDF for SQL, stateful streaming operators and the state store, and broadcast variables and accumulators.

Study it

  • User-defined functions and stateful streaming operators

    Lesson coming

  • Broadcast variables and accumulators

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A developer runs: from pyspark.sql import functions as F @F.udf def word_count(text): return len(text.split()) df = spark.createDataFrame([("a b c",)], "text STRING") print(df.select(word_count("text").alias("n")).dtypes) What does it print?

  1. A[('n', 'string')]
  2. B[('n', 'int')]
  3. C[('n', 'bigint')]
  4. DAn error, because the decorator does not name a return type
Show the answer and why
  • A[('n', 'string')]

    Correct

    A UDF created without a returnType returns StringType, whatever the Python function returns.

  • B[('n', 'int')]

    Incorrect

    Spark does not infer the UDF's type from the Python code: without a returnType it is StringType.

  • C[('n', 'bigint')]

    Incorrect

    The returnType is not taken from the Python int; it defaults to StringType.

  • DAn error, because the decorator does not name a return type

    Incorrect

    udf can be used as a bare decorator; the returnType then takes its default, StringType.

pyspark.sql.functions.udf creates a user-defined function whose result type is returnType, a DataType or a DDL type string, defaulting to StringType. Used as @udf without arguments, the column the UDF produces is therefore a string column; dtypes reads that from the schema. Declaring @F.udf("int") would give an integer column.

Question 2 · choose 1

A developer runs: from pyspark.sql import functions as F @F.udf("double") def ratio(a, b): return a / b df = spark.createDataFrame([(10, 2), (5, 0)], "a INT, b INT") df.withColumn("r", F.when(F.col("b") != 0, ratio("a", "b"))).show() show() fails with a ZeroDivisionError raised inside ratio. Which change returns NULL for the second row and keeps both rows?

  1. AAdd .otherwise(None) after the when(...) call
  2. BFilter out the rows with b = 0 before calling withColumn
  3. CRegister ratio with spark.udf.register and call it inside a SQL CASE WHEN
  4. DHandle b = 0 inside ratio, returning None in that case
Show the answer and why
  • AAdd .otherwise(None) after the when(...) call

    Incorrect

    UDFs do not support conditional expressions or short circuiting, so ratio still runs on the row with b = 0.

  • BFilter out the rows with b = 0 before calling withColumn

    Incorrect

    filter keeps only the rows that meet the condition, so the second row would be removed, while both rows must stay.

  • CRegister ratio with spark.udf.register and call it inside a SQL CASE WHEN

    Incorrect

    Inside SQL the UDF still does not support conditional expressions, so it can run on every row.

  • DHandle b = 0 inside ratio, returning None in that case

    Correct

    UDFs are evaluated without short circuiting, so the workaround is to incorporate the condition into the function.

Python UDFs do not support conditional expressions or short circuiting in boolean expressions; they end up being executed on all rows internally. When a function can fail on special rows, the documented workaround is to incorporate the condition into the function itself, for example return None when b is 0, which Spark stores as NULL.

Question 3 · choose 1

A Python UDF assigns each row a random test group with random.randint, and the developer notices that the optimizer may call it a different number of times than it appears in the query. What does the udf documentation say to do for such a function?

  1. ADeclare its returnType as StringType
  2. BCache the DataFrame before the UDF is applied
  3. CCall asNondeterministic() on the UDF
  4. DWrap the UDF call in F.when(...)
Show the answer and why
  • ADeclare its returnType as StringType

    Incorrect

    The return type does not tell Spark whether the function is deterministic.

  • BCache the DataFrame before the UDF is applied

    Incorrect

    Caching stores data for reuse; it does not tell Spark that the function is not deterministic.

  • CCall asNondeterministic() on the UDF

    Correct

    UDFs are considered deterministic by default, so a nondeterministic function should be marked with asNondeterministic.

  • DWrap the UDF call in F.when(...)

    Incorrect

    Conditional expressions do not change how Spark treats a UDF's determinism.

User-defined functions are considered deterministic by default. Because of optimization, duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query. If a function is not deterministic, call asNondeterministic on the user-defined function, as the udf documentation shows for a random-number UDF.

Question 4 · choose 1

On Spark 4.3, where Python UDFs use Arrow for (de)serialization by default, a team wants one particular UDF to run without Arrow while the others keep the default. Which definition does that?

  1. A@udf(returnType=IntegerType(), useArrow=False)
  2. B@pandas_udf("int")
  3. C@udf(returnType=IntegerType()).asNondeterministic()
  4. Dspark.conf.set("spark.sql.execution.pythonUDF.arrow.enabled", "true")
Show the answer and why
  • A@udf(returnType=IntegerType(), useArrow=False)

    Correct

    useArrow sets per UDF whether Arrow optimizes the (de)serialization; when it is None, the session configuration decides.

  • B@pandas_udf("int")

    Incorrect

    A pandas UDF is a vectorized UDF built on Arrow; it does not turn Arrow off.

  • C@udf(returnType=IntegerType()).asNondeterministic()

    Incorrect

    asNondeterministic marks the UDF as nondeterministic; it does not change serialization.

  • Dspark.conf.set("spark.sql.execution.pythonUDF.arrow.enabled", "true")

    Incorrect

    This setting applies to every UDF whose useArrow is None, and true keeps Arrow on.

pyspark.sql.functions.udf has a useArrow parameter that decides whether Arrow optimizes the (de)serialization of that UDF; when it is None, the configuration spark.sql.execution.pythonUDF.arrow.enabled takes effect, and since Spark 4.2 Python UDFs use Arrow by default. Passing useArrow=False to one UDF opts it out without changing the others.

Question 5 · choose 1

On a classic cluster, a job counts invalid rows with an accumulator created by spark.sparkContext.accumulator(0) and incremented inside a function that runs on the executors. Where can the accumulator's value be read?

  1. AIn any task, which sees the running total of all tasks
  2. BIn each executor's logs, written at the end of each task
  3. COnly in the driver program, through its value
  4. DIn each task, once the stage that updates it has finished
Show the answer and why
  • AIn any task, which sees the running total of all tasks

    Incorrect

    Tasks can add to an accumulator but cannot read its value.

  • BIn each executor's logs, written at the end of each task

    Incorrect

    The value is not published to executor logs; it is read by the driver.

  • COnly in the driver program, through its value

    Correct

    Tasks running on the cluster can only add to an accumulator; only the driver program can read its value, using its value method.

  • DIn each task, once the stage that updates it has finished

    Incorrect

    Tasks can only add to an accumulator, whether or not the stage has finished; reading the value is reserved to the driver.

Accumulators are variables that tasks only add to, such as counters and sums. Tasks running on a cluster can add to an accumulator with add or +=, but they cannot read its value; only the driver program can read the accumulator's value, using its value method.

Practice domain 3 →Practice all domains →