Associate Developer for Apache Spark · Domain 1: Apache Spark Architecture and Components · 20% of the exam
Task 1.4: Describe the execution patterns of the Apache Spark™ engine, including actions, transformations, and lazy evaluation
Which calls are transformations and which are actions, when work actually runs, narrow against wide dependencies, and reading the plans that explain() prints.
Study it
Transformations, actions, lazy evaluation and explain()
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
On Spark 4.3, a developer runs this code in a Spark Classic session with
default settings:
from pyspark.sql import functions as F
df = spark.createDataFrame([(1, 10, 2), (2, 7, 0)], "id INT, total INT, units INT")
priced = df.withColumn("unit_price", F.col("total") / F.col("units"))
recent = priced.filter(F.col("id") > 0)
print("built")
recent.show()
What happens?
AThe withColumn line raises an error for the zero divisor, and "built" is never printed
B"built" is printed, then show() prints both rows, with NULL as unit_price for id 2
C"built" is printed, then show() raises an error because id 2 divides by zero
D"built" is printed, then show() prints Infinity as unit_price for id 2
Show the answer and why
AThe withColumn line raises an error for the zero divisor, and "built" is never printed
Incorrect
withColumn and filter are transformations, which are lazy: nothing is computed until an action needs a result.
B"built" is printed, then show() prints both rows, with NULL as unit_price for id 2
Incorrect
With ANSI mode on by default, an invalid input to an operator raises an exception at runtime instead of returning NULL.
C"built" is printed, then show() raises an error because id 2 divides by zero
Correct
The transformations only run when show() needs rows, and under the default ANSI mode the division by zero then raises.
D"built" is printed, then show() prints Infinity as unit_price for id 2
Incorrect
Under the default ANSI mode the division by zero raises at runtime; no value is produced for the row.
Transformations in Spark are lazy: withColumn and filter only record what to do, and the work runs when an action such as show() needs a result, so "built" is printed first. spark.sql.ansi.enabled is true by default, so Spark throws an exception at runtime instead of returning NULL when the inputs to an operator are invalid, such as a division by 0. The error therefore surfaces at show(), not at the line that defined the column.
A DataFrame sales has a non-null integer column amount, and pyspark.sql.functions is imported as F. A developer needs the largest amount as a Python value to use in a later filter. Which expression returns it?
With n supplied, head returns a list of Row, and a list cannot be indexed by a column name.
Dsales.orderBy(F.desc("amount")).take(1).amount
Incorrect
take returns a list of Row objects, and a list has no amount attribute.
limit(n) returns a DataFrame, so no value comes back until an action runs. Among the actions, first() returns the first row as a Row (or None when the DataFrame is empty), head() without n returns a single Row, and head(n) and take(n) return a list of Row. Here first() on the sorted DataFrame gives the row with the largest amount, and Row["amount"] reads the value.
A developer calls df.show() with no arguments on a DataFrame of 5,000 rows with long text columns. What does it print?
AAll 5,000 rows, with full column values
BThe first row only
CThe first 100 rows, with values cut at 100 characters
DThe first 20 rows, with strings longer than 20 characters truncated
Show the answer and why
AAll 5,000 rows, with full column values
Incorrect
show prints the first n rows, and n defaults to 20.
BThe first row only
Incorrect
The default number of rows is 20.
CThe first 100 rows, with values cut at 100 characters
Incorrect
The defaults are 20 rows and truncation at 20 characters.
DThe first 20 rows, with strings longer than 20 characters truncated
Correct
show defaults to n=20 and truncate=True, which truncates strings longer than 20 characters.
DataFrame.show(n=20, truncate=True, vertical=False) prints the first n rows to the console. With truncate=True it truncates strings longer than 20 characters; a number greater than one truncates to that length instead, and vertical=True prints one line per column value.
A developer calls df.tail(5000000) on a large DataFrame to inspect its last rows, and the driver fails with an OutOfMemoryError. What explains it?
Atail is a transformation, so Spark builds a 5,000,000-row plan in memory
Btail keeps the requested rows on the executors, which run out of memory before returning them
Ctail moves the rows into the driver process, and a very large number can crash it
Dtail always returns the whole DataFrame, whatever number is given
Show the answer and why
Atail is a transformation, so Spark builds a 5,000,000-row plan in memory
Incorrect
tail returns a list of rows to the driver; it is not a lazy transformation.
Btail keeps the requested rows on the executors, which run out of memory before returning them
Incorrect
The rows are moved to the driver, which is the process that failed.
Ctail moves the rows into the driver process, and a very large number can crash it
Correct
Running tail requires moving data into the application's driver process, and a very large num can crash the driver with OutOfMemoryError.
Dtail always returns the whole DataFrame, whatever number is given
Incorrect
tail returns that number of records, or all records if the DataFrame holds fewer.
DataFrame.tail(num) returns the last num rows as a list of Row. Running tail requires moving data into the application's driver process, and doing so with a very large num can crash the driver process with OutOfMemoryError. It returns num records, or all records if the DataFrame contains fewer.
A developer wants to print a message when df has more than 10 departments and
writes:
per_dept = df.groupBy("dept").count()
if per_dept > 10:
print("many departments")
The if line fails. Why?
AgroupBy("dept").count() runs an action that returns one integer per department
Bcount() must be given the column to count
CThe comparison runs before Spark has computed the counts
DGroupedData.count returns a DataFrame of counts per group, not an integer
Show the answer and why
AgroupBy("dept").count() runs an action that returns one integer per department
Incorrect
GroupedData.count returns a DataFrame with a count per group, not integers.
Bcount() must be given the column to count
Incorrect
GroupedData.count takes no arguments; it counts the records of each group.
CThe comparison runs before Spark has computed the counts
Incorrect
The problem is the type of per_dept, not when it is computed.
DGroupedData.count returns a DataFrame of counts per group, not an integer
Correct
GroupedData.count counts the records for each group and returns a DataFrame, which cannot be compared with 10; DataFrame.count() returns an int.
DataFrame.count() is an action that returns the number of rows as an int. GroupedData.count(), called after groupBy, counts the records for each group and returns a DataFrame with the grouping column and a count column. To count departments, the developer can call df.select("dept").distinct().count() or count the rows of per_dept with per_dept.count().