Skip to content

Associate Developer for Apache Spark · Domain 1: Apache Spark Architecture and Components · 20% of the exam

Task 1.4: Describe the execution patterns of the Apache Spark™ engine, including actions, transformations, and lazy evaluation

Which calls are transformations and which are actions, when work actually runs, narrow against wide dependencies, and reading the plans that explain() prints.

Study it

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

On Spark 4.3, a developer runs this code in a Spark Classic session with default settings: from pyspark.sql import functions as F df = spark.createDataFrame([(1, 10, 2), (2, 7, 0)], "id INT, total INT, units INT") priced = df.withColumn("unit_price", F.col("total") / F.col("units")) recent = priced.filter(F.col("id") > 0) print("built") recent.show() What happens?

  1. AThe withColumn line raises an error for the zero divisor, and "built" is never printed
  2. B"built" is printed, then show() prints both rows, with NULL as unit_price for id 2
  3. C"built" is printed, then show() raises an error because id 2 divides by zero
  4. D"built" is printed, then show() prints Infinity as unit_price for id 2
Show the answer and why
  • AThe withColumn line raises an error for the zero divisor, and "built" is never printed

    Incorrect

    withColumn and filter are transformations, which are lazy: nothing is computed until an action needs a result.

  • B"built" is printed, then show() prints both rows, with NULL as unit_price for id 2

    Incorrect

    With ANSI mode on by default, an invalid input to an operator raises an exception at runtime instead of returning NULL.

  • C"built" is printed, then show() raises an error because id 2 divides by zero

    Correct

    The transformations only run when show() needs rows, and under the default ANSI mode the division by zero then raises.

  • D"built" is printed, then show() prints Infinity as unit_price for id 2

    Incorrect

    Under the default ANSI mode the division by zero raises at runtime; no value is produced for the row.

Transformations in Spark are lazy: withColumn and filter only record what to do, and the work runs when an action such as show() needs a result, so "built" is printed first. spark.sql.ansi.enabled is true by default, so Spark throws an exception at runtime instead of returning NULL when the inputs to an operator are invalid, such as a division by 0. The error therefore surfaces at show(), not at the line that defined the column.

Question 2 · choose 1

A DataFrame sales has a non-null integer column amount, and pyspark.sql.functions is imported as F. A developer needs the largest amount as a Python value to use in a later filter. Which expression returns it?

  1. Asales.orderBy(F.desc("amount")).limit(1).amount
  2. Bsales.orderBy(F.desc("amount")).first()["amount"]
  3. Csales.orderBy(F.desc("amount")).head(n=1)["amount"]
  4. Dsales.orderBy(F.desc("amount")).take(1).amount
Show the answer and why
  • Asales.orderBy(F.desc("amount")).limit(1).amount

    Incorrect

    limit is a transformation that returns a DataFrame, so .amount refers to a Column, not a value.

  • Bsales.orderBy(F.desc("amount")).first()["amount"]

    Correct

    first() is an action that returns the first row as a Row, and indexing the Row by name gives the value.

  • Csales.orderBy(F.desc("amount")).head(n=1)["amount"]

    Incorrect

    With n supplied, head returns a list of Row, and a list cannot be indexed by a column name.

  • Dsales.orderBy(F.desc("amount")).take(1).amount

    Incorrect

    take returns a list of Row objects, and a list has no amount attribute.

limit(n) returns a DataFrame, so no value comes back until an action runs. Among the actions, first() returns the first row as a Row (or None when the DataFrame is empty), head() without n returns a single Row, and head(n) and take(n) return a list of Row. Here first() on the sorted DataFrame gives the row with the largest amount, and Row["amount"] reads the value.

Question 3 · choose 1

A developer calls df.show() with no arguments on a DataFrame of 5,000 rows with long text columns. What does it print?

  1. AAll 5,000 rows, with full column values
  2. BThe first row only
  3. CThe first 100 rows, with values cut at 100 characters
  4. DThe first 20 rows, with strings longer than 20 characters truncated
Show the answer and why
  • AAll 5,000 rows, with full column values

    Incorrect

    show prints the first n rows, and n defaults to 20.

  • BThe first row only

    Incorrect

    The default number of rows is 20.

  • CThe first 100 rows, with values cut at 100 characters

    Incorrect

    The defaults are 20 rows and truncation at 20 characters.

  • DThe first 20 rows, with strings longer than 20 characters truncated

    Correct

    show defaults to n=20 and truncate=True, which truncates strings longer than 20 characters.

DataFrame.show(n=20, truncate=True, vertical=False) prints the first n rows to the console. With truncate=True it truncates strings longer than 20 characters; a number greater than one truncates to that length instead, and vertical=True prints one line per column value.

Question 4 · choose 1

A developer calls df.tail(5000000) on a large DataFrame to inspect its last rows, and the driver fails with an OutOfMemoryError. What explains it?

  1. Atail is a transformation, so Spark builds a 5,000,000-row plan in memory
  2. Btail keeps the requested rows on the executors, which run out of memory before returning them
  3. Ctail moves the rows into the driver process, and a very large number can crash it
  4. Dtail always returns the whole DataFrame, whatever number is given
Show the answer and why
  • Atail is a transformation, so Spark builds a 5,000,000-row plan in memory

    Incorrect

    tail returns a list of rows to the driver; it is not a lazy transformation.

  • Btail keeps the requested rows on the executors, which run out of memory before returning them

    Incorrect

    The rows are moved to the driver, which is the process that failed.

  • Ctail moves the rows into the driver process, and a very large number can crash it

    Correct

    Running tail requires moving data into the application's driver process, and a very large num can crash the driver with OutOfMemoryError.

  • Dtail always returns the whole DataFrame, whatever number is given

    Incorrect

    tail returns that number of records, or all records if the DataFrame holds fewer.

DataFrame.tail(num) returns the last num rows as a list of Row. Running tail requires moving data into the application's driver process, and doing so with a very large num can crash the driver process with OutOfMemoryError. It returns num records, or all records if the DataFrame contains fewer.

Question 5 · choose 1

A developer wants to print a message when df has more than 10 departments and writes: per_dept = df.groupBy("dept").count() if per_dept > 10: print("many departments") The if line fails. Why?

  1. AgroupBy("dept").count() runs an action that returns one integer per department
  2. Bcount() must be given the column to count
  3. CThe comparison runs before Spark has computed the counts
  4. DGroupedData.count returns a DataFrame of counts per group, not an integer
Show the answer and why
  • AgroupBy("dept").count() runs an action that returns one integer per department

    Incorrect

    GroupedData.count returns a DataFrame with a count per group, not integers.

  • Bcount() must be given the column to count

    Incorrect

    GroupedData.count takes no arguments; it counts the records of each group.

  • CThe comparison runs before Spark has computed the counts

    Incorrect

    The problem is the type of per_dept, not when it is computed.

  • DGroupedData.count returns a DataFrame of counts per group, not an integer

    Correct

    GroupedData.count counts the records for each group and returns a DataFrame, which cannot be compared with 10; DataFrame.count() returns an int.

DataFrame.count() is an action that returns the number of rows as an int. GroupedData.count(), called after groupBy, counts the records for each group and returns a DataFrame with the grouping column and a count column. To count departments, the developer can call df.select("dept").distinct().count() or count the rows of per_dept with per_dept.count().

Practice domain 1 →Practice all domains →