Associate Developer for Apache Spark · Domain 1: Apache Spark Architecture and Components · 20% of the exam
Task 1.3: Explain the Apache Spark™ Architecture execution hierarchy; Configure Spark partitioning in distributed data processing, including shuffles and partitions
How an action becomes a job, stages split at shuffles and one task per partition, and controlling partitions with repartition, coalesce and the shuffle partition setting.
Study it
Jobs, stages and tasks
Lesson coming
Partitions and shuffles: repartition, coalesce and the shuffle partition setting
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A job runs this code:
from pyspark.sql import functions as F
orders = spark.read.table("sales.orders")
paid = orders.filter(F.col("status") == "PAID")
paid = paid.withColumn("net", F.col("amount") - F.col("discount"))
totals = paid.groupBy("customer_id").agg(F.sum("net").alias("total"))
totals.write.mode("overwrite").parquet("/data/out/customer_totals")
The write is the only action. At which operation does a new stage start?
AAt filter, because it removes rows from each partition
BAt withColumn, because it adds a column to every row
CAt write, because the action starts a new stage of its own
DAt groupBy(...).agg(...), because grouping shuffles the rows
Show the answer and why
AAt filter, because it removes rows from each partition
Incorrect
A filter works on each partition where it is; it is not a distributed shuffle, so it stays in the stage that reads the table.
BAt withColumn, because it adds a column to every row
Incorrect
Adding a computed column works row by row inside each partition and involves no shuffle.
CAt write, because the action starts a new stage of its own
Incorrect
Stages are separated by distributed shuffle operations, not by the action itself.
DAt groupBy(...).agg(...), because grouping shuffles the rows
Correct
Spark actions run as stages separated by distributed shuffle operations, and grouping or aggregating by key is one.
Spark actions, here the Parquet write, are executed through a set of stages separated by distributed shuffle operations, such as grouping or aggregating by key. The read, filter and withColumn run inside each partition of the first stage; grouping by customer_id moves rows with the same key together, so the aggregation starts a new stage.
A Spark 4.3 standalone cluster runs jobs with large joins and aggregations. On every worker, /tmp sits on a small, slow disk, while two large, fast local disks are mounted at /disk1 and /disk2, and no SPARK_LOCAL_DIRS variable is set. The shuffles of these jobs should write their intermediate files to the fast disks. Which setting does that?
Aspark.sql.warehouse.dir set to /disk1,/disk2
Bspark.local.dir set to /disk1,/disk2
Cspark.sql.streaming.checkpointLocation set to /disk1
Dspark.files set to /disk1,/disk2
Show the answer and why
Aspark.sql.warehouse.dir set to /disk1,/disk2
Incorrect
This is the default location for managed databases and tables, not the scratch space for shuffle files.
Bspark.local.dir set to /disk1,/disk2
Correct
spark.local.dir is the scratch space for map output files and RDDs stored on disk, and it takes a comma-separated list of directories on different disks.
Cspark.sql.streaming.checkpointLocation set to /disk1
Incorrect
This is the default location for checkpoint data of streaming queries; batch shuffles do not write there.
Dspark.files set to /disk1,/disk2
Incorrect
spark.files lists files to be placed in the working directory of each executor; it does not choose where shuffle files go.
A shuffle writes a large number of intermediate files on disk, and the temporary storage directory is set by spark.local.dir (default /tmp), the scratch space for map output files and RDDs stored on disk. It should be on a fast local disk and can be a comma-separated list of directories on different disks. On a standalone cluster a SPARK_LOCAL_DIRS variable would override it; here none is set.
Before writing 16 output files, a developer wants each partition of a DataFrame to hold one contiguous range of event_date values, so that the files cover dates that do not overlap. Which call prepares the partitions?
Adf.repartitionByRange(16, "event_date")
Bdf.repartition(16, "event_date")
Cdf.coalesce(16)
Ddf.repartition(16, "event_date", "event_id")
Show the answer and why
Adf.repartitionByRange(16, "event_date")
Correct
repartitionByRange returns a range-partitioned DataFrame, so each partition covers its own range of event_date.
Bdf.repartition(16, "event_date")
Incorrect
repartition by a column returns a hash-partitioned DataFrame: each date goes to one partition, but a partition holds dates scattered across the calendar.
Cdf.coalesce(16)
Incorrect
coalesce only merges existing partitions; it does not reorganize rows by event_date.
Ddf.repartition(16, "event_date", "event_id")
Incorrect
Hash partitioning on event_date and event_id scatters each date's rows, and the dates, across the 16 partitions.
repartition(n, cols) and repartitionByRange(n, cols) both shuffle into n partitions by the given columns: repartition hash partitions, so a partition mixes unrelated dates, while repartitionByRange range partitions, so each partition holds a range of values. repartitionByRange estimates the ranges by sampling, so the exact boundaries may differ between runs. coalesce does not reorganize rows by a column.
AThe process that runs the application's main() function and creates the SparkContext
BA smaller set of tasks that a job is divided into, depending on other stages
CA unit of work sent to one executor
DA process on a worker node that runs tasks for one application
Show the answer and why
AThe process that runs the application's main() function and creates the SparkContext
Incorrect
That is the driver program, which runs the application's main() function.
BA smaller set of tasks that a job is divided into, depending on other stages
Correct
Each job gets divided into smaller sets of tasks called stages that depend on each other.
CA unit of work sent to one executor
Incorrect
That is a task.
DA process on a worker node that runs tasks for one application
Incorrect
That is an executor.
Each job is divided into smaller sets of tasks called stages that depend on each other, similar to the map and reduce stages in MapReduce. A task is a unit of work sent to one executor, an executor is a process launched for an application on a worker node, and the driver program is the process running the application's main() function and creating the SparkContext.
A team works only in Spark SQL and wants the result of a query to be redistributed into 100 partitions before the next step, as DataFrame.repartition(100) would do. What can they add to the query?
ASET spark.sql.files.maxPartitionBytes=100
BA LIMIT 100 clause
CA GROUP BY over 100 columns
DThe hint /*+ REPARTITION(100) */ in the SELECT
Show the answer and why
ASET spark.sql.files.maxPartitionBytes=100
Incorrect
That setting sizes file read partitions in bytes; it does not repartition a query result.
BA LIMIT 100 clause
Incorrect
LIMIT caps the number of rows returned; it does not change partitioning.
CA GROUP BY over 100 columns
Incorrect
Grouping aggregates rows; the number of grouping columns does not set the number of partitions.
DThe hint /*+ REPARTITION(100) */ in the SELECT
Correct
The REPARTITION hint repartitions to the given number of partitions, like the repartition API.
Spark SQL supports partitioning hints that let users suggest a partitioning strategy: COALESCE, REPARTITION and REPARTITION_BY_RANGE correspond to the coalesce, repartition and repartitionByRange APIs. The REPARTITION hint takes a partition number, column names or both, for example SELECT /*+ REPARTITION(100) */ * FROM t.
A Spark SQL query starts with SELECT /*+ REPARTITION(100), COALESCE(500) */ * FROM t. Which partitioning hint does the optimizer pick?
AREPARTITION(100), the leftmost hint
BCOALESCE(500), the last hint written
CCOALESCE(500), because it asks for more partitions
DNeither: two partitioning hints in one query raise an error
Show the answer and why
AREPARTITION(100), the leftmost hint
Correct
When multiple partitioning hints are specified, the leftmost hint is picked by the optimizer.
BCOALESCE(500), the last hint written
Incorrect
The optimizer picks the leftmost partitioning hint, not the last one.
CCOALESCE(500), because it asks for more partitions
Incorrect
The choice depends on the position of the hints, not on the numbers.
DNeither: two partitioning hints in one query raise an error
Incorrect
Several partitioning hints are allowed; they insert several nodes into the logical plan.
When multiple partitioning hints are specified, multiple nodes are inserted into the logical plan, but the leftmost hint is picked by the optimizer. In the Hints page example, REPARTITION(100) followed by COALESCE(500) leaves only a repartition into 100 partitions in the optimized plan.