Skip to content

Associate Developer for Apache Spark · Domain 1: Apache Spark Architecture and Components · 20% of the exam

Task 1.3: Explain the Apache Spark™ Architecture execution hierarchy; Configure Spark partitioning in distributed data processing, including shuffles and partitions

How an action becomes a job, stages split at shuffles and one task per partition, and controlling partitions with repartition, coalesce and the shuffle partition setting.

Study it

  • Jobs, stages and tasks

    Lesson coming

  • Partitions and shuffles: repartition, coalesce and the shuffle partition setting

    Partly covered by: Database Sharding

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A job runs this code: from pyspark.sql import functions as F orders = spark.read.table("sales.orders") paid = orders.filter(F.col("status") == "PAID") paid = paid.withColumn("net", F.col("amount") - F.col("discount")) totals = paid.groupBy("customer_id").agg(F.sum("net").alias("total")) totals.write.mode("overwrite").parquet("/data/out/customer_totals") The write is the only action. At which operation does a new stage start?

  1. AAt filter, because it removes rows from each partition
  2. BAt withColumn, because it adds a column to every row
  3. CAt write, because the action starts a new stage of its own
  4. DAt groupBy(...).agg(...), because grouping shuffles the rows
Show the answer and why
  • AAt filter, because it removes rows from each partition

    Incorrect

    A filter works on each partition where it is; it is not a distributed shuffle, so it stays in the stage that reads the table.

  • BAt withColumn, because it adds a column to every row

    Incorrect

    Adding a computed column works row by row inside each partition and involves no shuffle.

  • CAt write, because the action starts a new stage of its own

    Incorrect

    Stages are separated by distributed shuffle operations, not by the action itself.

  • DAt groupBy(...).agg(...), because grouping shuffles the rows

    Correct

    Spark actions run as stages separated by distributed shuffle operations, and grouping or aggregating by key is one.

Spark actions, here the Parquet write, are executed through a set of stages separated by distributed shuffle operations, such as grouping or aggregating by key. The read, filter and withColumn run inside each partition of the first stage; grouping by customer_id moves rows with the same key together, so the aggregation starts a new stage.

Question 2 · choose 1

A Spark 4.3 standalone cluster runs jobs with large joins and aggregations. On every worker, /tmp sits on a small, slow disk, while two large, fast local disks are mounted at /disk1 and /disk2, and no SPARK_LOCAL_DIRS variable is set. The shuffles of these jobs should write their intermediate files to the fast disks. Which setting does that?

  1. Aspark.sql.warehouse.dir set to /disk1,/disk2
  2. Bspark.local.dir set to /disk1,/disk2
  3. Cspark.sql.streaming.checkpointLocation set to /disk1
  4. Dspark.files set to /disk1,/disk2
Show the answer and why
  • Aspark.sql.warehouse.dir set to /disk1,/disk2

    Incorrect

    This is the default location for managed databases and tables, not the scratch space for shuffle files.

  • Bspark.local.dir set to /disk1,/disk2

    Correct

    spark.local.dir is the scratch space for map output files and RDDs stored on disk, and it takes a comma-separated list of directories on different disks.

  • Cspark.sql.streaming.checkpointLocation set to /disk1

    Incorrect

    This is the default location for checkpoint data of streaming queries; batch shuffles do not write there.

  • Dspark.files set to /disk1,/disk2

    Incorrect

    spark.files lists files to be placed in the working directory of each executor; it does not choose where shuffle files go.

A shuffle writes a large number of intermediate files on disk, and the temporary storage directory is set by spark.local.dir (default /tmp), the scratch space for map output files and RDDs stored on disk. It should be on a fast local disk and can be a comma-separated list of directories on different disks. On a standalone cluster a SPARK_LOCAL_DIRS variable would override it; here none is set.

Question 3 · choose 1

Before writing 16 output files, a developer wants each partition of a DataFrame to hold one contiguous range of event_date values, so that the files cover dates that do not overlap. Which call prepares the partitions?

  1. Adf.repartitionByRange(16, "event_date")
  2. Bdf.repartition(16, "event_date")
  3. Cdf.coalesce(16)
  4. Ddf.repartition(16, "event_date", "event_id")
Show the answer and why
  • Adf.repartitionByRange(16, "event_date")

    Correct

    repartitionByRange returns a range-partitioned DataFrame, so each partition covers its own range of event_date.

  • Bdf.repartition(16, "event_date")

    Incorrect

    repartition by a column returns a hash-partitioned DataFrame: each date goes to one partition, but a partition holds dates scattered across the calendar.

  • Cdf.coalesce(16)

    Incorrect

    coalesce only merges existing partitions; it does not reorganize rows by event_date.

  • Ddf.repartition(16, "event_date", "event_id")

    Incorrect

    Hash partitioning on event_date and event_id scatters each date's rows, and the dates, across the 16 partitions.

repartition(n, cols) and repartitionByRange(n, cols) both shuffle into n partitions by the given columns: repartition hash partitions, so a partition mixes unrelated dates, while repartitionByRange range partitions, so each partition holds a range of values. repartitionByRange estimates the ranges by sampling, so the exact boundaries may differ between runs. coalesce does not reorganize rows by a column.

Question 4 · choose 1

In Spark's execution hierarchy, what is a stage?

  1. AThe process that runs the application's main() function and creates the SparkContext
  2. BA smaller set of tasks that a job is divided into, depending on other stages
  3. CA unit of work sent to one executor
  4. DA process on a worker node that runs tasks for one application
Show the answer and why
  • AThe process that runs the application's main() function and creates the SparkContext

    Incorrect

    That is the driver program, which runs the application's main() function.

  • BA smaller set of tasks that a job is divided into, depending on other stages

    Correct

    Each job gets divided into smaller sets of tasks called stages that depend on each other.

  • CA unit of work sent to one executor

    Incorrect

    That is a task.

  • DA process on a worker node that runs tasks for one application

    Incorrect

    That is an executor.

Each job is divided into smaller sets of tasks called stages that depend on each other, similar to the map and reduce stages in MapReduce. A task is a unit of work sent to one executor, an executor is a process launched for an application on a worker node, and the driver program is the process running the application's main() function and creating the SparkContext.

Question 5 · choose 1

A team works only in Spark SQL and wants the result of a query to be redistributed into 100 partitions before the next step, as DataFrame.repartition(100) would do. What can they add to the query?

  1. ASET spark.sql.files.maxPartitionBytes=100
  2. BA LIMIT 100 clause
  3. CA GROUP BY over 100 columns
  4. DThe hint /*+ REPARTITION(100) */ in the SELECT
Show the answer and why
  • ASET spark.sql.files.maxPartitionBytes=100

    Incorrect

    That setting sizes file read partitions in bytes; it does not repartition a query result.

  • BA LIMIT 100 clause

    Incorrect

    LIMIT caps the number of rows returned; it does not change partitioning.

  • CA GROUP BY over 100 columns

    Incorrect

    Grouping aggregates rows; the number of grouping columns does not set the number of partitions.

  • DThe hint /*+ REPARTITION(100) */ in the SELECT

    Correct

    The REPARTITION hint repartitions to the given number of partitions, like the repartition API.

Spark SQL supports partitioning hints that let users suggest a partitioning strategy: COALESCE, REPARTITION and REPARTITION_BY_RANGE correspond to the coalesce, repartition and repartitionByRange APIs. The REPARTITION hint takes a partition number, column names or both, for example SELECT /*+ REPARTITION(100) */ * FROM t.

Question 6 · choose 1

A Spark SQL query starts with SELECT /*+ REPARTITION(100), COALESCE(500) */ * FROM t. Which partitioning hint does the optimizer pick?

  1. AREPARTITION(100), the leftmost hint
  2. BCOALESCE(500), the last hint written
  3. CCOALESCE(500), because it asks for more partitions
  4. DNeither: two partitioning hints in one query raise an error
Show the answer and why
  • AREPARTITION(100), the leftmost hint

    Correct

    When multiple partitioning hints are specified, the leftmost hint is picked by the optimizer.

  • BCOALESCE(500), the last hint written

    Incorrect

    The optimizer picks the leftmost partitioning hint, not the last one.

  • CCOALESCE(500), because it asks for more partitions

    Incorrect

    The choice depends on the position of the hints, not on the numbers.

  • DNeither: two partitioning hints in one query raise an error

    Incorrect

    Several partitioning hints are allowed; they insert several nodes into the logical plan.

When multiple partitioning hints are specified, multiple nodes are inserted into the logical plan, but the leftmost hint is picked by the optimizer. In the Hints page example, REPARTITION(100) followed by COALESCE(500) leaves only a repartition into 100 partitions in the optimized plan.

Databricks documentation

Practice domain 1 →Practice all domains →