Task 6.3: Identify common performance bottlenecks such as data skew, shuffling, and disk spilling by interpreting stage-level metrics in the Spark UI.

• AAdd a broadcast hint to the larger of the two inputs

  Incorrect

  Broadcasting a large input adds memory pressure; a large broadcast is itself a common cause of memory problems.

• BSalt the grouping key with a random suffix

  Incorrect

  Salting fixes skew, but even task durations show there is no skew here.

• CUse worker instances with more memory per core

  Correct

  Spill happens when execution memory runs low during shuffles and aggregations; more memory per core lets more data stay in memory.

• DRemove workers while keeping the same instance type

  Incorrect

  Fewer workers of the same size means less total executor memory, so each task has more data to hold and spills more.

Spill on the stage page points to execution memory or partition sizing. Giving each core more memory is the most direct of the fixes offered here.

• ADisk spill caused by too little execution memory for the tasks of the join

  Incorrect

  The stage page shows no spill statistics, so the stage did not spill.

• BToo few shuffle partitions across the whole stage

  Incorrect

  Too few partitions makes every task large; here most tasks are short and one is very long.

• CAn overloaded driver

  Incorrect

  An overloaded driver shows as gaps between Spark jobs and idle executors, not as one long task inside a stage.

• DData skew: one or a few tasks process far more data than the others

  Correct

  When the max duration exceeds the 75th percentile by 50% or more, the stage might have skew.

Compare max with the 75th percentile in Summary Metrics to spot skew, and look for spill statistics at the top of the stage page.

• AAdd more workers to the cluster so the stage gets more cores

  Incorrect

  A single-task stage has no parallelism, so extra workers cannot help.

• BAdd a PARTITION BY clause to the window, if the logic allows

  Correct

  Without PARTITION BY, the window spans the whole dataset, so one task processes every row.

• CRaise spark.sql.shuffle.partitions so the stage gets more tasks

  Incorrect

  A window without PARTITION BY still sends all rows to one partition.

• DEnable Photon on the cluster

  Incorrect

  Photon speeds up execution but does not split one task's work across the cluster.

A one-task stage usually comes from a window without PARTITION BY, repartition(1) or coalesce(1), an unsplittable file, or a UDF on small input.

• ABroadcast the table whenever it is used in joins

  Incorrect

  Broadcasting does not change how many files must be opened when the table is scanned.

• BRaise spark.sql.shuffle.partitions

  Incorrect

  Shuffle partitions apply after the scan; the cost here is reading tiny files.

• CRun OPTIMIZE and enable predictive optimization

  Correct

  Tens of thousands of small files point to a small file problem; OPTIMIZE compacts them, and predictive optimization keeps the layout healthy.

• DRun VACUUM on the table to clean up the old files

  Incorrect

  VACUUM deletes files the table no longer references; it does not compact the current files.

Files should be at least 8 MB. Small files often come from partitioning on too many or high-cardinality columns; also reconsider the table layout.

• ARewrite the loop with Spark DataFrame operations

  Correct

  Non-Spark Python runs on the driver; expressing the work in Spark spreads it across executors.

• BAdd more workers to the cluster to share the load

  Incorrect

  The workers are already idle, because the work is not running in Spark.

• CIncrease executor memory

  Incorrect

  Executor memory does not help code that runs only on the driver.

• DRaise spark.sql.shuffle.partitions

  Incorrect

  The gaps come from driver-side Python, not from shuffle stages.

An overloaded driver shows as timeline gaps and idle executors. Move driver-side work into Spark, reduce concurrency, or use a larger driver.