Associate Developer for Apache Spark · Domain 7: Using Pandas API on Spark · 5% of the exam
Task 7.1: Explain the advantages of using Pandas API on Spark; Create and invoke Pandas UDF
What pandas-on-Spark DataFrames give a pandas user and what they cost, the default index types, and pandas UDFs with type hints, applyInPandas and mapInPandas.
Study it
The pandas API on Spark: advantages, default index and costs
Lesson coming
Pandas UDFs, applyInPandas and mapInPandas
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A pipeline built with pandas API on Spark (import pyspark.pandas as ps) holds 500 GB in the pandas-on-Spark DataFrame psdf. One step needs a PySpark-only call, DataFrameWriter.partitionBy. Which call gives a PySpark DataFrame without moving the data to a single machine?
Apsdf.to_spark()
Bpsdf.spark.local_checkpoint()
Cps.from_pandas(psdf)
Dpsdf.spark.explain()
Show the answer and why
Apsdf.to_spark()
Correct
to_spark returns a Spark DataFrame, giving access to the full PySpark API; the two kinds of DataFrame are virtually interchangeable.
Bpsdf.spark.local_checkpoint()
Incorrect
local_checkpoint stores the result in the executors and truncates the plan; it still returns a pandas-on-Spark DataFrame.
Cps.from_pandas(psdf)
Incorrect
from_pandas creates a pandas-on-Spark object from a pandas object; it does not return a PySpark DataFrame.
Dpsdf.spark.explain()
Incorrect
spark.explain prints the Spark plan behind the DataFrame; it returns no DataFrame.
Pandas API on Spark uses Spark under the hood, and pandas-on-Spark and Spark DataFrames are virtually interchangeable: DataFrame.to_spark() gives the full PySpark API, for example df.write with partitionBy, and sdf.pandas_api() goes back. The data stays distributed. The spark accessor offers helpers such as explain and local_checkpoint, which keep the pandas-on-Spark DataFrame.
On Spark 4.3, a job computes per-customer totals with:
import pandas as pd
def totals(key, pdf: pd.DataFrame) -> pd.DataFrame:
return pd.DataFrame({"customer_id": [key[0]], "total": [pdf["amount"].sum()]})
result = orders.groupBy("customer_id").applyInPandas(totals, "customer_id STRING, total DOUBLE")
One customer_id holds 30% of the rows, and the task that processes it fails
with out-of-memory errors. Which change addresses the failure while keeping
one total per customer?
AAdd worker nodes so the large customer's rows are spread over more executors
BCall orders.mapInPandas with the same function and output schema
CRewrite totals to take an iterator of pandas DataFrames for the group and add up the amount batch by batch
DRepartition orders by customer_id into more partitions before the groupBy
Show the answer and why
AAdd worker nodes so the large customer's rows are spread over more executors
Incorrect
All data of a group is loaded into memory together before the function runs, so the large customer stays in one task.
BCall orders.mapInPandas with the same function and output schema
Incorrect
mapInPandas passes batches of rows that are not grouped by customer, so the function no longer sees each customer's rows together.
CRewrite totals to take an iterator of pandas DataFrames for the group and add up the amount batch by batch
Correct
applyInPandas also accepts a function over an iterator of pandas DataFrames per group, the documented way to avoid loading a large group into memory at once.
DRepartition orders by customer_id into more partitions before the groupBy
Incorrect
Whatever the partitioning, each group is passed to one function call, with all its data loaded into memory first.
With the pandas.DataFrame form, groupBy().applyInPandas() loads all data of a group into memory before calling the function, which risks out-of-memory errors when groups are skewed. Since Spark 4.1 the function can instead take an iterator of pandas.DataFrame for each group (optionally with the grouping key first) and yield the result, which the API reference names as the mitigation. Summing batch by batch keeps one total per customer without holding the large group at once.
A notebook already has a configured SparkSession, spark, when a data scientist starts using import pyspark.pandas as ps. What does pandas API on Spark do about the session?
AIt starts a separate local pandas engine that ignores the session
BIt uses the existing Spark context and session out of the box
CIt requires calling ps.set_session(spark) before each operation
DIt creates a second SparkSession with default settings, discarding the notebook's configuration
Show the answer and why
AIt starts a separate local pandas engine that ignores the session
Incorrect
No separate engine is started; the session that is already running is used.
BIt uses the existing Spark context and session out of the box
Correct
Existing Spark contexts and sessions are used out of the box by pandas API on Spark.
CIt requires calling ps.set_session(spark) before each operation
Incorrect
No extra call is needed; the running session is used automatically.
DIt creates a second SparkSession with default settings, discarding the notebook's configuration
Incorrect
If a configured session is already running, pandas API on Spark uses it.
Existing Spark contexts and sessions are used out of the box by pandas API on Spark: if a configured context or session is already running, pandas API on Spark uses it, and if none exists, the one it creates can be configured in advance.
A developer writes df.groupBy("store").agg(F.sum("amount"), trimmed_mean("amount")), where trimmed_mean is a group aggregate pandas UDF, and the call fails. What does the API reference say?
AGroup aggregate pandas UDFs can be used only in select, never inside groupBy().agg() calls
Bagg accepts at most one aggregate expression per call
Csum must be written as a pandas UDF as well
DBuilt-in aggregates and group aggregate pandas UDFs cannot be mixed in one agg call
Show the answer and why
AGroup aggregate pandas UDFs can be used only in select, never inside groupBy().agg() calls
Incorrect
GroupedData.agg accepts group aggregate pandas UDFs; the problem is mixing them with built-in functions.
Bagg accepts at most one aggregate expression per call
Incorrect
agg accepts a list of aggregate Column expressions.
Csum must be written as a pandas UDF as well
Incorrect
Built-in functions stay built-in; they just cannot share one agg call with group aggregate pandas UDFs.
DBuilt-in aggregates and group aggregate pandas UDFs cannot be mixed in one agg call
Correct
GroupedData.agg notes that built-in aggregation functions and group aggregate pandas UDFs cannot be mixed in a single call.
GroupedData.agg computes aggregates with built-in aggregation functions, such as avg, max, min, sum and count, or with group aggregate pandas UDFs created with pandas_udf. The two kinds cannot be mixed in a single call to agg, so the developer computes them in separate agg calls, for example joining the results on store.
A job runs sales.groupBy("store_id").applyInPandas(fit_store, schema) to fit one small model per store, and the Spark UI shows a large exchange of data before the pandas function runs. What does the API reference say about this?
AapplyInPandas requires a full shuffle of the data by the grouping key
BThe exchange happens only because sales is not cached
CThe exchange is a broadcast of fit_store's code to every executor
DapplyInPandas collects the whole DataFrame to the driver before running the function
Show the answer and why
AapplyInPandas requires a full shuffle of the data by the grouping key
Correct
The API reference for GroupedData.applyInPandas notes that this function requires a full shuffle.
BThe exchange happens only because sales is not cached
Incorrect
The shuffle comes from grouping the data for the function, not from a missing cache.
CThe exchange is a broadcast of fit_store's code to every executor
Incorrect
The exchange moves the rows into their groups; it is not a broadcast of the function.
DapplyInPandas collects the whole DataFrame to the driver before running the function
Incorrect
The function runs per group on the executors after the shuffle, not on the driver.
GroupedData.applyInPandas maps each group of the DataFrame with a Python function and combines the returned pandas DataFrames into a new DataFrame. The API reference notes that this function requires a full shuffle, which is the exchange the Spark UI shows before the function runs.