Skip to content

Associate Developer for Apache Spark · Domain 7: Using Pandas API on Spark · 5% of the exam

Task 7.1: Explain the advantages of using Pandas API on Spark; Create and invoke Pandas UDF

What pandas-on-Spark DataFrames give a pandas user and what they cost, the default index types, and pandas UDFs with type hints, applyInPandas and mapInPandas.

Study it

  • The pandas API on Spark: advantages, default index and costs

    Lesson coming

  • Pandas UDFs, applyInPandas and mapInPandas

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A pipeline built with pandas API on Spark (import pyspark.pandas as ps) holds 500 GB in the pandas-on-Spark DataFrame psdf. One step needs a PySpark-only call, DataFrameWriter.partitionBy. Which call gives a PySpark DataFrame without moving the data to a single machine?

  1. Apsdf.to_spark()
  2. Bpsdf.spark.local_checkpoint()
  3. Cps.from_pandas(psdf)
  4. Dpsdf.spark.explain()
Show the answer and why
  • Apsdf.to_spark()

    Correct

    to_spark returns a Spark DataFrame, giving access to the full PySpark API; the two kinds of DataFrame are virtually interchangeable.

  • Bpsdf.spark.local_checkpoint()

    Incorrect

    local_checkpoint stores the result in the executors and truncates the plan; it still returns a pandas-on-Spark DataFrame.

  • Cps.from_pandas(psdf)

    Incorrect

    from_pandas creates a pandas-on-Spark object from a pandas object; it does not return a PySpark DataFrame.

  • Dpsdf.spark.explain()

    Incorrect

    spark.explain prints the Spark plan behind the DataFrame; it returns no DataFrame.

Pandas API on Spark uses Spark under the hood, and pandas-on-Spark and Spark DataFrames are virtually interchangeable: DataFrame.to_spark() gives the full PySpark API, for example df.write with partitionBy, and sdf.pandas_api() goes back. The data stays distributed. The spark accessor offers helpers such as explain and local_checkpoint, which keep the pandas-on-Spark DataFrame.

Question 2 · choose 1

On Spark 4.3, a job computes per-customer totals with: import pandas as pd def totals(key, pdf: pd.DataFrame) -> pd.DataFrame: return pd.DataFrame({"customer_id": [key[0]], "total": [pdf["amount"].sum()]}) result = orders.groupBy("customer_id").applyInPandas(totals, "customer_id STRING, total DOUBLE") One customer_id holds 30% of the rows, and the task that processes it fails with out-of-memory errors. Which change addresses the failure while keeping one total per customer?

  1. AAdd worker nodes so the large customer's rows are spread over more executors
  2. BCall orders.mapInPandas with the same function and output schema
  3. CRewrite totals to take an iterator of pandas DataFrames for the group and add up the amount batch by batch
  4. DRepartition orders by customer_id into more partitions before the groupBy
Show the answer and why
  • AAdd worker nodes so the large customer's rows are spread over more executors

    Incorrect

    All data of a group is loaded into memory together before the function runs, so the large customer stays in one task.

  • BCall orders.mapInPandas with the same function and output schema

    Incorrect

    mapInPandas passes batches of rows that are not grouped by customer, so the function no longer sees each customer's rows together.

  • CRewrite totals to take an iterator of pandas DataFrames for the group and add up the amount batch by batch

    Correct

    applyInPandas also accepts a function over an iterator of pandas DataFrames per group, the documented way to avoid loading a large group into memory at once.

  • DRepartition orders by customer_id into more partitions before the groupBy

    Incorrect

    Whatever the partitioning, each group is passed to one function call, with all its data loaded into memory first.

With the pandas.DataFrame form, groupBy().applyInPandas() loads all data of a group into memory before calling the function, which risks out-of-memory errors when groups are skewed. Since Spark 4.1 the function can instead take an iterator of pandas.DataFrame for each group (optionally with the grouping key first) and yield the result, which the API reference names as the mitigation. Summing batch by batch keeps one total per customer without holding the large group at once.

Question 3 · choose 1

A notebook already has a configured SparkSession, spark, when a data scientist starts using import pyspark.pandas as ps. What does pandas API on Spark do about the session?

  1. AIt starts a separate local pandas engine that ignores the session
  2. BIt uses the existing Spark context and session out of the box
  3. CIt requires calling ps.set_session(spark) before each operation
  4. DIt creates a second SparkSession with default settings, discarding the notebook's configuration
Show the answer and why
  • AIt starts a separate local pandas engine that ignores the session

    Incorrect

    No separate engine is started; the session that is already running is used.

  • BIt uses the existing Spark context and session out of the box

    Correct

    Existing Spark contexts and sessions are used out of the box by pandas API on Spark.

  • CIt requires calling ps.set_session(spark) before each operation

    Incorrect

    No extra call is needed; the running session is used automatically.

  • DIt creates a second SparkSession with default settings, discarding the notebook's configuration

    Incorrect

    If a configured session is already running, pandas API on Spark uses it.

Existing Spark contexts and sessions are used out of the box by pandas API on Spark: if a configured context or session is already running, pandas API on Spark uses it, and if none exists, the one it creates can be configured in advance.

Databricks documentation

Question 4 · choose 1

A developer writes df.groupBy("store").agg(F.sum("amount"), trimmed_mean("amount")), where trimmed_mean is a group aggregate pandas UDF, and the call fails. What does the API reference say?

  1. AGroup aggregate pandas UDFs can be used only in select, never inside groupBy().agg() calls
  2. Bagg accepts at most one aggregate expression per call
  3. Csum must be written as a pandas UDF as well
  4. DBuilt-in aggregates and group aggregate pandas UDFs cannot be mixed in one agg call
Show the answer and why
  • AGroup aggregate pandas UDFs can be used only in select, never inside groupBy().agg() calls

    Incorrect

    GroupedData.agg accepts group aggregate pandas UDFs; the problem is mixing them with built-in functions.

  • Bagg accepts at most one aggregate expression per call

    Incorrect

    agg accepts a list of aggregate Column expressions.

  • Csum must be written as a pandas UDF as well

    Incorrect

    Built-in functions stay built-in; they just cannot share one agg call with group aggregate pandas UDFs.

  • DBuilt-in aggregates and group aggregate pandas UDFs cannot be mixed in one agg call

    Correct

    GroupedData.agg notes that built-in aggregation functions and group aggregate pandas UDFs cannot be mixed in a single call.

GroupedData.agg computes aggregates with built-in aggregation functions, such as avg, max, min, sum and count, or with group aggregate pandas UDFs created with pandas_udf. The two kinds cannot be mixed in a single call to agg, so the developer computes them in separate agg calls, for example joining the results on store.

Question 5 · choose 1

A job runs sales.groupBy("store_id").applyInPandas(fit_store, schema) to fit one small model per store, and the Spark UI shows a large exchange of data before the pandas function runs. What does the API reference say about this?

  1. AapplyInPandas requires a full shuffle of the data by the grouping key
  2. BThe exchange happens only because sales is not cached
  3. CThe exchange is a broadcast of fit_store's code to every executor
  4. DapplyInPandas collects the whole DataFrame to the driver before running the function
Show the answer and why
  • AapplyInPandas requires a full shuffle of the data by the grouping key

    Correct

    The API reference for GroupedData.applyInPandas notes that this function requires a full shuffle.

  • BThe exchange happens only because sales is not cached

    Incorrect

    The shuffle comes from grouping the data for the function, not from a missing cache.

  • CThe exchange is a broadcast of fit_store's code to every executor

    Incorrect

    The exchange moves the rows into their groups; it is not a broadcast of the function.

  • DapplyInPandas collects the whole DataFrame to the driver before running the function

    Incorrect

    The function runs per group on the executors after the shuffle, not on the driver.

GroupedData.applyInPandas maps each group of the DataFrame with a Python function and combines the returned pandas DataFrames into a new DataFrame. The API reference notes that this function requires a full shuffle, which is the exchange the Spark UI shows before the function runs.

Practice domain 7 →Practice all domains →