Skip to content

Associate Developer for Apache Spark · Domain 1: Apache Spark Architecture and Components · 20% of the exam

Task 1.1: Identify the advantages and challenges of implementing Spark; Identify the role of core components of Apache Spark™'s Architecture, including cluster, driver node, worker nodes/executors, CPU cores, and memory; Identify the features of the Apache Spark Modules, including Core, Spark SQL, DataFrames, Pandas API on Spark, Structured Streaming, and MLib

Where distributed in-memory processing helps and where it costs, what the driver, executors, cores, memory and the cluster manager each do, and which Spark module fits a job.

Study it

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

On a YARN cluster, an analyst's interactive PySpark session cleans 40 GB of raw orders into a DataFrame named clean, then runs clean.cache() and clean.count(). A nightly job, submitted with spark-submit to the same cluster as a separate Spark application with its own SparkContext, must start from the same cleaned data without repeating the cleaning. What should the analyst do?

  1. ASchedule the job on the same worker nodes, so its tasks read the session's cached partitions
  2. BPersist clean with StorageLevel.DISK_ONLY, so the cached blocks are on the workers' disks where the job can find them
  3. CCall clean.cache() in the job's code before its first action, so the job attaches to the blocks that are already cached
  4. DWrite clean as a table with saveAsTable and have the job read that table instead of the raw orders
Show the answer and why
  • ASchedule the job on the same worker nodes, so its tasks read the session's cached partitions

    Incorrect

    Each application gets its own executor processes, and tasks of different applications run in different JVMs, so the job cannot read the session's cached partitions.

  • BPersist clean with StorageLevel.DISK_ONLY, so the cached blocks are on the workers' disks where the job can find them

    Incorrect

    Persisted blocks are kept by the session application's own executors, in memory or on disk; that is not storage another application can read.

  • CCall clean.cache() in the job's code before its first action, so the job attaches to the blocks that are already cached

    Incorrect

    Calling cache() in the job marks the job's own DataFrame for caching; it computes it from the job's lineage and does not reach blocks of another application.

  • DWrite clean as a table with saveAsTable and have the job read that table instead of the raw orders

    Correct

    Data is shared between Spark applications only through external storage; a saved table persists outside the session's executors.

Every Spark application has its own executors, which run its tasks and keep its data in memory or disk storage; applications are isolated from each other, so data cannot be shared across applications without writing it to an external storage system. cache() and persist() keep the partitions a DataFrame computes on the computing application's executors for reuse by later actions of that application, whatever the storage level. The job therefore needs the cleaned data written out, for example as a table, and reads it from there.

Question 2 · choose 1

A data scientist's pandas script (groupby, merge and fillna calls) works on a 2 GB sample. It must now run on the full 300 GB table in Unity Catalog, on a cluster whose driver has 64 GB of memory and whose eight workers have 128 GB each, with as little rewriting as possible. Which approach fits?

  1. ARead the table with spark.read.table, call toPandas() on it, and run the script on the result unchanged
  2. BRead the table's files on the driver with pandas and run the script there, with a larger spark.driver.memory
  3. CImport pyspark.pandas instead of pandas, read the table with it, and keep the pandas-style calls
  4. DCall toPandas() on each of eight filtered slices of the table and run the script once per slice
Show the answer and why
  • ARead the table with spark.read.table, call toPandas() on it, and run the script on the result unchanged

    Incorrect

    toPandas() loads all the data into the driver's memory and is meant for small results, so 300 GB does not fit on a 64 GB driver.

  • BRead the table's files on the driver with pandas and run the script there, with a larger spark.driver.memory

    Incorrect

    A pandas DataFrame lives on a single machine, so the workers' memory is never used and the driver must hold all 300 GB.

  • CImport pyspark.pandas instead of pandas, read the table with it, and keep the pandas-style calls

    Correct

    Pandas API on Spark runs pandas-style code distributed across the cluster's nodes, so the workload scales past one machine with few changes.

  • DCall toPandas() on each of eight filtered slices of the table and run the script once per slice

    Incorrect

    Each toPandas() call still loads its slice into the driver's memory, and a groupby or merge run per slice does not see the rows of the other slices.

toPandas() collects the whole DataFrame into the driver's memory, so it is meant for small results; a pandas DataFrame lives on a single machine, while a pandas-on-Spark DataFrame is distributed. Pandas API on Spark lets you scale a pandas workload to any size by running it distributed across multiple nodes, so swapping the import keeps most of the script and uses the workers.

Question 3 · choose 1

A company already runs a Kubernetes cluster for its other services and wants to start using Apache Spark without building a separate cluster. What does the Spark documentation say about this?

  1. ASpark needs its own standalone cluster manager, so a separate cluster must be built next to Kubernetes
  2. BSpark is agnostic to the cluster manager and can run on a shared Kubernetes cluster
  3. CSpark runs only on Hadoop YARN, so a YARN cluster must be installed first
  4. DSpark can run on Kubernetes only after every other service is moved off the cluster
Show the answer and why
  • ASpark needs its own standalone cluster manager, so a separate cluster must be built next to Kubernetes

    Incorrect

    Standalone is one of several supported cluster managers; Spark can also run on YARN or Kubernetes.

  • BSpark is agnostic to the cluster manager and can run on a shared Kubernetes cluster

    Correct

    As long as Spark can acquire executor processes, it can run on a cluster manager that also serves other applications, for example YARN or Kubernetes.

  • CSpark runs only on Hadoop YARN, so a YARN cluster must be installed first

    Incorrect

    YARN is one option among standalone, YARN and Kubernetes.

  • DSpark can run on Kubernetes only after every other service is moved off the cluster

    Incorrect

    Spark can share a cluster manager with other applications.

Spark applications run as independent sets of processes coordinated by the driver's SparkContext, which can connect to several types of cluster managers: Spark's standalone manager, Hadoop YARN or Kubernetes. Spark is agnostic to the underlying cluster manager: as long as it can acquire executor processes that communicate with each other, it is relatively easy to run even on a cluster manager that also supports other applications.

Question 4 · choose 1

Several Spark applications share an open-source cluster that runs Hadoop YARN. Which component allocates the cluster's resources across the applications?

  1. AThe driver program of the application that started first
  2. BThe executors, which share out the worker nodes among themselves
  3. CThe cluster manager, here YARN
  4. DThe worker node with the most free memory
Show the answer and why
  • AThe driver program of the application that started first

    Incorrect

    Each driver schedules the tasks of its own application; it does not allocate resources across applications.

  • BThe executors, which share out the worker nodes among themselves

    Incorrect

    Executors run tasks and store data for one application each.

  • CThe cluster manager, here YARN

    Correct

    The cluster manager is the external service that acquires resources on the cluster and allocates them across applications.

  • DThe worker node with the most free memory

    Incorrect

    A worker node is any node that can run application code; it does not allocate resources.

The SparkContext in the driver connects to a cluster manager, such as Spark's standalone manager, YARN or Kubernetes, which allocates resources across applications. Once connected, Spark acquires executors on worker nodes: processes that run computations and store data for that application. The driver runs the application's main() function and schedules its tasks.

Question 5 · choose 1

A team wants to build a machine learning pipeline on DataFrames, with feature transformers and a classifier, trained in a distributed way with Spark. Which Spark module provides this?

  1. AMLlib, Spark's machine learning library (the guide's MLib)
  2. BSpark SQL, through SQL queries over the training table
  3. CStructured Streaming, through a streaming query over the training data
  4. DPandas API on Spark, through pandas-style calls
Show the answer and why
  • AMLlib, Spark's machine learning library (the guide's MLib)

    Correct

    MLlib is Spark's machine learning library; its DataFrame-based API (pyspark.ml) provides pipelines, transformers and estimators.

  • BSpark SQL, through SQL queries over the training table

    Incorrect

    Spark SQL processes structured data with SQL and DataFrames; it has no classifiers or ML pipelines.

  • CStructured Streaming, through a streaming query over the training data

    Incorrect

    Structured Streaming is the stream processing engine; it does not provide ML algorithms.

  • DPandas API on Spark, through pandas-style calls

    Incorrect

    Pandas API on Spark offers pandas-like data APIs, not Spark's ML pipelines.

PySpark lists Spark SQL and DataFrames, pandas API on Spark, Structured Streaming, MLlib and Spark Core among its features. MLlib is Spark's machine learning library; its pyspark.ml API provides pipelines that chain feature transformers and estimators such as classifiers.

Practice domain 1 →Practice all domains →