Associate Developer for Apache Spark · Domain 1: Apache Spark Architecture and Components · 20% of the exam
Task 1.2: Describe the architecture of Apache Spark™, including DataFrame and Dataset concepts, SparkSession lifecycle, caching, storage levels, and garbage collection
DataFrames against Datasets in Python, creating and stopping a SparkSession, cache and persist with their storage levels, and reading garbage collection pressure.
Study it
DataFrames, Datasets and the SparkSession
Lesson coming
Caching, persist and storage levels; garbage collection
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
On a classic cluster, a notebook prepares a view for several dashboards:
orders = spark.read.parquet("/data/orders")
orders.createOrReplaceTempView("orders_v")
The first dashboard query against orders_v must already find the data in the
cache. The developer adds one more statement, run on its own right after these
two lines, with no query or action after it. Which statement fills the cache
at that point?
Aspark.sql("CACHE LAZY TABLE orders_v")
Borders.cache()
Corders.persist()
Dspark.sql("CACHE TABLE orders_v")
Show the answer and why
Aspark.sql("CACHE LAZY TABLE orders_v")
Incorrect
With LAZY, the table is cached only when it is first used, so the first dashboard query does the caching.
Borders.cache()
Incorrect
cache() persists the DataFrame, and a persisted DataFrame keeps its contents only after the first time it is computed, which needs an action.
Corders.persist()
Incorrect
persist sets a storage level that keeps the contents after the first time the DataFrame is computed; nothing is computed yet.
Dspark.sql("CACHE TABLE orders_v")
Correct
CACHE TABLE caches the view's contents immediately; only the LAZY keyword would defer it to the first use.
The CACHE TABLE statement caches the contents of a table or view, and its LAZY keyword means "only cache the table when it is first used, instead of immediately", so without LAZY the caching happens when the statement runs. DataFrame.persist (and cache, which persists with the default storage level) keeps the contents across operations after the first time the DataFrame is computed, so on its own it fills nothing until an action runs.
Developers moving from Scala ask for typed Datasets in a new PySpark job: the names and types of the order columns must be declared in the code rather than worked out from the data at runtime. Which approach meets the requirement in PySpark?
With a list of column names as the schema, the type of each column is inferred from the data rather than declared.
The Dataset API is available in Scala and Java; Python does not have support for it, and in PySpark the DataFrame is the structured abstraction. To fix names and types up front, pass createDataFrame a schema: a StructType or a DDL-formatted string. When the schema is None or a list of column names, the types are inferred from the data.
A standalone PySpark script, run with spark-submit, must read Parquet files into DataFrames and run SQL on them. Which object is the entry point it should create first?
AA DataFrameReader, created with DataFrameReader()
BA SparkSession, created with SparkSession.builder ... getOrCreate()
CA pandas-on-Spark DataFrame, created with pyspark.pandas
DA Row object, created with Row()
Show the answer and why
AA DataFrameReader, created with DataFrameReader()
Incorrect
A DataFrameReader is obtained from a session as spark.read; it is not the entry point.
BA SparkSession, created with SparkSession.builder ... getOrCreate()
Correct
The entry point into all functionality in Spark is the SparkSession class, created with SparkSession.builder.
CA pandas-on-Spark DataFrame, created with pyspark.pandas
Incorrect
Pandas API on Spark is one of the APIs used through a session; it is not the entry point.
DA Row object, created with Row()
Incorrect
A Row represents one record of a DataFrame; it does not start Spark.
The entry point into all functionality in Spark is the SparkSession class. A script creates one with the builder pattern, for example SparkSession.builder.appName("app").getOrCreate(), and then uses it for spark.read, spark.sql and the other DataFrame APIs; notebooks and the pyspark shell provide one as spark.
A developer checks a persisted DataFrame and df.storageLevel returns StorageLevel(True, False, False, False, 2). The constructor arguments are useDisk, useMemory, useOffHeap, deserialized and replication. How is df stored?
AOn disk only, with each partition replicated on two nodes
BIn memory only, with two copies of each partition
CIn memory and on disk, deserialized, in two partitions
DOff-heap, replicated on two nodes
Show the answer and why
AOn disk only, with each partition replicated on two nodes
Correct
useDisk is True, useMemory False and replication 2: the level of DISK_ONLY_2.
BIn memory only, with two copies of each partition
Incorrect
useMemory is False, so the blocks are not kept in memory.
CIn memory and on disk, deserialized, in two partitions
Incorrect
useMemory and deserialized are False, and the last argument is replication, not a partition count.
DOff-heap, replicated on two nodes
Incorrect
useOffHeap is False.
A StorageLevel records whether to use memory, whether to drop data to disk, whether to keep data in a JVM-specific serialized format and whether to replicate partitions on multiple nodes; its constructor takes useDisk, useMemory, useOffHeap, deserialized and replication. StorageLevel(True, False, False, False, 2) is DISK_ONLY_2, which the API reference shows for persist(StorageLevel.DISK_ONLY_2).
GC statistics of a Spark job show too many minor collections but not many major collections. Following the Spark tuning guide, what would help?
ALowering spark.memory.fraction so that fewer objects are cached in the Old generation
BCalling cache() on every DataFrame the job reuses
CAllocating more memory for Eden, the young generation's main region
DNothing: minor collections never affect a Spark job
Show the answer and why
ALowering spark.memory.fraction so that fewer objects are cached in the Old generation
Incorrect
The guide suggests that when the Old generation is close to full, not for frequent minor collections.
BCalling cache() on every DataFrame the job reuses
Incorrect
Caching more objects adds to memory pressure; it is not the guide's step for frequent minor collections.
CAllocating more memory for Eden, the young generation's main region
Correct
With many minor collections and few major ones, the guide says allocating more memory for Eden would help.
DNothing: minor collections never affect a Spark job
Incorrect
The guide treats too many collections, minor or full, as something to tune.
The JVM heap has a Young generation, with Eden and two Survivor regions, and an Old generation. A minor GC runs when Eden is full; a full GC runs when Old is close to full. The tuning guide says that if there are too many minor collections but not many major GCs, allocating more memory for Eden would help, and that if Old is close to full, you should reduce the memory used for caching by lowering spark.memory.fraction.