Skip to content

Associate Developer for Apache Spark · Domain 6: Using Spark Connect to Deploy Applications · 5% of the exam

Task 6.1: Describe the features of Spark Connect; Describe the different deployment mode types (Client, Cluster, Local) in the Apache Spark™ environment

The Spark Connect client and server, what it supports and what it does not, how its lazy analysis differs from Spark Classic, and where the driver runs in client, cluster and local mode.

Study it

  • Spark Connect: client, server, supported APIs and lazy analysis

    Lesson coming

  • Deploy modes: client, cluster and local

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A platform team runs open-source Spark 4.3 for many application teams whose PySpark applications follow their own release cycles. The platform team wants to upgrade the Spark driver for performance and security fixes on its own schedule, without every application having to be rebuilt and redeployed together with the driver. Which way of running the applications does the Spark documentation describe for this?

  1. AAs clients of a Spark Connect server that the platform team upgrades
  2. BWith spark-submit in cluster deploy mode, so that the cluster starts each driver
  3. CWith spark-submit in client deploy mode from a gateway machine near the cluster
  4. DWith --master local[*] on each application's own server
Show the answer and why
  • AAs clients of a Spark Connect server that the platform team upgrades

    Correct

    With Spark Connect the client does not run in the same process as the driver, and the driver can be upgraded independently of applications.

  • BWith spark-submit in cluster deploy mode, so that the cluster starts each driver

    Incorrect

    In cluster mode the driver is still the process that runs the application's main() function.

  • CWith spark-submit in client deploy mode from a gateway machine near the cluster

    Incorrect

    Wherever it is submitted from, the driver is the process that runs the application's main() function, so driver and application are one program.

  • DWith --master local[*] on each application's own server

    Incorrect

    Local mode runs Spark locally on the application's own server, and the driver is still the application's own main() process.

In a classic Spark application the driver program is the process that runs the application's main() function, so the driver and the application are one program in every deploy mode. Spark Connect separates the client from the server: the client does not run in the same process as the Spark driver, and one of its operational benefits is upgradability, since the Spark driver can be upgraded independently of applications, as long as the server-side RPC definitions stay backwards compatible.

Question 2 · choose 1

From an edge node of a YARN cluster, a developer runs spark-submit --master yarn report.py, with no other options. The first thing report.py does is set spark.driver.memory to 16g in the configuration of the session it then creates. What memory does the driver get?

  1. A16 GB, because the script sets the property before its session exists
  2. BThe default, as the driver JVM is already running; pass --driver-memory 16g
  3. C16 GB, because a YARN master makes cluster mode the default deploy mode
  4. DThe default, because driver memory can change only through spark.conf.set
Show the answer and why
  • A16 GB, because the script sets the property before its session exists

    Incorrect

    In client mode this property must not be set through the SparkConf in the application, because the driver JVM has already started at that point.

  • BThe default, as the driver JVM is already running; pass --driver-memory 16g

    Correct

    Without --deploy-mode, spark-submit runs in client mode and launches the driver within its own process, so the setting comes too late; the command-line option sets it before the JVM starts.

  • C16 GB, because a YARN master makes cluster mode the default deploy mode

    Incorrect

    The default deploy mode is client whatever the master, so the driver runs in the spark-submit process on the edge node.

  • DThe default, because driver memory can change only through spark.conf.set

    Incorrect

    The reference says to set it with the --driver-memory command-line option or in the default properties file; deploy-related properties may not take effect when set at run time.

spark-submit's --deploy-mode is client by default, and in client mode the driver is launched directly within the spark-submit process. The spark.driver.memory entry of the configuration reference notes that in client mode the property must not be set through the SparkConf directly in the application, because the driver JVM has already started at that point; set it with --driver-memory or in the default properties file instead.

Question 3 · choose 1

In Spark Connect, what does the client send to the Spark server to describe the operations to run?

  1. ACompiled Python bytecode of the whole application
  2. BUnresolved logical plans, encoded as protocol buffers and sent over gRPC
  3. CPhysical plans that the client has already optimized
  4. DSQL strings only, so all DataFrame code must first be rewritten as SQL queries
Show the answer and why
  • ACompiled Python bytecode of the whole application

    Incorrect

    The protocol describes operations as logical plans, not as compiled code.

  • BUnresolved logical plans, encoded as protocol buffers and sent over gRPC

    Correct

    Spark Connect uses unresolved logical plans as its protocol, sent with protocol buffers over gRPC.

  • CPhysical plans that the client has already optimized

    Incorrect

    The client sends unresolved logical plans; the server optimizes and runs them.

  • DSQL strings only, so all DataFrame code must first be rewritten as SQL queries

    Incorrect

    DataFrame operations are translated into logical plans by the client.

Spark Connect decouples the client from the Spark server. The client translates DataFrame operations into unresolved logical query plans, encoded with protocol buffers and sent to the server over gRPC; the server optimizes and runs them, and results stream back to the client as Apache Arrow-encoded row batches.

Question 4 · choose 1

A Python application must connect to a Spark Connect server running at host spark.example.com on port 15002. Which code creates the session?

  1. ASparkSession.builder.remote("sc://spark.example.com:15002").getOrCreate()
  2. BSparkSession.builder.master("spark://spark.example.com:15002").getOrCreate()
  3. CSparkContext("sc://spark.example.com:15002")
  4. DSparkSession.builder.config("spark.connect.port", "15002").getOrCreate()
Show the answer and why
  • ASparkSession.builder.remote("sc://spark.example.com:15002").getOrCreate()

    Correct

    SparkSession.builder.remote takes the sc:// connection string of the Spark Connect server.

  • BSparkSession.builder.master("spark://spark.example.com:15002").getOrCreate()

    Incorrect

    master sets a cluster manager URL for a classic session; a Spark Connect server is reached through remote and sc://.

  • CSparkContext("sc://spark.example.com:15002")

    Incorrect

    A Spark Connect server is reached through a remote SparkSession built with builder.remote, not by constructing a SparkContext.

  • DSparkSession.builder.config("spark.connect.port", "15002").getOrCreate()

    Incorrect

    This sets a configuration value in a local session; it does not connect to the remote server.

A Spark Connect session is created programmatically with SparkSession.builder.remote("sc://host:port").getOrCreate(); the same connection string can be set with the SPARK_REMOTE environment variable or passed to pyspark --remote. By default the client connects to port 15002.

Question 5 · choose 1

An open-source PySpark application is submitted with spark-submit --master yarn. How does Spark find the YARN cluster to connect to?

  1. AFrom the host name written after yarn in the master URL
  2. BFrom a broadcast of all hosts on the local network
  3. CFrom the spark.driver.host property alone
  4. DFrom the configuration files in the directory that HADOOP_CONF_DIR or YARN_CONF_DIR points to
Show the answer and why
  • AFrom the host name written after yarn in the master URL

    Incorrect

    The master URL is just yarn; the cluster location comes from the Hadoop configuration.

  • BFrom a broadcast of all hosts on the local network

    Incorrect

    Spark does not search the network; it reads the Hadoop configuration files.

  • CFrom the spark.driver.host property alone

    Incorrect

    spark.driver.host is the driver's own address; it does not locate the YARN cluster.

  • DFrom the configuration files in the directory that HADOOP_CONF_DIR or YARN_CONF_DIR points to

    Correct

    With --master yarn, the cluster location is found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable.

To run on YARN, HADOOP_CONF_DIR or YARN_CONF_DIR must point to the directory that contains the client-side configuration files for the Hadoop cluster. With the master URL yarn, Spark connects to the cluster found through those files, in client or cluster mode depending on --deploy-mode.

Practice domain 6 →Practice all domains →