Skip to content

Associate Developer for Apache Spark · Domain 4: Troubleshooting and Tuning Apache Spark DataFrame API Applications · 10% of the exam

Task 4.2: Perform logging and monitoring of Spark applications - publish, customize, and analyze Driver logs and Executor logs to diagnose out-of-memory errors, cluster underutilization, etc

Where driver and executor logs and the Spark UI live, setting log levels, and reading them for out-of-memory errors and idle cores.

Study it

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A notebook on a classic Databricks cluster runs this cell: from pyspark.sql import functions as F @F.udf("string") def norm(s): print(f"normalizing {s}") return s.strip().lower() print("starting load") spark.read.table("raw.customers").select(norm("name").alias("name")).write.mode("overwrite").saveAsTable("silver.customers") Where can the developer find the two kinds of printed lines?

  1. ABoth in the notebook's driver logs, because the whole cell runs in the notebook
  2. BBoth in the executors' stdout, because the cell runs as a Spark job
  3. COnly "starting load": Spark discards print output from inside a UDF
  4. D"starting load" in the driver logs; the "normalizing" lines in executor stdout, under the Spark UI's Executors tab
Show the answer and why
  • ABoth in the notebook's driver logs, because the whole cell runs in the notebook

    Incorrect

    The UDF runs on the executors, so its prints are not visible on the driver side.

  • BBoth in the executors' stdout, because the cell runs as a Spark job

    Incorrect

    A print in the notebook code itself runs on the driver and goes to the driver logs.

  • COnly "starting load": Spark discards print output from inside a UDF

    Incorrect

    The UDF's prints are kept; they appear on the executors that ran the function.

  • D"starting load" in the driver logs; the "normalizing" lines in executor stdout, under the Spark UI's Executors tab

    Correct

    Notebook prints go to the Spark driver logs, and prints inside a Python UDF run on the executors and show under stdout in the Executors tab.

Direct print and log statements from notebooks, jobs and libraries go to the Spark driver logs, reached from the Driver logs tab of the compute. A Python UDF runs on the executors, so print statements inside it are not visible on the client or driver side; they appear in the Spark UI under stdout and stderr in the Executors tab. Worker logs are reached through the Spark UI tab of the compute.

Question 2 · choose 1

On a classic cluster, the driver output of a PySpark job is flooded with INFO messages. The developer wants only warnings and more severe messages from then on. Which call does that?

  1. Aspark.conf.set("spark.log.level", "quiet")
  2. Bprint("WARN")
  3. Cspark.sparkContext.setLogLevel("INFO")
  4. Dspark.sparkContext.setLogLevel("WARN")
Show the answer and why
  • Aspark.conf.set("spark.log.level", "quiet")

    Incorrect

    quiet is not one of the valid log levels.

  • Bprint("WARN")

    Incorrect

    This prints a string; it does not change the log level.

  • Cspark.sparkContext.setLogLevel("INFO")

    Incorrect

    INFO is the level of the messages that flood the output, so setting it keeps them; WARN is the valid level the developer needs.

  • Dspark.sparkContext.setLogLevel("WARN")

    Correct

    setLogLevel controls the log level, overriding user-defined log settings; WARN is one of the valid levels.

SparkContext.setLogLevel(logLevel) controls the log level and overrides any user-defined log settings. Valid log levels include ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE and WARN; setting WARN keeps warnings and more severe messages.

Question 3 · choose 1

A developer wants to confirm which Spark properties a running open-source application actually uses, for example the value of spark.executor.memory. Where in the application web UI are they listed?

  1. AIn the Storage tab
  2. BIn the Environment tab
  3. CIn the Jobs tab
  4. DIn the SQL tab
Show the answer and why
  • AIn the Storage tab

    Incorrect

    The Storage tab shows persisted RDDs and DataFrames.

  • BIn the Environment tab

    Correct

    The application web UI at http://<driver>:4040 lists Spark properties in the Environment tab.

  • CIn the Jobs tab

    Incorrect

    The Jobs tab lists the jobs of the application, not its properties.

  • DIn the SQL tab

    Incorrect

    The SQL tab lists SQL queries and their plans.

The application web UI, at http://<driver>:4040, lists Spark properties in the Environment tab, a useful place to check that properties were set correctly. Only values set explicitly through spark-defaults.conf, SparkConf or the command line appear there; for the others, the default value is used.

Question 4 · choose 1

A Databricks notebook submits eight Spark jobs from eight Python threads at once. In the Spark UI's event timeline, all eight jobs appear to start at the same moment, although the cluster runs their tasks one after another. What does the Spark UI guide say about this?

  1. AThe timeline is broken for jobs submitted from threads, so it should be read from the Stages tab instead
  2. BThe jobs share one task, which the timeline shows eight times
  3. CThe cluster is overloaded, which forces all jobs to start together
  4. DIt is expected: a job shows as running from its submission, not from its first task
Show the answer and why
  • AThe timeline is broken for jobs submitted from threads, so it should be read from the Stages tab instead

    Incorrect

    The timeline works as designed; it shows a job from its submission.

  • BThe jobs share one task, which the timeline shows eight times

    Incorrect

    Each job has its own tasks; the timeline shows when each job was submitted.

  • CThe cluster is overloaded, which forces all jobs to start together

    Incorrect

    Simultaneous starts come from parallel submission, not from an overloaded cluster.

  • DIt is expected: a job shows as running from its submission, not from its first task

    Correct

    Jobs submitted in parallel, for example from multiple threads, can all appear to start at the same time; this is expected and not a problem on its own.

The jobs timeline shows which jobs run, how long each takes and whether jobs or executors fail. A job appears as running from the moment it is submitted, not from when its first task starts, so jobs submitted in parallel, for example with multiple threads and futures, can all appear to start at the same time. This is expected and does not indicate a problem on its own.

Databricks documentation

Practice domain 4 →Practice all domains →