Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A notebook on a classic Databricks cluster runs this cell:
from pyspark.sql import functions as F
@F.udf("string")
def norm(s):
print(f"normalizing {s}")
return s.strip().lower()
print("starting load")
spark.read.table("raw.customers").select(norm("name").alias("name")).write.mode("overwrite").saveAsTable("silver.customers")
Where can the developer find the two kinds of printed lines?
ABoth in the notebook's driver logs, because the whole cell runs in the notebook
BBoth in the executors' stdout, because the cell runs as a Spark job
COnly "starting load": Spark discards print output from inside a UDF
D"starting load" in the driver logs; the "normalizing" lines in executor stdout, under the Spark UI's Executors tab
Show the answer and why
ABoth in the notebook's driver logs, because the whole cell runs in the notebook
Incorrect
The UDF runs on the executors, so its prints are not visible on the driver side.
BBoth in the executors' stdout, because the cell runs as a Spark job
Incorrect
A print in the notebook code itself runs on the driver and goes to the driver logs.
COnly "starting load": Spark discards print output from inside a UDF
Incorrect
The UDF's prints are kept; they appear on the executors that ran the function.
D"starting load" in the driver logs; the "normalizing" lines in executor stdout, under the Spark UI's Executors tab
Correct
Notebook prints go to the Spark driver logs, and prints inside a Python UDF run on the executors and show under stdout in the Executors tab.
Direct print and log statements from notebooks, jobs and libraries go to the Spark driver logs, reached from the Driver logs tab of the compute. A Python UDF runs on the executors, so print statements inside it are not visible on the client or driver side; they appear in the Spark UI under stdout and stderr in the Executors tab. Worker logs are reached through the Spark UI tab of the compute.
On a classic cluster, the driver output of a PySpark job is flooded with INFO messages. The developer wants only warnings and more severe messages from then on. Which call does that?
Aspark.conf.set("spark.log.level", "quiet")
Bprint("WARN")
Cspark.sparkContext.setLogLevel("INFO")
Dspark.sparkContext.setLogLevel("WARN")
Show the answer and why
Aspark.conf.set("spark.log.level", "quiet")
Incorrect
quiet is not one of the valid log levels.
Bprint("WARN")
Incorrect
This prints a string; it does not change the log level.
Cspark.sparkContext.setLogLevel("INFO")
Incorrect
INFO is the level of the messages that flood the output, so setting it keeps them; WARN is the valid level the developer needs.
Dspark.sparkContext.setLogLevel("WARN")
Correct
setLogLevel controls the log level, overriding user-defined log settings; WARN is one of the valid levels.
SparkContext.setLogLevel(logLevel) controls the log level and overrides any user-defined log settings. Valid log levels include ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE and WARN; setting WARN keeps warnings and more severe messages.
A developer wants to confirm which Spark properties a running open-source application actually uses, for example the value of spark.executor.memory. Where in the application web UI are they listed?
AIn the Storage tab
BIn the Environment tab
CIn the Jobs tab
DIn the SQL tab
Show the answer and why
AIn the Storage tab
Incorrect
The Storage tab shows persisted RDDs and DataFrames.
BIn the Environment tab
Correct
The application web UI at http://<driver>:4040 lists Spark properties in the Environment tab.
CIn the Jobs tab
Incorrect
The Jobs tab lists the jobs of the application, not its properties.
DIn the SQL tab
Incorrect
The SQL tab lists SQL queries and their plans.
The application web UI, at http://<driver>:4040, lists Spark properties in the Environment tab, a useful place to check that properties were set correctly. Only values set explicitly through spark-defaults.conf, SparkConf or the command line appear there; for the others, the default value is used.
A Databricks notebook submits eight Spark jobs from eight Python threads at once. In the Spark UI's event timeline, all eight jobs appear to start at the same moment, although the cluster runs their tasks one after another. What does the Spark UI guide say about this?
AThe timeline is broken for jobs submitted from threads, so it should be read from the Stages tab instead
BThe jobs share one task, which the timeline shows eight times
CThe cluster is overloaded, which forces all jobs to start together
DIt is expected: a job shows as running from its submission, not from its first task
Show the answer and why
AThe timeline is broken for jobs submitted from threads, so it should be read from the Stages tab instead
Incorrect
The timeline works as designed; it shows a job from its submission.
BThe jobs share one task, which the timeline shows eight times
Incorrect
Each job has its own tasks; the timeline shows when each job was submitted.
CThe cluster is overloaded, which forces all jobs to start together
Incorrect
Simultaneous starts come from parallel submission, not from an overloaded cluster.
DIt is expected: a job shows as running from its submission, not from its first task
Correct
Jobs submitted in parallel, for example from multiple threads, can all appear to start at the same time; this is expected and not a problem on its own.
The jobs timeline shows which jobs run, how long each takes and whether jobs or executors fail. A job appears as running from the moment it is submitted, not from when its first task starts, so jobs submitted in parallel, for example with multiple threads and futures, can all appear to start at the same time. This is expected and does not indicate a problem on its own.