Associate Developer for Apache Spark · Domain 5: Structured Streaming · 10% of the exam
Task 5.1: Explain the Structured Streaming engine in Spark, including its functions, programming model, micro-batch processing, exactly-once semantics, and fault tolerance mechanisms; Create and write Streaming DataFrames and Streaming Datasets, including the basic output modes and output sinks
The unbounded table model, micro-batches and triggers, checkpoints and exactly-once delivery, readStream and writeStream, and which output modes and sinks a query supports.
Study it
The Structured Streaming model: micro-batches, triggers, checkpoints and exactly-once
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A Structured Streaming query has kept a running count of events per country, read from a Kafka topic, for several months. What does the engine keep from one trigger to the next to produce the counts?
AEvery input row received so far, held as an unbounded input table
BEvery input row, read again from the source at each trigger to recompute the counts
COnly the running counts per country, updated with each trigger's new rows
DNothing: each trigger counts only the rows that arrived during that trigger
Show the answer and why
AEvery input row received so far, held as an unbounded input table
Incorrect
Structured Streaming does not materialize the entire input table; it discards the source data after processing it.
BEvery input row, read again from the source at each trigger to recompute the counts
Incorrect
Each trigger reads the latest available data and processes it incrementally; earlier rows are not read again.
COnly the running counts per country, updated with each trigger's new rows
Correct
The engine keeps only the minimal intermediate state needed to update the result, here the running counts.
DNothing: each trigger counts only the rows that arrived during that trigger
Incorrect
The incremental query combines the previous running counts with the new data, so the counts carry over between triggers.
In the Structured Streaming model the input is treated as an unbounded table and the query as an incremental query on it, but Spark does not materialize the entire table. It reads the latest available data from the source, processes it incrementally to update the result, then discards the source data, keeping only the minimal intermediate state needed, such as the running counts.
On a small test stream, a developer wants to inspect the output of a streaming filter query with spark.sql in the same notebook while the query runs, without writing files or tables. Which sink fits?
AThe memory sink: writeStream.format("memory").queryName("test_out").start(), then query test_out
BThe console sink: writeStream.format("console").start(), then query the console output with spark.sql
CThe foreach sink: writeStream.foreach(handler).start(), then query the handler's rows
DThe Kafka sink: writeStream.format("kafka").start(), then query the topic with spark.sql
Show the answer and why
AThe memory sink: writeStream.format("memory").queryName("test_out").start(), then query test_out
Correct
The memory sink stores the output as an in-memory table, and the query name is the table name to query with SQL.
BThe console sink: writeStream.format("console").start(), then query the console output with spark.sql
Incorrect
The console sink prints the output to the console at every trigger; it creates no table to query.
CThe foreach sink: writeStream.foreach(handler).start(), then query the handler's rows
Incorrect
foreach runs custom code on each output row; it does not register any table for SQL.
DThe Kafka sink: writeStream.format("kafka").start(), then query the topic with spark.sql
Incorrect
The Kafka sink writes the output to Kafka topics, outside Spark's catalog.
The console and memory sinks are meant for debugging on low data volumes, because their output is collected into the driver's memory. The memory sink stores the output as an in-memory table named by queryName, so spark.sql("SELECT * FROM test_out") reads it interactively; the console sink only prints each trigger's output.
A production streaming job starts with this line, with default settings:
events = spark.readStream.format("json").load("/Volumes/main/raw/events/")
The line fails because no schema is given. The schema must stay the same
across restarts and failures. What should the developer do?
ASet spark.sql.streaming.schemaInference to true for the session
BAdd .option("checkpointLocation", ckpt) to the reader
CReplace spark.readStream with spark.read
DAdd .schema("id STRING, ts TIMESTAMP, amount DOUBLE") before load
Show the answer and why
ASet spark.sql.streaming.schemaInference to true for the session
Incorrect
Re-enabling inference is meant for ad-hoc use; requiring a schema is what keeps it consistent across failures.
BAdd .option("checkpointLocation", ckpt) to the reader
Incorrect
A checkpoint location belongs to the writing side of a query and does not provide a schema.
CReplace spark.readStream with spark.read
Incorrect
That makes a batch read, which no longer processes new files as a stream.
DAdd .schema("id STRING, ts TIMESTAMP, amount DOUBLE") before load
Correct
File-based streaming sources require the schema to be specified by default, which ensures a consistent schema even in the case of failures.
By default, Structured Streaming from file-based sources requires you to specify the schema rather than rely on inference. The restriction ensures that a consistent schema is used for the streaming query, even after failures; spark.sql.streaming.schemaInference can re-enable inference for ad-hoc use cases.
By default, how does Structured Streaming process a streaming query?
AAs one long-running task per input record, processing each record on arrival with latencies of about 1 millisecond
BAs a nightly batch job over the data collected during the day
CWith a micro-batch engine that runs small batch jobs, with latencies as low as 100 milliseconds
DBy converting the query to RDD operations on the Spark Streaming (DStream) engine
Show the answer and why
AAs one long-running task per input record, processing each record on arrival with latencies of about 1 millisecond
Incorrect
By default queries are processed as a series of small batch jobs, not record by record.
BAs a nightly batch job over the data collected during the day
Incorrect
The engine processes data continuously as it arrives, in small batches.
CWith a micro-batch engine that runs small batch jobs, with latencies as low as 100 milliseconds
Correct
By default, queries are processed by a micro-batch engine that runs small batch jobs, achieving end-to-end latencies as low as 100 milliseconds.
DBy converting the query to RDD operations on the Spark Streaming (DStream) engine
Incorrect
By default, Structured Streaming queries are processed by its own micro-batch processing engine, not by another streaming engine.
Structured Streaming is a scalable, fault-tolerant stream processing engine. Internally, by default, its queries are processed using a micro-batch processing engine, which processes data streams as a series of small batch jobs, achieving end-to-end latencies as low as 100 milliseconds.
A Python script submitted with spark-submit ends with these lines, and the
process exits right after printing started, while the stream should keep
running. What should the developer add?
query = (events.writeStream
.option("checkpointLocation", ckpt)
.toTable("bronze.events"))
print("started")
Aquery.stop() after the print
Bquery.awaitTermination() after the print
CA trigger(processingTime="0 seconds") before toTable
DA second call to toTable with the same table
Show the answer and why
Aquery.stop() after the print
Incorrect
stop ends the query, the opposite of what is needed.
Bquery.awaitTermination() after the print
Correct
awaitTermination waits for the query to terminate, by stop() or by an exception, so the script keeps running while the stream runs.
CA trigger(processingTime="0 seconds") before toTable
Incorrect
The trigger decides when micro-batches run; it does not keep the script alive.
DA second call to toTable with the same table
Incorrect
Starting another query does not keep the process waiting for either query.
DataStreamWriter.toTable starts the execution of the streaming query, which keeps writing results to the table as new data arrives, and returns a StreamingQuery object. StreamingQuery.awaitTermination() waits for the termination of the query, either by query.stop() or by an exception (which it then throws), so calling it keeps the driver process alive.
A developer wants to test a streaming query on a laptop with a steady flow of generated rows, without Kafka or files. Which built-in source fits?
AThe Kafka source with a topic named test
BThe file source on an empty directory
CThe rate source, readStream.format("rate")
DThe console sink
Show the answer and why
AThe Kafka source with a topic named test
Incorrect
The Kafka source reads from a Kafka cluster, which the developer wants to avoid.
BThe file source on an empty directory
Incorrect
A file source on an empty directory produces no rows until files arrive.
CThe rate source, readStream.format("rate")
Correct
The rate source, meant for testing, generates data at a specified number of rows per second, each with a timestamp and a value.
DThe console sink
Incorrect
The console sink prints a query's output; it is not a source of input rows.
Structured Streaming has built-in sources such as the file source and Kafka, plus sources meant for testing: the socket source, which reads text from a socket and gives no end-to-end fault-tolerance guarantees, the rate source, which generates rows with a timestamp and a value at a specified number of rows per second, and the rate per micro-batch source.