Skip to content

Associate Developer for Apache Spark · Domain 5: Structured Streaming · 10% of the exam

Task 5.1: Explain the Structured Streaming engine in Spark, including its functions, programming model, micro-batch processing, exactly-once semantics, and fault tolerance mechanisms; Create and write Streaming DataFrames and Streaming Datasets, including the basic output modes and output sinks

The unbounded table model, micro-batches and triggers, checkpoints and exactly-once delivery, readStream and writeStream, and which output modes and sinks a query supports.

Study it

  • The Structured Streaming model: micro-batches, triggers, checkpoints and exactly-once

    Partly covered by: Message Queues

  • Output modes and sinks

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A Structured Streaming query has kept a running count of events per country, read from a Kafka topic, for several months. What does the engine keep from one trigger to the next to produce the counts?

  1. AEvery input row received so far, held as an unbounded input table
  2. BEvery input row, read again from the source at each trigger to recompute the counts
  3. COnly the running counts per country, updated with each trigger's new rows
  4. DNothing: each trigger counts only the rows that arrived during that trigger
Show the answer and why
  • AEvery input row received so far, held as an unbounded input table

    Incorrect

    Structured Streaming does not materialize the entire input table; it discards the source data after processing it.

  • BEvery input row, read again from the source at each trigger to recompute the counts

    Incorrect

    Each trigger reads the latest available data and processes it incrementally; earlier rows are not read again.

  • COnly the running counts per country, updated with each trigger's new rows

    Correct

    The engine keeps only the minimal intermediate state needed to update the result, here the running counts.

  • DNothing: each trigger counts only the rows that arrived during that trigger

    Incorrect

    The incremental query combines the previous running counts with the new data, so the counts carry over between triggers.

In the Structured Streaming model the input is treated as an unbounded table and the query as an incremental query on it, but Spark does not materialize the entire table. It reads the latest available data from the source, processes it incrementally to update the result, then discards the source data, keeping only the minimal intermediate state needed, such as the running counts.

Question 2 · choose 1

On a small test stream, a developer wants to inspect the output of a streaming filter query with spark.sql in the same notebook while the query runs, without writing files or tables. Which sink fits?

  1. AThe memory sink: writeStream.format("memory").queryName("test_out").start(), then query test_out
  2. BThe console sink: writeStream.format("console").start(), then query the console output with spark.sql
  3. CThe foreach sink: writeStream.foreach(handler).start(), then query the handler's rows
  4. DThe Kafka sink: writeStream.format("kafka").start(), then query the topic with spark.sql
Show the answer and why
  • AThe memory sink: writeStream.format("memory").queryName("test_out").start(), then query test_out

    Correct

    The memory sink stores the output as an in-memory table, and the query name is the table name to query with SQL.

  • BThe console sink: writeStream.format("console").start(), then query the console output with spark.sql

    Incorrect

    The console sink prints the output to the console at every trigger; it creates no table to query.

  • CThe foreach sink: writeStream.foreach(handler).start(), then query the handler's rows

    Incorrect

    foreach runs custom code on each output row; it does not register any table for SQL.

  • DThe Kafka sink: writeStream.format("kafka").start(), then query the topic with spark.sql

    Incorrect

    The Kafka sink writes the output to Kafka topics, outside Spark's catalog.

The console and memory sinks are meant for debugging on low data volumes, because their output is collected into the driver's memory. The memory sink stores the output as an in-memory table named by queryName, so spark.sql("SELECT * FROM test_out") reads it interactively; the console sink only prints each trigger's output.

Question 3 · choose 1

A production streaming job starts with this line, with default settings: events = spark.readStream.format("json").load("/Volumes/main/raw/events/") The line fails because no schema is given. The schema must stay the same across restarts and failures. What should the developer do?

  1. ASet spark.sql.streaming.schemaInference to true for the session
  2. BAdd .option("checkpointLocation", ckpt) to the reader
  3. CReplace spark.readStream with spark.read
  4. DAdd .schema("id STRING, ts TIMESTAMP, amount DOUBLE") before load
Show the answer and why
  • ASet spark.sql.streaming.schemaInference to true for the session

    Incorrect

    Re-enabling inference is meant for ad-hoc use; requiring a schema is what keeps it consistent across failures.

  • BAdd .option("checkpointLocation", ckpt) to the reader

    Incorrect

    A checkpoint location belongs to the writing side of a query and does not provide a schema.

  • CReplace spark.readStream with spark.read

    Incorrect

    That makes a batch read, which no longer processes new files as a stream.

  • DAdd .schema("id STRING, ts TIMESTAMP, amount DOUBLE") before load

    Correct

    File-based streaming sources require the schema to be specified by default, which ensures a consistent schema even in the case of failures.

By default, Structured Streaming from file-based sources requires you to specify the schema rather than rely on inference. The restriction ensures that a consistent schema is used for the streaming query, even after failures; spark.sql.streaming.schemaInference can re-enable inference for ad-hoc use cases.

Question 4 · choose 1

By default, how does Structured Streaming process a streaming query?

  1. AAs one long-running task per input record, processing each record on arrival with latencies of about 1 millisecond
  2. BAs a nightly batch job over the data collected during the day
  3. CWith a micro-batch engine that runs small batch jobs, with latencies as low as 100 milliseconds
  4. DBy converting the query to RDD operations on the Spark Streaming (DStream) engine
Show the answer and why
  • AAs one long-running task per input record, processing each record on arrival with latencies of about 1 millisecond

    Incorrect

    By default queries are processed as a series of small batch jobs, not record by record.

  • BAs a nightly batch job over the data collected during the day

    Incorrect

    The engine processes data continuously as it arrives, in small batches.

  • CWith a micro-batch engine that runs small batch jobs, with latencies as low as 100 milliseconds

    Correct

    By default, queries are processed by a micro-batch engine that runs small batch jobs, achieving end-to-end latencies as low as 100 milliseconds.

  • DBy converting the query to RDD operations on the Spark Streaming (DStream) engine

    Incorrect

    By default, Structured Streaming queries are processed by its own micro-batch processing engine, not by another streaming engine.

Structured Streaming is a scalable, fault-tolerant stream processing engine. Internally, by default, its queries are processed using a micro-batch processing engine, which processes data streams as a series of small batch jobs, achieving end-to-end latencies as low as 100 milliseconds.

Question 5 · choose 1

A Python script submitted with spark-submit ends with these lines, and the process exits right after printing started, while the stream should keep running. What should the developer add? query = (events.writeStream .option("checkpointLocation", ckpt) .toTable("bronze.events")) print("started")

  1. Aquery.stop() after the print
  2. Bquery.awaitTermination() after the print
  3. CA trigger(processingTime="0 seconds") before toTable
  4. DA second call to toTable with the same table
Show the answer and why
  • Aquery.stop() after the print

    Incorrect

    stop ends the query, the opposite of what is needed.

  • Bquery.awaitTermination() after the print

    Correct

    awaitTermination waits for the query to terminate, by stop() or by an exception, so the script keeps running while the stream runs.

  • CA trigger(processingTime="0 seconds") before toTable

    Incorrect

    The trigger decides when micro-batches run; it does not keep the script alive.

  • DA second call to toTable with the same table

    Incorrect

    Starting another query does not keep the process waiting for either query.

DataStreamWriter.toTable starts the execution of the streaming query, which keeps writing results to the table as new data arrives, and returns a StreamingQuery object. StreamingQuery.awaitTermination() waits for the termination of the query, either by query.stop() or by an exception (which it then throws), so calling it keeps the driver process alive.

Question 6 · choose 1

A developer wants to test a streaming query on a laptop with a steady flow of generated rows, without Kafka or files. Which built-in source fits?

  1. AThe Kafka source with a topic named test
  2. BThe file source on an empty directory
  3. CThe rate source, readStream.format("rate")
  4. DThe console sink
Show the answer and why
  • AThe Kafka source with a topic named test

    Incorrect

    The Kafka source reads from a Kafka cluster, which the developer wants to avoid.

  • BThe file source on an empty directory

    Incorrect

    A file source on an empty directory produces no rows until files arrive.

  • CThe rate source, readStream.format("rate")

    Correct

    The rate source, meant for testing, generates data at a specified number of rows per second, each with a timestamp and a value.

  • DThe console sink

    Incorrect

    The console sink prints a query's output; it is not a source of input rows.

Structured Streaming has built-in sources such as the file source and Kafka, plus sources meant for testing: the socket source, which reads text from a socket and gives no end-to-end fault-tolerance guarantees, the rate source, which generates rows with a timestamp and a value at a specified number of rows per second, and the rate per micro-batch source.

Practice domain 5 →Practice all domains →