Associate Developer for Apache Spark · Domain 5: Structured Streaming · 10% of the exam
Task 5.2: Perform basic operations on Streaming DataFrames and Streaming Datasets, such as selection, projection, window and aggregation; Perform Streaming Deduplication in Structured Streaming, both with and without watermark usage
Event-time windows and watermarks, what each output mode emits for a windowed aggregate, operations streams do not support, and deduplication with and without a watermark.
Streaming deduplication with and without a watermark
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A streaming query over the event stream events runs this code:
from pyspark.sql import functions as F
counts = (events
.withWatermark("event_time", "10 minutes")
.groupBy(F.window("event_time", "5 minutes"))
.count())
(counts.writeStream.outputMode("update")
.option("checkpointLocation", ckpt).toTable("ops.window_counts"))
Which statement describes what the query writes and keeps?
AEach trigger writes only the windows that the watermark has finalized, and their state is then dropped
BEach trigger writes the windows whose counts changed, and state for windows older than the watermark is dropped
CEach trigger writes the windows whose counts changed, but no state is ever dropped, because only Append mode uses the watermark
DThe query fails to start, because an aggregation with a watermark can be written only in Append mode
Show the answer and why
AEach trigger writes only the windows that the watermark has finalized, and their state is then dropped
Incorrect
Update mode writes the rows that changed since the last trigger, so a window's partial counts are written before the window is final.
BEach trigger writes the windows whose counts changed, and state for windows older than the watermark is dropped
Correct
Update mode writes the result rows that changed since the last trigger, and a watermark on the aggregation's event-time column, set before the aggregation, cleans old state in Update mode.
CEach trigger writes the windows whose counts changed, but no state is ever dropped, because only Append mode uses the watermark
Incorrect
Watermarking cleans aggregation state when the output mode is Append or Update.
DThe query fails to start, because an aggregation with a watermark can be written only in Append mode
Incorrect
A watermarked aggregation supports more than Append; Update is allowed and uses the watermark to drop old state.
In Update mode, only the rows of the result table that were updated since the last trigger are written, so the current windows appear with partial counts at every trigger. A watermark cleans aggregation state when the output mode is Append or Update, the aggregation uses the event-time column or a window on it, and withWatermark is called on that same column before the aggregation, as here.
A streaming query runs this code:
deduped = spark.readStream.table("raw.events").dropDuplicates(["event_id"])
(deduped.writeStream
.option("checkpointLocation", ckpt)
.toTable("silver.events"))
The first micro-batch brings events with event_id a, b and a; the second
brings a and c. Which event_id values reach silver.events?
Aa and b from the first micro-batch, then a and c from the second
Ba, a and b from the first micro-batch, then a and c from the second
Ca and b from the first micro-batch, then only c from the second
DNone: the query fails to start, because streaming deduplication requires a watermark
Show the answer and why
Aa and b from the first micro-batch, then a and c from the second
Incorrect
Streaming deduplication stores data from previous records to filter duplicates, so it also catches a duplicate from an earlier micro-batch.
Ba, a and b from the first micro-batch, then a and c from the second
Incorrect
dropDuplicates removes duplicates in a streaming query just as it does on a static DataFrame.
Ca and b from the first micro-batch, then only c from the second
Correct
The query filters records whose event_id it has seen in earlier micro-batches, so the second a is not written again.
DNone: the query fails to start, because streaming deduplication requires a watermark
Incorrect
Streaming deduplication can be used with or without a watermark.
Streaming deduplication with a unique identifier works like deduplication on a static DataFrame: the query stores the data it needs from previous records so that it can filter duplicates, also across micro-batches, and it can be used with or without a watermark. Here a and b are written in the first micro-batch and only c in the second.
devices is a streaming DataFrame with the columns device, signal and time. Which statement about df = devices.select("device").where("signal > 10") is true?
AIt defines a streaming DataFrame of devices with signal above 10
BIt fails, because select is not supported on streaming DataFrames
CIt returns a static DataFrame holding the rows received so far
DIt works only after a watermark is defined on time
Show the answer and why
AIt defines a streaming DataFrame of devices with signal above 10
Correct
Most common DataFrame operations, such as select and where, are supported on streaming DataFrames.
BIt fails, because select is not supported on streaming DataFrames
Incorrect
Selection and projection are supported on streaming DataFrames.
CIt returns a static DataFrame holding the rows received so far
Incorrect
Applying operations to a streaming DataFrame gives another streaming DataFrame.
DIt works only after a watermark is defined on time
Incorrect
A watermark is needed to clean state in aggregations; a select and where keep no state.
You can apply all kinds of operations on streaming DataFrames, from untyped SQL-like operations such as select, where and groupBy to typed ones. The programming guide uses this very example, selecting the devices whose signal is above 10, as a basic selection and projection; only a few operations are unsupported on streams.
A streaming aggregation groups events with F.window("event_time", "10 minutes") and no slide duration. Into which window does an event with event_time 12:10:00 fall?
A[12:00, 12:10) only
BBoth [12:00, 12:10) and [12:10, 12:20)
C[12:05, 12:15)
D[12:10, 12:20) only
Show the answer and why
A[12:00, 12:10) only
Incorrect
Window ends are exclusive, so 12:10:00 is not in the window that ends at 12:10.
BBoth [12:00, 12:10) and [12:10, 12:20)
Incorrect
Without a slide duration the windows are tumbling and do not overlap, and window ends are exclusive.
C[12:05, 12:15)
Incorrect
Tumbling 10-minute windows start at 12:00, 12:10 and so on, without an offset.
D[12:10, 12:20) only
Correct
Window starts are inclusive and ends exclusive, so 12:10:00 belongs to the window that starts at 12:10.
pyspark.sql.functions.window bucketizes rows into time windows on a timestamp column. Window starts are inclusive and window ends exclusive: 12:05 is in [12:05, 12:10) but not in [12:00, 12:05). Without slideDuration the windows are tumbling, so an event at 12:10:00 falls only into [12:10, 12:20).
On Databricks, a streaming query removes duplicate events and writes in append output mode. A developer proposes switching to update mode so that duplicates are handled differently. What does Databricks say about output modes and deduplication?
AUpdate mode makes deduplication re-emit each duplicate as an update
BThe output mode doesn't impact deduplication
CDeduplication requires complete output mode
DAppend mode disables deduplication
Show the answer and why
AUpdate mode makes deduplication re-emit each duplicate as an update
Incorrect
The output mode does not change how deduplication treats duplicates.
BThe output mode doesn't impact deduplication
Correct
Databricks states that output mode doesn't impact deduplication.
CDeduplication requires complete output mode
Incorrect
The output mode does not decide how deduplication works.
DAppend mode disables deduplication
Incorrect
Deduplication runs the same way in append mode.
The output mode controls which rows of the result are written at each trigger. Databricks notes that joins support only append mode and that output mode doesn't impact deduplication; the arbitrary stateful operators also emit records with their own logic, whatever the output mode.