Data Engineer Professional · Domain 1: Developing Code for Data Processing using Python and SQL · 23% of the exam
Task 1.12: Apply Structured Streaming stateful-processing semantics, including watermarks, output modes, foreachBatch, and checkpoints for fault-tolerant exactly-once state recovery.
Watermarks and late data, append, update and complete output modes, idempotent foreachBatch writes, checkpoint contents and the changes a checkpoint does not allow.
Study it
Stateful streaming: watermarks, output modes, foreachBatch and checkpoints
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A streaming query uses foreachBatch to MERGE each micro-batch into one Delta table and then append the same rows to a second Delta table for auditing. After a failure, the query restarts from its checkpoint and replays the last micro-batch, and the audit table gets duplicate rows. The data engineer must make both writes safe to retry. What should they do?
AWrap each write in try/except and retry it locally until it succeeds
BSwitch the query to a continuous processing trigger
CSet txnAppId and txnVersion, bound to batchId, on each write
DMove the checkpoint to a new location after every failure
Show the answer and why
AWrap each write in try/except and retry it locally until it succeeds
Incorrect
Databricks advises against local retry loops; errors should propagate so the orchestrator retries the whole batch cleanly.
BSwitch the query to a continuous processing trigger
Incorrect
foreachBatch relies on micro-batches and does not work in continuous processing mode.
CSet txnAppId and txnVersion, bound to batchId, on each write
Correct
Delta uses the application ID and version to skip a write it has already committed, which makes each write in foreachBatch idempotent.
DMove the checkpoint to a new location after every failure
Incorrect
A new checkpoint makes the query start fresh, reprocessing data and losing its progress.
foreachBatch gives at-least-once guarantees. For Delta targets, Databricks requires the txnAppId and txnVersion options, with txnVersion set to the batchId, so that a replayed batch does not write twice; failures should be left to propagate so the orchestrator retries the batch.
A streaming query reads the Kafka topic orders, aggregates by store_id over event-time windows with a watermark, and writes to a Delta table. The team plans several changes and wants to restart from the existing checkpoint where possible. Which TWO changes require a new checkpoint? (Choose TWO.)
AAdding a filter that drops test stores before the aggregation
BChanging the trigger from every minute to every five minutes
CAdding maxOffsetsPerTrigger to the options of the orders topic
DAdding region as a second grouping key of the aggregation
ESubscribing to the topic orders_v2 instead of orders
Show the answer and why
AAdding a filter that drops test stores before the aggregation
Incorrect
Adding or removing filters is listed among the generally safe changes.
BChanging the trigger from every minute to every five minutes
Incorrect
Trigger intervals can be changed between runs.
CAdding maxOffsetsPerTrigger to the options of the orders topic
Incorrect
Changing rate limits such as maxOffsetsPerTrigger is allowed.
DAdding region as a second grouping key of the aggregation
Correct
Changing the number or type of grouping keys changes the state schema, which is not allowed with the same checkpoint.
ESubscribing to the topic orders_v2 instead of orders
Correct
Changing the subscribed topics or paths is generally not allowed, because the results are unpredictable.
The checkpoint stores offsets, commits and state. Changes to stateful operations or their state schema, to the number or type of sources, to subscribed topics or paths and to the sink type generally need a new checkpoint, while filters, rate limits and triggers can change.
A windowed count with a 10-minute watermark writes to a Delta table in append mode. When does a row for a window appear in the table?
AAt every trigger, with the latest partial count for the window
BOnce the watermark passes and the window is final
COnly when the query is stopped
DAt every trigger, with all windows rewritten in the table
Show the answer and why
AAt every trigger, with the latest partial count for the window
Incorrect
Append mode does not emit partial counts for windows that are still open.
BOnce the watermark passes and the window is final
Correct
In append mode, operators emit only rows that will not change, using the watermark to decide when.
COnly when the query is stopped
Incorrect
Append mode emits finished windows while the query runs.
DAt every trigger, with all windows rewritten in the table
Incorrect
Append mode does not rewrite earlier windows at each trigger.
Append is the default output mode. For windowed aggregations it writes a row after the watermark threshold passes, so writes are delayed by the watermark.
A streaming aggregation counts events per session_id, and new session IDs arrive without end. A teammate proposes complete output mode. Why is that a poor choice here?
AIt cannot be used with streaming aggregations such as this count
BIt requires a stream-stream join with a second input
CIt drops late rows before they reach the aggregation, losing counts
DIt keeps all aggregation state and rewrites every row each trigger
Show the answer and why
AIt cannot be used with streaming aggregations such as this count
Incorrect
Complete mode works only with streaming aggregations.
BIt requires a stream-stream join with a second input
Incorrect
Joins between streams support only append mode, not complete.
CIt drops late rows before they reach the aggregation, losing counts
Incorrect
Late data handling depends on the watermark, not on complete mode.
DIt keeps all aggregation state and rewrites every row each trigger
Correct
In complete mode aggregation state is not dropped, so state and output grow with every new key.
Complete mode emits all result rows ever produced by the operator at each trigger and keeps all aggregation state, so it suits small result sets, not aggregations whose keys grow without bound.
A team shortens the watermark of a windowed aggregation from 2 hours to 10 minutes. What trade-off should they expect?
ALess state and earlier results, but more late events left out
BMore state and later results, but no late events left out
CNo change, because the output mode decides how late data is handled
DExactly-once guarantees are lost
Show the answer and why
ALess state and earlier results, but more late events left out
Correct
Shorter watermarks lower latency and state but tolerate less late data.
BMore state and later results, but no late events left out
Incorrect
That describes a longer watermark.
CNo change, because the output mode decides how late data is handled
Incorrect
The watermark sets the lateness threshold.
DExactly-once guarantees are lost
Incorrect
Checkpoints provide recovery guarantees; the watermark sets lateness.
Shorter watermarks store less state and write results sooner but have low tolerance for late data. Longer watermarks tolerate more lateness at the cost of more state and higher latency.
Two streams, impressions and clicks, are joined with a left outer join so that impressions without clicks are also written. What must the query have?
AComplete output mode so that unmatched rows are rewritten later
BUpdate output mode so that rows can be corrected when clicks arrive
CWatermarks; unmatched rows are written after the lateness threshold
DNo watermark, so that every impression keeps waiting for its click
Show the answer and why
AComplete output mode so that unmatched rows are rewritten later
Incorrect
Stream-stream joins support only append mode.
BUpdate output mode so that rows can be corrected when clicks arrive
Incorrect
Joins between streams do not support update mode.
CWatermarks; unmatched rows are written after the lateness threshold
Correct
Watermarking is mandatory for outer joins, and because joins only support append mode, unmatched rows are written after the threshold.
DNo watermark, so that every impression keeps waiting for its click
Incorrect
Without watermarks an outer join cannot know when to emit nulls.
Stream-stream joins support only append mode. Watermarks are recommended for inner joins and mandatory for outer joins; unmatched rows are written with nulls once the lateness threshold passes.