Skip to content
Cheat sheets

Data Engineer Professional · Domain 1: Developing Code for Data Processing using Python and SQL · 23% of the exam

Task 1.8: Choose between Spark Structured Streaming and Apache Spark™ Declarative Pipelines for scalable ETL given operational constraints.

What a declarative pipeline manages for you (dependencies, checkpoints, retries, expectations, autoscaling) and when custom Structured Streaming code is still the right tool.

Study it

  • Declarative pipelines or hand-written Structured Streaming

    Partly covered by: Message Queues

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A team runs 25 hand-written Structured Streaming notebooks that feed each other through Delta tables. Each has its own checkpoint path, retry loop and row-count logging, and renaming a table regularly breaks a checkpoint. They want the platform to work out dependencies, manage checkpoints and retries, and report data quality metrics, while writing as little orchestration code as possible. What should they move to?

  1. AOne notebook that starts all 25 streams with writeStream
  2. BA job with one task per stream and task retries enabled
  3. CStreaming tables and flows in a Lakeflow pipeline
  4. DforeachBatch writers that log counts to an audit table
Show the answer and why

  • AOne notebook that starts all 25 streams with writeStream

    Incorrect

    A single notebook still needs a checkpoint and retry logic per stream, and gives no dependency handling or quality metrics.

  • BA job with one task per stream and task retries enabled

    Incorrect

    Task retries help, but checkpoints, ordering between streams and quality metrics remain hand-written.

  • CStreaming tables and flows in a Lakeflow pipeline

    Correct

    Pipelines infer the dependency graph, manage checkpoints, retry failures and record expectation metrics in the event log.

  • DforeachBatch writers that log counts to an audit table

    Incorrect

    foreachBatch adds custom per-batch logic; it does not manage dependencies or checkpoints for the team.

Lakeflow pipelines extend Structured Streaming with declarative datasets: the pipeline resolves dependencies, manages checkpoints and state, retries on failure and records data quality in the event log. Hand-written Structured Streaming remains the choice when a workload needs control that pipelines do not give.

Databricks documentation

Task 1.8Report a problem with this question

Question 2 · choose 1

A data engineer moves a notebook that calls DataFrame.groupBy().pivot() into a Lakeflow pipeline, and the pipeline rejects it. Why?

  1. APipelines support only SQL, so Python DataFrame code like this is rejected
  2. BPipelines do not allow groupBy aggregations in any dataset they define
  3. CPipelines do not support pivot(), as it eagerly loads data for its schema
  4. Dpivot() needs a cluster in dedicated access mode, not the pipeline's compute
Show the answer and why

  • APipelines support only SQL, so Python DataFrame code like this is rejected

    Incorrect

    Pipelines support both Python and SQL.

  • BPipelines do not allow groupBy aggregations in any dataset they define

    Incorrect

    Aggregations are supported, for example in materialized views.

  • CPipelines do not support pivot(), as it eagerly loads data for its schema

    Correct

    pivot() requires eager loading of input data to compute its output schema, which pipelines do not support.

  • Dpivot() needs a cluster in dedicated access mode, not the pipeline's compute

    Incorrect

    The limitation comes from pipelines, not the access mode.

Pipeline limitations include pivot(): Spark's pivot operation eagerly loads input data to compute the output schema, which pipelines do not support.

Databricks documentation

Task 1.8Report a problem with this question

Question 3 · choose 1

Three regional Kafka topics must feed one streaming table in a Lakeflow pipeline. Defining the table three times fails. What should the data engineer use?

  1. AThree streaming tables with the same name in different files
  2. BOne streaming table with three append flows that target it
  3. CA materialized view that unions the three topics
  4. DThree separate pipelines that write to the same table
Show the answer and why

  • AThree streaming tables with the same name in different files

    Incorrect

    A dataset can be defined only once across pipelines.

  • BOne streaming table with three append flows that target it

    Correct

    Append flows let several streaming sources write to one streaming table.

  • CA materialized view that unions the three topics

    Incorrect

    Kafka keeps no full history to recompute from, so a streaming table is the right target.

  • DThree separate pipelines that write to the same table

    Incorrect

    A dataset can be the target of only one operation across pipelines.

Pipeline datasets can be defined only once, so they are the target of one operation. Streaming tables with append flows are the exception: several streaming sources can append to the same streaming table.

Databricks documentation

Task 1.8Report a problem with this question

Question 4 · choose 1

A Lakeflow pipeline must publish enriched events to a Kafka topic for an external fraud system, in addition to its Delta tables. How is this done inside the pipeline?

  1. ADefine a streaming table in SQL whose LOCATION points to the Kafka topic
  2. BUse an AUTO CDC flow that names the Kafka topic as its target table
  3. CAdd an expectation that forwards passing rows to the Kafka topic
  4. DA Kafka sink from create_sink() in Python, targeted by an append_flow
Show the answer and why

  • ADefine a streaming table in SQL whose LOCATION points to the Kafka topic

    Incorrect

    Sinks are available only in Python, and streaming tables are Delta tables.

  • BUse an AUTO CDC flow that names the Kafka topic as its target table

    Incorrect

    AUTO CDC flows cannot write to sinks.

  • CAdd an expectation that forwards passing rows to the Kafka topic

    Incorrect

    Expectations check data quality; they do not write to external systems.

  • DA Kafka sink from create_sink() in Python, targeted by an append_flow

    Correct

    Sinks write pipeline output outside managed tables, and only append and update flows can target them.

Sinks let pipelines write to Kafka, Event Hubs, external Delta tables or custom data sources. They are Python-only, take streaming queries, and are the target of append_flow or update_flow.

Databricks documentation

Task 1.8Report a problem with this question

Question 5 · choose 1

A team wants to run its declarative pipeline code on open-source Apache Spark Declarative Pipelines as well as on Databricks. Which feature that their pipeline uses is not available in the open-source framework?

  1. AThe AUTO CDC APIs
  2. BDefining datasets in Python
  3. CStreaming tables as datasets
  4. DDefining datasets in SQL
Show the answer and why

  • AThe AUTO CDC APIs

    Correct

    Databricks states that the AUTO CDC APIs are not supported by Apache Spark Declarative Pipelines.

  • BDefining datasets in Python

    Incorrect

    Python dataset definitions are part of the declarative framework.

  • CStreaming tables as datasets

    Incorrect

    Streaming tables are a core declarative dataset type.

  • DDefining datasets in SQL

    Incorrect

    SQL dataset definitions are part of the declarative framework.

Lakeflow pipelines build on Apache Spark Declarative Pipelines and add Databricks features. The AUTO CDC APIs are among those not supported by the open-source framework.

Databricks documentation

Task 1.8Report a problem with this question

Practice domain 1 →Practice all domains →