Skip to content
Cheat sheets

Data Engineer Professional · Domain 2: Data Ingestion & Acquisition · 12% of the exam

Task 2.2: Configure incremental CDC pipelines using Lakeflow Pipelines with Delta or Iceberg as the target table format.

Feeding AUTO CDC from a change feed or a CDC stream, snapshots, change data feed on the target, and writing targets that Iceberg clients can read.

Study it

  • Incremental CDC pipelines into Delta and Iceberg-readable targets

    Partly covered by: Database Replication

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

An AUTO CDC flow in a Lakeflow pipeline maintains the streaming table customers. An analytics team must read it from an external engine that uses an Iceberg REST catalog client supporting the Iceberg v3 specification. The dataset is large, so it must not be copied, and readers must see each update right after it completes. What should the data engineer configure?

  1. ATurn on compatibility mode for the streaming table for all clients
  2. BEnable external metadata with pipelines.externalMetadata.enabled and add the UniForm Iceberg v3 table properties
  3. CEnable Iceberg reads with IcebergCompatV2 on the streaming table
  4. DRun a nightly job that writes a copy of customers as a managed Iceberg table for external readers
Show the answer and why

  • ATurn on compatibility mode for the streaming table for all clients

    Incorrect

    Compatibility mode works with all clients but requires a full copy of the dataset.

  • BEnable external metadata with pipelines.externalMetadata.enabled and add the UniForm Iceberg v3 table properties

    Correct

    External data access exposes pipeline streaming tables to Delta and Iceberg clients without a full copy, with read-after-write consistency; Iceberg clients also need the UniForm Iceberg v3 properties.

  • CEnable Iceberg reads with IcebergCompatV2 on the streaming table

    Incorrect

    Iceberg reads cannot be enabled on streaming tables or materialized views with IcebergCompatV2.

  • DRun a nightly job that writes a copy of customers as a managed Iceberg table for external readers

    Incorrect

    A nightly copy duplicates the data and is stale between runs.

For pipeline-managed and standalone streaming tables and materialized views, external data access is enabled with the pipelines.externalMetadata.enabled setting, as a pipeline configuration or a table property. Modern Iceberg clients also need the UniForm Iceberg v3 properties, and the feature requires the default publishing mode.

Databricks documentation

Task 2.2Report a problem with this question

Question 2 · choose 1

An AUTO CDC flow keeps accounts, an SCD type 1 streaming table published to Unity Catalog. A faulty upstream batch wrote a wrong credit_tier for 40 accounts, and the source system will not resend them. The rows must be corrected today without a full refresh, which would reprocess three years of changes, and auditors want the reason for each fix recorded. Which approach is supported?

  1. AAdd an expectation to the AUTO CDC flow that rejects the wrong credit_tier values
  2. BUPDATE the rows from a SQL warehouse and record the reasons in a separate audit table
  3. CMERGE the fixes WITH SCHEMA EVOLUTION, adding a fix_reason column to accounts
  4. DUPDATE the rows from a notebook on the team's Databricks Runtime 12.2 LTS cluster
Show the answer and why

  • AAdd an expectation to the AUTO CDC flow that rejects the wrong credit_tier values

    Incorrect

    Expectations check records as the flow processes them; they do not change rows that are already in the target.

  • BUPDATE the rows from a SQL warehouse and record the reasons in a separate audit table

    Correct

    DML can modify AUTO CDC target streaming tables published to Unity Catalog, from a SQL warehouse or a shared cluster on Databricks Runtime 13.3 LTS or above, as long as the schema is unchanged.

  • CMERGE the fixes WITH SCHEMA EVOLUTION, adding a fix_reason column to accounts

    Incorrect

    DML statements that modify the schema of a streaming table are not supported, so the new column cannot be added this way.

  • DUPDATE the rows from a notebook on the team's Databricks Runtime 12.2 LTS cluster

    Incorrect

    DML that updates a streaming table runs only on a SQL warehouse or a shared Unity Catalog cluster using Databricks Runtime 13.3 LTS or above.

When a pipeline publishes to Unity Catalog, insert, update, delete and merge statements can modify the target streaming tables of AUTO CDC flows. The statements must not evolve the table schema, and they run on a SQL warehouse or a shared Unity Catalog cluster with Databricks Runtime 13.3 LTS or above.

Databricks documentation

Task 2.2Report a problem with this question

Question 3 · choose 1

Some change records in a CDC feed have a NULL value in the column used for SEQUENCE BY in an AUTO CDC flow. What does the documentation say?

  1. ANULL sequence values are applied last, after all other changes
  2. BNULL sequence values are treated as deletes of that key
  3. CNULL sequence values are applied in the order they arrive
  4. DThey are not supported; the sequencing column must be sortable
Show the answer and why

  • ANULL sequence values are applied last, after all other changes

    Incorrect

    NULLs are not given an order; they are not supported.

  • BNULL sequence values are treated as deletes of that key

    Incorrect

    Deletes come from APPLY AS DELETE WHEN, not from NULL sequences.

  • CNULL sequence values are applied in the order they arrive

    Incorrect

    AUTO CDC orders changes by the sequencing column, not by arrival.

  • DThey are not supported; the sequencing column must be sortable

    Correct

    AUTO CDC limitations state that the sequencing column must be a sortable type and NULL values are not supported.

AUTO CDC orders changes per key by the SEQUENCE BY column. The column must be sortable, and NULL sequencing values are not supported.

Databricks documentation

Task 2.2Report a problem with this question

Question 4 · choose 1

A pipeline will maintain a replica of an external database table with AUTO CDC. It needs one initial full load of the existing rows, followed by ongoing change processing. What pattern does Databricks describe?

  1. AA materialized view for the full load that the flow then updates
  2. BA once flow for the initial hydration, then AUTO CDC on the feed
  3. CA full refresh of the target table before every pipeline update
  4. DA separate job that writes the rows into the target table directly
Show the answer and why

  • AA materialized view for the full load that the flow then updates

    Incorrect

    An AUTO CDC target is a streaming table; a materialized view is refreshed from its own query.

  • BA once flow for the initial hydration, then AUTO CDC on the feed

    Correct

    For initial hydration from a source with a change feed, use AUTO CDC with a once flow and then continue processing the change feed.

  • CA full refresh of the target table before every pipeline update

    Incorrect

    That reloads everything each time instead of loading once.

  • DA separate job that writes the rows into the target table directly

    Incorrect

    Writing to a pipeline-managed target from outside the pipeline is not the documented pattern.

The AUTO CDC docs describe using a once flow for the initial hydration, then continuing with the change feed, as in replicating an external RDBMS table.

Databricks documentation

Task 2.2Report a problem with this question

Question 5 · choose 1

A data engineer wants the target streaming table of an AUTO CDC flow to generate a surrogate key with GENERATED ALWAYS AS IDENTITY. What happens?

  1. AIt works, and the values are regenerated on every pipeline update
  2. BIt works only when the flow is stored as SCD type 2
  3. CIt is not supported on tables that are AUTO CDC targets
  4. DIt works only when the pipeline runs in continuous mode
Show the answer and why

  • AIt works, and the values are regenerated on every pipeline update

    Incorrect

    Recomputation of identity values is a caveat for materialized views.

  • BIt works only when the flow is stored as SCD type 2

    Incorrect

    The limitation applies to AUTO CDC targets in general.

  • CIt is not supported on tables that are AUTO CDC targets

    Correct

    Pipeline limitations state that identity columns are not supported on tables that are the target of AUTO CDC processing.

  • DIt works only when the pipeline runs in continuous mode

    Incorrect

    Pipeline mode does not change this limitation.

Identity columns are not supported on AUTO CDC targets, and they might be recomputed during materialized view updates, so Databricks recommends them in pipelines only with streaming tables.

Databricks documentation

Task 2.2Report a problem with this question

Question 6 · choose 1

A downstream job must read the change data feed of an AUTO CDC target streaming table that a pipeline publishes to Unity Catalog. On the job's cluster, which runs Databricks Runtime 14.3 LTS, the read fails. What does Databricks require for this read?

  1. ADatabricks Runtime 15.2 or above on the compute that reads it
  2. BPublishing the pipeline to the Hive metastore instead of Unity Catalog
  3. CSCD type 2 storage, because SCD type 1 targets keep no change feed
  4. DA full refresh of the target table before each downstream read
Show the answer and why

  • ADatabricks Runtime 15.2 or above on the compute that reads it

    Correct

    Reading the change data feed of an AUTO CDC target needs Databricks Runtime 15.2 or above, also for a pipeline that reads it.

  • BPublishing the pipeline to the Hive metastore instead of Unity Catalog

    Incorrect

    The target streaming table must be published to Unity Catalog for its change data feed to be read.

  • CSCD type 2 storage, because SCD type 1 targets keep no change feed

    Incorrect

    Downstream queries can use the change feed of an SCD type 1 target to process only changed records.

  • DA full refresh of the target table before each downstream read

    Incorrect

    The change feed is read like that of any other Delta table; a full refresh would rebuild the target, not enable the read.

In Databricks Runtime 15.2 and above, the change data feed of a streaming table that is the target of AUTO CDC can be read like that of other Delta tables. The target must be published to Unity Catalog, and a pipeline that reads it must also use Databricks Runtime 15.2 or above.

Databricks documentation

Task 2.2Report a problem with this question

Practice domain 2 →Practice all domains →