Data Engineer Associate · Domain 2: Data Ingestion and Loading · 21% of the exam
Task 2.4: Configure Lakeflow Connect to reliably ingest data from diverse enterprise sources into Unity Catalog–governed tables.
Managed SaaS, database, query-based and file connectors, connections, ingestion gateways and pipelines, schedules, incremental and CDC reads, and history tracking in destination tables.
Study it
Lakeflow Connect managed connectors: connections, gateways, pipelines and schedules
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A team ingests a PostgreSQL database with a Lakeflow Connect managed database connector. Which component runs continuously on classic compute to extract snapshots and change logs before the source truncates them?
AThe ingestion pipeline
BThe ingestion gateway
CThe staging storage volume
DThe Unity Catalog connection
Show the answer and why
AThe ingestion pipeline
Incorrect
The ingestion pipeline runs on the schedule you choose and moves data from staging storage into the destination tables; it does not read the source's change logs.
BThe ingestion gateway
Correct
The gateway runs continuously on classic compute and captures snapshots, change logs and metadata so changes are not lost before the source truncates its logs.
CThe staging storage volume
Incorrect
Staging storage is a Unity Catalog volume that holds extracted data temporarily; it stores data but does not run or extract anything.
DThe Unity Catalog connection
Incorrect
The connection is a definition the other components use to reach the database; it is not a running process and extracts nothing.
Database connectors split the work: a continuously running gateway captures changes into a staging volume, and a scheduled ingestion pipeline applies them to the destination tables.
Three Lakeflow Connect ingestion pipelines pull different objects from the same Salesforce org. The credentials must be stored once, governed by Unity Catalog and reused by the pipelines. What holds them?
AA Unity Catalog connection
BA secret scope read by each pipeline's notebook
CA storage credential
DAn external location
Show the answer and why
AA Unity Catalog connection
Correct
A connection is a Unity Catalog securable object that stores the authentication details for the source application, and pipelines reference it.
BA secret scope read by each pipeline's notebook
Incorrect
Managed ingestion pipelines are not notebooks; secret scopes serve custom code, while managed connectors use a connection.
CA storage credential
Incorrect
A storage credential authorizes access to cloud object storage; it does not hold SaaS application credentials.
DAn external location
Incorrect
An external location pairs a cloud storage path with a storage credential; it is unrelated to Salesforce authentication.
Managed connectors separate authentication from ingestion: the connection holds credentials as a governed object, and each ingestion pipeline uses it.
A Lakeflow Connect pipeline ingests customer records from a SaaS application. Auditors need every past version of each record, with when it was valid, kept in the destination table indefinitely. Which history tracking setting should the pipeline use?
ASCD type 1
BSCD type 1 plus Delta time travel on the destination
CSCD type 2
DA full refresh of the destination on each run
Show the answer and why
ASCD type 1
Incorrect
SCD type 1 overwrites the existing row with the latest values, so earlier versions are lost.
BSCD type 1 plus Delta time travel on the destination
Incorrect
Time travel depends on retention settings, seven days by default, and is not meant as long-term history.
CSCD type 2
Correct
SCD type 2 keeps the full history of changes by adding a new row version with validity metadata for each change.
DA full refresh of the destination on each run
Incorrect
A full refresh reloads the current source state; it does not retain past versions.
History tracking modes decide what the destination keeps: SCD type 1 keeps the latest state, SCD type 2 keeps every version with start and end markers.
A managed connector that supports automated schema evolution ingests an object whose source column region is deleted. What happens to region in the destination table on the next runs?
AThe column is dropped from the destination table on the next run
BThe column stays, and a table property marks it as inactive
CThe pipeline fails until someone runs a full refresh of the table
DExisting values in the column are overwritten with NULL
Show the answer and why
AThe column is dropped from the destination table on the next run
Incorrect
Databricks does not delete the column automatically when it disappears from the source.
BThe column stays, and a table property marks it as inactive
Correct
The connector uses a table property to set the deleted column to inactive in the destination instead of removing it.
CThe pipeline fails until someone runs a full refresh of the table
Incorrect
A failure happens only if a new column later appears with the same name; deleting a column alone does not fail the pipeline.
DExisting values in the column are overwritten with NULL
Incorrect
Existing values are not rewritten; the column is kept and marked inactive.
Managed connectors that evolve schemas add new columns automatically and mark deleted ones inactive. Some connectors use a fixed schema; check each connector's feature list.