Task 2.1: Enable and detail data ingestion patterns, including batch, streaming, and incremental loading, and import data from sources such as local files, Lakeflow Connect standard connectors, and Lakeflow Connect managed connectors.

• AIn continuous mode, with pipelines.trigger.interval set to one hour

  Incorrect

  Continuous execution keeps compute running between updates; the trigger interval only spaces out flow updates on an always-on pipeline, so idle cost remains.

• BAs a continuous job that wraps the pipeline task

  Incorrect

  A continuous job keeps the pipeline running all the time, which suits latency of seconds to minutes, not an hourly refresh that should stop between runs.

• CIn triggered mode, with a job that starts an update each hour

  Correct

  A triggered update processes the data available when it starts and then stops, so the compute runs only for each hourly refresh.

• DIn triggered mode, started by hand from the pipeline UI when needed

  Incorrect

  Triggered mode is right, but a manual start gives no reliable hourly refresh; a schedule or trigger is still needed.

Triggered pipelines refresh what is available and stop, which keeps cost down when freshness of an hour or a day is enough. Continuous pipelines keep compute running for latency in the seconds-to-minutes range.

• AUpload it with Create or modify a table using file upload

  Correct

  The file upload page creates or overwrites a managed Delta table from small CSV, TSV, JSON, Avro, Parquet or text files on a local machine.

• BRun COPY INTO with the laptop's local file path as the source

  Incorrect

  COPY INTO loads from a file location in cloud object storage or a Unity Catalog volume, not from a path on a user's machine.

• CStart an Auto Loader stream on the laptop's Downloads folder

  Incorrect

  Auto Loader discovers files in cloud storage and volumes such as s3:// or /Volumes/; it cannot watch a folder on a personal computer.

• DCreate a Lakeflow Connect managed connector for the CSV file

  Incorrect

  Managed connectors ingest from SaaS applications, databases and enterprise file services, not a single file on a laptop.

The file upload UI is meant for small local files: it stages the upload and creates a managed Delta table. COPY INTO and Auto Loader read from cloud storage or volumes.

• ACREATE OR REPLACE TABLE bronze AS SELECT * FROM read_files('/Volumes/raw/app/events')

  Incorrect

  A CTAS reads the whole directory again on each run and rebuilds the table; it is not incremental.

• BCREATE OR REFRESH STREAMING TABLE bronze AS SELECT * FROM STREAM read_files('/Volumes/raw/app/events')

  Correct

  A streaming table over STREAM read_files uses Auto Loader underneath and processes only new files on each refresh.

• CCREATE VIEW bronze AS SELECT * FROM read_files('/Volumes/raw/app/events')

  Incorrect

  A view stores the query and reads every file again whenever someone queries it; nothing is loaded.

• DINSERT INTO bronze SELECT * FROM read_files('/Volumes/raw/app/events'), run hourly

  Incorrect

  Each run inserts every file again, so rows are duplicated.

For SQL users, a streaming table over STREAM read_files is the recommended scalable way to ingest from cloud storage incrementally, and the documentation presents it as the alternative to COPY INTO.

• AA streaming table that appends each change record to silver as it arrives

  Incorrect

  Appending keeps every change as a separate row; it does not apply updates and deletes to produce the current state.

• BA notebook that MERGEs each micro-batch into silver in the order records arrive

  Incorrect

  Applying changes in arrival order lets a late, older change overwrite a newer one; ordering by seq needs extra logic.

• CCOPY INTO the silver table with mergeSchema set, run each time a new export of changes lands

  Incorrect

  COPY INTO appends file contents; it does not apply keyed updates or deletes.

• DAUTO CDC INTO silver with KEYS (id), SEQUENCE BY seq, APPLY AS DELETE WHEN op = 'DELETE'

  Correct

  AUTO CDC applies inserts, updates and deletes by key, orders them by the sequence column and handles out-of-order records, here as SCD type 1.

For a feed of change events, AUTO CDC turns the changes into a current-state (SCD type 1) or history (SCD type 2) table. It replaces the older APPLY CHANGES API with the same syntax.

• AThe managed connector, dropping to a lower layer if it falls short

  Correct

  Databricks recommends starting with the most managed layer and dropping down only when it does not meet a requirement, such as source support.

• BWith Structured Streaming, because it gives the most control from the start

  Incorrect

  Structured Streaming is the most customizable layer and needs the most code; it is the fallback, not the starting point.

• CWith custom Python using open-source connector libraries

  Incorrect

  DIY connectors are possible but put long-term maintenance on the team; managed connectors handle authentication, CDC, retries and schema evolution.

• DWith a Lakeflow pipeline that calls the source's API from SQL

  Incorrect

  Pipelines sit between the two; for a source with a managed connector, the connector adds source-specific automation on top of pipelines.

Ingestion options form layers: Structured Streaming, then Lakeflow pipelines, then managed connectors, each adding automation. Start at the managed end.

• ABoth streams share one checkpoint location so the table has a single progress record

  Incorrect

  Each ingestion workload needs its own checkpoint; sharing one mixes the progress of two different sources.

• BNeither stream needs a checkpoint, because the Delta table records which files were loaded

  Incorrect

  Auto Loader keeps discovered-file state in the checkpoint; without it, there is no exactly-once tracking.

• CEach stream gets its own checkpoint location in Unity Catalog-governed storage

  Correct

  When more than one source location loads into a table, each Auto Loader workload requires a separate streaming checkpoint.

• DEach checkpoint goes in a subdirectory of the table's own storage directory

  Incorrect

  Unity Catalog does not allow checkpoint or schema files to be nested under the table directory.

The checkpoint holds a stream's progress, including the files Auto Loader has processed. One stream, one checkpoint, stored in a location Unity Catalog governs.