Skip to content
Cheat sheets

Data Engineer Professional · Domain 1: Developing Code for Data Processing using Python and SQL · 23% of the exam

Task 1.4: Create production-ready data pipelines for streaming data using Lakeflow Pipelines and Auto Loader.

Streaming tables over Auto Loader, file detection with file events, schema evolution and the rescued data column, triggered and continuous modes, and full refresh.

Study it

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

An Auto Loader stream in a Lakeflow pipeline uses classic file notification mode because the use case is very latency-sensitive. An audit found that a handful of files a month were never ingested because their notifications were lost. The business requires every file to be loaded within a day, without giving up the low latency. What should the data engineer configure?

  1. AcloudFiles.maxFileAge set to 1 day on the stream
  2. BDirectory listing mode with a continuous processing-time trigger
  3. CcloudFiles.backfillInterval set to 1 day on the stream
  4. DFile events on the location, with backfillInterval set to 1 day
Show the answer and why

  • AcloudFiles.maxFileAge set to 1 day on the stream

    Incorrect

    maxFileAge does not relist the directory, so it does not find missed files.

  • BDirectory listing mode with a continuous processing-time trigger

    Incorrect

    Listing would catch the files, but continuous listing of a large directory is expensive and adds latency compared with notifications.

  • CcloudFiles.backfillInterval set to 1 day on the stream

    Correct

    A backfill relists the directory asynchronously on that cadence and ingests only files that were missed, which suits classic notification mode with a completeness SLA.

  • DFile events on the location, with backfillInterval set to 1 day

    Incorrect

    File events backfill automatically and do not support the interval setting; the stem also keeps classic mode for its lower latency.

Cloud notification systems deliver events at least once, but a file can still be missed. cloudFiles.backfillInterval schedules periodic asynchronous directory listings that pick up only unprocessed files. With file events, Databricks backfills automatically and the interval is unsupported.

Databricks documentation

Task 1.4Report a problem with this question

Question 2 · choose 1

A streaming table in a Lakeflow pipeline ingests files with Auto Loader from a landing location whose lifecycle policy deletes files after 7 days. The table holds three years of history. Engineers sometimes start full refreshes of whole pipelines while debugging, and a full refresh of this table would lose everything older than 7 days. Other tables in the pipeline must still allow full refresh. What should the data engineer do?

  1. ASet cloudFiles.includeExistingFiles to false on the Auto Loader source
  2. BRun the pipeline in continuous mode instead of triggered mode
  3. CSet the table property pipelines.reset.allowed to false on this table
  4. DTurn on deletion vectors on the streaming table
Show the answer and why

  • ASet cloudFiles.includeExistingFiles to false on the Auto Loader source

    Incorrect

    That option controls whether files already present at the first start are loaded; a full refresh still truncates the table.

  • BRun the pipeline in continuous mode instead of triggered mode

    Incorrect

    Continuous mode keeps the pipeline running, but a full refresh can still be started and would reset the table.

  • CSet the table property pipelines.reset.allowed to false on this table

    Correct

    With this property, full refreshes are not run on the table, while other datasets in the pipeline can still be fully refreshed.

  • DTurn on deletion vectors on the streaming table

    Incorrect

    Deletion vectors change how deletes are written; they do not stop a full refresh or bring back source files that no longer exist.

A full refresh clears a streaming table and reprocesses what is still in the source, so data whose source files were removed by a lifecycle policy is lost. Setting pipelines.reset.allowed to false on the table prevents full refreshes on it; an append flow is another way to add data without needing a full refresh.

Databricks documentation

Task 1.4Report a problem with this question

Question 3 · choose 1

A data engineer must check whether a specific file in the landing folder has been discovered by an Auto Loader stream. Which SQL call shows the files the stream has discovered?

  1. ADESCRIBE HISTORY on the table the stream writes to
  2. BSELECT * FROM read_files('<landing path>')
  3. CSELECT * FROM cloud_files_state('<checkpoint path>')
  4. DLIST '<landing path>' to see the landing folder's files
Show the answer and why

  • ADESCRIBE HISTORY on the table the stream writes to

    Incorrect

    Table history lists commits, not the files the stream has tracked.

  • BSELECT * FROM read_files('<landing path>')

    Incorrect

    read_files reads the files' contents; it does not show stream state.

  • CSELECT * FROM cloud_files_state('<checkpoint path>')

    Correct

    cloud_files_state returns metadata about files an Auto Loader stream has discovered, read from its checkpoint.

  • DLIST '<landing path>' to see the landing folder's files

    Incorrect

    LIST shows what is in storage, not what the stream has discovered.

cloud_files_state, given the checkpoint location of an Auto Loader stream, returns metadata about the files the stream has discovered.

Databricks documentation

Task 1.4Report a problem with this question

Question 4 · choose 1

Operators want to alert when an Auto Loader stream falls behind, measured as the number and size of files still waiting to be processed. Which metrics report this?

  1. AnumFilesOutstanding and numBytesOutstanding source metrics
  2. BnumOutputRows and inputRowsPerSecond in the stream's progress metrics
  3. CThe numFiles value of DESCRIBE DETAIL on the target table
  4. DThe file count of LIST on the landing folder, run on a schedule
Show the answer and why

  • AnumFilesOutstanding and numBytesOutstanding source metrics

    Correct

    Auto Loader reports its backlog in these source metrics at every batch.

  • BnumOutputRows and inputRowsPerSecond in the stream's progress metrics

    Incorrect

    These measure throughput, not the remaining backlog.

  • CThe numFiles value of DESCRIBE DETAIL on the target table

    Incorrect

    That counts the target table's files, not unprocessed source files.

  • DThe file count of LIST on the landing folder, run on a schedule

    Incorrect

    LIST shows every file in storage, processed or not, so it does not measure the backlog.

Auto Loader reports metrics to the Streaming Query Listener at each batch, including numFilesOutstanding and numBytesOutstanding for the backlog.

Databricks documentation

Task 1.4Report a problem with this question

Question 5 · choose 1

To save money, an administrator adds a 30-day object lifecycle policy to the bucket path that holds Auto Loader checkpoints. What risk does Databricks warn about?

  1. AThe stream switches to directory listing mode once the 30 days pass
  2. BFiles older than 30 days are ingested a second time each month
  3. CThe policy has no effect because checkpoints are kept in memory
  4. DRemoved files corrupt the stream state and force a fresh start
Show the answer and why

  • AThe stream switches to directory listing mode once the 30 days pass

    Incorrect

    Lifecycle rules do not change the file detection mode.

  • BFiles older than 30 days are ingested a second time each month

    Incorrect

    The stated risk is corrupted state that requires a fresh start.

  • CThe policy has no effect because checkpoints are kept in memory

    Incorrect

    Checkpoints are stored in the checkpoint location, not only in memory.

  • DRemoved files corrupt the stream state and force a fresh start

    Correct

    Databricks recommends a checkpoint location without a lifecycle policy because removed files corrupt the stream state.

The checkpoint stores the stream's state and progress. If a lifecycle policy removes files from it, the state is corrupted and the stream must be restarted from scratch.

Databricks documentation

Task 1.4Report a problem with this question

Question 6 · choose 1

An Auto Loader stream that runs with an availableNow trigger after an outage creates very large micro-batches. The team wants a hard cap on the number of files per micro-batch. Which option sets it?

  1. AcloudFiles.maxBytesPerTrigger
  2. BcloudFiles.maxFilesPerTrigger
  3. CcloudFiles.maxFileAge
  4. DcloudFiles.backfillInterval
Show the answer and why

  • AcloudFiles.maxBytesPerTrigger

    Incorrect

    The byte limit is a soft limit, so a batch can exceed it.

  • BcloudFiles.maxFilesPerTrigger

    Correct

    The file limit is a hard limit; the default is 1,000 files per micro-batch.

  • CcloudFiles.maxFileAge

    Incorrect

    maxFileAge expires tracked file state; it does not size batches.

  • DcloudFiles.backfillInterval

    Incorrect

    This option does not limit the size of a micro-batch.

By default Auto Loader processes at most 1,000 files per micro-batch. cloudFiles.maxFilesPerTrigger sets a hard file limit and cloudFiles.maxBytesPerTrigger a soft byte limit.

Databricks documentation

Task 1.4Report a problem with this question

Practice domain 1 →Practice all domains →