Data Engineer Associate · Domain 4: Working with Lakeflow Jobs · 16% of the exam
Task 4.4: Choose between time-based and data-driven triggers based on data availability and pipeline dependencies.
Running when the data is there rather than at a guessed time, cross-team table dependencies, and when a fixed schedule is still the better fit.
Study it
Time-based or data-driven: picking the trigger
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
Another team's Lakeflow pipeline refreshes the materialized view main.gold.daily_kpis two to five times a day, each time that team's loads finish, at no set time. A separate export job reads that view and nothing else. It should run soon after each refresh, and runs between refreshes waste money. Which trigger should the export job use?
AA table update trigger on main.gold.daily_kpis
BA table update trigger on the view's source tables
CA scheduled trigger every 15 minutes, day and night
DA scheduled trigger once a day at 23:30
Show the answer and why
AA table update trigger on main.gold.daily_kpis
Correct
Table update triggers can monitor materialized views, so each update that the other team's pipeline makes to main.gold.daily_kpis starts a run.
BA table update trigger on the view's source tables
Incorrect
The loads that change the sources come before each refresh, so runs would start ahead of the refreshes they are meant to follow.
CA scheduled trigger every 15 minutes, day and night
Incorrect
A schedule starts runs by the clock, so most runs export a view that has not changed since the run before.
DA scheduled trigger once a day at 23:30
Incorrect
A daily run exports the view once, so most refreshes wait for hours instead of being exported soon after they land.
A table update trigger can watch a materialized view directly, which ties the export to each update of the view rather than to the loads that come before it. A schedule either runs between refreshes or lags far behind them.
A partner drops files into a Unity Catalog volume at irregular times, a few times a week. They must be processed soon after they arrive, without paying for runs or compute when nothing has arrived. Which trigger fits?
AA file arrival trigger on the volume path
BA scheduled trigger each hour
CA continuous job with an Auto Loader stream
DA table update trigger on the volume
Show the answer and why
AA file arrival trigger on the volume path
Correct
The trigger starts a run when new files land, so there are no empty runs and no always-on compute.
BA scheduled trigger each hour
Incorrect
Most hourly runs would find nothing to process, and files could wait up to an hour.
CA continuous job with an Auto Loader stream
Incorrect
A continuous job keeps a run going all the time, paying for compute between deliveries.
DA table update trigger on the volume
Incorrect
A table update trigger watches tables for new commits; a volume holds files, not a table.
Data-driven triggers fit irregular arrivals: a file arrival trigger starts a run only when new files land.
A finance summary must be produced at 08:00 every Monday for a standing meeting. The underlying tables are refreshed several times over the weekend at varying times. Which trigger should the summary job use?
AA table update trigger on the underlying tables
BA file arrival trigger on the landing volume
CA scheduled trigger for 08:00 on Mondays
DA continuous trigger
Show the answer and why
AA table update trigger on the underlying tables
Incorrect
It would start a run after every weekend refresh, at unpredictable times, instead of once for the meeting.
BA file arrival trigger on the landing volume
Incorrect
File arrival follows data deliveries, not the fixed time the consumers need.
CA scheduled trigger for 08:00 on Mondays
Correct
The deliverable is tied to a fixed time, so a time-based schedule matches the requirement.
DA continuous trigger
Incorrect
A continuous job runs all the time and does not align output with a weekly meeting.
Pick time-based triggers when the consumer's deadline drives the run, and data-driven triggers when data arrival does.
A team owns an ingestion job and a transformation job in the same workspace. Transformation must start right after ingestion succeeds, in the same run, and must not start if ingestion fails. What should the team do?
ASchedule the transformation job one hour after the ingestion job's schedule
BMake transformation a task that depends on ingestion in the same job
CAdd a table update trigger on one of the tables that ingestion writes
DAdd a file arrival trigger on the ingestion job's source folder
Show the answer and why
ASchedule the transformation job one hour after the ingestion job's schedule
Incorrect
A guessed offset runs on stale or incomplete data when ingestion is slow or fails.
BMake transformation a task that depends on ingestion in the same job
Correct
A task dependency starts transformation when ingestion succeeds and stops it when ingestion fails, within the same run; a Run Job task can call a separate job the same way.
CAdd a table update trigger on one of the tables that ingestion writes
Incorrect
A table changing does not mean the whole ingestion job succeeded, and the trigger runs separately from the ingestion run.
DAdd a file arrival trigger on the ingestion job's source folder
Incorrect
That would start transformation when source files arrive, before ingestion has even run.
When both steps are under the same team's control, express the order as dependencies in Lakeflow Jobs instead of matching schedules or triggers.
Each night a partner writes about 200 data files to /Volumes/raw/partner/data/ over 20 to 40 minutes, sometimes pausing for several minutes between files, starting at any time after midnight. When the delivery is complete, it writes one empty file to /Volumes/raw/partner/done/. The load job must start once per night, only after the whole delivery has landed. Which trigger setup fits?
AFile arrival on /Volumes/raw/partner/data/, Wait after last change 60
BFile arrival on /Volumes/raw/partner/, the parent of both folders
CA scheduled trigger at 02:00, after the usual end of delivery
DFile arrival on /Volumes/raw/partner/done/, the completion folder
Show the answer and why
AFile arrival on /Volumes/raw/partner/data/, Wait after last change 60
Incorrect
Each new file resets the wait, but a pause of several minutes inside the delivery lets the 60 seconds expire and starts a run before the delivery is complete.
BFile arrival on /Volumes/raw/partner/, the parent of both folders
Incorrect
A file arrival trigger checks all subdirectories of its path recursively, so the first data files in data/ would start a run during the delivery.
CA scheduled trigger at 02:00, after the usual end of delivery
Incorrect
Deliveries can start at any time after midnight and take up to 40 minutes, so a fixed time can run before the delivery ends or wait long after it.
DFile arrival on /Volumes/raw/partner/done/, the completion folder
Correct
A file arrival trigger starts a run when a new file appears in the path it monitors, and the done folder receives one file per night, written after the whole delivery.
When data lands in pieces at unpredictable times, the most reliable signal is the one that marks completion. Pointing the file arrival trigger at a path that receives only that signal makes the job data-driven without starting on a partial delivery.