Data Engineer Associate · Domain 4: Working with Lakeflow Jobs · 16% of the exam
Task 4.3: Implement job schedules using Lakeflow Jobs with an understanding of trigger types (scheduled, file arrival, and table update).
Simple and cron schedules with time zones, file arrival triggers on volumes and external locations, table update triggers, continuous jobs, and pausing a trigger.
Study it
Schedules, file arrival triggers, table update triggers and continuous jobs
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 2
A job uses a file arrival trigger on /Volumes/raw/sales/landing/. Which TWO statements about this trigger are true? (Choose TWO.)
AIt needs a separate trigger for each subdirectory under landing/
BIt can watch the root or a subpath of a Unity Catalog volume or external location
CThe path can include a wildcard such as landing/*.csv to select files
DIt checks for new files about once a minute on a best-effort basis
EIt requires the job to also have a continuous trigger
Show the answer and why
AIt needs a separate trigger for each subdirectory under landing/
Incorrect
A file arrival trigger checks for new files recursively in all subdirectories of the configured location, so one trigger covers them.
BIt can watch the root or a subpath of a Unity Catalog volume or external location
Correct
The storage location must be a volume or external location in Unity Catalog, at its root or a subpath.
CThe path can include a wildcard such as landing/*.csv to select files
Incorrect
The trigger path cannot contain wildcards such as * or ?.
DIt checks for new files about once a minute on a best-effort basis
Correct
File arrival triggers make a best effort to check for new files every minute, depending on the storage performance.
EIt requires the job to also have a continuous trigger
Incorrect
A continuous trigger cannot be combined with other triggers; a file arrival trigger works on its own.
File arrival triggers watch a Unity Catalog storage location recursively and start a run when new files appear.
A job must start at the top of every hour all year, with no skipped or doubled hourly runs when clocks change for daylight saving time. How should its schedule be set?
AAn advanced schedule at minute 0 of each hour, with time zone UTC
BAn advanced schedule at minute 0 of each hour, with time zone America/New_York
CA simple schedule with an interval of 1 hour
DA continuous trigger, with the notebook sleeping one hour before it ends
Show the answer and why
AAn advanced schedule at minute 0 of each hour, with time zone UTC
Correct
UTC has no daylight saving changes, and the documentation says to choose UTC when a job must run at every hour in absolute time.
BAn advanced schedule at minute 0 of each hour, with time zone America/New_York
Incorrect
In a time zone that observes daylight saving time, an hourly job can be skipped or appear delayed when the change happens.
CA simple schedule with an interval of 1 hour
Incorrect
A simple schedule runs at an interval from a start the scheduler chooses; you cannot pick the first run time, so runs are not on the hour.
DA continuous trigger, with the notebook sleeping one hour before it ends
Incorrect
A continuous job starts a new run whenever one ends; a sleep adds cost and drift instead of a clock-based schedule.
Advanced schedules set the period, start time and time zone. Choose UTC when absolute timing matters across daylight saving transitions.
During a source system's maintenance week, a scheduled job must not start new runs, but its schedule and settings must be kept, and a run already in progress may finish. What should the data engineer do?
ADelete the trigger and recreate it after the maintenance
BPause the trigger, then resume it after the maintenance
CSet the job's maximum concurrent runs to 0
DCancel each new run as soon as it starts
Show the answer and why
ADelete the trigger and recreate it after the maintenance
Incorrect
Deleting removes the configured schedule, which then has to be rebuilt.
BPause the trigger, then resume it after the maintenance
Correct
A paused trigger starts no new runs while active runs continue, and resuming restores the same schedule.
CSet the job's maximum concurrent runs to 0
Incorrect
This changes a job setting that must be kept, and limiting concurrency is not how a schedule is suspended.
DCancel each new run as soon as it starts
Incorrect
Manual cancellation is error-prone and still starts compute for each run.
Pausing a trigger in the Schedules & Triggers section stops new runs without changing its configuration, and resuming restores the schedule.
A job joins main.sales.orders and main.sales.customers, which different processes update. It should start only after both tables have been updated since its last run. How should the trigger be set?
AA table update trigger on main.sales.orders, with Wait after last change set to one hour
BOne trigger on both tables, with Trigger when set to Any table is updated
COne trigger on both tables, with Trigger when set to All tables are updated
DA file arrival trigger on the cloud storage paths that hold the two tables
Show the answer and why
AA table update trigger on main.sales.orders, with Wait after last change set to one hour
Incorrect
This watches orders alone; waiting after its last change does not make the run wait for customers to be updated.
BOne trigger on both tables, with Trigger when set to Any table is updated
Incorrect
Any starts a run as soon as one of the tables changes.
COne trigger on both tables, with Trigger when set to All tables are updated
Correct
With several tables in one trigger, All waits until each of them has been updated before starting a run.
DA file arrival trigger on the cloud storage paths that hold the two tables
Incorrect
File arrival triggers cannot watch paths that contain tables; table changes are what table update triggers detect.
A table update trigger can watch up to 10 tables and start a run when any or all of them change.
Files arrive in bursts: about 50 files within one minute, a few times a day. A job with a file arrival trigger should start once per burst, after the burst has fully landed. Which advanced option should be set?
AWait after last change in seconds, set to about 60
BMinimum time between triggers in seconds, set to about 60
CMaximum concurrent runs set to 1
DA scheduled trigger each minute in addition
Show the answer and why
AWait after last change in seconds, set to about 60
Correct
The trigger waits this long after the most recent file arrives, and each new file resets the timer, so a run starts once the burst is complete.
BMinimum time between triggers in seconds, set to about 60
Incorrect
This is a cooldown after a run completes; the first file of a burst can still start a run while the rest are landing.
CMaximum concurrent runs set to 1
Incorrect
Concurrency caps how many runs are active at once; it does not make the trigger wait for the burst to finish.
DA scheduled trigger each minute in addition
Incorrect
A schedule starts runs by time, whether or not a burst has finished.
The two advanced options of file arrival and table update triggers are a cooldown (minimum time between triggers) and debouncing (wait after last change).