Data Engineer Associate · Domain 6: Troubleshooting, Monitoring, and Optimization · 10% of the exam
Task 6.2: Use the Lakeflow Jobs UI to monitor pipeline health by interpreting job statuses, viewing DAG-based task graphs to spot upstream blockers, and tracking pipeline run times and failure rates.
Run and task states, upstream failures, the run graph, notifications and duration thresholds, and the pipeline event log.
Study it
Job and task states, run graphs, notifications and duration alerts
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 2
The on-call engineer opens a failed job run. Its graph shows extract as Succeeded, validate as Failed, and transform and publish, which follow validate in a chain with the default Run if condition, as Upstream failed. Which TWO statements about reading this run are true? (Choose TWO.)
Avalidate is the task that blocked the run, so its cause comes first
Btransform also failed while it ran, on the rows that validate passed
Ctransform and publish were disabled for this run by the scheduler
DThe run shows the status Skipped in the job's runs list
EClicking validate opens its task run with its output and error message
Show the answer and why
Avalidate is the task that blocked the run, so its cause comes first
Correct
transform and publish are marked Upstream failed because their All succeeded condition was not met; the one task that failed while running is validate.
Btransform also failed while it ran, on the rows that validate passed
Incorrect
Upstream failed marks a task whose Run if condition was unmet because a dependency failed, so transform did not run at all.
Ctransform and publish were disabled for this run by the scheduler
Incorrect
Disabled is the status of tasks turned off in the job settings, or of tasks downstream of a disabled task; these tasks were not disabled.
DThe run shows the status Skipped in the job's runs list
Incorrect
Skipped is for a run that was skipped, for example because the maximum concurrent runs was exceeded; this run started, and validate ran and failed.
EClicking validate opens its task run with its output and error message
Correct
Clicking a failed task opens the Task run details page, which shows the task's output, error message and metadata, the place to start the diagnosis.
In a run graph, a failure shows once as Failed and then spreads downstream as Upstream failed on the tasks that never ran. The task to investigate is the Failed one, and its task run details hold the output and the error message that explain the failure.
In a job run, a non-leaf task failed, a downstream cleanup task with Run if set to All done succeeded, and every leaf task succeeded. Which status does the job run get?
AFailed
BSucceeded
CSucceeded with failures
DSkipped
Show the answer and why
AFailed
Incorrect
A run is Failed when one or more leaf tasks failed; here all leaf tasks succeeded.
BSucceeded
Incorrect
Succeeded means every task succeeded, and one task failed in this run.
CSucceeded with failures
Correct
When some tasks fail but all leaf tasks succeed, the run is marked Succeeded with failures.
DSkipped
Incorrect
Skipped is for runs that never started, which does not match a run whose tasks ran.
Databricks decides a run's status from its leaf tasks. Notifications treat Succeeded with failures as a success, so select Success to be told about it.
A task has three retries. The on-call engineer wants an email each time an attempt of that task fails, not just when the job finally fails. How should notifications be configured?
AAdd a failure notification on the task itself
BAdd a failure notification at the job level
CTurn on Mute notifications until the last retry
DAdd a duration warning notification on the job
Show the answer and why
AAdd a failure notification on the task itself
Correct
Job-level notifications are not sent when failed tasks are retried; task notifications report each failed attempt.
BAdd a failure notification at the job level
Incorrect
Job-level failure notifications are not sent for retried task failures.
CTurn on Mute notifications until the last retry
Incorrect
That option does the opposite, holding notifications back until the final retry.
DAdd a duration warning notification on the job
Incorrect
Duration warnings fire when a run exceeds its expected time, not when an attempt fails.
Notifications can be set on jobs and on tasks. For visibility into every failed attempt, use task-level notifications.
A job normally finishes in 30 minutes. If it is still running after 45 minutes, the team wants to be told, but the run must keep going. What should be configured?
AA Timeout of 45 minutes set in the job's duration thresholds
BA failure notification sent to the team's email address
CA maximum of one concurrent run set in the job's advanced settings
DA 45-minute Warning threshold with a duration warning notification
Show the answer and why
AA Timeout of 45 minutes set in the job's duration thresholds
Incorrect
A timeout ends the run and sets its status to Timed Out, which the team does not want.
BA failure notification sent to the team's email address
Incorrect
A failure notification fires only when the run fails, not when it is slow.
CA maximum of one concurrent run set in the job's advanced settings
Incorrect
Concurrency limits overlapping runs; it does not detect slow ones.
DA 45-minute Warning threshold with a duration warning notification
Correct
The warning threshold sets the expected completion time; exceeding it triggers an event that a Duration warning notification can report.
Duration thresholds have two settings: Warning, which raises an event and can notify, and Timeout, which stops the run.
A job created in the Jobs UI this year runs every 10 minutes and keeps its default settings, including a maximum of 1 concurrent run and the default queue setting. On a busy day one run takes 25 minutes. What happens to the runs that the schedule starts while that run is still active?
AThey are skipped and listed with the status Skipped
BThey are queued and start once the active run finishes
CThey start at once and run in parallel with the active run
DThey fail with a maximum concurrent runs error
Show the answer and why
AThey are skipped and listed with the status Skipped
Incorrect
Runs are skipped because of concurrency limits when queueing is off; for jobs created in the UI after April 15, 2024, queueing is on by default.
BThey are queued and start once the active run finishes
Correct
With queueing on, a run that hits the job's maximum concurrent runs is queued for up to 48 hours, shown in the runs list, and dequeued when capacity is available.
CThey start at once and run in parallel with the active run
Incorrect
By default the maximum concurrent runs for a new job is 1, so a second run cannot be active at the same time.
DThey fail with a maximum concurrent runs error
Incorrect
Exceeding the maximum concurrent runs can cause a failure, but with queueing on the run waits in the queue instead.
Two job settings decide what happens to overlapping runs. Maximum concurrent runs, 1 by default, caps how many run at once. Queueing, on by default for jobs created in the UI since April 15, 2024, makes a run that hits that cap wait as Queued instead of being skipped.