Skip to content

Data Engineer Associate · Domain 4: Working with Lakeflow Jobs · 16% of the exam

Task 4.1: Implement control flows (retries and conditional tasks such as branching and looping) using Lakeflow Jobs for pipeline orchestration.

Task retries and timeouts, If/else condition tasks, Run if dependencies, For each tasks, task values, and repairing a failed run.

Study it

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A job has two parallel tasks, load_orders and load_returns. A third task, drop_staging, removes temporary staging tables and must run after both load tasks finish, whether they succeed or fail. Which Run if condition should drop_staging use?

  1. AAll succeeded
  2. BAll done
  3. CAt least one failed
  4. DNone failed
Show the answer and why
  • AAll succeeded

    Incorrect

    This is the default, and it runs drop_staging only when both load tasks succeed, so a failed load leaves the staging tables behind.

  • BAll done

    Correct

    All done runs the task after all its dependencies have run, regardless of their outcome, which is what a cleanup step needs.

  • CAt least one failed

    Incorrect

    This runs only when a load task fails; when both succeed, drop_staging is excluded and never cleans up.

  • DNone failed

    Incorrect

    This requires that no dependency failed, so a failed load task would stop the cleanup.

Run if conditions decide whether a task runs based on its dependencies' outcomes. All done is the choice for cleanup or teardown that must happen every time.

Question 2 · choose 1

The same notebook must refresh 30 bronze tables, receiving one table name per run, with at most five tables processed at the same time. The list of tables changes every month. How should the job be built?

  1. AA For each task over a JSON array of table names, concurrency 5, nesting the notebook
  2. BAn If/else condition task that compares the table name and then routes to the notebook task
  3. CA Run Job task that triggers a second job holding the notebook
  4. DThirty separate notebook tasks in one job, each with its table name hard-coded
Show the answer and why
  • AA For each task over a JSON array of table names, concurrency 5, nesting the notebook

    Correct

    A For each task runs its nested task once per input value, passes the value with {{input}}, and runs up to the configured concurrency in parallel.

  • BAn If/else condition task that compares the table name and then routes to the notebook task

    Incorrect

    If/else adds branching on a boolean expression; it runs one branch, not one run per item in a list.

  • CA Run Job task that triggers a second job holding the notebook

    Incorrect

    Run Job starts another job once; it does not loop over a list or control per-item concurrency.

  • DThirty separate notebook tasks in one job, each with its table name hard-coded

    Incorrect

    This works but must be edited by hand every time the list changes and has no single concurrency setting.

Looping in Lakeflow Jobs is done with the For each task: one nested task, a list of inputs, and a concurrency value. The list can also come from task values or job parameters.

Question 3 · choose 1

A notebook task sometimes fails because a partner API returns transient errors, and occasionally it hangs. The task must be attempted up to three more times after a failure, and each attempt must be stopped if it runs longer than 30 minutes. How should the task be configured?

  1. AA 30-minute Timeout on the job, with no task retries
  2. BRetries set to 3 on the task, plus a 30-minute task Timeout
  3. CA continuous trigger on the job instead of a schedule
  4. DThree copies of the task chained with Run if set to At least one failed
Show the answer and why
  • AA 30-minute Timeout on the job, with no task retries

    Incorrect

    A job-level timeout bounds the whole run and gives no retries, so a transient error ends the run.

  • BRetries set to 3 on the task, plus a 30-minute task Timeout

    Correct

    A retry policy re-runs a failed task, and when both Timeout and Retries are configured, the timeout applies to each retry.

  • CA continuous trigger on the job instead of a schedule

    Incorrect

    A continuous trigger decides when the job runs; it sets neither a retry count nor a time limit for each attempt of this task.

  • DThree copies of the task chained with Run if set to At least one failed

    Incorrect

    Chaining copies imitates retries with extra tasks and still gives no limit on how long each attempt runs.

Task retries handle transient failures, and the task timeout caps each attempt.

Question 4 · choose 1

A job is run with a job parameter env set to 'dev' or 'prod'. An extra validation task must run only when env is 'prod'; otherwise the job continues without it. How should the job express this?

  1. AA For each task that loops over the possible values of the env parameter
  2. BRun if set to None failed on the validation task, with env passed to it
  3. CTwo separate jobs, one for dev and one for prod, each with its own tasks
  4. DAn If/else task on env == prod, with validation on its true branch
Show the answer and why
  • AA For each task that loops over the possible values of the env parameter

    Incorrect

    For each runs a nested task once per input; it does not decide whether a task should run.

  • BRun if set to None failed on the validation task, with env passed to it

    Incorrect

    Run if conditions look at the outcome of upstream tasks, not at parameter values.

  • CTwo separate jobs, one for dev and one for prod, each with its own tasks

    Incorrect

    This duplicates the job definition instead of branching inside one job.

  • DAn If/else task on env == prod, with validation on its true branch

    Correct

    If/else evaluates a boolean expression on job parameters, task values or dynamic values and runs the tasks that depend on the matching branch.

Branch on values with an If/else condition task, and on upstream outcomes with Run if. Both are part of control flow in Lakeflow Jobs.

Question 5 · choose 1

A six-task job run failed at task 4 because an external database was down. Tasks 1 to 3 succeeded and took two hours. The database is back. How should the data engineer finish the run without repeating tasks 1 to 3?

  1. AUse Repair run on the failed run
  2. BClick Run now on the job
  3. CRaise the task retries to 5 and click Run now
  4. DAdd a Run Job task that triggers the same job
Show the answer and why
  • AUse Repair run on the failed run

    Correct

    A repair re-runs only the unsuccessful tasks and the tasks that depend on them, so successful tasks are not repeated.

  • BClick Run now on the job

    Incorrect

    Run now starts a new run of the whole job, repeating the two hours of successful work.

  • CRaise the task retries to 5 and click Run now

    Incorrect

    Retries apply to future attempts within a run; Run now still starts from the beginning.

  • DAdd a Run Job task that triggers the same job

    Incorrect

    A Run Job task starts another full run of the job, and a job that triggers itself creates a circular dependency.

Repair run saves time and cost after a failure: fix the cause, then re-run the failed and skipped tasks with the current job settings. Make tasks idempotent, because a repaired task starts over.

Question 6 · choose 1

An optional leaf task in a scheduled job is broken while its source is being rebuilt. The rest of the job must keep running on schedule, and the broken task's configuration and run history must be kept so it can come back next week. What should the data engineer do?

  1. ADelete the task from the job
  2. BPause the job's schedule
  3. CDisable the task in the job
  4. DSet the task's retries to 0
Show the answer and why
  • ADelete the task from the job

    Incorrect

    Deleting removes the task's configuration and history, which must be kept.

  • BPause the job's schedule

    Incorrect

    Pausing stops every scheduled run, including the tasks that should keep running.

  • CDisable the task in the job

    Correct

    A disabled task is skipped at run time but keeps its configuration and run history, and can be re-enabled later.

  • DSet the task's retries to 0

    Incorrect

    The task would still run and fail on every scheduled run.

Disabling skips a task without removing it. Downstream tasks are evaluated against their Run if conditions, and a leaf task has none to affect.

Practise domain 4 →Practise all domains →