Data Engineer Associate · Domain 4: Working with Lakeflow Jobs · 16% of the exam
Task 4.2: Configure common tasks (notebook, SQL query, dashboard, and pipeline tasks) and their dependencies using Lakeflow Jobs and its DAG-based task graph.
Task types and their compute, depends_on and the task graph, job and task parameters, and passing values between tasks.
Study it
Task types, dependencies, parameters and task values
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A Python notebook task named validate counts invalid rows and calls dbutils.jobs.taskValues.set(key="bad_rows", value=n). A downstream If/else condition task must branch to a quarantine task when n is greater than zero. What should the left operand of the condition be?
A{{tasks.validate.values.bad_rows}}
B{{job.parameters.bad_rows}}
C{{tasks.validate.result_state}}
D{{tasks.validate.output.first_row.bad_rows}}
Show the answer and why
A{{tasks.validate.values.bad_rows}}
Correct
Task values are referenced with the dynamic value reference tasks.<task_name>.values.<value_name>, which If/else conditions accept as operands.
B{{job.parameters.bad_rows}}
Incorrect
This references a job parameter defined on the job, not a value that a task sets at run time.
C{{tasks.validate.result_state}}
Incorrect
This returns the task's result state, such as success or failed, not the number the notebook computed.
D{{tasks.validate.output.first_row.bad_rows}}
Incorrect
Output references such as first_row read the result of a SQL task; validate is a notebook task that publishes a task value instead.
dbutils.jobs.taskValues.set publishes a value from a Python notebook; downstream tasks read it with {{tasks.<task>.values.<key>}}, including in If/else conditions.
After a pipeline task, a job must run several SQL statements kept in the file etl/gold/transform.sql of the team's remote Git repository, and each run should use the committed version of that file. The job has no Git folder in the workspace. How should the SQL task be configured?
AType Query, with the file's statements pasted into a saved query
BType File, source Workspace, with the path etl/gold/transform.sql
CType File, source Git provider, with the path etl/gold/transform.sql
DType File, source Git provider, with the path /etl/gold/transform.sql
Show the answer and why
AType Query, with the file's statements pasted into a saved query
Incorrect
A Query task runs a saved SQL query against the warehouse. Pasting the statements makes a copy that commits to the repository never reach.
BType File, source Workspace, with the path etl/gold/transform.sql
Incorrect
The Workspace source runs a SQL file stored as a workspace file. This file lives in the remote repository, which the Git provider source reads.
CType File, source Git provider, with the path etl/gold/transform.sql
Correct
The Git provider source runs a .sql file from a remote Git repository, with the path entered relative to the repository root, and the file may hold several statements separated by semicolons.
DType File, source Git provider, with the path /etl/gold/transform.sql
Incorrect
The relative path must not begin with / or ./; for a file at /etl/gold/transform.sql the path to enter is etl/gold/transform.sql.
A SQL task can run a query, an alert or a .sql file. For a file versioned in a remote repository, the File type with the Git provider source runs it from that repository, and Databricks recommends that source for versioning assets scheduled with jobs. The path is relative, without a leading / or ./.
A job has the tasks ingest_orders and ingest_customers, which are independent of each other, and transform, which needs the output of both. The job should finish as soon as possible. How should the dependencies be set?
Aingest_orders, then ingest_customers, then transform, chained in sequence
Btransform depends on ingest_orders and not on ingest_customers
Ctransform depends on both ingest tasks; the ingests depend on nothing
DThe three steps combined into one notebook task
Show the answer and why
Aingest_orders, then ingest_customers, then transform, chained in sequence
Incorrect
Chaining makes the two independent ingest tasks wait for each other, which lengthens the run.
Btransform depends on ingest_orders and not on ingest_customers
Incorrect
transform could then start before ingest_customers has finished.
Ctransform depends on both ingest tasks; the ingests depend on nothing
Correct
Databricks runs as many tasks in parallel as dependencies allow, so both ingests run together and transform starts after both.
DThe three steps combined into one notebook task
Incorrect
One task runs the steps one after another and loses per-step retries, status and repair.
The task graph is a DAG: tasks without dependencies start together, and a task with several dependencies waits for all of them under the default All succeeded condition.
A job runs a Lakeflow pipeline as a pipeline task. After a change to the transformation logic, the pipeline's streaming tables must be recomputed from all source data once, in the next run of the job. What should the data engineer do?
ATurn on full refresh in the pipeline task for that run
BDelete the pipeline and create it again from the same code
CRepair the last successful run of the job
DIncrease the retries on the pipeline task
Show the answer and why
ATurn on full refresh in the pipeline task for that run
Correct
A pipeline task can trigger a full refresh of the pipeline, which reprocesses all source data instead of only new data.
BDelete the pipeline and create it again from the same code
Incorrect
Rebuilding the whole pipeline is a disruptive workaround when the pipeline task can request a full refresh directly.
CRepair the last successful run of the job
Incorrect
Repair re-runs failed or skipped tasks; it does not change how the pipeline processes its data.
DIncrease the retries on the pipeline task
Incorrect
Retries repeat a failed update with the same incremental behavior.
Streaming tables process each row once, so logic changes affect only new rows until a full refresh reprocesses everything.
Five notebook tasks in a job all need the same run_date value, and an operator must be able to change it for a single manual run. Where should run_date be defined?
AAs a task parameter on each of the five tasks
BAs a task value set by the first task
CAs a Spark configuration property on the job cluster
DAs a job parameter defined on the job
Show the answer and why
AAs a task parameter on each of the five tasks
Incorrect
Five copies must be kept in sync, and static task parameters change only by editing the task definitions.
BAs a task value set by the first task
Incorrect
Task values carry results computed during a run; they are not inputs an operator sets.
CAs a Spark configuration property on the job cluster
Incorrect
Spark properties configure the compute, not inputs for notebook code.
DAs a job parameter defined on the job
Correct
Job parameters are defined once at the job level, pushed down to tasks that take key-value parameters, and can be overridden with Run now with different parameters.
Use job parameters for values shared across tasks and changeable per run, task parameters for task-specific inputs, and task values for results passed between tasks.