Skip to content
Cheat sheets

Data Engineer Professional · Domain 3: Data Manipulation · 12% of the exam

Task 3.4: Implement data quality expectations in Lakeflow Pipelines to quarantine, drop, or fail on bad records.

Warn, drop and fail actions, grouped expectations, quarantine patterns, reusable rule tables, and what each action does to the update.

Study it

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A pipeline has a validation materialized view with expect_or_fail that compares row counts between bronze and silver. Gold tables in the same pipeline still updated on a day when the validation failed. Gold must not update unless validation passes. What should the data engineer do?

  1. AMake the gold tables read from the validation view
  2. BChange the validation expectation from expect_or_fail to expect_or_drop
  3. CRun the pipeline in continuous mode
  4. DMove gold to its own pipeline, chained after validation in a job
Show the answer and why

  • AMake the gold tables read from the validation view

    Incorrect

    Reading a validation table does not make a dataset wait for the validation result.

  • BChange the validation expectation from expect_or_fail to expect_or_drop

    Incorrect

    Changing the action of the validation expectation does not make other tables wait for it.

  • CRun the pipeline in continuous mode

    Incorrect

    In continuous mode a failure stops the flow and its dependents, and gold does not depend on the validation view.

  • DMove gold to its own pipeline, chained after validation in a job

    Correct

    A pipeline task fails when its update fails, so the dependent task with the gold pipeline does not run.

Expectations enforce data quality, not orchestration. A validation table with expect_or_fail fails its own flow, but it does not gate other datasets. To stop downstream work, split it into separate pipelines and orchestrate them with a job.

Databricks documentation

Task 3.4Report a problem with this question

Question 2 · choose 2

In the streaming table orders_silver, rows from the test store (store_id = 0) are known noise and must be left out. A negative amount, however, means an upstream bug, and the update must stop so that someone investigates. Which TWO constraint clauses implement this? (Choose TWO.)

  1. ACONSTRAINT real_store EXPECT (store_id <> 0) ON VIOLATION DROP ROW
  2. BCONSTRAINT real_store EXPECT (store_id <> 0) ON VIOLATION FAIL UPDATE
  3. CCONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION DROP ROW
  4. DCONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION QUARANTINE
  5. ECONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION FAIL UPDATE
Show the answer and why

  • ACONSTRAINT real_store EXPECT (store_id <> 0) ON VIOLATION DROP ROW

    Correct

    DROP ROW removes invalid records before they are written and logs the count.

  • BCONSTRAINT real_store EXPECT (store_id <> 0) ON VIOLATION FAIL UPDATE

    Incorrect

    This would stop the update for known noise that should only be dropped.

  • CCONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION DROP ROW

    Incorrect

    Dropping the rows hides the upstream bug instead of stopping the update.

  • DCONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION QUARANTINE

    Incorrect

    There is no QUARANTINE action; quarantining is a pattern built from other datasets, and it would not stop the update.

  • ECONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION FAIL UPDATE

    Correct

    FAIL UPDATE makes an invalid record stop the update, so it can be investigated.

Each expectation names its own action: DROP ROW removes invalid records before the write, and FAIL UPDATE stops the update. Several expectations with different actions can be combined on one dataset.

Databricks documentation

Task 3.4Report a problem with this question

Question 3 · choose 1

A dataset has CONSTRAINT valid_email EXPECT (email LIKE '%@%') with no ON VIOLATION clause. What happens to rows that fail it?

  1. AThey are dropped before the write, and their count is logged
  2. BThey are written to the target, and violations are counted
  3. CThe update fails and stops at the first violating row
  4. DThey are moved to a quarantine table automatically by the pipeline
Show the answer and why

  • AThey are dropped before the write, and their count is logged

    Incorrect

    Rows are not removed without an explicit action clause.

  • BThey are written to the target, and violations are counted

    Correct

    warn is the default action: invalid records are written and metrics are recorded.

  • CThe update fails and stops at the first violating row

    Incorrect

    The update does not stop without an explicit action clause.

  • DThey are moved to a quarantine table automatically by the pipeline

    Incorrect

    Quarantine is a pattern you build; it is not an action.

The default action, warn, writes invalid records to the target and records metrics. drop removes them and fail stops the update.

Databricks documentation

Task 3.4Report a problem with this question

Question 4 · choose 1

An update stopped because an expect_or_fail expectation was violated. The team looks for the violation counts in the Data quality tab. What should they expect?

  1. AA count of the failing rows, as for drop expectations
  2. BThe failing rows written to the target with a flag
  3. CNo metrics for it, since a failed update records none
  4. DAn automatic retry that skips the failing rows next time
Show the answer and why

  • AA count of the failing rows, as for drop expectations

    Incorrect

    Violation counts are not kept for this action.

  • BThe failing rows written to the target with a flag

    Incorrect

    Rows that violate a fail expectation are not written.

  • CNo metrics for it, since a failed update records none

    Correct

    Because fail stops the update, its metrics are not recorded, and manual intervention is needed before reprocessing.

  • DAn automatic retry that skips the failing rows next time

    Incorrect

    A fail expectation does not skip rows; it stops the update.

Tracking metrics are shown for warn and drop actions. A fail action stops the update when an invalid record is found, so no metrics are recorded and someone must intervene before the data is reprocessed.

Databricks documentation

Task 3.4Report a problem with this question

Question 5 · choose 1

In a Python pipeline, the streaming table payments has @dp.expect_or_fail("positive", "amount > 0"). A triggered update reads a batch of 10,000 new rows, and one of them has amount = -5. What does payments hold from that batch after the update fails?

  1. ANone of its rows; the failed update is rolled back atomically
  2. BThe rows that were processed before the invalid row was reached
  3. CEvery row except the invalid one, which is dropped from the target
  4. DAll 10,000 rows, with the invalid row marked as failing the rule
Show the answer and why

  • ANone of its rows; the failed update is rolled back atomically

    Correct

    expect_or_fail stops execution when a record fails validation, and for a table update the transaction is rolled back atomically, so no rows of the batch are committed.

  • BThe rows that were processed before the invalid row was reached

    Incorrect

    Execution stops at the invalid record, but the whole transaction is rolled back, so the earlier rows of the batch are not kept either.

  • CEvery row except the invalid one, which is dropped from the target

    Incorrect

    Removing only the invalid rows is what the drop action does; a fail expectation stops the update instead.

  • DAll 10,000 rows, with the invalid row marked as failing the rule

    Incorrect

    A fail expectation does not write or mark violating rows; it stops the update and rolls back its transaction.

expect_or_fail stops execution as soon as a record fails validation. When the operation is a table update, the transaction is rolled back atomically, so the table keeps none of the rows from that update.

Databricks documentation

Task 3.4Report a problem with this question

Question 6 · choose 1

Data stewards keep quality rules in a Delta table with name, constraint and tag columns. A pipeline should apply the rules for a tag without code changes when rules change. How is this implemented?

  1. AIn SQL, with a CONSTRAINT whose expression queries the rules table
  2. BIn SQL, by loading the rules from a file at the top of the source
  3. CWith CHECK constraints added to the rules table
  4. DIn Python, read the rules into a dict for expect_all_or_drop
Show the answer and why

  • AIn SQL, with a CONSTRAINT whose expression queries the rules table

    Incorrect

    In SQL, expectations cannot be loaded from a rules table at run time.

  • BIn SQL, by loading the rules from a file at the top of the source

    Incorrect

    Dynamically loading expectations from a file is not supported in SQL.

  • CWith CHECK constraints added to the rules table

    Incorrect

    CHECK constraints apply to the table they are on, not to pipeline datasets.

  • DIn Python, read the rules into a dict for expect_all_or_drop

    Correct

    Rules can be read from a table and applied with expect_all decorators in Python; dynamic loading is not supported in SQL.

Databricks recommends storing expectation definitions separately from pipeline code, for example in a Delta table with tags, and applying them in Python with functions such as expect_all_or_drop.

Databricks documentation

Task 3.4Report a problem with this question

Practice domain 3 →Practice all domains →