Skip to content
Cheat sheets

Data Engineer Professional · Domain 4: Monitoring and Alerting · 10% of the exam

Task 4.3: Use Pipeline event logs to monitor pipeline health and data quality.

The event log schema, event types such as flow_progress and update_progress, expectation metrics, lineage and backlog, and publishing the log.

Study it

  • The pipeline event log

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A data engineer must report, for the latest update of a pipeline, how many records passed and failed each expectation on each dataset, by querying the pipeline event log. Which events and field hold these numbers?

  1. Astream_progress events, in the metrics of each streaming source
  2. Bflow_progress events, under data_quality.expectations
  3. CDESCRIBE HISTORY of each dataset, in its operationMetrics column
  4. Dsystem.lakeflow.pipeline_update_timeline, one row per update
Show the answer and why

  • Astream_progress events, in the metrics of each streaming source

    Incorrect

    Source metrics describe streaming progress, not results per expectation.

  • Bflow_progress events, under data_quality.expectations

    Correct

    Expectation metrics, with passed and failed records per expectation, are in the data_quality object of flow_progress events.

  • CDESCRIBE HISTORY of each dataset, in its operationMetrics column

    Incorrect

    Table history records write metrics per commit, not results per expectation, and it is not the event log.

  • Dsystem.lakeflow.pipeline_update_timeline, one row per update

    Incorrect

    This system table tracks update times and compute, not expectation results.

Data quality events have the type flow_progress. Passed and failed records per expectation are in details:flow_progress.data_quality.expectations, and dropped records in details:flow_progress.data_quality.

Databricks documentation

Task 4.3Report a problem with this question

Question 2 · choose 1

A report must show whether each pipeline update ended as COMPLETED, FAILED or CANCELED, from the pipeline's event log. Which event type records these states?

  1. Aflow_progress
  2. Bflow_definition
  3. Cupdate_progress
  4. Dplanning_information
Show the answer and why

  • Aflow_progress

    Incorrect

    These events do not carry the update's final state.

  • Bflow_definition

    Incorrect

    These events are not where update states are recorded.

  • Cupdate_progress

    Correct

    update_progress events carry the update's state, including terminal states.

  • Dplanning_information

    Incorrect

    These events do not report whether an update completed or failed.

The documented queries take the latest update_progress event per update to get its state; COMPLETED, FAILED and CANCELED are terminal states.

Databricks documentation

Task 4.3Report a problem with this question

Question 3 · choose 1

A data engineer needs the input datasets of every dataset in a pipeline, taken from the event log. Which event type and fields hold this?

  1. Acluster_resources, with avg_task_slot_utilization
  2. Boperation_progress, with auto_loader_details per dataset
  3. Ccreate_update, with origin.update_id
  4. Dflow_definition, with output_dataset and input_datasets
Show the answer and why

  • Acluster_resources, with avg_task_slot_utilization

    Incorrect

    These events report task slot use on classic compute, not input datasets.

  • Boperation_progress, with auto_loader_details per dataset

    Incorrect

    These events report Auto Loader listing and backfill, not dataset lineage.

  • Ccreate_update, with origin.update_id

    Incorrect

    That marks the start of an update.

  • Dflow_definition, with output_dataset and input_datasets

    Correct

    Lineage events have the type flow_definition and list each relationship's input and output datasets.

Events with lineage information have the type flow_definition; the details:flow_definition object contains output_dataset and input_datasets.

Databricks documentation

Task 4.3Report a problem with this question

Question 4 · choose 1

A materialized view seems to be fully recomputed on some updates. Where in the event log can the data engineer see which refresh technique each update chose and why?

  1. AIn planning_information events, listing techniques and the chosen one
  2. BIn cluster_resources events, which report task slot use and queued tasks
  3. CIn operation_progress events, which report the progress of operations
  4. DIn user_action events, under the details:user_action field
Show the answer and why

  • AIn planning_information events, listing techniques and the chosen one

    Correct

    planning_information holds the technique information, including which technique was chosen and incrementalization issues.

  • BIn cluster_resources events, which report task slot use and queued tasks

    Incorrect

    These events describe compute use, not how a view was refreshed.

  • CIn operation_progress events, which report the progress of operations

    Incorrect

    These events do not report how a materialized view was refreshed.

  • DIn user_action events, under the details:user_action field

    Incorrect

    These events do not record refresh techniques.

The event log's planning_information details list the maintenance techniques considered for a materialized view, their cost, whether each was chosen, and the issues that prevented incremental refresh.

Databricks documentation

Task 4.3Report a problem with this question

Question 5 · choose 1

A pipeline's event log has never been published. The data engineer cannot find it in Catalog Explorer. How is it accessed?

  1. AFrom the pipeline's checkpoint folder in DBFS, as Delta files
  2. BQuery it with the event_log() table-valued function
  3. CFrom the system.lakeflow.pipelines system table
  4. DWith DESCRIBE HISTORY on the pipeline's streaming tables
Show the answer and why

  • AFrom the pipeline's checkpoint folder in DBFS, as Delta files

    Incorrect

    The event log is a Delta table read through event_log() or its published name.

  • BQuery it with the event_log() table-valued function

    Correct

    The hidden event log is not visible in Catalog Explorer and must be read with event_log().

  • CFrom the system.lakeflow.pipelines system table

    Incorrect

    That table describes pipelines, not their events.

  • DWith DESCRIBE HISTORY on the pipeline's streaming tables

    Incorrect

    Table history lists commits, not pipeline events.

By default the event log is a hidden table named event_log_ followed by the pipeline ID. It appears in information_schema but not in Catalog Explorer, and it is read with the event_log() function unless it is published.

Databricks documentation

Task 4.3Report a problem with this question

Practice domain 4 →Practice all domains →