Skip to content

Data Engineer Professional · Domain 8: Debugging and Deploying · 10% of the exam

Task 8.1: Identify pertinent diagnostic information using Spark UI, cluster logs, system tables, and query profiles to troubleshoot errors.

Driver and executor logs, compute event logs, the Spark UI stages and executors tabs, query profiles, and the system tables that hold the history.

Study it

  • Diagnostics: Spark UI, compute logs, system tables and query profiles

    Partly covered by: Tracing a Request

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

An audit requires that the driver and worker logs of a nightly job's classic job clusters stay available for 90 days, so that failures can be investigated long after the clusters are gone. Today the engineers often find no logs for old failures. What should the data engineer configure?

  1. AA larger driver node with more local disk for logs
  2. BCompute log delivery to a volume or storage path the team keeps
  3. CA query on system.lakeflow.job_task_run_timeline for the error text of each failed task
  4. DThe Spark UI of each failed run, opened from the run page months later
Show the answer and why
  • AA larger driver node with more local disk for logs

    Incorrect

    Local disk disappears with the cluster, whatever its size.

  • BCompute log delivery to a volume or storage path the team keeps

    Correct

    Log delivery copies driver, worker and event logs to a destination that outlives the cluster.

  • CA query on system.lakeflow.job_task_run_timeline for the error text of each failed task

    Incorrect

    The timeline records run times and compute IDs, not driver log output.

  • DThe Spark UI of each failed run, opened from the run page months later

    Incorrect

    The Spark UI depends on the cluster's logs and history, which are gone with the cluster.

Compute log delivery sends a cluster's driver, worker and event logs to a destination you choose, so they can be read after the compute is terminated or removed.

Question 2 · choose 1

In the Spark UI, the job details page for one micro-batch of a Structured Streaming query shows several stages as grayed boxes in its DAG. The engineer wonders whether those stages failed. What do the grayed boxes mean?

  1. AStages that failed and are now waiting for Spark to retry them
  2. BStages that ran on the driver instead of the executors
  3. CSkipped stages, whose data came from a checkpoint or cache
  4. DStages still queued because no task slots are free yet
Show the answer and why
  • AStages that failed and are now waiting for Spark to retry them

    Incorrect

    Gray boxes mark stages that Spark skipped, not failed ones.

  • BStages that ran on the driver instead of the executors

    Incorrect

    The gray color does not show where a stage ran.

  • CSkipped stages, whose data came from a checkpoint or cache

    Correct

    Spark skips stages it does not need to recompute, for example when the data is checkpointed or cached, and shows them as gray boxes.

  • DStages still queued because no task slots are free yet

    Incorrect

    Gray boxes are stages that Spark skipped, not stages waiting to run.

The job details page shows a DAG of each batch. Grayed boxes are skipped stages: Spark skips stages that do not need to be recomputed when the data is checkpointed or cached. Structured Streaming checkpoints internally and reads from the checkpoint instead of depending on previous batches, so those stages appear gray.

Question 3 · choose 1

A cluster terminated unexpectedly yesterday. Where does the data engineer look first for the lifecycle event and its reason?

  1. AThe Event log tab of the compute
  2. BThe Spark UI SQL tab
  3. CThe notebook's cell output
  4. DThe table history of the job's output table
Show the answer and why
  • AThe Event log tab of the compute

    Correct

    The compute event log records lifecycle events such as creation, termination and configuration edits.

  • BThe Spark UI SQL tab

    Incorrect

    The Spark UI covers query execution, not compute lifecycle.

  • CThe notebook's cell output

    Incorrect

    Cell output shows code results, not compute events.

  • DThe table history of the job's output table

    Incorrect

    Table history records writes to the table.

Databricks keeps three kinds of compute logs: event logs for lifecycle events, Spark driver and worker logs, and init script logs. Event log entries are kept for 60 days and can be downloaded.

Question 4 · choose 1

A team suspects that a job cluster's workers ran out of memory during last week's runs. Which system table shows the utilization of each node over time?

  1. Asystem.compute.clusters
  2. Bsystem.compute.warehouse_events
  3. Csystem.query.history
  4. Dsystem.compute.node_timeline
Show the answer and why
  • Asystem.compute.clusters

    Incorrect

    That table does not record node utilization over time.

  • Bsystem.compute.warehouse_events

    Incorrect

    Warehouse events record SQL warehouse starts, stops and scaling, not job cluster node utilization.

  • Csystem.query.history

    Incorrect

    That table does not hold node utilization metrics.

  • Dsystem.compute.node_timeline

    Correct

    Node timeline captures utilization metrics of all-purpose and jobs compute.

system.compute.node_timeline records utilization metrics for classic all-purpose and jobs compute, with a 90-day retention.

Question 5 · choose 1

An engineer wants to look at the configuration of an all-purpose cluster that was terminated six weeks ago, but it no longer appears. What happened?

  1. ATerminated compute is hidden from users without CAN MANAGE
  2. BUnpinned compute is deleted 30 days after termination
  3. CAll-purpose compute is deleted when its notebook is detached
  4. DThe compute moved to the system.compute.clusters table
Show the answer and why
  • ATerminated compute is hidden from users without CAN MANAGE

    Incorrect

    Deletion, not visibility, explains it.

  • BUnpinned compute is deleted 30 days after termination

    Correct

    Compute that is not pinned or restarted is permanently deleted 30 days after it terminates.

  • CAll-purpose compute is deleted when its notebook is detached

    Incorrect

    Detaching does not delete compute.

  • DThe compute moved to the system.compute.clusters table

    Incorrect

    The compute object is deleted; it is not moved anywhere.

Unless compute is pinned or restarted, it is permanently deleted 30 days after termination; an admin can pin up to 100 compute resources to keep their configuration.

Practice domain 8 →Practice all domains →