Skip to content

Data Engineer Professional · Domain 4: Monitoring and Alerting · 10% of the exam

Task 4.1: Use Databricks system tables (billing, compute, access, lakeflow) for cost analysis, auditing, and workload monitoring.

Billable usage and list prices, custom tags and budget policies, compute and warehouse tables, audit logs, and the Lakeflow jobs and pipelines tables.

Study it

  • System tables for cost, audit and workload monitoring

    Partly covered by: AWS Cost Levers

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A FinOps query joins 30 days of system.billing.usage for JOBS with system.lakeflow.jobs on workspace_id and job_id to show each job's name and list cost. Totals for jobs that were edited often are several times higher than the billing total. What should the data engineer change?

  1. AJoin on job_id only and drop workspace_id from the join
  2. BReduce system.lakeflow.jobs to the row with the latest change_time per workspace_id and job_id before joining
  3. CFilter system.billing.usage to record_type = 'ORIGINAL' before the join
  4. DRead the job names from system.lakeflow.job_run_timeline instead
Show the answer and why
  • AJoin on job_id only and drop workspace_id from the join

    Incorrect

    Job IDs are unique only within a workspace, and dropping workspace_id does not remove the duplicate history rows.

  • BReduce system.lakeflow.jobs to the row with the latest change_time per workspace_id and job_id before joining

    Correct

    The jobs table is a slowly changing dimension with a row per change, so joining every row multiplies the usage rows of edited jobs.

  • CFilter system.billing.usage to record_type = 'ORIGINAL' before the join

    Incorrect

    record_type separates original rows from corrections; it does not remove the repeated history rows of the jobs table.

  • DRead the job names from system.lakeflow.job_run_timeline instead

    Incorrect

    The run timeline has one row per run period, so joining it multiplies the usage rows as well.

The jobs, job_tasks and pipelines system tables are SCD2 tables that keep a row for each change. The documented job cost queries pick the most recent row per workspace_id and job_id with ROW_NUMBER over change_time before joining to usage.

Question 2 · choose 1

An account has workspaces in us-east-1 and eu-west-1. From a us-east-1 workspace, a dashboard joins system.billing.usage with system.lakeflow.jobs to name the most expensive jobs. Usage rows appear for both regions, but EU jobs show no name. What explains this, and what is the fix?

  1. AThe EU workspaces have system tables turned off, so an admin must turn on the lakeflow schema for them
  2. BThe EU jobs ran on serverless compute, so they must be moved to classic compute to appear
  3. CThe jobs tables are regional, so the EU job names must be read from a workspace in eu-west-1
  4. DThe EU usage rows are retractions, so they should be filtered out first
Show the answer and why
  • AThe EU workspaces have system tables turned off, so an admin must turn on the lakeflow schema for them

    Incorrect

    Usage for the EU workspaces is present; the lakeflow tables are regional, so they never show other regions' jobs.

  • BThe EU jobs ran on serverless compute, so they must be moved to classic compute to appear

    Incorrect

    The jobs tables record jobs regardless of their compute.

  • CThe jobs tables are regional, so the EU job names must be read from a workspace in eu-west-1

    Correct

    System tables hold data for workspaces in the same region, while the billing usage table includes global data.

  • DThe EU usage rows are retractions, so they should be filtered out first

    Incorrect

    Record types describe corrections; they do not explain missing names for a whole region.

System tables contain operational data for all workspaces of the account in the same cloud region; some tables, such as billable usage, include global data. The lakeflow schema is regional.

Question 3 · choose 1

A query must turn DBUs in system.billing.usage into list cost in dollars, using the price that was valid when the usage happened. How should it join system.billing.list_prices?

  1. AOn sku_name only, taking the first matching price for each SKU
  2. BOn workspace_id, because list prices are set per workspace
  3. COn sku_name, with usage_end_time inside the price's time range
  4. DOn sku_name and usage_date matching price_start_time
Show the answer and why
  • AOn sku_name only, taking the first matching price for each SKU

    Incorrect

    A SKU can have several prices over time; the first one may not apply.

  • BOn workspace_id, because list prices are set per workspace

    Incorrect

    Prices are recorded per SKU, not per workspace.

  • COn sku_name, with usage_end_time inside the price's time range

    Correct

    list_prices keeps a record for each price change, so the join must pick the price valid at the usage time.

  • DOn sku_name and usage_date matching price_start_time

    Incorrect

    Prices start on change dates, not on each usage date.

The pricing table is a historical log of SKU prices with a new record for each change. Cost queries join it on sku_name and keep the price whose time range contains the usage end time.

Question 4 · choose 1

A platform team wants to find the slowest SQL statements of the past week on SQL warehouses and on serverless notebooks and jobs. Which system table holds these records?

  1. Asystem.query.history
  2. Bsystem.compute.warehouse_events
  3. Csystem.billing.usage
  4. Dsystem.compute.node_timeline
Show the answer and why
  • Asystem.query.history

    Correct

    Query history captures queries run on SQL warehouses and on serverless compute for notebooks and jobs.

  • Bsystem.compute.warehouse_events

    Incorrect

    Warehouse events record starts, stops and scaling, not queries.

  • Csystem.billing.usage

    Incorrect

    Usage records DBUs, not individual statements.

  • Dsystem.compute.node_timeline

    Incorrect

    Node timeline does not record SQL statements.

system.query.history records queries run on SQL warehouses and serverless compute for notebooks and jobs; other system tables cover warehouse events, usage and node metrics.

Question 5 · choose 1

An incident review needs the worker type and autoscaling settings that a classic cluster had three weeks ago, before someone edited it. Which system table answers this?

  1. Asystem.compute.node_timeline
  2. Bsystem.billing.list_prices
  3. Csystem.lakeflow.job_run_timeline
  4. Dsystem.compute.clusters
Show the answer and why
  • Asystem.compute.node_timeline

    Incorrect

    Node timeline does not keep cluster configurations.

  • Bsystem.billing.list_prices

    Incorrect

    Pricing lists SKU prices, not cluster settings.

  • Csystem.lakeflow.job_run_timeline

    Incorrect

    This table does not hold cluster settings.

  • Dsystem.compute.clusters

    Correct

    The clusters table is a slowly changing dimension with the full history of each cluster's configuration.

system.compute.clusters is a slow-changing dimension table that contains the full history of compute configurations over time for any cluster.

Question 6 · choose 1

A job has twelve tasks, and its runs have slowed down. The team wants to know which task's duration grew over the last month, using system tables. Which table should they query?

  1. Asystem.lakeflow.job_run_timeline
  2. Bsystem.lakeflow.job_task_run_timeline
  3. Csystem.lakeflow.job_tasks
  4. Dsystem.compute.clusters
Show the answer and why
  • Asystem.lakeflow.job_run_timeline

    Incorrect

    job_run_timeline tracks whole job runs, not the duration of each task.

  • Bsystem.lakeflow.job_task_run_timeline

    Correct

    This table tracks start and end times and compute for each job task run.

  • Csystem.lakeflow.job_tasks

    Incorrect

    job_tasks lists the tasks of jobs, not how long their runs took.

  • Dsystem.compute.clusters

    Incorrect

    Cluster settings do not show how long tasks took.

In the lakeflow schema, job_run_timeline tracks job runs and job_task_run_timeline tracks task runs, while jobs and job_tasks describe the definitions.

Practice domain 4 →Practice all domains →