Data Engineer Professional · Domain 4: Monitoring and Alerting · 10% of the exam
Task 4.2: Use Databricks REST APIs / CLI and SDK for monitoring and analyzing jobs and pipelines.
Listing and getting runs, run output, pipeline updates and events, pagination, and authenticating scripts as service principals.
Study it
Monitoring jobs and pipelines with the REST API, CLI and SDK
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A monitoring script must collect the dbutils.notebook.exit value of every notebook task in the latest run of a multi-task job. Calling runs/get-output with the job run's run_id returns an error. What should the script do?
ACall runs/get-output with the job_id instead of the run_id
BCall runs/export with the job run's run_id
Cruns/get for the task list, then runs/get-output per task run_id
DCall jobs/list-runs with expand_tasks set to true and read the outputs
Show the answer and why
ACall runs/get-output with the job_id instead of the run_id
Incorrect
get-output takes a run ID, not a job ID.
BCall runs/export with the job run's run_id
Incorrect
export returns exported views of a run, not the task exit values.
Cruns/get for the task list, then runs/get-output per task run_id
Correct
get-output returns the output of a single task run, and runs/get lists the tasks with their own run IDs.
DCall jobs/list-runs with expand_tasks set to true and read the outputs
Incorrect
This lists runs with task details, but not the tasks' output values.
Each task in a run has its own run_id. runs/get-output retrieves the output and metadata of a single task run, including a value returned through dbutils.notebook.exit().
A notebook task returns a large JSON document with dbutils.notebook.exit, and the monitoring script that reads it through runs/get-output receives a truncated value. What does Databricks recommend?
ACall get-output again with a larger page size parameter
BWrite the result to cloud storage and return a reference
CCall get-output twice and concatenate the results
DReturn the value as a task value instead of exiting with it
Show the answer and why
ACall get-output again with a larger page size parameter
Incorrect
get-output has no page size; the 5 MB limit applies.
BWrite the result to cloud storage and return a reference
Correct
get-output returns the first 5 MB of output; larger results should be stored in cloud storage.
CCall get-output twice and concatenate the results
Incorrect
Repeated calls return the same truncated output.
DReturn the value as a task value instead of exiting with it
Incorrect
Task values are small values for passing between tasks, not a store for large results.
runs/get-output retrieves a single task run's output, such as a value from dbutils.notebook.exit(), and returns at most the first 5 MB. Store larger results in a cloud storage service.
A script calls runs/get for a run of a job with 250 tasks and sees only 100 tasks in the response. How does it get the rest?
ASet limit to 250 on the runs/get request for that run
BCall runs/get-output for the job run to list its tasks
CUse API version 2.0, which returns every task in one response
DCall runs/get again with page_token set to next_page_token
Show the answer and why
ASet limit to 250 on the runs/get request for that run
Incorrect
runs/get pages its array properties with tokens, not a limit.
BCall runs/get-output for the job run to list its tasks
Incorrect
get-output does not return the list of tasks.
CUse API version 2.0, which returns every task in one response
Incorrect
Use pagination; older API versions are not the documented approach.
DCall runs/get again with page_token set to next_page_token
Correct
Array properties beyond 100 elements are returned on later pages requested with page_token.
A single-run request returns the first 100 elements of array properties such as tasks. Use next_page_token from the response as page_token to get the next page.
A script must fetch only error-level events of a pipeline through the Databricks CLI, without downloading the whole event log. Which command supports this?
get-update returns one update, not a filtered list of events.
Ddatabricks jobs list-runs --filter level=ERROR
Incorrect
jobs list-runs lists job runs and has no event filter.
The pipelines command group mirrors the Pipelines API. list-pipeline-events retrieves a pipeline's events, with --filter for a SQL-like condition, --max-results and --page-token.