Skip to content

Data Engineer Associate · Domain 1: Databricks Intelligence Platform · 6% of the exam

Task 1.2: Understand Databricks Data Intelligence Platform's compute services, including their characteristics, limitations, and cost models, and select the most suitable option for each workload use case.

Serverless compute for notebooks, jobs and pipelines, classic all-purpose and jobs compute, standard and dedicated access modes, SQL warehouse types, pools and Photon, and which one each workload should run on.

Study it

  • Choosing compute: serverless, classic jobs and all-purpose compute, SQL warehouses

    Partly covered by: Vertical vs Horizontal Scaling

  • Access modes, pools, Photon, and what each compute type costs

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A team schedules a Python notebook that transforms bronze tables into silver tables every night. It needs no RDD APIs, no R, no GPUs and no custom Spark settings, and the team wants the least compute configuration and fast start-up. Which compute should the job task use?

  1. AA classic all-purpose cluster shared with the team's interactive work
  2. BA pro SQL warehouse sized for the nightly data volume
  3. CServerless compute for jobs
  4. DA classic jobs cluster attached to an instance pool
Show the answer and why
  • AA classic all-purpose cluster shared with the team's interactive work

    Incorrect

    Databricks advises against all-purpose compute for automated workloads; it is built for interactive use and is not optimized for scheduled jobs.

  • BA pro SQL warehouse sized for the nightly data volume

    Incorrect

    SQL warehouses run SQL, dbt and similar tasks; a Python notebook task runs on jobs compute, not on a warehouse.

  • CServerless compute for jobs

    Correct

    Serverless compute is the recommended choice for most automated workloads: fewer settings, faster start-up, automatic scaling, and Databricks manages the compute.

  • DA classic jobs cluster attached to an instance pool

    Incorrect

    Nothing in this workload needs a classic cluster, so choosing instance types, sizing the cluster and managing a pool is configuration the team does not need.

For a scheduled notebook with no special requirements, serverless compute for jobs is the default and recommended choice: Databricks manages the compute, and it starts quickly.

Question 2 · choose 1

Analysts use a classic all-purpose cluster during working hours. After the last notebook command of the day it often keeps running through the night and the weekend, and the team pays for that idle time. Which setting stops this cost without anyone having to remember to stop the cluster?

  1. AEnable autoscaling with a minimum of one worker
  2. BSet automatic termination after 60 minutes of inactivity
  3. CUse spot instances for the cluster's worker nodes
  4. DChoose a smaller instance type for the driver node
Show the answer and why
  • AEnable autoscaling with a minimum of one worker

    Incorrect

    Autoscaling moves the worker count between the minimum and the maximum while the cluster runs. An idle cluster keeps its driver and its minimum workers, so it is still billed all night.

  • BSet automatic termination after 60 minutes of inactivity

    Correct

    Automatic termination ends the compute once the time since the last command run on it exceeds the inactivity period, so an idle cluster stops on its own.

  • CUse spot instances for the cluster's worker nodes

    Incorrect

    Spot instances can lower the price of the workers, and the driver stays on-demand, but the cluster keeps running and billing while idle.

  • DChoose a smaller instance type for the driver node

    Incorrect

    A smaller driver lowers the hourly rate while the cluster runs; it does not stop an idle cluster from running.

The automatic termination setting takes an inactivity period in minutes and terminates the compute once no command has run on it for longer than that. Autoscaling, spot workers and node sizes change what a running cluster costs per hour, not whether an idle one keeps running.

Question 3 · choose 1

About 200 analysts run dashboards that spike at 09:00, drop off during the day and sit idle overnight. The team wants dashboards to start fast, scale up quickly under load and cost little when idle. Which compute fits best?

  1. AA pro SQL warehouse
  2. BA classic all-purpose cluster with autoscaling
  3. CA serverless SQL warehouse
  4. DA classic SQL warehouse
Show the answer and why
  • AA pro SQL warehouse

    Incorrect

    A pro warehouse supports Photon and Predictive IO but takes several minutes to start and scales less responsively, without Intelligent Workload Management.

  • BA classic all-purpose cluster with autoscaling

    Incorrect

    All-purpose clusters suit interactive notebooks; for SQL BI, SQL warehouses are the optimized option.

  • CA serverless SQL warehouse

    Correct

    Serverless warehouses typically start in 2 to 6 seconds, scale up rapidly with Intelligent Workload Management and scale down quickly when demand falls.

  • DA classic SQL warehouse

    Incorrect

    A classic warehouse gives entry-level performance and starts in minutes, without Predictive IO or Intelligent Workload Management.

For BI with demand that varies a lot over time, serverless SQL warehouses give the fastest start-up and the most responsive scaling. Pro and classic warehouses remain for cases serverless cannot serve.

Question 4 · choose 1

A set of classic jobs clusters must stay on classic compute because of custom cluster settings, but each run loses several minutes waiting for cloud instances. How can the team cut cluster start time?

  1. AAttach the clusters to an instance pool with idle instances
  2. BIncrease the number of workers in each cluster
  3. CEnable autoscaling with a higher maximum worker count on each cluster
  4. DEnable Photon on each cluster
Show the answer and why
  • AAttach the clusters to an instance pool with idle instances

    Correct

    Clusters built from a pool's idle instances start and scale faster, and Databricks charges no DBUs for idle pool instances, only the cloud provider's cost.

  • BIncrease the number of workers in each cluster

    Incorrect

    More workers means more instances to acquire at start-up; it adds capacity, not speed of launch.

  • CEnable autoscaling with a higher maximum worker count on each cluster

    Incorrect

    Autoscaling adds and removes workers while the cluster runs; it does not shorten the initial launch.

  • DEnable Photon on each cluster

    Incorrect

    Photon speeds up query execution; it has no effect on how long instances take to start.

Pools hold ready instances so classic clusters skip most of the instance acquisition wait. Where a workload can run on serverless, Databricks recommends serverless instead of pools.

Question 5 · choose 1

Eight data engineers want to share one classic all-purpose cluster for Python and SQL notebooks, with each person's own Unity Catalog permissions enforced and their code isolated from each other. Which access mode should the cluster use?

  1. ADedicated access mode assigned to the team lead
  2. BNo isolation shared access mode
  3. CDedicated access mode assigned to a group of the eight
  4. DStandard access mode
Show the answer and why
  • ADedicated access mode assigned to the team lead

    Incorrect

    Dedicated compute assigned to one user can be used only by that user, so the others cannot attach.

  • BNo isolation shared access mode

    Incorrect

    No isolation shared is a legacy configuration that Databricks does not recommend; it does not provide the user isolation needed.

  • CDedicated access mode assigned to a group of the eight

    Incorrect

    On compute assigned to a group, members run with the group's permissions instead of their own, so each person's grants are not enforced.

  • DStandard access mode

    Correct

    Standard compute lets many users attach and run workloads concurrently, with Lakeguard isolating user code and enforcing each user's permissions.

Standard access mode is the recommended default for shared compute, with Lakeguard isolating each user's code. Dedicated access mode serves a single user or a single group.

Question 6 · choose 1

Photon instance types consume DBUs at a different rate, so the team wants Photon only where it pays off. Which job is most likely to benefit?

  1. AA job that runs many small queries that each finish in under two seconds
  2. BA nightly job joining and aggregating billions of rows in large Delta tables
  3. CA Python job that loops over REST API calls and processes results on the driver
  4. DA Structured Streaming job with a stateful windowed aggregation
Show the answer and why
  • AA job that runs many small queries that each finish in under two seconds

    Incorrect

    For queries that finish in under two seconds, planning and scheduling dominate, so Photon shows no meaningful gain.

  • BA nightly job joining and aggregating billions of rows in large Delta tables

    Correct

    Photon gives the biggest benefit on long-running queries over large data, with faster scans, hash joins, aggregations and writes.

  • CA Python job that loops over REST API calls and processes results on the driver

    Incorrect

    Plain Python code on the driver does not run through the Spark engine, so Photon cannot accelerate it.

  • DA Structured Streaming job with a stateful windowed aggregation

    Incorrect

    Photon supports stateless streaming; stateful streaming is not supported.

Photon is a vectorized engine for SQL and DataFrame work. It pays off on large scans, joins, aggregations and writes, not on tiny queries or non-Spark code.

Practise domain 1 →Practise all domains →