Skip to content
Cheat sheets

Data Engineer Professional · Domain 1: Developing Code for Data Processing using Python and SQL · 23% of the exam

Task 1.3: Develop User-Defined Functions (UDFs) using Pandas, Python, and SQL, including Unity Catalog functions for the given constraints.

SQL UDFs, Python and batch Python UDFs in Unity Catalog, pandas UDFs and their type hints, session-scoped UDFs, and choosing a built-in function first.

Study it

  • SQL, Python, batch and pandas UDFs, and Unity Catalog functions

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A data engineer must turn a Python function that normalizes product codes into one governed definition in Unity Catalog. Dashboards on a serverless SQL warehouse will call it, and so will a nightly job on a classic job cluster in standard access mode that must stay on Databricks Runtime 15.4 LTS this quarter. Which kind of function works in both places?

  1. AA Unity Catalog Python UDF created with CREATE FUNCTION and LANGUAGE PYTHON
  2. BA batch Unity Catalog Python UDF declared with PARAMETER STYLE PANDAS
  3. CA pandas UDF that each job registers in its own code with spark.udf.register
  4. DA Unity Catalog Python UDTF that returns one row for each input value
Show the answer and why

  • AA Unity Catalog Python UDF created with CREATE FUNCTION and LANGUAGE PYTHON

    Correct

    Unity Catalog Python UDFs run on pro and serverless SQL warehouses and on standard access compute from Databricks Runtime 13.3 LTS, so 15.4 LTS qualifies.

  • BA batch Unity Catalog Python UDF declared with PARAMETER STYLE PANDAS

    Incorrect

    On classic compute, batch Unity Catalog Python UDFs require Databricks Runtime 16.3 or above, so the 15.4 LTS job cluster cannot call it.

  • CA pandas UDF that each job registers in its own code with spark.udf.register

    Incorrect

    A session-scoped UDF exists only in its SparkSession; it is not governed by Unity Catalog or callable from the SQL warehouse.

  • DA Unity Catalog Python UDTF that returns one row for each input value

    Incorrect

    A UDTF returns a table rather than one value per row, and on classic compute Unity Catalog Python UDTFs need Databricks Runtime 17.1 or above.

Unity Catalog governed UDFs are persisted and permissioned, and each type lists the compute it runs on. The Unity Catalog Python UDF runs on pro and serverless SQL warehouses and on standard access compute from Databricks Runtime 13.3 LTS. On classic compute, the batch variant needs Databricks Runtime 16.3 or above and Python UDTFs need 17.1 or above.

Databricks documentation

Task 1.3Report a problem with this question

Question 2 · choose 1

A nightly ETL job over billions of rows needs to trim, lowercase and concatenate two string columns. A teammate proposes a Python UDF. What does Databricks recommend for this job?

  1. AUse a Python UDF registered in Unity Catalog for governance
  2. BUse built-in functions such as trim, lower and concat
  3. CUse a session-scoped Python UDF applied to each column
  4. DUse a pandas UDF with an Iterator of Series for batching
Show the answer and why

  • AUse a Python UDF registered in Unity Catalog for governance

    Incorrect

    Governance does not remove the overhead of a UDF for logic that built-ins already express.

  • BUse built-in functions such as trim, lower and concat

    Correct

    Built-in functions are optimized for distributed processing and are recommended for large datasets and regular ETL.

  • CUse a session-scoped Python UDF applied to each column

    Incorrect

    Row-by-row Python adds overhead that built-ins avoid.

  • DUse a pandas UDF with an Iterator of Series for batching

    Incorrect

    Any UDF adds overhead that built-in functions avoid when they can express the logic.

Use UDFs for logic that is hard to express with built-in functions. Built-in Apache Spark functions are optimized and recommended for large datasets and for workloads that run regularly, such as ETL.

Databricks documentation

Task 1.3Report a problem with this question

Question 3 · choose 1

A data engineer needs a custom aggregation, written in Python, that returns one value per group in a groupBy().agg() call. Which kind of function do they write?

  1. AA Unity Catalog Python scalar UDF called inside agg()
  2. BA pandas UDF that maps a Series to a Series
  3. CA pandas UDF from a pandas Series to a scalar
  4. DA Python UDTF called once for each group
Show the answer and why

  • AA Unity Catalog Python scalar UDF called inside agg()

    Incorrect

    A scalar UDF returns one value per row, not one per group.

  • BA pandas UDF that maps a Series to a Series

    Incorrect

    Series to Series keeps one output per input row.

  • CA pandas UDF from a pandas Series to a scalar

    Correct

    A Series to scalar pandas UDF works as an aggregate in groupBy().agg(); UDAFs are session-scoped.

  • DA Python UDTF called once for each group

    Incorrect

    A UDTF returns rows for each input row; it does not aggregate.

UDAFs operate on multiple rows and return a single aggregated result, and they are session-scoped only. In Python, a pandas UDF from Series to scalar can be used in groupBy().agg().

Databricks documentation

Task 1.3Report a problem with this question

Question 4 · choose 1

Analysts get a permission error when they call main.util.normalize_code() in a query. They already have USE CATALOG on main and USE SCHEMA on main.util. What else do they need?

  1. AEXECUTE on the function main.util.normalize_code
  2. BSELECT on the function main.util.normalize_code
  3. CMODIFY on the schema main.util, which holds the function
  4. DCREATE FUNCTION on the schema main.util
Show the answer and why

  • AEXECUTE on the function main.util.normalize_code

    Correct

    Running a Unity Catalog UDF requires EXECUTE on the function, along with usage on its catalog and schema.

  • BSELECT on the function main.util.normalize_code

    Incorrect

    SELECT applies to tables and views; functions need EXECUTE.

  • CMODIFY on the schema main.util, which holds the function

    Incorrect

    MODIFY is for changing table data, not for calling functions.

  • DCREATE FUNCTION on the schema main.util

    Incorrect

    That lets users create functions; it is not needed to call one.

To run a Unity Catalog UDF, a user needs EXECUTE on the function and usage privileges on its schema and catalog. Creating one needs CREATE FUNCTION on the schema.

Databricks documentation

Task 1.3Report a problem with this question

Question 5 · choose 1

A session-scoped Python UDF computes a numeric score from two columns one row at a time and is slow on a large DataFrame. The logic uses NumPy and cannot be written with built-in functions. Which change usually speeds it up the most?

  1. ARegister the same row-at-a-time UDF in Unity Catalog
  2. BCall the UDF twice, once per input column, and add the results
  3. CCache the input DataFrame in memory before calling the UDF
  4. DRewrite it as a pandas UDF that takes and returns Series
Show the answer and why

  • ARegister the same row-at-a-time UDF in Unity Catalog

    Incorrect

    Registration adds governance; the function still runs per row.

  • BCall the UDF twice, once per input column, and add the results

    Incorrect

    More calls add work instead of removing overhead.

  • CCache the input DataFrame in memory before calling the UDF

    Incorrect

    Caching helps repeated reads, not per-row Python execution.

  • DRewrite it as a pandas UDF that takes and returns Series

    Correct

    pandas UDFs process data in Arrow batches as vectors, avoiding per-row Python overhead.

pandas UDFs, also called vectorized UDFs, use Apache Arrow to move data and pandas to work on it, so they run much faster than row-at-a-time Python UDFs.

Databricks documentation

Task 1.3Report a problem with this question

Practice domain 1 →Practice all domains →