Skip to content
Cheat sheets

Data Engineer Professional · Domain 1: Developing Code for Data Processing using Python and SQL · 23% of the exam

Task 1.11: Develop unit and integration tests using assertDataFrameEqual, assertSchemaEqual, DataFrame.transform, and testing frameworks to ensure code correctness.

Writing transformations as functions, chaining them with transform, the pyspark.testing helpers and their options, pytest with Databricks Connect, and integration tests in a bundle target.

Study it

  • Testing transformations: transform, assertDataFrameEqual, assertSchemaEqual and pytest

    Lesson coming

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A unit test compares the output of a transformation with an expected DataFrame. The transformation runs a shuffle, so row order changes from run to run, and an average column differs from the expected value in the sixth decimal place. The test must pass for these differences but fail on any real mismatch in values or schema. Which assertion fits?

  1. AassertDataFrameEqual(actual, expected, rtol=1e-3)
  2. BassertDataFrameEqual(actual, expected, checkRowOrder=True)
  3. CassertSchemaEqual(actual.schema, expected.schema)
  4. Dassert actual.collect() == expected.collect()
Show the answer and why

  • AassertDataFrameEqual(actual, expected, rtol=1e-3)

    Correct

    Row order is ignored by default (checkRowOrder=False), and rtol sets the relative tolerance for float comparisons; schema and other values are still checked.

  • BassertDataFrameEqual(actual, expected, checkRowOrder=True)

    Incorrect

    This makes row order matter, so a result reshuffled by the shuffle fails even when every value matches.

  • CassertSchemaEqual(actual.schema, expected.schema)

    Incorrect

    This compares only the schemas, so wrong values would pass.

  • Dassert actual.collect() == expected.collect()

    Incorrect

    Plain list equality depends on row order and on exact float values.

pyspark.testing.assertDataFrameEqual compares schema and rows. By default it ignores row order, and rtol and atol control approximate equality for float columns. assertSchemaEqual is for schema-only checks.

Databricks documentation

Task 1.11Report a problem with this question

Question 2 · choose 1

A transformation returns the right columns and values, but in a different column order than the expected DataFrame in the test. The test should compare columns by name. Which call fits?

  1. AassertDataFrameEqual(actual, expected, ignoreColumnName=True)
  2. BassertDataFrameEqual(actual, expected, ignoreColumnOrder=True)
  3. CassertDataFrameEqual(actual, expected, checkRowOrder=True)
  4. DassertDataFrameEqual(actual, expected, ignoreNullable=False)
Show the answer and why

  • AassertDataFrameEqual(actual, expected, ignoreColumnName=True)

    Incorrect

    That ignores names and compares columns by position, the opposite of what is needed.

  • BassertDataFrameEqual(actual, expected, ignoreColumnOrder=True)

    Correct

    With ignoreColumnOrder=True, each expected column is compared to the actual column with the same name.

  • CassertDataFrameEqual(actual, expected, checkRowOrder=True)

    Incorrect

    This affects row order, not column order.

  • DassertDataFrameEqual(actual, expected, ignoreNullable=False)

    Incorrect

    Nullability is not the problem here; the column order is.

By default assertDataFrameEqual compares columns in the order they appear. ignoreColumnOrder=True compares them by name instead; ignoreColumnName, ignoreNullable and checkRowOrder control other parts of the comparison.

Databricks documentation

Task 1.11Report a problem with this question

Question 3 · choose 1

A pipeline is written as small functions that each take and return a DataFrame, so each can be unit tested. Which code chains them in order?

  1. Adf.transform(clean).transform(enrich)
  2. Bdf.apply(clean).apply(enrich)
  3. Cdf.map(clean).map(enrich)
  4. Ddf.foreach(clean).foreach(enrich)
Show the answer and why

  • Adf.transform(clean).transform(enrich)

    Correct

    DataFrame.transform applies a function that takes and returns a DataFrame, which allows concise chaining.

  • Bdf.apply(clean).apply(enrich)

    Incorrect

    Spark DataFrames have no apply method for this.

  • Cdf.map(clean).map(enrich)

    Incorrect

    map is not a DataFrame method for chaining DataFrame functions.

  • Ddf.foreach(clean).foreach(enrich)

    Incorrect

    foreach runs a function on each row and returns nothing.

DataFrame.transform(func) returns func(df), so transformations written as DataFrame-in, DataFrame-out functions can be chained and tested one by one.

Databricks documentation

Task 1.11Report a problem with this question

Question 4 · choose 1

Developers want to run their pytest suite from a local IDE, with the Spark code in the tests executing on Databricks compute. What do they use?

  1. AThe Jobs API, submitting each test function as a one-time run
  2. BA Git folder, which runs the test suite on each commit
  3. CDatabricks Connect, so their SparkSession uses remote compute
  4. DA notebook that calls pytest.main() when run from the IDE
Show the answer and why

  • AThe Jobs API, submitting each test function as a one-time run

    Incorrect

    That runs code remotely but does not give pytest a SparkSession.

  • BA Git folder, which runs the test suite on each commit

    Incorrect

    Git folders sync code; they do not run tests.

  • CDatabricks Connect, so their SparkSession uses remote compute

    Correct

    Databricks Connect connects local Python code and IDEs to Databricks compute.

  • DA notebook that calls pytest.main() when run from the IDE

    Incorrect

    A notebook runs in the workspace, not from the local IDE.

Databricks Connect lets IDEs and local Python code run Spark code on Databricks compute, so tools like pytest can run locally.

Databricks documentation

Task 1.11Report a problem with this question

Question 5 · choose 1

Functions defined in Python notebooks are hard to unit test. Where does Databricks recommend keeping the functions and their unit tests?

  1. AIn the same notebook, in a cell below the code they test
  2. BIn a job that runs the notebooks and checks their output
  3. CIn SQL UDFs that wrap the Python functions in Unity Catalog
  4. DIn Python files outside the notebooks, imported by them
Show the answer and why

  • AIn the same notebook, in a cell below the code they test

    Incorrect

    That mixes tests with production code and is not the recommendation.

  • BIn a job that runs the notebooks and checks their output

    Incorrect

    That is an end-to-end run, not unit testing of functions.

  • CIn SQL UDFs that wrap the Python functions in Unity Catalog

    Incorrect

    SQL UDFs are the recommendation for SQL notebooks, not Python ones.

  • DIn Python files outside the notebooks, imported by them

    Correct

    For Python notebooks, Databricks recommends storing functions and their tests outside notebooks.

For Python and R notebooks, Databricks recommends storing functions and their unit tests outside of notebooks. For SQL notebooks, store functions as SQL UDFs; for Scala, use separate notebooks.

Databricks documentation

Task 1.11Report a problem with this question

Practice domain 1 →Practice all domains →