Skip to content

Associate Developer for Apache Spark · Domain 3: Developing Apache Spark DataFrame/DataSet API Applications · 30% of the exam

Task 3.5: Manage input and output operations by writing, overwriting, and reading DataFrames with schemas; Perform operations on DataFrames such as sorting, iterating, printing schema, and conversion between DataFrame and sequence/list formats

Explicit schemas against inference on read, write modes, sort and orderBy with null ordering, printSchema, and collect, take, toLocalIterator and toPandas.

Study it

  • Reading and writing with schemas

    Lesson coming

  • Sorting, printing the schema and collecting rows to Python

    Partly covered by: ORDER BY and LIMIT

Sample questions

Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.

Question 1 · choose 1

A folder /data/in/orders holds CSV files without a header line; each line has an integer id and a decimal amount. Spark uses default settings. Which read returns a DataFrame with the columns id INT and amount DOUBLE?

  1. Aspark.read.load("/data/in/orders")
  2. Bspark.read.format("csv").schema("id INT, amount DOUBLE").load("/data/in/orders")
  3. Cspark.read.schema("id INT, amount DOUBLE").load("/data/in/orders")
  4. Dspark.read.option("sep", ",").schema("id INT, amount DOUBLE").load("/data/in/orders")
Show the answer and why
  • Aspark.read.load("/data/in/orders")

    Incorrect

    Without a format, load uses the default data source, which is parquet unless configured otherwise, so the CSV files are not parsed as CSV.

  • Bspark.read.format("csv").schema("id INT, amount DOUBLE").load("/data/in/orders")

    Correct

    format names the CSV source, and schema gives the column names and types as a DDL string.

  • Cspark.read.schema("id INT, amount DOUBLE").load("/data/in/orders")

    Incorrect

    A schema does not choose the source: without a format, load still uses the default parquet source.

  • Dspark.read.option("sep", ",").schema("id INT, amount DOUBLE").load("/data/in/orders")

    Incorrect

    Setting a CSV option does not choose the source either; load still reads with the default parquet source.

In the simplest form, load and save use the default data source, parquet unless spark.sql.sources.default says otherwise, so reading CSV needs format("csv") or the csv() method. DataFrameReader.schema takes a StructType or a DDL-formatted string such as "id INT, amount DOUBLE", which sets the names and types of the columns read.

Question 2 · choose 1

A DataFrame df has the columns dept, salary and name, and pyspark.sql.functions is imported as F. A report must list the rows by dept in ascending order and, within each dept, by salary from the highest to the lowest. Which call does that?

  1. Adf.orderBy(["dept", "salary"], ascending=False)
  2. Bdf.orderBy(0, -1)
  3. Cdf.orderBy(F.col("dept").asc(), F.col("salary").asc())
  4. Ddf.orderBy(["dept", "salary"], ascending=[True, False])
Show the answer and why
  • Adf.orderBy(["dept", "salary"], ascending=False)

    Incorrect

    A single False applies to every column, so dept is sorted in descending order too.

  • Bdf.orderBy(0, -1)

    Incorrect

    Column ordinals start from 1, so 0 does not name a column.

  • Cdf.orderBy(F.col("dept").asc(), F.col("salary").asc())

    Incorrect

    salary is sorted in ascending order, from the lowest to the highest.

  • Ddf.orderBy(["dept", "salary"], ascending=[True, False])

    Correct

    A list for ascending gives one sort order per column, and its length must equal the number of columns.

DataFrame.orderBy takes columns, names or 1-based column ordinals; a negative ordinal sorts descending. The ascending parameter is a boolean or a list of booleans: a list gives one sort order per column and must be as long as the list of columns. ascending=[True, False] sorts dept upward and salary downward within each dept.

Question 3 · choose 1

A DataFrame whose id column is declared non-nullable is written with df.write.parquet(path) and read back with spark.read.parquet(path). What is the nullability of id in the schema of the DataFrame read back?

  1. Anullable = true, because Parquet columns are converted to nullable when read
  2. Bnullable = false, because Parquet preserves the schema exactly
  3. CThe read fails, because a non-nullable column cannot be stored in Parquet files at all
  4. Dnullable = false only if inferSchema is set on the read
Show the answer and why
  • Anullable = true, because Parquet columns are converted to nullable when read

    Correct

    When reading Parquet files, all columns are automatically converted to be nullable for compatibility reasons.

  • Bnullable = false, because Parquet preserves the schema exactly

    Incorrect

    Parquet preserves the schema of the data, but on read all columns are converted to nullable.

  • CThe read fails, because a non-nullable column cannot be stored in Parquet files at all

    Incorrect

    Writing works; nullability changes only on read.

  • Dnullable = false only if inferSchema is set on the read

    Incorrect

    inferSchema is a CSV option; Parquet reads convert every column to nullable.

Spark SQL reads and writes Parquet files and automatically preserves the schema of the original data. When Parquet files are read, all columns are automatically converted to be nullable for compatibility reasons, so id comes back as nullable.

Databricks documentation

Question 4 · choose 1

A developer needs the column names of a DataFrame df as a Python list, in the order of the DataFrame, to loop over them. Which expression returns that list?

  1. Adf.printSchema()
  2. Bdf.schema
  3. Cdf.columns
  4. Ddf.dtypes
Show the answer and why
  • Adf.printSchema()

    Incorrect

    printSchema prints the schema as a tree; it does not return the names.

  • Bdf.schema

    Incorrect

    schema returns a StructType of fields, not a plain list of names.

  • Cdf.columns

    Correct

    columns retrieves the names of all columns as a list, in the order of the DataFrame.

  • Ddf.dtypes

    Incorrect

    dtypes returns pairs of column name and data type, not names alone.

DataFrame.columns retrieves the names of all columns as a list, and the order of the names reflects their order in the DataFrame, for example ['age', 'name', 'state']. schema returns the StructType, dtypes returns (name, type) pairs and printSchema prints the schema tree.

Question 5 · choose 1

What does spark.range(1, 10, 3) return?

  1. AA DataFrame with a column value holding 1, 4 and 7
  2. BA DataFrame with a column id holding 1, 4, 7 and 10
  3. CA Python list [1, 4, 7]
  4. DA DataFrame with a column id holding 1, 4 and 7
Show the answer and why
  • AA DataFrame with a column value holding 1, 4 and 7

    Incorrect

    The single column created by range is named id.

  • BA DataFrame with a column id holding 1, 4, 7 and 10

    Incorrect

    The end value is exclusive, so 10 is not included.

  • CA Python list [1, 4, 7]

    Incorrect

    SparkSession.range returns a DataFrame, not a list.

  • DA DataFrame with a column id holding 1, 4 and 7

    Correct

    range creates a DataFrame with a single LongType column named id, from start to end (exclusive) with the given step.

SparkSession.range(start, end, step, numPartitions) creates a DataFrame with a single LongType column named id, containing elements in a range from start to end, exclusive, with step value step. range(1, 10, 3) therefore holds 1, 4 and 7.

Practice domain 3 →Practice all domains →