Associate Developer for Apache Spark · Domain 3: Developing Apache Spark DataFrame/DataSet API Applications · 30% of the exam
Task 3.5: Manage input and output operations by writing, overwriting, and reading DataFrames with schemas; Perform operations on DataFrames such as sorting, iterating, printing schema, and conversion between DataFrame and sequence/list formats
Explicit schemas against inference on read, write modes, sort and orderBy with null ordering, printSchema, and collect, take, toLocalIterator and toPandas.
Study it
Reading and writing with schemas
Lesson coming
Sorting, printing the schema and collecting rows to Python
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A folder /data/in/orders holds CSV files without a header line; each line has an integer id and a decimal amount. Spark uses default settings. Which read returns a DataFrame with the columns id INT and amount DOUBLE?
Setting a CSV option does not choose the source either; load still reads with the default parquet source.
In the simplest form, load and save use the default data source, parquet unless spark.sql.sources.default says otherwise, so reading CSV needs format("csv") or the csv() method. DataFrameReader.schema takes a StructType or a DDL-formatted string such as "id INT, amount DOUBLE", which sets the names and types of the columns read.
A DataFrame df has the columns dept, salary and name, and pyspark.sql.functions is imported as F. A report must list the rows by dept in ascending order and, within each dept, by salary from the highest to the lowest. Which call does that?
A list for ascending gives one sort order per column, and its length must equal the number of columns.
DataFrame.orderBy takes columns, names or 1-based column ordinals; a negative ordinal sorts descending. The ascending parameter is a boolean or a list of booleans: a list gives one sort order per column and must be as long as the list of columns. ascending=[True, False] sorts dept upward and salary downward within each dept.
A DataFrame whose id column is declared non-nullable is written with df.write.parquet(path) and read back with spark.read.parquet(path). What is the nullability of id in the schema of the DataFrame read back?
Anullable = true, because Parquet columns are converted to nullable when read
Bnullable = false, because Parquet preserves the schema exactly
CThe read fails, because a non-nullable column cannot be stored in Parquet files at all
Dnullable = false only if inferSchema is set on the read
Show the answer and why
Anullable = true, because Parquet columns are converted to nullable when read
Correct
When reading Parquet files, all columns are automatically converted to be nullable for compatibility reasons.
Bnullable = false, because Parquet preserves the schema exactly
Incorrect
Parquet preserves the schema of the data, but on read all columns are converted to nullable.
CThe read fails, because a non-nullable column cannot be stored in Parquet files at all
Incorrect
Writing works; nullability changes only on read.
Dnullable = false only if inferSchema is set on the read
Incorrect
inferSchema is a CSV option; Parquet reads convert every column to nullable.
Spark SQL reads and writes Parquet files and automatically preserves the schema of the original data. When Parquet files are read, all columns are automatically converted to be nullable for compatibility reasons, so id comes back as nullable.
A developer needs the column names of a DataFrame df as a Python list, in the order of the DataFrame, to loop over them. Which expression returns that list?
Adf.printSchema()
Bdf.schema
Cdf.columns
Ddf.dtypes
Show the answer and why
Adf.printSchema()
Incorrect
printSchema prints the schema as a tree; it does not return the names.
Bdf.schema
Incorrect
schema returns a StructType of fields, not a plain list of names.
Cdf.columns
Correct
columns retrieves the names of all columns as a list, in the order of the DataFrame.
Ddf.dtypes
Incorrect
dtypes returns pairs of column name and data type, not names alone.
DataFrame.columns retrieves the names of all columns as a list, and the order of the names reflects their order in the DataFrame, for example ['age', 'name', 'state']. schema returns the StructType, dtypes returns (name, type) pairs and printSchema prints the schema tree.
AA DataFrame with a column value holding 1, 4 and 7
BA DataFrame with a column id holding 1, 4, 7 and 10
CA Python list [1, 4, 7]
DA DataFrame with a column id holding 1, 4 and 7
Show the answer and why
AA DataFrame with a column value holding 1, 4 and 7
Incorrect
The single column created by range is named id.
BA DataFrame with a column id holding 1, 4, 7 and 10
Incorrect
The end value is exclusive, so 10 is not included.
CA Python list [1, 4, 7]
Incorrect
SparkSession.range returns a DataFrame, not a list.
DA DataFrame with a column id holding 1, 4 and 7
Correct
range creates a DataFrame with a single LongType column named id, from start to end (exclusive) with the given step.
SparkSession.range(start, end, step, numPartitions) creates a DataFrame with a single LongType column named id, containing elements in a range from start to end, exclusive, with step value step. range(1, 10, 3) therefore holds 1, 4 and 7.