Data Engineer Professional · Domain 3: Data Manipulation · 12% of the exam
Task 3.2: Develop a model and query semi-structured data using the VARIANT data type and related functions (e.g., parse_json, variant_get).
parse_json and try_parse_json, path extraction and casting, variant_get and try_variant_get, variant_explode, schema_of_variant, and when to shred fields into columns.
Study it
VARIANT: parse_json, paths, variant_get and shredding
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A silver query extracts price from the VARIANT column payload as DECIMAL(10,2). Most events carry a number, but some partner feeds send the string "N/A", and the job currently fails on those rows. The query must keep running and leave price NULL for such rows. Which expression fits?
try_variant_get returns NULL when the path is missing or the value cannot be cast to the requested type.
Bvariant_get(payload, '$.price', 'decimal(10,2)')
Incorrect
variant_get raises an error when the cast fails, which is the current failure.
Cpayload:price::decimal(10,2)
Incorrect
The :: cast also raises an error for a value that cannot be cast.
Dtry_parse_json(payload:price) to reparse the extracted field
Incorrect
Reparsing the extracted value does not cast it to DECIMAL, so price would still not be a number.
variant_get and the :: operator raise errors when a value cannot be cast. try_variant_get takes the same path and type arguments and returns NULL instead.
Before promoting frequently used fields of a VARIANT column into typed columns, a data engineer wants one schema that merges the structure of all rows in the table, including fields that appear only in some rows. Which function returns it?
Aschema_of_variant_agg(payload) over the table
Bschema_of_variant(payload) on the first row
Cschema_of_json(to_json(payload)) on the first row
DDESCRIBE TABLE on the table
Show the answer and why
Aschema_of_variant_agg(payload) over the table
Correct
This aggregate merges the schemas of all VARIANT values by field name and returns one DDL schema.
Bschema_of_variant(payload) on the first row
Incorrect
This returns the schema of a single value, so fields missing from that row are left out.
Cschema_of_json(to_json(payload)) on the first row
Incorrect
This also infers from one value and misses fields of other rows.
DDESCRIBE TABLE on the table
Incorrect
DESCRIBE shows the column type VARIANT, not the structure inside it.
schema_of_variant_agg returns the combined schema of all VARIANT values in a group, merging fields by name and using the least common type when types differ. schema_of_variant works on a single value.
In a VARIANT column v, some documents contain "discount": null. The filter WHERE v:discount IS NULL misses them. Which predicate finds them?
Ais_variant_null(v:discount)
Bv:discount = 'null'
Cisnull(v:discount)
Dv:discount <=> NULL
Show the answer and why
Ais_variant_null(v:discount)
Correct
is_variant_null tests for a VARIANT-encoded null, which IS NULL does not treat as SQL NULL.
Bv:discount = 'null'
Incorrect
That compares with the string 'null', not a JSON null value.
Cisnull(v:discount)
Incorrect
isnull checks for SQL NULL, like IS NULL.
Dv:discount <=> NULL
Incorrect
Null-safe equality with NULL is true only for a SQL NULL, and a VARIANT-encoded null is not a SQL NULL.
A JSON null inside a VARIANT is a VARIANT-encoded null, not a SQL NULL. is_variant_null returns true for it, while IS NULL checks whether the expression itself is SQL NULL.
A bronze STRING column contains JSON documents, and a few of them are malformed. A silver query must convert the column to VARIANT without failing. Which function should it use?
Aparse_json(raw)
Bschema_of_json(raw)
Ctry_parse_json(raw)
Dto_json(raw)
Show the answer and why
Aparse_json(raw)
Incorrect
parse_json raises an error on malformed JSON.
Bschema_of_json(raw)
Incorrect
schema_of_json returns the schema of a JSON string in DDL format, not a VARIANT value.
Ctry_parse_json(raw)
Correct
try_parse_json returns NULL for strings that cannot be parsed.
Dto_json(raw)
Incorrect
to_json converts values to JSON strings, the opposite direction.
parse_json returns an error when the string is not valid JSON, while try_parse_json returns NULL.
A VARIANT column order holds an items array. A query needs one row per item, with the item as a VARIANT value. Which function produces the rows?
Aexplode(order:items)
Bvariant_explode(order:items)
Cschema_of_variant(order:items)
Dvariant_get(order, '$.items', 'string')
Show the answer and why
Aexplode(order:items)
Incorrect
explode works on ARRAY and MAP types, not on a VARIANT.
Bvariant_explode(order:items)
Correct
variant_explode returns pos, key and value columns, one row per array element or object field.
Cschema_of_variant(order:items)
Incorrect
That returns a schema string, not rows.
Dvariant_get(order, '$.items', 'string')
Incorrect
That extracts one value per row as a string.
variant_explode un-nests a VARIANT array or object into rows with pos, key and value. For arrays the key is NULL; variant_explode_outer keeps a NULL row when the input is not an array or object.