Data Engineer Associate · Domain 2: Data Ingestion and Loading · 21% of the exam
Task 2.2: Use the COPY INTO command to incrementally load files from cloud object storage (ADLS/S3/GCS) into Unity Catalog–governed tables.
COPY INTO syntax, its idempotent file tracking, FILES and PATTERN, FORMAT_OPTIONS and COPY_OPTIONS such as mergeSchema and force, VALIDATE, and the privileges on volumes and external locations it needs.
Study it
COPY INTO: idempotent loads, options, validation and privileges
Lesson coming
Sample questions
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 2
A SQL user runs the same COPY INTO statement each day to load JSON files from an S3 prefix, covered by a Unity Catalog external location, into the table raw.events. Which TWO statements about how the statement treats the files under the prefix are true? (Choose TWO.)
AFiles loaded on an earlier day are skipped when it runs again
BA file rewritten in place with corrected rows is loaded again
CWith COPY_OPTIONS ('force' = 'true'), it loads files already loaded
DWith VALIDATE ALL, it writes the rows that parse and reports the rest
EFiles in another format under the prefix are read in their own format
Show the answer and why
AFiles loaded on an earlier day are skipped when it runs again
Correct
COPY INTO is retryable and idempotent: files in the source location that have already been loaded are skipped on later runs, so each daily run loads only the new files.
BA file rewritten in place with corrected rows is loaded again
Incorrect
A file that was already loaded is skipped even if it has been modified since it was loaded, so the corrected rows are not picked up by a normal run.
CWith COPY_OPTIONS ('force' = 'true'), it loads files already loaded
Correct
The force copy option disables idempotency, so files are loaded whether or not they were loaded before.
DWith VALIDATE ALL, it writes the rows that parse and reports the rest
Incorrect
VALIDATE checks whether the data parses, matches the schema and meets the constraints, but the validated data is not written to the table.
EFiles in another format under the prefix are read in their own format
Incorrect
Files in the source location must have the format named in FILEFORMAT; COPY INTO does not switch readers file by file.
COPY INTO remembers the files it has loaded from the source location and skips them on later runs, even when a file has changed since. To load files again, the force copy option turns that idempotency off. VALIDATE checks data without writing it, and every file is read with the format that FILEFORMAT names.
A user runs COPY INTO main.bronze.events FROM '/Volumes/main/landing/raw/events' FILEFORMAT = JSON. The user has USE CATALOG on main, USE SCHEMA on main.bronze and main.landing, and MODIFY on the target table, but the command fails because the user cannot read the source files. Which privilege is missing?
AREAD FILES on the volume main.landing.raw
BWRITE VOLUME on the volume main.landing.raw
CREAD VOLUME on the volume main.landing.raw
DCREATE EXTERNAL TABLE on the external location behind the volume
Show the answer and why
AREAD FILES on the volume main.landing.raw
Incorrect
READ FILES is the privilege for paths in an external location; a Unity Catalog volume uses volume privileges.
BWRITE VOLUME on the volume main.landing.raw
Incorrect
WRITE VOLUME allows adding or changing files in the volume; COPY INTO only needs to read them.
CREAD VOLUME on the volume main.landing.raw
Correct
To load from a Unity Catalog volume, COPY INTO requires READ VOLUME on the volume, which applies to all directories under it.
DCREATE EXTERNAL TABLE on the external location behind the volume
Incorrect
That privilege is for creating external tables in an external location; it does not grant reading a volume's files.
COPY INTO needs read access to its source: READ VOLUME for a volume, or READ FILES for an external location path, plus the usage privileges on the parents.
CSV files in /Volumes/main/landing/raw/orders start with a header row of column names and separate fields with semicolons, for example order_id;amount;country. COPY INTO main.bronze.orders FROM that path with FILEFORMAT = CSV must read the fields correctly and match them to the table's columns by the header names. Which clause completes the statement?
FORMAT_OPTIONS passes reader options to the CSV data source: header uses the first line as column names, and sep sets the field separator.
BCOPY_OPTIONS ('header' = 'true', 'sep' = ';')
Incorrect
COPY_OPTIONS controls the COPY INTO operation itself, with options such as force and mergeSchema; reader options for the file format belong in FORMAT_OPTIONS.
lineSep defines the line separator, so each semicolon would end a record while the fields would still be split on the default comma.
DFORMAT_OPTIONS ('header' = 'false', 'sep' = ';')
Incorrect
With header set to false, the first line is read as data rather than as column names, so the header row is loaded as a record.
COPY INTO hands the options in FORMAT_OPTIONS to the Apache Spark reader of the chosen format, so CSV options such as header and sep go there. The CSV separator defaults to a comma and header defaults to false, so a headered, semicolon-separated file needs both set.
A data engineer wants COPY INTO to create the schema of a new table from JSON files, without writing the column list by hand. What should the engineer run?
ACOPY INTO main.bronze.reviews FROM the source path, letting COPY INTO create the table
BCREATE TABLE main.bronze.reviews listing each column as STRING, then COPY INTO it
CCREATE TABLE IF NOT EXISTS main.bronze.reviews, then INSERT INTO it from the JSON files
DCREATE TABLE IF NOT EXISTS, then COPY INTO it with mergeSchema in both option clauses
Show the answer and why
ACOPY INTO main.bronze.reviews FROM the source path, letting COPY INTO create the table
Incorrect
COPY INTO needs an existing Delta table as its target; it does not create the table.
BCREATE TABLE main.bronze.reviews listing each column as STRING, then COPY INTO it
Incorrect
This means writing the column list by hand, which the engineer wants to avoid.
CCREATE TABLE IF NOT EXISTS main.bronze.reviews, then INSERT INTO it from the JSON files
Incorrect
INSERT INTO and MERGE INTO are not supported for writing to a schemaless placeholder table.
DCREATE TABLE IF NOT EXISTS, then COPY INTO it with mergeSchema in both option clauses
Correct
An empty placeholder Delta table gets its schema inferred during COPY INTO when mergeSchema is set in both options.
A schemaless placeholder table plus mergeSchema lets COPY INTO infer the schema on the first load. The table becomes queryable after COPY INTO writes data to it.
During a COPY INTO load from JSON files, the amount field must be stored as DECIMAL(10,2), and each row must record the path of the file it came from. Both must happen in the same statement. Which form of COPY INTO does this?
ACOPY INTO t FROM 's3://landing/orders' FILEFORMAT = JSON FORMAT_OPTIONS ('inferSchema' = 'true')
BCOPY INTO t FROM (SELECT CAST(amount AS DECIMAL(10,2)) AS amount, _metadata.file_path AS source_file FROM 's3://landing/orders') FILEFORMAT = JSON
CCOPY INTO t FROM 's3://landing/orders' FILEFORMAT = JSON COPY_OPTIONS ('mergeSchema' = 'true')
DCOPY INTO t FROM 's3://landing/orders' FILEFORMAT = JSON, followed by an UPDATE that casts amount
Show the answer and why
ACOPY INTO t FROM 's3://landing/orders' FILEFORMAT = JSON FORMAT_OPTIONS ('inferSchema' = 'true')
Incorrect
Schema inference picks types from the data; it neither forces DECIMAL(10,2) nor adds the file path.
BCOPY INTO t FROM (SELECT CAST(amount AS DECIMAL(10,2)) AS amount, _metadata.file_path AS source_file FROM 's3://landing/orders') FILEFORMAT = JSON
Correct
The SELECT form of the source lets you cast columns and add expressions such as the hidden _metadata.file_path column during the load.
CCOPY INTO t FROM 's3://landing/orders' FILEFORMAT = JSON COPY_OPTIONS ('mergeSchema' = 'true')
Incorrect
mergeSchema concerns the table's columns, not the values; it neither casts amount nor records the file path.
DCOPY INTO t FROM 's3://landing/orders' FILEFORMAT = JSON, followed by an UPDATE that casts amount
Incorrect
A second statement rewrites the data after loading, and the file path is no longer available to it.
COPY INTO accepts a SELECT over the source, so you can cast, rename and add columns, including file metadata from _metadata, as part of the load.