Task 2.3: Use Auto Loader with schema enforcement and schema evolution in batch modes (for example, directory listing or file notification) to land data into Unity Catalog–governed tables.

• ARun the stream as a job task with retries so it restarts on the new schema

  Correct

  In the default addNewColumns mode Auto Loader merges the new column into the schema location before failing; a restart resumes with the updated schema, so automatic restarts keep it running.

• BSet cloudFiles.schemaEvolutionMode to failOnNewColumns

  Incorrect

  failOnNewColumns stops the stream and does not update the schema, so it needs manual intervention on every new field.

• CDelete the checkpoint and schema location directories, then restart the stream

  Incorrect

  Removing the checkpoint loses the record of processed files, so files are read again and duplicated.

• DSet cloudFiles.schemaEvolutionMode to none

  Incorrect

  In none mode new columns are ignored and not rescued unless rescuedDataColumn is set, so the field would never become a column.

With addNewColumns, the default when no schema is given, the failure is the expected signal that the schema has grown. Databricks recommends running such streams in Lakeflow Jobs so they restart automatically.

• AEnable file events on the location and set cloudFiles.useManagedFileEvents

  Correct

  File notification mode with file events is more performant and scalable than listing, and Databricks recommends migrating directory listing streams to it.

• BSet cloudFiles.backfillInterval to one day on the existing stream

  Incorrect

  A backfill interval schedules periodic full listings to catch missed files; it adds listing work rather than replacing it.

• CIncrease cloudFiles.maxFilesPerTrigger on the existing stream

  Incorrect

  This caps how many files each micro-batch processes; it does not change how new files are discovered.

• DReplace the stream with a COPY INTO statement scheduled at five-minute intervals

  Incorrect

  COPY INTO also has to work out which files in the directory are new, so the discovery cost stays; the change is to stop listing, not to swap the loading command.

Directory listing is the default and needs only read access, but its cost grows with the listing. File notification mode with file events uses cloud change notifications through one queue per external location.

• AaddNewColumns

  Incorrect

  In this mode new fields are added to the table schema as columns, which the requirement rules out.

• Bnone

  Incorrect

  In none mode new columns are ignored, and their data is not kept unless a rescued data column is set separately.

• CfailOnNewColumns

  Incorrect

  In this mode the stream fails and does not restart until the schema or the offending file is fixed.

• Drescue

  Correct

  In rescue mode the schema never evolves and the stream does not fail; new columns are recorded in the rescued data column.

Schema evolution modes trade stream stability against schema changes. rescue keeps the schema fixed and puts unexpected data into _rescued_data as JSON.

• A.schema("event_ts TIMESTAMP, amount DOUBLE")

  Incorrect

  Providing a schema replaces inference; the other fields would not be part of it, and new columns would not be added by default.

• B.option("cloudFiles.schemaHints", "event_ts TIMESTAMP, amount DOUBLE")

  Correct

  Schema hints set the types you know on top of the inferred schema, leaving inference and evolution for the other fields.

• C.option("cloudFiles.partitionColumns", "event_ts,amount")

  Incorrect

  partitionColumns tells Auto Loader which key=value directory names to parse as columns; it does not set field types.

• D.option("spark.databricks.cloudFiles.schemaInference.sampleSize.numFiles", "5000")

  Incorrect

  This changes how many files are sampled for inference; a larger sample does not make JSON fields typed.

Use schema hints when you know the type of some columns but still want Auto Loader to infer and evolve the rest.

• AcloudFiles.useStrictGlobber set to true

  Incorrect

  This changes how glob patterns match file paths; it does not affect type inference.

• BcloudFiles.schemaLocation set to a new directory

  Incorrect

  The schema location stores the inferred schema over time; moving it does not change how types are inferred.

• CcloudFiles.inferColumnTypes set to true

  Correct

  For formats that do not encode types, such as JSON, CSV and XML, this makes Auto Loader infer column types from the sample instead of using strings.

• DcloudFiles.maxFilesPerTrigger set to 1000

  Incorrect

  This caps how many new files each micro-batch processes; it does not affect type inference.

Auto Loader infers JSON, CSV and XML columns as strings by default to avoid type mismatches during evolution. inferColumnTypes turns on type inference.

• A.writeStream.option("checkpointLocation", cp).trigger(availableNow=True).toTable("main.bronze.events")

  Correct

  availableNow processes all data available when the run starts and then stops, and it is the trigger Databricks recommends on serverless.

• B.writeStream.option("checkpointLocation", cp).trigger(processingTime="1 hour").toTable("main.bronze.events")

  Incorrect

  A processing-time trigger keeps the stream running between micro-batches, and serverless compute does not support it.

• C.writeStream.option("checkpointLocation", cp).trigger(continuous="1 second").toTable("main.bronze.events")

  Incorrect

  Continuous processing keeps running and is not supported on serverless compute.

• D.writeStream.option("checkpointLocation", cp).toTable("main.bronze.events") with no trigger

  Incorrect

  Without a trigger, Spark uses a processing-time trigger of 0 seconds, so the query keeps running; on serverless it is rejected.

Scheduled incremental batches use Auto Loader with availableNow: the checkpoint remembers what was processed, and each run drains the backlog and ends.