Task 2.1: Develop data ingestion pipelines to ingest a variety of data formats, including Delta Lake, Parquet, JSON, Iceberg, CSV, and Binary from diverse sources such as message buses (Kafka, Kinesis, Pub/Sub) and cloud storage.

• AUse SASL/PLAIN with a username and password read from a secret scope

  Incorrect

  That stores a long-lived password in a secret scope, which the policy forbids, and is not IAM authentication.

• BAttach an instance profile with MSK permissions to the job's compute

  Incorrect

  Serverless compute is managed by Databricks; the job cannot attach an instance profile, and this is not governed in Unity Catalog.

• CRead the topic with Auto Loader using a Unity Catalog external location

  Incorrect

  Auto Loader reads files from cloud storage, not Kafka topics.

• DPass a Unity Catalog service credential as databricks.serviceCredential

  Correct

  Kafka sources can authenticate with Unity Catalog service credentials, which are governed securables and hold no secret in the job.

Databricks supports several Kafka authentication methods, including Unity Catalog service credentials, SASL/SSL and cloud-specific options for MSK. The documented examples pass databricks.serviceCredential with the service credential's name in the Kafka options.

• AcloudFiles.format set to text with wholetext set to true

  Incorrect

  The text source decodes files as strings, which corrupts binary PDF content.

• BmultiLine set to true

  Incorrect

  multiLine is a JSON and CSV option for records that span lines.

• CcloudFiles.format set to binaryFile

  Correct

  The binary file source turns each file into one record with path, modificationTime, length and content columns.

• DpathGlobFilter set to *.pdf

  Correct

  pathGlobFilter loads only the files whose names match the glob, so the .txt notes are skipped.

• EcloudFiles.inferColumnTypes set to true

  Incorrect

  The binary file schema is fixed; there are no column types to infer.

Use the binaryFile format to load unstructured files such as PDFs or images as records with the raw content and file metadata. pathGlobFilter limits the files read to those that match a pattern while keeping partition discovery.

• ACast value to string and parse it with from_json

  Correct

  Kafka keys and values are always byte arrays; deserialize them explicitly, for example cast to string and parse the JSON.

• BSet the Kafka option valueFormat to json

  Incorrect

  The Kafka source has no such option; deserialization is done with DataFrame operations.

• CAdd the option inferSchema set to true on the reader

  Incorrect

  The Kafka source returns a fixed schema with binary key and value.

• DSwitch the reader to format json with the topic as the path

  Incorrect

  The JSON file source reads files, not Kafka topics.

The Kafka source returns key and value as binary, deserialized with the ByteArrayDeserializer. Use operations such as cast("string"), from_json or from_avro to turn them into usable columns.

• AdurationMs.triggerExecution in each query progress event

  Incorrect

  durationMs reports how long the stages of each micro-batch take, not how far the query is behind the topic.

• BThe avgOffsetsBehindLatest and maxOffsetsBehindLatest metrics

  Correct

  These source metrics report the average and maximum offset lag across the subscribed topic partitions, next to a minimum, minOffsetsBehindLatest.

• CinputRowsPerSecond and processedRowsPerSecond of every batch

  Incorrect

  These describe throughput, not how far behind the topic the query is.

• DThe numOutputRows value of the last batch

  Incorrect

  Output rows say what was written, not what remains unread.

To monitor lag behind Kafka, use the avgOffsetsBehindLatest, maxOffsetsBehindLatest and minOffsetsBehindLatest metrics in the streaming query progress.

• AThe connector is at-least-once, so the checkpoint must be deleted and rebuilt

  Incorrect

  The connector has exactly-once semantics; deleting the checkpoint would reprocess data.

• BDuplicates mean the subscription was created twice by the connector

  Incorrect

  The connector creates a subscription only if one with the ID does not exist.

• CSetting Trigger.AvailableNow removes the duplicates that Pub/Sub delivers

  Incorrect

  The trigger controls batching; it does not deduplicate messages.

• DPub/Sub itself can publish duplicates, so the code must handle them

  Correct

  The connector is exactly-once for rows it receives, but Pub/Sub itself might publish duplicates or deliver rows out of order.

The Pub/Sub connector has exactly-once processing semantics for rows from the subscriber, but Pub/Sub might publish duplicate rows or deliver them out of order, so pipelines must deduplicate and handle ordering themselves.

• AfailOnDataLoss set to false

  Incorrect

  That changes how the query reacts to missing data, not consumer registration.

• BstreamARN instead of streamName in the options

  Incorrect

  Both identify the stream; switching them mid-stream is not supported and does not deregister consumers.

• CrequireConsumerDeregistration set to true

  Correct

  By default, removing a Kinesis source does not deregister its EFO consumer; this option deregisters it when the query stops.

• DA new checkpoint location for the query

  Incorrect

  A fresh checkpoint restarts reading; it leaves the old EFO consumer registered.

Removing a Kinesis stream source does not deregister its enhanced fan-out consumer by default, and the consumer can keep incurring cloud costs. Set requireConsumerDeregistration to true to deregister it when the query stops.