Task 2.3: Ingest and transform streaming data

• AAn eventstream with Filter and Group by operators on the streaming data

  Incorrect

  The no-code operators don't run the team's own Python libraries.

• BAn eventhouse update policy written in KQL

  Incorrect

  The team's logic lives in Python libraries, which an update policy can't run.

• CA Dataflow Gen2 scheduled every five minutes

  Incorrect

  Dataflows run batch Power Query refreshes, not continuous streams.

• DSpark structured streaming in a notebook or Spark job definition

  Correct

  Structured streaming runs code-first stateful processing with Python libraries and writes Delta.

Fabric offers several streaming engines. Eventstreams give no-code ingestion, transformation and routing; an eventhouse processes data with KQL; and Spark structured streaming suits code-first teams that need custom, stateful processing with their own libraries and Delta sinks.

• AIngest the catalog into a native KQL table with a daily pipeline

  Incorrect

  A native copy needs the ingestion pipeline and sync the team wants to avoid.

• BA OneLake shortcut to the catalog table in the KQL database

  Correct

  Shortcuts expose the lakehouse table to KQL without moving or copying the data.

• CTurn on OneLake availability for the eventhouse's KQL database

  Incorrect

  This setting doesn't give the KQL database access to the lakehouse's catalog table.

• DUse an eventstream to stream the catalog table into the eventhouse

  Incorrect

  An eventstream would still ingest a second copy of the catalog.

A OneLake shortcut in a KQL database references data in internal Fabric items or external storage without moving or copying it. Shortcut queries can be slower than queries over native tables, which matters less for occasional use.

• AIngest the Delta tables into native tables every five minutes with a pipeline

  Incorrect

  This is the ingestion pipeline and second copy you want to avoid.

• BAsk the other team to partition the shortcut's Delta tables by day

  Incorrect

  Partitioning someone else's files doesn't give KQL the caching and indexing that native data has.

• CTurn on query acceleration for the OneLake shortcut in the eventhouse

  Correct

  Acceleration caches and indexes shortcut data, giving performance comparable to ingested data.

• DRaise the dashboard's refresh interval to every 30 minutes

  Incorrect

  Refreshing less often hides the latency instead of fixing query speed.

Query acceleration lets you set a policy on external Delta tables, including OneLake shortcuts in an eventhouse, that caches and indexes their data. It removes the need to manage ingestion pipelines or duplicate copies while giving performance comparable to ingested data.

• ATumbling

  Correct

  Fixed, back-to-back intervals put every reading in exactly one window.

• BHopping

  Incorrect

  Not the window for this requirement: a reading could be counted in more than one result.

• CSliding

  Incorrect

  Not the window for this requirement: its results aren't tied to fixed clock intervals.

• DSession

  Incorrect

  Not the window for this requirement: its length follows the gaps between readings.

A tumbling window divides a stream into fixed-size, non-overlapping, contiguous time intervals, so each event belongs to exactly one window, which is what counts per clock-aligned five-minute interval need. The eventstream Group by operator supports tumbling windows alongside the other window patterns.

• AManage fields on the readings array

  Incorrect

  Manage fields doesn't turn the items of an array into separate rows.

• BExpand, applied to the readings array

  Correct

  Expand creates a new row for each value within an array.

• CUnion of the event with itself

  Incorrect

  Union works across streams; it doesn't turn array items into rows.

• DGroup by readings with a tumbling window

  Incorrect

  Group by aggregates events; it doesn't turn array items into rows.

Eventstream's no-code operators include Filter, Manage fields, Aggregate, Group by, Union, Join and Expand. Expand creates a new row for each value in an array, which flattens nested readings before they reach a destination such as a lakehouse.

• AThe data is already maintained as Delta tables by another team

  Incorrect

  Data another team already maintains as Delta tables doesn't call for native tables.

• BStorage of a second copy of the data must be avoided

  Incorrect

  Avoiding a second copy is no reason to ingest into native tables.

• CUpdate policies and materialized views must be built on the data

  Correct

  Accelerated or not, shortcuts behave like external tables: materialized views aren't supported, and update policies over them are only a preview capability.

• DThe data is queried only a few times a month

  Incorrect

  Infrequent queries don't need the performance of native tables.

• EEvents must be queryable within seconds of being produced

  Correct

  Native eventhouse tables take events in near real time, so they're queryable within seconds.

Native eventhouse tables give the best query performance, take streaming data in near real time and allow update policies and materialized views. OneLake shortcuts behave like external tables: queries can be slower, materialized views aren't supported, and update policies over accelerated shortcuts are only in preview, so a design that depends on them needs native tables.

• Atable('DeviceRegistry')

  Incorrect

  table() references tables in the database, not shortcut external tables.

• Bexternal_table('DeviceRegistry')

  Correct

  Shortcuts in a KQL database are queried as external tables.

• Cmaterialized_view('DeviceRegistry')

  Incorrect

  materialized_view() queries a materialized view.

• DDeviceRegistry | as external

  Incorrect

  as names a tabular result; it doesn't read external data.

Each OneLake shortcut in a KQL database appears under Shortcuts and is queried as an external table in a KQL queryset by using the external_table() function.

• AExpand

  Incorrect

  Expand doesn't rename fields or change their types.

• BJoin

  Incorrect

  Join doesn't rename fields or change their types.

• CFilter

  Incorrect

  Filter doesn't rename fields or change their types.

• DManage fields

  Correct

  Manage fields adds, removes, renames or changes the data type of fields.

The Manage fields operator adds, removes, changes the data type of, or renames fields that come from an input or another transformation.

• AComplete

  Incorrect

  Complete isn't the default; it must be set explicitly.

• BUpdate

  Incorrect

  Update must be set explicitly.

• CAppend

  Correct

  The default output mode is append, which suits stateless ingestion.

• DOverwrite

  Incorrect

  Overwrite is a batch save mode, not a streaming output mode.

Output modes tell Spark which rows to write after a trigger. The default is append, which writes only new rows and works well for stateless transformations and event ingestion into Delta tables.

• AoutputMode("update")

  Incorrect

  An output mode chooses which rows reach the sink after a trigger; it doesn't decide when microbatches start.

• Btrigger(processingTime="1 hour")

  Incorrect

  A one-hour interval waits between microbatches.

• Ctrigger(once=True) on the streaming query

  Incorrect

  The legacy once trigger runs a single microbatch and stops.

• Dtrigger(processingTime="0 seconds")

  Correct

  A zero interval runs the next microbatch as soon as possible.

A processing-time trigger runs microbatches continuously: "0 seconds" starts the next microbatch as soon as possible, and a longer interval sets a fixed cadence.