Task 3.2: Combine DataFrames with operations such as Inner join, left join, broadcast join, multiple keys, cross join, union, and union all.

• Aorders.join(customers, 'customer_id', 'inner')

  Incorrect

  An inner join keeps only orders whose customer_id has a match, so orders without a customer disappear.

• Borders.join(customers, 'customer_id', 'left_anti')

  Incorrect

  A left anti join filters orders instead of looking customers up: it adds no customer columns, and some orders are left out.

• Corders.join(customers, 'customer_id', 'left')

  Correct

  A left join keeps every row of the left DataFrame, orders, and fills customer columns with nulls where there is no match.

• Dorders.join(customers, 'customer_id', 'left_semi')

  Incorrect

  A left semi join is also a filter on orders: the report would get no customer details, and orders without a customer would be missing.

The join type decides which unmatched rows survive. left (left_outer) keeps all left rows and fills nulls; inner keeps matches only; semi and anti joins filter the left side without adding the right side's columns.

• Adf_a.unionByName(df_b)

  Correct

  unionByName resolves columns by name rather than position, so city and name line up correctly.

• Bdf_a.union(df_b)

  Incorrect

  union resolves columns by position, so df_b's city values would land in name and its name values in city.

• Cdf_a.join(df_b, 'id', 'outer')

  Incorrect

  A join matches rows on id and places the columns side by side; it does not stack rows.

• Ddf_a.union(df_b.select('id', 'name', 'city'))

  Correct

  Selecting df_b's columns in df_a's order first makes the positional union line up, so each value lands under its own column.

• Edf_a.crossJoin(df_b)

  Incorrect

  A cross join places the columns of the two DataFrames side by side instead of stacking their rows.

DataFrame union matches columns by position, while unionByName matches them by name. Either align the columns yourself before a union or let unionByName do it.

• Asales.repartition('store_id').join(stores.repartition('store_id'), 'store_id')

  Incorrect

  Repartitioning both sides by the key is still a shuffle of the 2 TB table; it does not broadcast anything.

• Bsales.crossJoin(stores).filter(sales.store_id == stores.store_id)

  Incorrect

  This asks for every pairing of rows and then filters them; it does not request a broadcast of stores.

• Csales.join(stores, 'store_id', 'left_semi')

  Incorrect

  A left semi join returns only the sales columns, so the store attributes needed from the dimension are lost.

• Dsales.join(broadcast(stores), 'store_id')

  Correct

  The broadcast function marks the small DataFrame for a broadcast join, so the large side is not shuffled.

Broadcast joins copy a small table to every executor so the large table can be joined in place. Spark broadcasts automatically below a size threshold; a hint or broadcast() requests it explicitly.

• Aorder_lines.join(shipments, 'order_id')

  Incorrect

  Joining on order_id alone matches every line of an order with every shipment of that order, multiplying rows.

• Border_lines.join(shipments, ['order_id', 'line_no'])

  Correct

  A list of column names performs an equi-join on all of them, so rows match only when both columns are equal.

• Corder_lines.join(shipments, (order_lines.order_id == shipments.order_id) | (order_lines.line_no == shipments.line_no))

  Incorrect

  The | operator is OR, so rows match when either column is equal, not both.

• Dorder_lines.join(shipments, 'order_id').join(shipments, 'line_no')

  Incorrect

  This joins shipments twice on single keys and still multiplies rows instead of matching on the pair.

For a composite key, pass a list of column names or combine equality conditions with &.

• Astores.union(dates)

  Incorrect

  union stacks the rows of two DataFrames one after the other; it does not pair them.

• Bstores.join(dates, 'store_id', 'left')

  Incorrect

  dates has no store_id column, and a left join keeps stores, not every pairing.

• Cstores.crossJoin(dates)

  Correct

  A cross join returns the Cartesian product, every row of stores paired with every row of dates.

• Dstores.unionByName(dates, allowMissingColumns=True)

  Incorrect

  This stacks the rows of both DataFrames and fills missing columns with nulls; it does not combine them.

Use a cross join when you need every combination of two sets. On large inputs it explodes in size, so keep it for small dimension sets like calendars.

• A3 rows: 1, 2 and 3

  Correct

  UNION defaults to DISTINCT, so duplicate rows are removed across the combined result.

• B5 rows: 1, 2, 2, 2 and 3

  Incorrect

  This is the result of UNION ALL, which keeps duplicates.

• C4 rows: 1, 2, 2 and 3

  Incorrect

  UNION does not only remove matches between the two sides; it removes duplicates within the combined result too.

• D1 row: 2

  Incorrect

  That is the result of INTERSECT, which returns rows present in both inputs.

In SQL, UNION means UNION DISTINCT unless ALL is given. INTERSECT and EXCEPT also default to DISTINCT.