Task 3.3: Manipulate columns, rows, and table structures by adding, dropping, splitting, renaming column names, applying filters, and exploding arrays.

• Aorders.select('order_id', explode('items').alias('item'))

  Incorrect

  explode produces no rows for a NULL or empty array, so orders without items are dropped.

• Borders.select('order_id', split('items', ',').alias('item'))

  Incorrect

  split turns a string into an array; it does not create one row per element and does not accept an array column.

• Corders.select('order_id', posexplode('items').alias('pos', 'item'))

  Incorrect

  posexplode adds each element's position but, like explode, returns no rows for NULL or empty arrays.

• Dorders.select('order_id', explode_outer('items').alias('item'))

  Correct

  explode_outer returns a row with a NULL element when the array is NULL or empty, so every order is kept.

explode and posexplode drop rows whose array is NULL or empty. Their _outer variants keep such rows with a NULL value.

• Asplit('full_name', ' ').getItem(0)

  Correct

  split returns an array of the parts, and getItem(0) picks the first element into the new column.

• Bexplode(split('full_name', ' ')).alias('first_name')

  Incorrect

  explode turns the array into separate rows, so each name part lands on its own row.

• Csplit('full_name', ' ')

  Incorrect

  This stores the whole array, ['Ada', 'Lovelace'], in first_name instead of its first element.

• Dtrim('full_name')

  Incorrect

  trim only removes leading and trailing spaces; the full name stays together.

Splitting a string into columns is split plus getItem. explode is the tool when each part should become its own row.

• Adf.select(col('cust_id').alias('customer_id'))

  Incorrect

  select returns only the listed columns, so the other 39 columns are dropped.

• Bdf.withColumn('customer_id', col('cust_id'))

  Incorrect

  withColumn adds a copy under the new name and keeps cust_id as well.

• Cdf.withColumnRenamed('cust_id', 'customer_id')

  Correct

  withColumnRenamed renames one column and returns all the others unchanged.

• Ddf.toDF('customer_id')

  Incorrect

  toDF assigns new names to all columns by position and needs one name per column.

withColumnRenamed (or withColumnsRenamed for several) renames in place; alias works inside a select that lists the columns you keep.

• Adf.select('_rescued_data', 'ingest_ts')

  Incorrect

  select keeps the listed columns and removes the rest, the opposite of what is needed.

• Bdf.drop('_rescued_data', 'ingest_ts')

  Correct

  drop returns a DataFrame without the named columns, with every row kept.

• Cdf.dropna(subset=['_rescued_data', 'ingest_ts'])

  Incorrect

  dropna removes rows that have nulls in those columns; the columns themselves stay.

• Ddf.dropDuplicates(['_rescued_data', 'ingest_ts'])

  Incorrect

  dropDuplicates works on rows, not columns, so _rescued_data and ingest_ts would stay in the DataFrame.

drop removes columns. dropna and dropDuplicates remove rows.

• Adf.filter(col('status') == 'shipped' and col('amount') > 100)

  Incorrect

  Python's and keyword cannot combine Column conditions; PySpark uses the & operator for this.

• Bdf.filter(col('status') == 'shipped' & col('amount') > 100)

  Incorrect

  & binds tighter than == and >, so without parentheses the expression is grouped wrongly.

• Cdf.filter((col('status') == 'shipped') | (col('amount') > 100))

  Incorrect

  | is OR, so rows meeting either condition would be kept.

• Ddf.filter((col('status') == 'shipped') & (col('amount') > 100))

  Correct

  Each comparison is wrapped in parentheses and combined with &, which is how PySpark expresses AND.

In PySpark, combine Column conditions with &, | and ~, and put each comparison in parentheses. A SQL string such as "status = 'shipped' AND amount > 100" also works.