Task 3.1: Implement data cleaning by reading bronze tables with PySpark/SQL, cleaning nulls, standardizing data types, and writing to new silver tables.

• Adf.fillna('unknown', subset=['country', 'quantity'])

  Incorrect

  Columns in subset whose type does not match the value are ignored, so country is filled but quantity keeps its nulls.

• Bdf.dropna(subset=['country', 'quantity'])

  Incorrect

  dropna removes rows with nulls in those columns instead of filling them, so rows are lost.

• Cdf.na.fill({'country': 'unknown', 'quantity': 0})

  Correct

  A dictionary maps each column name to its own replacement value, so each column gets the right default and other columns are untouched.

• Ddf.na.fill(0).na.fill('unknown')

  Incorrect

  Without a subset, each call fills nulls in all columns of a matching type, so other numeric and string columns change too.

fillna and na.fill are aliases. A dictionary gives per-column replacement values; a single value applies to columns of a matching type, limited by subset if given.

• ACAST(signup_date AS DATE)

  Incorrect

  With ANSI mode on, an invalid cast raises a runtime error instead of returning NULL, which is the failure being seen.

• Bsignup_date::DATE

  Incorrect

  The :: operator is shorthand for cast, so it fails on 'n/a' in the same way.

• CCOALESCE(signup_date, '1900-01-01')

  Incorrect

  COALESCE replaces NULL strings only; 'n/a' is not NULL, so it stays and the cast still fails.

• Dtry_cast(signup_date AS DATE)

  Correct

  try_cast returns NULL when a value is not well formed for the target type instead of raising an error.

Serverless compute runs with ANSI mode on, so bad values make cast fail loudly. try_cast keeps the row and yields NULL, which you can count or quarantine later.

• Adf.dropna(how='any')

  Incorrect

  how='any' drops a row that has even one null, so many usable rows would be lost.

• Bdf.dropna(how='all')

  Incorrect

  how='all' drops a row only when every value is null, so rows with one or two values would stay.

• Cdf.dropna(thresh=3)

  Correct

  thresh drops rows that have fewer than the given number of non-null values, and it overrides how.

• Ddf.na.fill('')

  Incorrect

  Filling nulls keeps every row and changes values, instead of removing the unusable rows.

dropna removes rows; fillna replaces values. how chooses between any and all nulls, and thresh sets a minimum count of non-null values.

• Aupper(trim(col('country')))

  Correct

  trim removes leading and trailing spaces, and upper converts the result to upper case.

• Binitcap(col('country'))

  Incorrect

  initcap capitalizes the first letter of each word, giving 'Us', and keeps the spaces.

• Cregexp_replace(col('country'), ' ', '')

  Incorrect

  This removes the spaces but leaves the mixed case unchanged.

• Dlower(col('country'))

  Incorrect

  lower changes only the case, to the wrong case, and keeps the spaces.

Standardizing text usually combines functions: trim for whitespace, then upper or lower for case.

• Ato_date(col('order_day'))

  Incorrect

  Without a format, to_date follows the cast rules for DATE, which expect the year-month-day form, so these strings do not convert.

• Bcol('order_day').cast('date')

  Incorrect

  A plain cast also expects the standard year-month-day form and cannot read day/month/year strings.

• Cto_date(col('order_day'), 'MM/dd/yyyy')

  Incorrect

  This reads the first number as the month, so '19/10/2026' has no valid month and the dates come out wrong or fail.

• Dto_date(col('order_day'), 'dd/MM/yyyy')

  Correct

  Passing the datetime pattern that matches the data, day first, makes to_date parse each value into a DATE.

When strings are not in the standard date form, give to_date the matching datetime pattern. Use try_to_date instead when some values may not parse.

• ADROP TABLE main.silver.customers, then CREATE TABLE main.silver.customers AS SELECT ...

  Incorrect

  Between the drop and the create the table does not exist, concurrent readers can fail, and the dropped table's history is no longer linked to the name.

• BCREATE OR REPLACE TABLE main.silver.customers AS SELECT ... FROM main.bronze.customers

  Correct

  CREATE OR REPLACE replaces the contents in one transaction, keeps the table's identity and history, and lets concurrent readers continue.

• CINSERT INTO main.silver.customers SELECT ... FROM main.bronze.customers

  Incorrect

  INSERT INTO appends, so each nightly run adds another full copy of the data.

• DTRUNCATE TABLE main.silver.customers, then INSERT INTO it

  Incorrect

  These are two separate commits, so readers can see an empty table between them.

For a full rebuild of a table that others read, use CREATE OR REPLACE TABLE: it is atomic, keeps the table's history, and avoids the drop-and-recreate anti-pattern.