Data Engineer Professional · Domain 6: Ensuring Data Security and Compliance · 8% of the exam
Task 6.5: Implement a data purging strategy that satisfies data retention and deletion policies (e.g., GDPR right-to-erasure, GDPR right-to-be-forgotten deletion) using Delta Lake and Unity Catalog features.
Deletes and VACUUM, deletion vectors and REORG, retention properties, deletes in pipelines and streaming sources, and change feed and clone copies.
Try each one before opening the answer. Every option is explained, with the Databricks documentation page that proves it.
Question 1 · choose 1
A managed table without deletion vectors is maintained by predictive optimization and has delta.deletedFileRetentionDuration set to 60 days for long time travel. A new policy requires that erased customers' data be physically gone within 30 days of the DELETE. Rows are deleted daily from a requests table. What must change?
ALower delta.deletedFileRetentionDuration to well under 30 days
BRun REORG TABLE ... APPLY (PURGE) after each daily delete
CRun OPTIMIZE after each daily delete, keeping delta.deletedFileRetentionDuration at 60 days
DRaise delta.logRetentionDuration to 90 days for auditing
Show the answer and why
ALower delta.deletedFileRetentionDuration to well under 30 days
Correct
VACUUM keeps unreferenced files for the retention period, so a 60-day setting keeps deleted rows in storage too long.
BRun REORG TABLE ... APPLY (PURGE) after each daily delete
Incorrect
REORG PURGE rewrites files with soft-deleted rows; without deletion vectors, old files still wait for VACUUM.
CRun OPTIMIZE after each daily delete, keeping delta.deletedFileRetentionDuration at 60 days
Incorrect
OPTIMIZE compacts files but leaves the old versions in storage until VACUUM.
DRaise delta.logRetentionDuration to 90 days for auditing
Incorrect
Keeping more history does nothing to remove deleted rows' files.
A DELETE removes rows from the current version, but older files still hold them until VACUUM removes files older than the retention threshold. To meet an erasure deadline, the deleted-file retention must be shorter than the deadline.
Each day, hundreds of erasure requests are added to the table gdpr_requests(user_id). A job must remove these users from a large bronze table. Which approach does Databricks recommend?
AOne DELETE statement per request, run in a loop over the table
BOverwrite the bronze table without those users once each day
CMERGE INTO bronze USING gdpr_requests ON user_id WHEN MATCHED THEN DELETE
DFilter the users out in every downstream query and dashboard
Show the answer and why
AOne DELETE statement per request, run in a loop over the table
Incorrect
Many separate deletes are slower than one MERGE.
BOverwrite the bronze table without those users once each day
Incorrect
Rewriting the whole table is far more expensive.
CMERGE INTO bronze USING gdpr_requests ON user_id WHEN MATCHED THEN DELETE
Correct
For many records at once, Databricks recommends MERGE against a control table of requests.
DFilter the users out in every downstream query and dashboard
Incorrect
Filtering does not delete the data.
Drive deletions from a scheduled job that reads a table of requests, starting in bronze. For many records, use MERGE with WHEN MATCHED THEN DELETE, then propagate the deletions downstream.
An e-commerce team must honor erasure requests but still count erased users' clicks in its gold aggregates. Today the email address is the key that links users to their clicks. Which design does Databricks describe for this?
AKeep email as the key and replace it with its hash on each erasure
BLink users by a non-natural user_id, so PII can be deleted on its own
CDelete the erased users' clicks too, then rebuild the gold aggregates
DMove erased users' emails to a restricted table behind a column mask
Show the answer and why
AKeep email as the key and replace it with its hash on each erasure
Incorrect
A hash of the email still links the clicks to the person, so the PII is not separated from the activity data.
BLink users by a non-natural user_id, so PII can be deleted on its own
Correct
With a non-natural key such as user_id instead of a natural key like email, PII can be deleted while non-PII data stays in place.
CDelete the erased users' clicks too, then rebuild the gold aggregates
Incorrect
This loses the activity counts that the team wants to keep.
DMove erased users' emails to a restricted table behind a column mask
Incorrect
A mask hides the values from readers but keeps them stored, so they are not erased.
Depending on the lakehouse design, the link between PII and non-PII user data can be severed. With a non-natural key such as user_id instead of a natural key like email, the PII can be deleted while the non-PII data, such as click activity counted in downstream aggregations, stays in place.
A team proposes to answer erasure requests by replacing names and emails with hashes instead of deleting rows. What does Databricks say about this choice?
AHashing is equivalent to deletion for GDPR erasure requests
BMasking views must be used instead of deleting the rows outright
CEncryption with a shared team key counts as erasure under GDPR
DComplete deletion is safer; obfuscated data can be re-identified
Show the answer and why
AHashing is equivalent to deletion for GDPR erasure requests
Incorrect
Obfuscated data can still carry re-identification risk.
BMasking views must be used instead of deleting the rows outright
Incorrect
Masking hides data from some users but does not erase it.
CEncryption with a shared team key counts as erasure under GDPR
Incorrect
Encrypted data can be decrypted with the key.
DComplete deletion is safer; obfuscated data can be re-identified
Correct
Eliminating re-identification risk often requires complete deletion of PII.
Teams must choose between deleting and obfuscating data. Databricks notes that complete erasure is the safest option, because removing the risk of re-identification often requires deleting the PII.