Task 5.5: Maintaining awareness of failures and mitigating impact.

• AA read replica in another zone of the same region

  Incorrect

  A read replica must be promoted by someone to replace the primary, and replication to it is asynchronous.

• BHigh availability, with the standby in another zone

  Correct

  An HA instance writes synchronously to disks in two zones and fails over to the standby automatically, typically in about a minute.

• CA second instance kept in sync by a nightly import

  Incorrect

  A nightly import loses up to a day of data, and switching to the second instance is manual.

• DAutomated backups with point-in-time recovery

  Incorrect

  Restoring from backups is a manual recovery that takes far longer than a minute.

Cloud SQL high availability places a primary and a standby in two zones of the region, replicates every write synchronously to both, and switches to the standby automatically when the primary is unresponsive. Read replicas serve reads and need someone to promote them before they can take over.

• AEnable exactly-once delivery on the subscription before deploying

  Incorrect

  Exactly-once delivery prevents redelivery after an acknowledgment, which makes replaying wrongly acknowledged messages less likely, not more.

• BAttach a dead-letter topic to catch messages that the new code rejects

  Incorrect

  A dead-letter topic only receives messages that are not acknowledged; the risk here is messages that were wrongly acknowledged.

• CRaise the acknowledgment deadline to 600 seconds for the new version

  Incorrect

  A longer deadline gives slow processing more time; acknowledged messages are still gone.

• DCreate a snapshot of the subscription just before deploying

  Correct

  A snapshot keeps the subscription's unacknowledged messages at that moment and the messages published afterward, even if they are later acknowledged.

• EAfter rolling back, seek the subscription to the snapshot

  Correct

  Seeking to the snapshot marks those messages unacknowledged again, so the fixed subscriber processes them once more.

Seek replays messages on a subscription. Seeking to a timestamp needs retention of acknowledged messages, which is off by default, so for a planned deployment Google suggests creating a snapshot first and seeking back to it if the new code acknowledges messages it should not have. A snapshot lives at most seven days.

• AAsk Cloud Customer Care to restore it from the fail-safe copy

  Incorrect

  Fail-safe is for data past the time travel window; three hours ago is well within time travel, which needs no support case.

• BQuery the table FOR SYSTEM_TIME AS OF 13:55 and write the result back

  Correct

  Time travel reads a table as it was at any point in the window, seven days by default, so the earlier state can be selected and restored.

• CRun the reverse of the MERGE using the values in the audit logs

  Incorrect

  Audit logs record the statement, not the old values it replaced.

• DReload the table from the last nightly export in Cloud Storage

  Incorrect

  A nightly export loses the changes made between the export and 13:55.

BigQuery keeps changed and deleted data for the time travel window, seven days by default and configurable down to two. Within it, FOR SYSTEM_TIME AS OF reads the table at an earlier time, which can be copied over the damaged table; fail-safe keeps deleted data seven more days, reached only through Cloud Customer Care.

• AReload them from the source systems, since deletion is permanent

  Incorrect

  With default settings the objects are not gone yet; soft delete keeps them for seven days.

• BTurn on Object Versioning now and restore the noncurrent versions

  Incorrect

  Versioning only protects objects deleted after it is enabled.

• CRestore the soft-deleted objects within their retention window

  Correct

  Soft delete is on by default with seven-day retention, and soft-deleted objects can be restored as live objects during that window.

• DSeek the bucket's Pub/Sub notification subscription to three days ago

  Incorrect

  Notification messages describe object events; they do not contain the objects' data, so nothing can be restored from them.

Cloud Storage soft delete preserves deleted and overwritten objects for a retention period, seven days by default and up to 90, and they can be restored during that time. Soft-deleted data is billed as storage until the period ends.

• AMove the instance to a single zone to avoid cross-zone traffic

  Incorrect

  A single-zone instance loses everything when that zone fails.

• BDouble the number of shards in the instance

  Incorrect

  More shards spread the keyspace further, but each shard still has no standby to fail over to.

• CMove the shards to a larger node type with more memory each

  Incorrect

  Bigger nodes hold more data per shard, but a shard with no replica still has nothing to fail over to.

• DProvision at least one replica for each shard

  Correct

  Replicas in other zones allow automatic failover of a shard's primary, which is what high availability requires.

In Memorystore for Redis Cluster, high availability comes from replicas: with at least one replica per shard, primaries and replicas are spread over zones and a replica is promoted automatically when a primary or its zone fails. Without replicas, a zone outage takes that part of the keyspace offline.

• ATurn on high availability mode with three master nodes

  Incorrect

  High availability mode protects the cluster's master services; it does not resubmit a job whose driver failed.

• BSubmit the job with a maximum number of restarts per hour

  Correct

  Restartable jobs are resubmitted when the driver exits with a non-zero code, up to the configured per-hour and total limits.

• CMake the job handle restarts, for example with checkpointing

  Correct

  Google advises designing jobs to handle restarting gracefully; a checkpointed streaming job can resume after a restart.

• DAdd primary workers to the cluster to give the job more capacity

  Incorrect

  More workers add executor capacity, but the driver keeps the same memory and nothing resubmits the job after it stops.

• EUse Spot VMs for the secondary workers to add capacity

  Incorrect

  Spot VMs can be preempted and add their own failures; they do not restart the driver.

Restartable jobs resubmit a job automatically when its driver fails, up to 10 restarts an hour and 240 in total, and stop if the driver fails more than four times in ten minutes. The job itself must cope with being restarted, which for streaming means checkpointing its progress.