Task 3.2: Training models.

• ASave checkpoints to the VM's local disk every hour so they are close to the training process

  Incorrect

  Local disk does not survive the restart; progress must be exported to a location outside the VM, such as Cloud Storage.

• BWrite checkpoints to AIP_CHECKPOINT_DIR at least every four hours, and load the latest one when the code starts

  Correct

  Agent Platform reruns the code from the start after a restart, so the code must export progress and resume from it.

• CRaise the job's maximum running time so that maintenance restarts no longer happen during training

  Incorrect

  The time limit only bounds how long the job may run; Google Cloud can still restart the VMs for maintenance.

• DKeep the code unchanged, because a restarted job resumes from its last completed step on its own

  Incorrect

  When a VM restarts, Agent Platform starts running the code again from its start; it does not restore training state.

The VMs that run serverless training code restart occasionally, for example for maintenance, and Agent Platform then runs the code again from its start. For code that runs more than about four hours, export training progress to Cloud Storage at least every four hours and, at startup, load any saved state instead of starting from scratch. When a base output directory is set, the job gets AIP_CHECKPOINT_DIR, a Cloud Storage directory intended for checkpoints.

• ARun parallel trials at the quota limit of ten clusters to finish within the two days

  Incorrect

  More concurrency was the problem; it finishes sooner but further weakens the search.

• BSwitch the search algorithm to grid search with 40 parallel trials

  Incorrect

  Grid search does not learn from earlier trials either, so it does not fix the weak result.

• CTurn on early stopping and keep 40 parallel trials

  Incorrect

  Even where early stopping ends weak trials sooner, all 40 trials still start at once without any completed results to learn from.

• DRun fewer trials at once, for example five, so the later trials learn from the earlier ones

  Correct

  Bayesian tuning picks new values from completed trials; running everything at once leaves nothing to learn from.

Hyperparameter tuning on Agent Platform uses Bayesian optimization by default and uses the results of previous trials to choose values for the next. Running trials in parallel reduces wall-clock time but can reduce the effectiveness of the job, because some trials start without the results of trials still running.

• ASpot VMs for every job to lower the cost of each run

  Incorrect

  Spot VMs are cheaper but are still provisioned per job and can be harder to obtain, so startup does not improve.

• BLarger machine types to make each job finish faster

  Incorrect

  Training time is already short; the wait comes from provisioning, not from the machine size.

• CA persistent resource that the jobs run on

  Correct

  A persistent resource keeps compute provisioned, which cuts job startup time and helps ensure capacity; it bills while running.

• DA hyperparameter tuning job that groups the 200 runs into trials

  Incorrect

  Tuning trials are still provisioned as training clusters and are for searching hyperparameters, not for unrelated jobs.

A persistent resource is a long-running cluster for serverless training jobs that ensures compute availability and removes the job startup time otherwise needed to create resources. It suits jobs whose training time is shorter than their startup time, and it is billed for as long as it runs, whether or not a job is using it.

• ARun more epochs on the same examples to strengthen the tuning

  Incorrect

  More epochs fit the mismatched format even harder and can overfit, without teaching the production prompt shape.

• BRaise the adapter size and the learning rate multiplier for production

  Incorrect

  Larger adapters and faster learning change capacity, not the gap between tuning prompts and production prompts.

• CRebuild the tuning examples in the same prompt format, with the same instructions and context, as production requests

  Correct

  Tuning data should reflect the prompt distribution, format and context the model sees in production, which these examples did not.

• DSwitch the job to preference tuning with pairs of preferred and rejected answers from reviewers

  Incorrect

  Changing the tuning method does not fix examples whose prompts look nothing like production.

Data quality matters more than quantity for tuning, and the data used for fine-tuning should reflect the prompt distribution, format and context that the model encounters in production. A model tuned on prompts of a different shape can underperform when real prompts arrive.

• APick the learning rate and batch size at random inside the script

  Incorrect

  The tuning service chooses each trial's values; values picked in the script would ignore its search.

• BWrite the TF_CONFIG variable for each trial's cluster

  Incorrect

  Agent Platform sets the cluster configuration; tuning does not need the script to write it.

• CRead each tuned value from a command-line argument

  Correct

  Agent Platform passes each trial's hyperparameter values to the training application as command-line arguments.

• DReport the target metric with the cloudml-hypertune package

  Correct

  The tuning service learns each trial's result from the metric the code reports through cloudml-hypertune.

• ESave every trial's model to the same fixed Cloud Storage path

  Incorrect

  A shared fixed path makes trials overwrite each other; each trial should use the directory Agent Platform provides.

For hyperparameter tuning, define a command-line argument for each tuned hyperparameter and use its value in the code, and report the target metric with the cloudml-hypertune package so that Agent Platform can compare trials. Agent Platform takes the model from the best trial's AIP_MODEL_DIR.

• AInstall the library and framework from a requirements file and apt when each job starts

  Incorrect

  Installing at start is the 15-minute delay the team wants to remove.

• BBuild a custom container image with the library and framework pre-installed

  Correct

  Custom containers pre-install every dependency, which shortens start-up and allows frameworks no prebuilt container offers.

• CUse the closest prebuilt container and drop the GDAL code

  Incorrect

  Dropping the code the model needs is not a fix.

• DRun the training in a Workbench notebook instead of a training job

  Incorrect

  A notebook avoids the job, but loses managed, repeatable training.

Custom containers let you specify and pre-install all the dependencies your application needs, which gives faster start-up than installing them in each job, and they let you use an ML framework or version that no prebuilt container supports.

• ACloud Logging with a more detailed log level

  Incorrect

  More logs may help later, but they do not let you inspect the running container now.

• BA new job in the same container on a larger machine

  Incorrect

  A bigger machine changes the run; it does not show why this one stalls.

• CModel Monitoring on the training job

  Incorrect

  Model Monitoring watches deployed models' inputs; it does not inspect training containers.

• DAn interactive shell into the job's container

  Correct

  The interactive shell lets you browse the container's file system and run debugging utilities while the job runs.

Agent Platform lets you open an interactive shell into the container where your training code is running, in prebuilt or custom containers, to browse the file system and run debugging utilities.

• ASpot VMs for the job

  Incorrect

  Spot VMs change the price of the job, not where its step time goes.

• BA hyperparameter tuning job that searches over the batch size and learning rate

  Incorrect

  Tuning tries settings but does not show where step time goes.

• CCloud Profiler for the training job, viewed through TensorBoard

  Correct

  Profiler shows the resource consumption of training operations, which reveals bottlenecks such as a slow input pipeline.

• DMore GPUs in the training worker pool

  Incorrect

  Idle GPUs suggest they are waiting; adding more makes the waste bigger.

Profiler lets you monitor and optimize the training performance of serverless training jobs by showing the resource consumption of training operations, so you can find bottlenecks such as an input pipeline that keeps accelerators idle.

• ARun the job on private IP in the peered VPC network

  Correct

  Private IP connects training to resources on private addresses inside Google Cloud or on-premises, with lower latency.

• BCopy the database to Cloud Storage every hour for the job to read

  Incorrect

  A copy adds a pipeline and stale data instead of private access.

• CGive the database a public IP restricted by a firewall rule

  Incorrect

  Security forbids exposing the database publicly.

• DMount the private database through Cloud Storage FUSE in the peered network

  Incorrect

  FUSE mounts Cloud Storage buckets, not databases.

Serverless training jobs can run on private IP: with VPC Network Peering to your network, the job reaches private IP addresses inside Google Cloud or on-premises, with more network security and lower latency than public IP.