Task 2.3: Tracking and running ML experiments.

• AWrite the metrics to a TensorBoard instance after each run

  Incorrect

  TensorBoard stores time-series metrics, but writing to it is still a manual call in the code.

• BRecord a custom execution in ML Metadata for each run

  Incorrect

  Custom metadata needs explicit calls for every execution, which is what the team wants to avoid.

• CPrint the parameters and metrics and search for them later in Cloud Logging

  Incorrect

  Log lines are not experiment runs and cannot be compared as runs.

• DTurn on SDK autologging before the experiment runs start

  Correct

  Autologging records parameters and evaluation metrics from supported frameworks such as scikit-learn and XGBoost into an experiment run.

The Agent Platform SDK's autologging, built on MLflow autologging, logs parameters and metrics from frameworks such as scikit-learn, XGBoost, Keras, LightGBM and PyTorch Lightning to an experiment run without manual logging calls.

• ALog the loss of each epoch with log_params, which keeps every value it is given for the run

  Incorrect

  Parameters are also single key-value pairs of the run; they do not store a series of values.

• BWrite TensorBoard event files to a Cloud Storage folder and open them in open source TensorBoard

  Incorrect

  This draws curves, but they are not stored with the experiment runs or shown next to their parameters.

• CGive the experiment a backing Vertex AI TensorBoard instance and use log_time_series_metrics

  Correct

  Time series metrics are stored in the backing TensorBoard instance, and Experiments keeps a reference to it for the run.

• DStart a separate experiment run for every epoch so that each loss value gets a run of its own

  Incorrect

  This scatters one training run across many runs and still stores no curve.

Summary metrics in Experiments are single-value scalars. Time series metrics, longitudinal values for each step of the training routine, need a backing Vertex AI TensorBoard instance, set at the experiment or the run level with experiment_tensorboard, and are logged with log_time_series_metrics. Experiments stores a reference to the TensorBoard resource.

• AGive the best model the default alias in Model Registry

  Incorrect

  An alias marks which version to serve; it does not show how the version was produced or compare runs.

• BAdd the runs to an experiment and view them side by side

  Correct

  Pipeline runs added to an experiment can be selected and compared with each other and with other experiment runs.

• CBuild a Cloud Monitoring dashboard of the pipeline jobs

  Incorrect

  Monitoring shows resource and job metrics, not the model metrics and parameters of each run.

• DQuery ML Metadata for the lineage of the best run's model

  Correct

  A model's lineage includes the training, test and evaluation data used to create it, which ML Metadata records for pipeline artifacts.

• EAdd labels with the run's hyperparameters to each pipeline job

  Incorrect

  Labels help filter and attribute cost; they do not give a side by side comparison or artifact lineage.

Agent Platform Experiments lets you add pipeline runs to an experiment and compare them in the console. Pipelines record their artifacts in ML Metadata, and the lineage of a model artifact shows the data, hyperparameters and runs that produced it.

• ACall start_run with a new run name and copy the earlier metrics into it

  Incorrect

  This creates a second run, which is what she wants to avoid.

• BCall log_metrics straight away, because the run is still open after the restart

  Incorrect

  The restarted kernel has no current run to log to; the run has to be started again with resume.

• CDelete the run and start the whole training again from the beginning

  Incorrect

  This throws away what was already logged instead of continuing the run.

• DCall start_run with the same run name and resume set to True

  Correct

  With resume set to True, the previously started run resumes; without it, a new run is created.

Runs are started with start_run. When its optional resume parameter is True, the previously started run with that name resumes and further parameters and metrics are logged to it; when resume is not specified, it defaults to False and a new run is created.

• AParse the training output in Cloud Logging for the values that each run printed

  Incorrect

  Printed log lines are not the parameters and metrics logged to the runs, and parsing them is fragile.

• BCall aiplatform.get_experiment_df() for the experiment in the notebook

  Correct

  The SDK returns the data of the experiment's runs, with their parameters and metrics, as a DataFrame.

• CExport the BigQuery table to which Experiments writes the runs of each experiment

  Incorrect

  Experiments keeps runs in ML Metadata, not in a BigQuery table of its own.

• DRerun the 150 trainings with extra log_params and log_metrics calls added

  Incorrect

  The runs already hold their parameters and metrics; logging them again does not produce a DataFrame.

With the Agent Platform SDK for Python, aiplatform.get_experiment_df() retrieves the data associated with an experiment and returns the data for its runs in a DataFrame, ready to filter, sort or plot in the notebook.

• AAutoML tabular regression trained on the results of past simulation runs

  Incorrect

  AutoML would predict delivery time, but it does not choose the next settings to try.

• BModel Monitoring on the simulator's outputs

  Incorrect

  Model Monitoring watches deployed models for skew and drift; it does not optimize parameters.

• CAgent Platform Vizier, suggesting the parameter values for each run

  Correct

  Vizier performs black-box optimization of any system with configurable parameters that are expensive to evaluate.

• DAgent Platform Experiments, logging each run's settings and delivery time

  Incorrect

  Experiments record and compare runs, but they do not choose the next settings.

Agent Platform Vizier is a black-box optimization service. It tunes hyperparameters of ML models but can also optimize any system with configurable parameters when there is no known objective function or it is too costly to evaluate.

• AWhether training examples of those two classes are mislabeled

  Correct

  Confident errors between the most-confused classes can indicate mislabeled training examples.

• BWhether the model needs a much larger learning rate during training

  Incorrect

  A training setting is not suggested by confident confusion between two specific classes.

• CWhether the test set has too few examples of those classes

  Incorrect

  A small test set adds noise; it does not concentrate errors in one pair of classes.

• DWhether the serving endpoint has enough replicas

  Incorrect

  Serving capacity affects latency, not which classes the model confuses.

Google's guidelines for high-quality ML solutions suggest looking at instances predicted with high confidence and at the most-confused classes in a multi-class confusion matrix. These errors can indicate mislabeled training examples, or a need for preprocessing or new features that separate the classes.