Task 5.2: Designing automation and repeatability.

• ATemplate the run's logical date into the source path and target partition

  Correct

  With Jinja templating, each run reads only its own day's files and writes only its own partition, whenever it actually runs.

• BRaise task retries to five with exponential backoff

  Incorrect

  Retries repeat the same non-idempotent append, so they add duplicates rather than prevent them.

• CSet the DAG's start_date to datetime.now() so that reruns start fresh

  Incorrect

  Google advises a fixed start_date because it is evaluated at every parse; a moving date breaks scheduling and does nothing for duplicates.

• DReplace the target partition's data instead of appending to it

  Correct

  Overwriting the partition makes a rerun produce the same result as the first run instead of adding a second copy.

• ELoad in a PythonOperator that loops over the files in the bucket

  Incorrect

  Looping over the whole bucket is the opposite of scoping a run to its date, so reruns would load even more data.

Google's DAG guidance is that each task should be an idempotent unit of work. For daily loads that means the run's logical date decides both what is read and which partition is written, and the write replaces that partition, so a rerun or a backfill gives the same result as the original run.

• AAdd more Airflow workers to the environment configuration

  Incorrect

  Workers run tasks; the problem is parsing, which happens every time the file is processed, whatever the number of workers.

• BRaise dagbag_import_timeout from 30 seconds to 300 seconds

  Incorrect

  Google suggests raising the timeout only when the DAG cannot be optimized; it hides the slow parsing instead of removing it.

• CMove the API call into a task and read Variables with Jinja templates

  Correct

  Top-level code runs at every parse; moving the call into a task and using templates for Variables takes the slow work out of parsing.

• DSplit the DAG into SubDAGs so that each part is parsed separately

  Incorrect

  Google advises against SubDAGs, and they do not remove the top-level calls that slow parsing down.

Airflow parses DAG files continually. Code at module level, such as API calls and many Variable.get calls, runs at every parse and can exceed the import timeout. With CPU usage low, the fix is to optimize the DAG file: avoid top-level code, read Variables through Jinja templates, and keep heavy work inside tasks.

• AEnable task retries with a delay between attempts

  Correct

  Google recommends task retries to avoid failures of whole DAG runs, and with idempotent tasks a retry is safe.

• BIncrease the number of schedulers in the environment

  Incorrect

  More schedulers help with scheduling throughput, not with a task whose API call fails.

• CMove the API call into the DAG file's top-level code

  Incorrect

  Moving the call elsewhere in the DAG file does not make the partner API any more reliable, and a failed call still fails the run.

• DSet the DAG's catchup to run the missed day again

  Incorrect

  Catch-up runs missed schedule intervals; it does not retry a failed task within its run.

Transient failures are expected when calling external systems. Google's DAG guidance is to make each task an idempotent unit of work and to enable task retries, so that a brief failure is absorbed by a retry instead of failing the run.

• ALower the environment's worker concurrency to four

  Incorrect

  Worker concurrency limits all tasks on a worker, which throttles every DAG instead of only the exports.

• BGive each DAG a schedule offset so that they start at different times

  Incorrect

  Staggered starts help until runs drift or are rerun, and then the exports overlap again.

• CSet max_active_runs to one on each of the twelve DAGs

  Incorrect

  This limits runs per DAG, but twelve DAGs can still run twelve exports at the same time.

• DCreate a pool with four slots and assign the export tasks to it

  Correct

  Pools limit how many of their tasks run at once across all DAGs, so the exports share four slots while other tasks run freely.

Airflow pools cap concurrency for a set of tasks wherever they are defined. Pools are created in the Airflow UI and tasks are associated with them in the DAG code, which protects shared resources such as a fragile source database without slowing unrelated work.

• AUse SubDAGs, one for each of the five stages

  Incorrect

  Google advises against SubDAGs; they add complexity and scheduling problems for no benefit over grouping.

• BGroup each stage's tasks into a task group

  Correct

  Task groups organize tasks visually inside one DAG without separate DAG runs, which is Google's recommendation instead of SubDAGs.

• CSplit the stages into five DAGs triggered one after another

  Incorrect

  Separate DAGs lose the direct dependencies and add cross-DAG triggering to manage.

• DName each task with a stage prefix so they sort together

  Incorrect

  Naming helps search but does not group anything in the graph.

For organizing large DAGs, Google's guidance is plain: do not use SubDAGs, group tasks inside the DAG instead. Task groups give a collapsible structure in the UI while the DAG keeps one run and ordinary task dependencies.

• AEdit the query to hard-code each date and run it ten times by hand

  Incorrect

  Hand edits work but risk mistakes and leave the scheduled query changed; the backfill feature does this for each date.

• BDelete and recreate the scheduled query with an earlier start date

  Incorrect

  A new schedule's start date does not run past intervals for it.

• CSchedule a backfill of the scheduled query for 1 to 11 October

  Correct

  A backfill runs the scheduled query for each date in the range with the matching @run_date and @run_time values; the range includes the first date and excludes the last.

• DRestore the summaries with time travel to before the bug started

  Incorrect

  The bad summaries were written on the days they cover, so time travel cannot return correct versions of them.

Scheduled queries are parameterized by @run_date and @run_time, which makes them repeatable for any past date. A backfill, from the console, the bq tool or the API, runs them for a date range, with the first date included and the last excluded, in UTC.