Task 3.1: Automate data processing by using AWS services

• AThe task logs in CloudWatch Logs for the new DAG

  Incorrect

  Task logs are written when a DAG's tasks run. A DAG that was never parsed has no task runs to log.

• BThe webserver logs in CloudWatch Logs

  Incorrect

  The webserver log group holds logs from the Airflow web interface. The UI only shows DAGs after the scheduler has processed them.

• CThe CloudTrail event history for the S3 bucket

  Incorrect

  CloudTrail records API calls such as the upload itself. It does not show how Airflow processed the file afterwards.

• DThe DAG processing logs in CloudWatch Logs

  Correct

  MWAA can send the logs of the DAG processor manager, the part of the scheduler that processes DAG files, to a CloudWatch log group. Errors while parsing a new file show up there.

A missing DAG is a parsing problem until proven otherwise. Turn on the DAG processing log group (it is off unless enabled) and look for the import or syntax error of the new file.

• APass a ClientRequestToken so that StartQueryExecution waits for the query to finish

  Incorrect

  The token makes the start request idempotent so a retry does not run the query twice. It does not make the call wait.

• BCall GetQueryResults with a larger MaxResults value so that it waits for the rows

  Incorrect

  MaxResults sets how many rows come back per page. GetQueryResults does not run or wait for the query.

• CCall GetQueryExecution until the state is SUCCEEDED, then call GetQueryResults

  Correct

  StartQueryExecution only starts the query. GetQueryExecution returns its current state (QUEUED, RUNNING, SUCCEEDED and so on), and results can be read once it has succeeded.

• DRead the result file from the S3 output location as soon as StartQueryExecution returns

  Incorrect

  Athena writes results to the output location as the query completes, so right after the start call there is nothing complete to read.

Athena queries are asynchronous: start, check the state, then fetch. In a Step Functions workflow the athena:startQueryExecution.sync integration does the waiting for you.

• AAn AWS Glue DataBrew profile job that runs on every new extract

  Incorrect

  A profile job evaluates a dataset and reports statistics. It does not change the data or write cleaned output.

• BAn AWS Glue crawler that runs on the extract folder every day

  Incorrect

  A crawler records the schema in the Data Catalog. It does not clean or transform any values.

• CAn Amazon Athena CTAS query that the analysts edit for each extract

  Incorrect

  CTAS can write Parquet, but the cleaning would be SQL that someone has to write and run, which the analysts cannot do.

• DAn AWS Glue DataBrew recipe, run daily by a scheduled recipe job

  Correct

  DataBrew offers more than 250 point-and-click transformations saved as steps in a recipe. A recipe job applies the recipe to the data and writes the output, including Parquet, and jobs can be scheduled.

DataBrew is the no-code preparation tool: build the steps once as a recipe on a sample, then let a scheduled recipe job apply them to each new file.

• AA scheduled rule in the default EventBridge event bus with a cron expression

  Incorrect

  Scheduled rules use the UTC+0 time zone, so a fixed cron expression drifts by an hour in local time when daylight saving time changes.

• BAn EventBridge Scheduler cron schedule with the Europe/Berlin time zone

  Correct

  Scheduler evaluates cron and rate expressions in the time zone you specify and handles daylight saving time, and it can start the Glue job directly as a target.

• CA time-based trigger on the AWS Glue job with a cron expression

  Incorrect

  Glue time-based schedules are defined in UTC, so the local start time moves with daylight saving time.

• DA Lambda function that runs every minute and starts the job at 06:00

  Incorrect

  This is custom code and wastes thousands of invocations a day for one start.

For schedules tied to a local clock, use EventBridge Scheduler and give it the time zone. Legacy scheduled rules and Glue triggers work in UTC.

• AA Task state that invokes a Lambda function that calls StartCrawler

  Incorrect

  This works, but it adds a function whose code the team would have to write and maintain.

• BA Task state that uses the aws-sdk:glue:startCrawler resource

  Correct

  AWS SDK service integrations let a Task state call an AWS API action directly, with a resource such as arn:aws:states:::aws-sdk:glue:startCrawler.

• CAn activity task with a worker that polls and starts the crawler

  Incorrect

  With activities, a worker program running outside Step Functions does the work, which is code the team would have to run.

• DAn HTTP Task that calls the AWS Glue API endpoint for the crawler

  Incorrect

  HTTP Tasks are for calling HTTPS APIs such as third-party SaaS applications. AWS APIs have SDK integrations.

Step Functions reaches most AWS APIs through SDK integrations, so glue code in Lambda is needed only when a step does real work of its own.

• ATurn auto-stop back on, with a suitable idle timeout

  Correct

  Auto-stop stops an idle application, 15 minutes by default, and a stopped application releases its pre-initialized capacity.

• BLower the application's maximum capacity for CPU and memory

  Incorrect

  Maximum capacity caps how far the application can scale. The pre-initialized workers are still paid for while they sit idle.

• CTurn off auto-start so jobs cannot start the application at night

  Incorrect

  Auto-start starts the application when a job is submitted. It does not stop an application that is already running and idle.

• DAdd more pre-initialized workers so each job finishes sooner at night

  Incorrect

  Pre-initialized workers are paid for even when the application is idle, so adding more raises the overnight cost.

Pre-initialized capacity trades idle cost for fast starts; keeping auto-stop on bounds that idle cost.

• ACREATE TABLE AS SELECT, writing a new table each day

  Incorrect

  CTAS creates a new table from the query results, so each run would produce a separate table instead of adding to the existing one.

• BUNLOAD the SELECT query results to the table's S3 path

  Incorrect

  UNLOAD writes query results to S3 in a chosen format. It is not a statement for adding rows to a table.

• CINSERT INTO the summary table with the SELECT query

  Correct

  INSERT INTO adds new rows to an existing destination table based on a SELECT statement.

• DCREATE VIEW over the orders table with the SELECT query

  Incorrect

  A view is a logical table whose query runs each time it is used. It stores no daily summary rows.

CTAS creates, INSERT INTO appends: scheduled jobs that grow an existing table use INSERT INTO with a SELECT.

• AOne BatchExecuteStatement call with the AUTO_COMMIT run mode

  Incorrect

  With AUTO_COMMIT, each statement is committed on its own, and one failure does not affect the others.

• BThree ExecuteStatement calls, issued one after another in the same order

  Incorrect

  Each call runs its own statement, so a failure in the third leaves the first two committed.

• COne BatchExecuteStatement call with the default TRANSACTION run mode

  Correct

  By default, the statements in a batch run as a single transaction, and if any statement fails, all of the work is rolled back.

• DThree ExecuteStatement calls, issued in parallel from three threads

  Incorrect

  Parallel, separate statements are not one transaction, and the delete and insert could also run out of order.

Use BatchExecuteStatement in its default transaction mode when several statements must succeed or fail together.

• AAWS Batch

  Correct

  AWS Batch runs batch computing workloads and provisions the compute resources for submitted jobs, removing the work of managing that infrastructure.

• BAWS Lambda with container images

  Incorrect

  Lambda functions can run for at most 900 seconds, far short of 2 to 3 hours.

• CAn Amazon ECS service with a set desired count

  Incorrect

  An ECS service keeps a specified number of tasks running and replaces failed ones, which suits long-running services, not a queue of batch jobs.

• DStep Functions Express workflows that run each job

  Incorrect

  Express workflows run for at most 5 minutes, so they cannot wait for a job that takes hours.

Queued, long-running container work is AWS Batch's job: submit jobs to a queue, and Batch provisions the compute to run them.