Amazon Web Services Data-Engineer-Associate Exam With Confidence Using Practice Dumps

Option C is correct because the company only needs to keep the original files for 90 days and then delete them. AWS states that S3 Lifecycle can be used to delete expired objects automatically, which directly matches this requirement. There is no need to transition the objects to another storage class because the files are not needed beyond day 90. Adding a transition step would add complexity and, in this case, would not improve cost-effectiveness.

Option A is not optimal because S3 Glacier Flexible Retrieval has a 90-day minimum storage duration and is intended for longer-term archival. Transitioning objects right before deleting them is unnecessary overhead and can add minimum-duration charges. Option B is incorrect because Object Lock is for retention and write-once-read-many protection, not for normal temporary retention with lowest cost. Option D is also less cost-effective because S3 Intelligent-Tiering is useful when access patterns are uncertain, but here the retention period and deletion timing are already known. The simplest and cheapest solution is to keep the originals in S3 Standard for the required 90 days and then expire them with Lifecycle.

Problem Analysis:

The pipeline processes compressed files in S3 and must support incremental data processing.

AWS Glue features must facilitate tracking progress to avoid reprocessing the same data.

Key Considerations:

Incremental data processing requires tracking which files or partitions have already been processed.

The solution must be automated and efficient for large-scale ETL jobs.

Solution Analysis:

Option A: Workflows

Workflows organize and orchestrate multiple Glue jobs but do not track progress for incremental data processing.

Option B: Triggers

Triggers initiate Glue jobs based on a schedule or events but do not track which data has been processed.

Option C: Job Bookmarks

Job bookmarks track the state of the data that has been processed, enabling incremental processing.

Automatically skip files or partitions that were previously processed in Glue jobs.

Option D: Classifiers

Classifiers determine the schema of incoming data but do not handle incremental processing.

Final Recommendation:

Job bookmarks are specifically designed to enable incremental data processing in AWS Glue ETL pipelines.

AWS Glue Job Bookmarks Documentation

AWS Glue ETL Features

AWS Step Functions is a fully managed, serverless orchestration service designed to coordinate multiple AWS services into reliable workflows. Step Functions natively provide automatic retries, error handling, state-level visibility, and execution history, which directly satisfy the requirements for retry logic and monitoring individual pipeline stages.

In this solution, AWS Lambda functions are used for batch data ingestion, AWS Glue jobs handle transformation, and Amazon S3 is used for storage. Step Functions orchestrate these stages while maintaining clear separation of responsibilities. Each step can be configured with retry and catch policies, allowing the workflow to automatically recover from transient failures without manual intervention.

Chaining AWS Glue jobs alone does not provide detailed stage-level visibility or flexible retry control, and setting MaxRetries to 0 contradicts the retry requirement. EventBridge-based pipelines lack native workflow state tracking. Amazon MWAA introduces unnecessary operational overhead, including environment management and Airflow maintenance.

Therefore, using AWS Step Functions to orchestrate Lambda and Glue provides the most scalable, observable, and low-maintenance solution.