Overview
We are seeking an experienced AWS Data Engineer to join a dynamic team on a 6-month contract basis. This role is crucial for designing, building, and optimizing modern data platforms and pipelines that support analytics, reporting, and AI-driven initiatives. The ideal candidate will leverage their expertise in AWS cloud services and modern data tooling to help the organization make strategic decisions based on data.
Responsibilities
- Design and develop scalable ETL/ELT data pipelines utilizing Python and PySpark.
- Build and maintain robust data transformation frameworks using dbt to ensure data quality.
- Develop and orchestrate complex workflows using Apache Airflow to automate data processes.
- Deliver high-performance data warehouse solutions within Amazon Redshift, optimizing for cost and scalability.
- Support the implementation of Apache Iceberg lakehouse architectures for reliable data storage.
- Enhance search and indexing capabilities using OpenSearch or Elasticsearch.
- Collaborate closely with Data Analysts and Business Stakeholders to deliver tailored data solutions.
- Implement CI/CD best practices using CircleCI to ensure reliable delivery pipelines.
Requirements
- Extensive commercial experience as a Data Engineer within the AWS ecosystem.
- Proficiency in Python, SQL, and PySpark.
- Hands-on experience with Apache Airflow for managing workflows.
- Familiarity with Amazon Redshift and data lakehouse architectures like Apache Iceberg.
- Experience with search technologies such as OpenSearch or Elasticsearch.
- Knowledge of data modeling and performance optimization best practices.
- Experience implementing CI/CD pipelines, preferably with CircleCI.
- Exposure to AWS services such as S3, Lambda, Glue, and ECS is beneficial.