Overview
We are looking for a Senior Data Engineer with strong AWS expertise to conduct a three-month fully remote project focused on delivering cloud-based data engineering and integration solutions. The role will involve close collaboration with technical and delivery teams to build data pipelines, develop lakehouse capabilities, and integrate data efficiently across various systems.
Responsibilities
- Design, build and optimise data pipelines using AWS Glue, S3, Lambda and Apache Spark.
- Develop and maintain lakehouse solutions using Apache Iceberg, supporting efficient data ingestion, transformation and access.
- Build workflows for ingesting, processing and transforming unstructured data.
- Develop integrations using API Gateway, Lambda and EventBridge, applying appropriate integration patterns.
- Work with Amazon Bedrock to enable AI processing of unstructured data as needed.
- Utilise AWS Systems Manager to support configuration and operational tasks.
- Develop infrastructure as code with Terraform, alongside CI/CD pipelines using GitHub Actions.
- Implement data quality checks, monitoring, error handling and recovery processes.
Requirements
- Strong hands-on AWS data engineering experience, especially with Glue, Bedrock, SSM, S3, Lambda, API Gateway and EventBridge.
- Proven track record of building lakehouse solutions with Apache Iceberg and Apache Spark.
- Strong understanding of unstructured data processing, including ingestion, transformation and metadata handling.
- Proficient in integration patterns, with practical experience in delivering systems integrations.
- Working knowledge of Terraform, CI/CD practices and GitHub Actions for deployment and automation.
- Experience in creating reliable, maintainable data solutions with a focus on performance, security and operational support.
- Ability to work independently and contribute effectively within a remote team environment.
- Strong communication skills to articulate technical requirements into actionable solutions.