Overview
We are looking for three SC Cleared Data Engineers to support the design and development of cloud-based data integration and analytics solutions specifically in the defence sector. The successful candidates will focus on building scalable ETL pipelines and data lakes within AWS, collaborating closely with other engineering teams to deliver high-quality data products and insights.
Responsibilities
- Support the design and development of cloud-based data integration solutions using AWS Glue.
- Build scalable ETL pipelines and data processing frameworks within AWS.
- Implement and manage data lakes on Amazon S3.
- Develop Glue Jobs utilizing PySpark and Python.
- Utilize AWS Glue Data Catalog, Crawlers, Workflows, and triggers effectively.
- Ensure adherence to security best practices using AWS IAM roles and policies.
- Collaborate with other engineering teams to enhance data workflows and analytics capabilities.
Requirements
- Must hold current SC clearance and be a sole British national.
- Extensive commercial experience with AWS Glue and related technologies.
- Proficiency in developing Glue Jobs using PySpark and Python.
- Experience with AWS Glue Data Catalog, Crawlers, Workflows, and triggers.
- Knowledge of data lake implementation and management on Amazon S3.
- Familiarity with Amazon Redshift, AWS Lambda, and Amazon Athena.
- Understanding of data processing using Amazon EMR and Amazon Kinesis.
- Experience with AWS Step Functions, Amazon SQS/SNS, and AWS CloudWatch.