Overview
The Data Engineer position is a contract role focused on enhancing high-quality digital services within the UK public sector. The contractor will collaborate with a team of Data Engineers, Data Architects, and Analysts to design and develop scalable data infrastructures and pipelines, ensuring robust data solutions that improve operational processes and citizen outcomes. As part of a technology solutions provider, the role involves technical leadership and mentorship to support data engineering best practices in a secure and governed environment.
Responsibilities
- Design, build, and maintain robust, scalable, and secure data pipelines using AWS services and Apache Spark.
- Develop and optimize data models for reporting and analytics in Redshift and other DWH platforms.
- Monitor, troubleshoot, and optimize ETL/ELT workflows to ensure data quality and pipeline efficiency.
- Lead and mentor junior data engineers, promoting a culture of technical excellence and continuous improvement.
- Collaborate with Data Scientists, Analysts, and Business Stakeholders to understand data requirements and deliver clean, validated datasets.
- Document code, systems, and processes to facilitate knowledge sharing and maintainability.
- Stay up to date with the latest in cloud and big data technologies, evaluating their potential for use in the platform.
Requirements
- Proven work experience as a Senior Data Engineer using cloud platform technologies.
- Strong expertise in AWS services including DMS, S3, Lambda, Step Functions, Glue, Athena, and Redshift.
- Proficient in Apache Spark and big data processing.
- Experience building ETL/ELT pipelines using tools such as AWS Glue, EMR, or Apache Airflow.
- Solid experience with SQL and performance tuning in data warehouse environments.
- Hands-on experience with Amazon Redshift including table design and workload management.
- Familiarity with version control systems (e.g., Git) and agile development methodologies.
- Must qualify for UK Security Clearance (SC) and possess proven UK government experience.