Overview
We are seeking a Data Engineer on a contract basis to enhance digital services within the UK public sector. The contractor will collaborate closely with existing Data Engineers, Data Architects, Analysts, and other delivery teams to design and implement scalable data infrastructure, ensuring high-quality data solutions that meet government standards. This role focuses on providing technical leadership throughout the data engineering lifecycle while understanding the unique security and governance requirements of the public sector.
Responsibilities
- Design, build, and maintain robust, scalable, and secure data pipelines using AWS services and Apache Spark.
- Develop and optimize data models for reporting and analytics in Redshift and other data warehouse platforms.
- Collaborate with Data Scientists, Analysts, and Business Stakeholders to understand data requirements and deliver clean, validated datasets.
- Monitor, troubleshoot, and optimize ETL/ELT workflows to ensure data quality and pipeline efficiency.
- Implement best practices in data governance, security, and compliance within cloud environments.
- Lead and mentor junior data engineers, promoting a culture of technical excellence and continuous improvement.
- Document code, systems, and processes to facilitate knowledge sharing and maintainability.
Requirements
- Proven work experience as a Senior Data Engineer using cloud platform technologies.
- Strong expertise in AWS services, including AWS DMS, S3, Lambda, Glue, and Redshift.
- Proficient in Apache Spark and big data processing.
- Solid experience with SQL and performance tuning in data warehouse environments.
- Hands-on experience with Amazon Redshift and ETL/ELT pipeline building using tools like AWS Glue or Apache Airflow.
- Proficiency in Java and familiarity with data modelling concepts.
- Experience working in the UK public sector and ability to qualify for UK Security Clearance (SC).