Overview
We are seeking a Data Engineer on a contract basis to enhance our digital services for the UK public sector. The contractor will design and develop data pipelines and infrastructure while collaborating with Data Engineers, Data Architects, and Analysts to deliver high-performance data solutions. This role demands extensive hands-on experience with cloud technologies and data engineering practices, ensuring secure and compliant data environments for government use.
Responsibilities
- Design, build, and maintain robust, scalable, and secure data pipelines using AWS services and Apache Spark.
- Develop and optimize data models for reporting and analytics in Redshift and other data warehouse platforms.
- Collaborate with Data Scientists, Analysts, and Business Stakeholders to understand data requirements and deliver clean, validated datasets.
- Monitor, troubleshoot, and optimize ETL/ELT workflows to ensure data quality and pipeline efficiency.
- Implement best practices in data governance, security, and compliance within cloud environments.
- Lead and mentor junior data engineers to promote technical excellence and continuous improvement.
- Write clean, maintainable, and efficient code following software development best practices.
- Stay up-to-date with industry trends and emerging technologies in cloud and big data.
Requirements
- Proven work experience as a Senior Data Engineer using cloud platform technologies.
- Strong expertise in AWS services including DMS, S3, Lambda, Step Functions, Glue, Athena, Redshift, and IAM.
- Proficient in Apache Spark and big data processing.
- Hands-on experience with Amazon Redshift, including table design and workload management.
- Experience building ETL/ELT pipelines with tools like AWS Glue or Apache Airflow.
- Excellent problem-solving skills with proficiency in SQL and performance tuning.
- Familiarity with version control systems like Git and agile development methodologies.
- Proven UK government/public sector experience and must qualify for UK Security Clearance (SC).