Overview
The Senior Data Engineer will play a pivotal role in enhancing data infrastructure for a leading client, focusing on the design and optimization of data pipelines within a Databricks Lakehouse environment. Collaborating with stakeholders, this position involves ensuring high-quality data delivery for analytics and reporting while supporting production environments. This contract role offers a hybrid working arrangement, requiring one day on-site each week for an initial 6-month term, with opportunities for extension.
Responsibilities
- Design, build, and optimise data pipelines using ELT/ETL best practices.
- Migrate and transform data into a Databricks Lakehouse architecture.
- Ensure data quality, reliability, and scalability for analytics and reporting.
- Collaborate with stakeholders to deliver robust solutions aligned with business needs.
- Support production environments and troubleshoot performance issues.
Requirements
- 7+ years in Data Engineering with strong experience in cloud-based data platforms.
- Proven ability to design and optimise pipelines for large-scale data processing.
- Hands-on experience with Databricks and Azure.
- Strong stakeholder communication and problem-solving skills.
- Experience with Databricks, DBT, Python, PySpark, SQL, and Azure.