Overview
The Senior Data Engineer will join a dynamic team focused on modernizing the data architecture within a well-established transport and travel sector organization. This role involves designing and building high-quality data pipelines and contributing to the broader data transformation efforts by leveraging Databricks and Spark technologies. Collaborating closely with various stakeholders, the contractor will play a crucial role in shaping the technical direction of the data platform while ensuring data integrity and security throughout the project lifecycle.
Responsibilities
- Design and implement high-quality end-to-end data pipelines.
- Lead data modeling initiatives in collaboration with business analysts and engineering teams.
- Manage projects across the full lifecycle from planning to delivery and operational run.
- Maintain a strong focus on security and data protection throughout development processes.
Requirements
- A minimum of 5 years of progressive experience in data engineering.
- Strong expertise in Databricks and Spark, including performance optimization and use of Delta Lake.
- Proficient in Python with a solid understanding of software development and DevOps practices.
- Excellent data modeling skills, demonstrating the ability to navigate technical and organizational trade-offs.
- Advanced SQL skills with experience in querying data lakes or lakehouses.
- Experience in data ingestion through APIs or database connections.
- Familiarity with Agile methodologies in a self-organizing team environment.