Overview
As a Senior Data Engineer, you will play a crucial role in a high-impact rail infrastructure project, collaborating with a leading consultancy to modernise data systems across the rail network. This fully remote position emphasizes your expertise in Databricks and real-time streaming data, allowing you to contribute to operational insights and decision-making processes in a dynamic environment.
Responsibilities
- Develop and optimise streaming data solutions in Databricks.
- Collaborate with cross-functional teams to enhance data systems architecture.
- Implement real-time telemetry data pipelines to support operational insights.
- Monitor and troubleshoot data workflows to ensure reliability and performance.
- Utilize Azure cloud services to manage large-scale datasets effectively.
- Contribute to the strategic approach of data infrastructure modernization.
Requirements
- Proven experience with Databricks, especially in streaming data pipelines.
- Strong understanding of real-time telemetry data processes.
- Solid background in Azure cloud services, with practical application.
- Experience working with large-scale, fast-moving datasets in production environments.
- Ability to adapt to new technologies and data engineering practices.