Overview
The Data Lead will play a crucial role in the transformation of an enterprise data platform for a large trading organization. This hands-on position focuses on guiding the development of event-streaming pipelines and production data pipelines while establishing standards for other engineers to follow. The role is primarily collaborative, working closely with teams to turn architectural designs into functional software.
Responsibilities
- Lead the design and implementation of streaming and production data pipelines.
- Ensure the integration of modern lakehouse technologies into the data platform.
- Establish and maintain best practices and standards for data engineering.
- Mentor and guide engineers across various teams in technical direction.
- Collaborate with cross-functional teams to deliver high-quality data solutions.
- Continuously evaluate and adopt emerging data technologies that enhance performance.
Requirements
- Proven experience in building or modernising production data platforms.
- Strong hands-on expertise with streaming technologies, particularly Kafka or similar.
- Experience with lakehouse and Spark-based analytics platforms in production environments.
- Proficiency in Python and SQL for data manipulation and analysis.
- Familiarity with public cloud services and solutions.
- Ability to mentor and provide technical guidance to engineering teams.