Overview
The Engineering Manager will lead the engineering efforts for a cutting-edge real-time data platform at a major commodity trading firm. Working closely with two dedicated teams, the manager will oversee the development of new capabilities, ensuring the platform delivers reliable real-time data across trading, risk management, and operations. This role demands expertise in large-scale data engineering and a focus on high performance and reliability in complex data environments.
Responsibilities
- Lead the engineering of the real-time data platform across two dedicated teams.
- Stand up the streaming backbone using Kafka (Amazon MSK) and establish change data capture feeds from trading systems.
- Ensure the platform's reliability with high service levels, including ordering, replay, and no lost events.
- Build out the data lakehouse infrastructure using Iceberg and Delta, as well as associated knowledge graph and search layers.
- Connect cloud and on-premises systems into a cohesive platform.
- Set engineering standards for CI/CD, infrastructure as code, testing, and code reviews to empower domain teams.
- Direct external partners to maintain consistent architecture and delivery plans.
- Recruit, hire, and develop the engineering team to drive future innovation.
Requirements
- Proven experience in leading engineering teams to deliver large-scale data platforms with streaming capabilities.
- Deep expertise in AWS and Databricks or a comparable lakehouse platform.
- Strong understanding of real-time data reliability and appropriate use of batch processing.
- Hands-on ability to prototype and review technical designs, including schema and data partitioning.
- Experience with AI agents in team deliverables and overall project execution.
- Demonstrated ability to direct vendors and systems integrators effectively.
- Experience working with fast-moving, complex data environments such as trading, energy, mining, or similar sectors.