Overview
The Senior Data Engineer will play a crucial role in building a Databricks lakehouse platform designed for a high-performance, business-critical Front Office Risk trading environment. This position involves hands-on engineering tasks to create and optimize large-scale distributed data systems in collaboration with front and middle office users, ensuring a strong focus on market and credit risk concerns. The engineer will be part of a highly technical team dedicated to implementing innovative data solutions within a fast-paced financial services setting.
Responsibilities
- Define technical approach for migrating historic data from SQL Server to Databricks.
- Build and optimize pipelines across bronze, silver, gold, and platinum layers.
- Enhance data products in market risk, credit risk, and historical market data.
- Design and develop a lakehouse platform utilizing Medallion architecture.
- Oversee data modeling, architecture, and pipeline design.
- Manage large-scale data operations, ranging from TB to PB.
- Ensure performance, scalability, and reliability of systems in production.
Requirements
- Strong experience in running Spark workloads in production environments.
- Proven ability to optimize Spark for large datasets (TB/PB).
- Solid programming skills in Python; knowledge of Scala is beneficial.
- Experience with data modeling and lakehouse architecture.
- Demonstrated capability to debug and enhance performance in distributed systems.
- Recent hands-on experience with Spark; Databricks experience preferred.
- Familiarity with AI/ML or advanced analytics platforms is a significant advantage.
- Background in financial services or trading environments is essential.