Overview
We are seeking a skilled Databricks/Spark Data Engineer to join a consultancy team, working on-site with a major financial services client in a live trading environment. The successful candidate will collaborate directly with the client’s trading and technology teams to build and optimize high-performance data pipelines that support trading operations. This role requires a deep understanding of trading environments and strong engineering capabilities within the Databricks platform.
Responsibilities
- Build and tune real-time and batch pipelines on Databricks.
- Optimize Spark jobs and queries for performance, latency, and cost.
- Collaborate with the client’s trading and technology teams on market and trade data.
- Represent the consultancy on-site, ensuring delivery meets production standards.
Requirements
- Advanced experience with Apache Spark, including performance tuning.
- Proficiency in Spark Structured Streaming in production environments.
- Strong knowledge of Spark SQL and query optimization.
- Core experience with the Databricks platform.
- Strong programming skills in Python/PySpark.
- Previous experience in trading or financial markets is essential.