Databricks / Spark Data Engineer

Overview

We are seeking a skilled Databricks/Spark Data Engineer to join a consultancy team, working on-site with a major financial services client in a live trading environment. The successful candidate will collaborate directly with the client’s trading and technology teams to build and optimize high-performance data pipelines that support trading operations. This role requires a deep understanding of trading environments and strong engineering capabilities within the Databricks platform.

Responsibilities

  • Build and tune real-time and batch pipelines on Databricks.
  • Optimize Spark jobs and queries for performance, latency, and cost.
  • Collaborate with the client’s trading and technology teams on market and trade data.
  • Represent the consultancy on-site, ensuring delivery meets production standards.

Requirements

  • Advanced experience with Apache Spark, including performance tuning.
  • Proficiency in Spark Structured Streaming in production environments.
  • Strong knowledge of Spark SQL and query optimization.
  • Core experience with the Databricks platform.
  • Strong programming skills in Python/PySpark.
  • Previous experience in trading or financial markets is essential.
OIR35+ members get up to 3 days early access to new jobs
SkillsData Engineering
TechnologiesDatabricks, Spark, SQL, Python, PySpark
LocationLondon Area
TypeOn-site
Rate
£600/day
PostedNEW