Overview
We are looking for an experienced Databricks Consultant to enhance and optimize our trading data platform for the Energy and Commodities Trading sector. The consultant will collaborate with trading desks, quantitative analysts, and engineering teams to develop scalable, real-time data pipelines that support trading decisions and risk management. This role primarily focuses on leveraging the capabilities of the Databricks Lakehouse architecture.
Responsibilities
- Build and optimize ETL/ELT pipelines using Databricks, Spark (PySpark/Scala), and Delta Lake for both structured and unstructured energy market data.
- Partner with Quants, Risk Managers, and Traders to implement analytical models and risk metrics including position tracking and curve generation.
- Tune clusters and SQL warehouses to ensure optimal performance for time-sensitive trading environments.
- Enforce data governance and compliance with UK and EU financial regulations through Unity Catalog.
- Provide best practices on architectural design and mentor engineering staff.
Requirements
- Proven experience in energy trading environments, such as firms, banks, utilities, or oil/gas corporations.
- Strong understanding of physical and financial energy markets, including Power, Gas, LNG, Crude, and Carbon/Renewables.
- Hands-on expertise with Databricks (Azure/AWS), including PySpark, Spark SQL, Delta Lake, and Unity Catalog.
- Experience with orchestration tools such as Airflow, Workflows, or ADF, and streaming technologies like Kafka or Event Hubs.
- Excellent communication skills, with the ability to interface effectively with traders and senior stakeholders.