Overview
We are seeking a skilled Lead Databricks Platform Engineer to join a global software vendor on a contract basis. In this role, you will take ownership of a modern data ecosystem while leading a team of data engineers, driving data engineering excellence and platform optimization. You will implement best practices and ensure the Databricks platform is reliable and scalable, enabling advanced AI capabilities.
Responsibilities
- Lead, coach, and mentor a team of 6+ data engineers in modern, AI-enabled practices.
- Own and optimize the Databricks platform, including Delta Lake, Unity Catalog, and MLflow.
- Redesign data pipelines for maximum scalability, performance, and cost efficiency.
- Champion Python engineering best practices such as CI/CD, testing, and code reviews.
- Drive DataOps and MLOps to ensure observability, governance, and seamless lifecycles.
Requirements
- 5+ years of experience in data engineering, with at least 2 years scaling Databricks environments.
- Deep expertise in Apache Spark, PySpark, Unity Catalog, and MLflow.
- Strong production Python background with robust CI/CD experience.
- Proven track record of optimizing Databricks environments for cost and performance.
- Experience leading engineering teams and establishing modern best practices.