Overview
We are seeking an experienced Databricks Consultant to lead the design, implementation, and optimization of a cloud-based data platform. This hands-on contractor role focuses on deploying enterprise-level Databricks solutions and will involve collaboration with internal teams to ensure best practices in data governance and technical documentation.
Responsibilities
- Design and implement scalable Databricks Lakehouse solutions.
- Build and optimize ETL/ELT pipelines using PySpark and SQL.
- Implement Delta Lake, Unity Catalog, and Medallion Architecture.
- Develop batch and streaming data ingestion pipelines.
- Configure CI/CD, Infrastructure as Code, and automated deployments.
- Optimize Spark performance, security, and cost.
- Implement data governance and monitoring best practices.
- Produce technical documentation and mentor internal teams.
Requirements
- Proven experience delivering enterprise Databricks implementations.
- Strong Databricks expertise including Delta Lake, Unity Catalog, Workflows, and Delta Live Tables.
- Expert knowledge of PySpark, Python, and SQL.
- Strong Apache Spark performance tuning experience.
- Experience designing modern cloud data platforms and Lakehouse architectures.
- Experience building batch and real-time data pipelines.
- Strong understanding of data modeling, ETL/ELT, and data warehousing concepts.
- Experience implementing CI/CD using Azure DevOps, GitHub, or GitLab.
- Familiarity with Infrastructure as Code using Terraform.
- Strong knowledge of data governance, security, and access control.