Overview
We are seeking an experienced Databricks Engineer to join our team and focus on building data pipelines and artificial intelligence solutions for a major UK client. This role involves transforming data from various sources into trustworthy datasets, which will be utilized for reporting and AI applications. The engineer will collaborate with architects and business stakeholders while employing advanced technologies in data engineering.
Responsibilities
- Build Databricks pipelines using the Medallion Architecture (Bronze, Silver and Gold layers).
- Configure Unity Catalog to manage permissions, data access and lineage.
- Develop AI solutions and deploy them across Development, Testing, and Production environments using Asset Bundles and CI/CD practices.
- Optimize pipeline performance and manage associated cluster costs.
- Implement data quality checks to identify issues early in the pipeline process.
- Collaborate with architects and business stakeholders to deliver effective technical solutions.
Requirements
- Demonstrated experience with Databricks, including PySpark, Spark SQL, Delta Lake, Delta Live Tables, and Workflows.
- Hands-on experience with Unity Catalog configuration, including schemas, groups, and permissions.
- Successful delivery of at least one Databricks AI solution.
- Strong proficiency in Python and SQL.
- Experience with Azure Data Factory and Azure Data Lake Storage (ADLS).
- Familiarity with Asset Bundles or Terraform and CI/CD deployment methodologies.
- Strong communication skills for collaboration with technical and business teams.