Overview
The Senior / Principal Data Engineer role focuses on enhancing the Databricks engineering capabilities within a dynamic Data & AI consultancy. The appointed contractors will work on various client projects, taking on responsibilities that range from hands-on engineering to strategic technical leadership, depending on their experience level. Collaboration with both technical teams and business stakeholders will be essential to develop robust data solutions.
Responsibilities
- Build production-grade data pipelines using Databricks, PySpark, and Spark SQL.
- Design Delta Lake and Medallion Architecture solutions.
- Develop scalable ingestion and transformation pipelines.
- Implement Unity Catalog for governance, lineage, and access control.
- Establish data engineering standards and best practices.
- Support and mentor junior engineers.
- At the Principal level, own platform architecture and set technical direction.
Requirements
- Strong commercial experience with Databricks, PySpark, and Spark SQL.
- Proficiency in Apache Spark, Delta Lake, and Medallion Architecture.
- Experience with Unity Catalog and Python.
- Familiarity with Terraform / Infrastructure-as-Code and CI/CD practices.
- Knowledge of Azure, AWS, or GCP is preferred.
- Experience with Databricks Asset Bundles, Delta Sharing, and Structured Streaming is beneficial.
- Prior experience as a Principal or Lead Data Engineer or Databricks Architect for Principal-level opportunities.