Overview
We are seeking a Principal Data Engineer with extensive experience in Databricks to join a dynamic Tech consultancy on a contract basis. In this role, you will collaborate with various teams to enhance and scale data platform capabilities, focusing on optimizing infrastructure and ensuring robust governance and management practices. The ideal candidate should be available immediately or within two weeks notice.
Responsibilities
- Develop and implement FinOps strategies for Databricks environments.
- Manage infrastructure as code ensuring scalability and efficiency.
- Optimize Spark applications and compute resources.
- Enhance Delta tables for performance and storage efficiency.
- Oversee Kafka infrastructure management and monitor observability at scale.
- Implement CI/CD processes and manage asset bundles deployment.
- Establish access management and platform governance protocols.
Requirements
- Expertise in Databricks with proven FinOps experience.
- Strong knowledge of infrastructure as code tools and practices.
- Experience in Spark compute optimization techniques.
- Proficiency in Delta table format optimization and tuning.
- Deep understanding of Kafka and MSK schema registry management.
- Ability to design and implement observability and monitoring solutions at scale.
- Experience in CI/CD processes for data engineering.