Overview
The Staff Data Engineer will play a crucial role in a technology consultancy, focusing on leveraging Databricks to enhance data processing and governance capabilities. They will work closely with cross-functional teams to implement and optimize data infrastructure and processes, ensuring efficient scaling and monitoring of systems.
Responsibilities
- Develop and optimize data infrastructure using Databricks and Spark.
- Implement Continuous Integration/Continuous Deployment (CI/CD) processes for asset deployment.
- Manage infrastructure as code for scalability and governance.
- Optimize Delta table formats for performance and efficiency.
- Oversee Kafka infrastructure management and monitoring at scale.
- Ensure effective access management and observability of data systems.
Requirements
- Proven experience with Databricks and FinOps methodologies.
- Strong skills in CI/CD processes and infrastructure as code.
- Expertise in Spark optimization techniques.
- Experience with Delta table format tuning.
- Familiarity with Kafka management, including MSK and schema registry.
- Capability in observability and scaling of data systems.