Overview
The Senior Data Engineer will play a critical role in the design, build, and delivery of innovative data solutions for enterprise-level clients, focusing on the Databricks platform. You will work collaboratively with various stakeholders, including data scientists and business teams, while providing technical leadership and mentoring to junior engineers. This role requires a hands-on approach and is ideal for a seasoned professional looking to make a significant impact in data engineering.
Responsibilities
- Design and build end-to-end data solutions on Databricks using Spark, Python, SQL, and Delta Lake.
- Apply software engineering best practices, including TDD, CI/CD, version control, automation, and clean coding principles.
- Work across the entire software development lifecycle, from design to deployment.
- Engage directly with stakeholders to shape requirements and influence technical direction.
- Lead delivery of complex data engineering initiatives, either independently or with small teams.
- Provide technical leadership, mentoring others, reviewing code, and driving best practices.
- Collaborate with data scientists, architects, and business teams to deliver production-grade outcomes.
Requirements
- Deep hands-on experience with Databricks, including SQL, PySpark, Delta Lake, Unity Catalog, and Workflows.
- Strong proficiency in Python and Spark.
- Solid understanding of CI/CD pipelines, DevOps, and Infrastructure as Code.
- Proven track record in designing and delivering solutions from scratch.
- Excellent communication and consulting skills, comfortable working with both technical and business stakeholders.
- Strong problem-solving ability, commercial awareness, and proactive approach to delivery.