Overview
We are seeking a Senior Databricks Data Engineer to join a dynamic, private equity-backed SaaS company on an initial three-month contract with the potential for extension. In this fully remote role, you will play a crucial part in designing, building, and optimizing a modern lakehouse architecture to enhance analytics and data-driven decision-making across the organization. You will collaborate closely with various stakeholders and take on significant responsibilities from day one.
Responsibilities
- Design and maintain scalable, reliable data pipelines on Databricks using PySpark and Spark SQL.
- Implement and evolve a lakehouse architecture on Delta Lake, adhering to Medallion principles.
- Ingest data from various sources in batch and near real time.
- Build and optimize ETL/ELT workflows using Databricks Workflows and Delta Live Tables.
- Ensure data quality through monitoring, alerting, and observability practices.
- Manage compute costs and performance tuning across clusters and jobs.
- Apply Unity Catalog for data governance, access control, and secure sharing.
- Contribute to CI/CD practices and document solutions for knowledge transfer.
Requirements
- Significant commercial experience as a Data Engineer with hands-on Databricks delivery.
- Strong proficiency in Python and PySpark, with advanced SQL capabilities.
- Deep understanding of Delta Lake and Spark performance optimization.
- Experience with Unity Catalog for governance and security.
- Proven experience building and orchestrating pipelines using Databricks or similar tools.
- Familiarity with major cloud platforms (Azure, AWS or GCP).
- Solid grounding in data modelling and data warehousing concepts.
- Experience with Git and CI/CD pipelines such as Azure DevOps or GitHub Actions.