Overview
The Senior Data Platform Lead will oversee the operational health and performance of a production-focused data environment, ensuring the stability and resilience of critical business applications. This role requires a technical leader who will take ownership of data flow incidents and drive swift diagnosis and recovery across complex data pipelines. The candidate will also collaborate with various teams to enhance service reliability while participating in a support model that ensures continuous operation.
Responsibilities
- Lead technical diagnosis and recovery for data flow incidents affecting business applications.
- Own and manage platform health, monitoring, performance, and operational resilience.
- Provide hands-on leadership in the use of PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
- Establish standards for Unity Catalog and access controls, ensuring data lineage and smooth production releases.
- Collaborate with engineering, application, service management, and vendor teams to resolve complex issues.
- Enhance runbooks, recovery procedures, and architectural documentation to facilitate operational readiness.
- Transfer knowledge to team members and support their professional growth.
- Participate in a 17x5 support model as per briefing arrangements.
Requirements
- Strong experience supporting Azure Databricks in a production environment, including incident management.
- Hands-on proficiency with PySpark, SQL, Delta Lake, Workflows, and Unity Catalog.
- Familiarity with Azure Data Lake Storage Gen2, Azure DevOps, Git, CI/CD, and controlled production changes.
- Capability to troubleshoot data flows from source systems through to end consumers.
- Experience integrating ERP data (e.g., JD Edwards, Oracle ERP, SAP, Dynamics).
- Confidence to lead major incidents, articulate business impacts, and make recovery decisions under pressure.
- Desirable experience in Qlik Replicate, CDC, and supply chain data management.
- A clear example of a past Databricks incident that was diagnosed and resolved independently is preferred.