Overview
We are seeking several experienced Senior Data Engineers with expertise in Databricks to join a consultancy on a fully remote, 12-month contract basis. The contractors will work collaboratively to design, implement, and maintain production pipelines while ensuring data quality and integrity through robust monitoring and logging practices. This role focuses on optimizing workflow orchestration and leveraging CI/CD practices to automate deployments.
Responsibilities
- Develop production pipeline solutions using Databricks, PySpark, and Python.
- Troubleshoot and maintain existing data pipelines to ensure reliability and performance.
- Implement workflow orchestration and manage dependencies effectively.
- Design and apply idempotency, retry/restart, and failure recovery patterns.
- Utilize CI/CD practices with Git for automated software deployment.
- Establish monitoring, logging, and alerting strategies for operational efficiency.
- Create data quality and reconciliation controls to ensure the accuracy of data flows.
- Integrate APIs and manage file/SFTP transfer patterns for data movement.
Requirements
- Proven experience with Databricks and related technologies.
- Strong proficiency in PySpark, Python, and SQL.
- Experience in production data pipeline engineering and troubleshooting.
- Familiarity with workflow orchestration and dependency management.
- Knowledge of CI/CD processes and tools, especially Git.
- Ability to implement robust monitoring and operational alerting mechanisms.
- Experience with data quality controls and reconciliation methodologies.
- Understanding of API, file, and SFTP integration practices.