Overview
We are seeking an experienced Data Engineer to contribute to a dynamic data transformation programme on a 12-month, fully remote contract. The successful candidate will be responsible for building and optimizing scalable data solutions while leveraging modern technologies such as Databricks, PySpark, Python, and SQL. This role offers the opportunity to engage with production-level data pipelines and implement best practices in data management and delivery.
Responsibilities
- Design, develop and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
- Build, optimise and troubleshoot production-grade ETL/ELT pipelines.
- Implement workflow orchestration and dependency management.
- Develop robust retry, restart and failure-recovery mechanisms.
- Implement CI/CD pipelines, Git version control, and automated deployments.
- Establish monitoring, logging, and operational alerting.
- Ensure data quality, reconciliation, and reliability.
- Develop integrations using APIs, file transfers, and SFTP.
Requirements
- Strong hands-on Databricks experience.
- Advanced skills in PySpark, Python, and SQL.
- Experience in production pipeline engineering and troubleshooting.
- Knowledge of workflow orchestration and dependency management.
- Familiarity with CI/CD processes, Git, and automated deployment.
- Understanding of data-quality controls, monitoring, and logging.
- Experience in API, file, and SFTP integrations.
- Desirable knowledge of Delta Lake, Azure DevOps, MLOps, and Salesforce Marketing Cloud.