Overview
We are seeking an experienced SC Cleared Data Engineer specializing in Azure Databricks to support a Central Government team in the design and development of high-volume datasets. The contractor will work remotely, utilizing their technical skills to enhance data processing and management capabilities within the Government context.
Responsibilities
- Design and develop high-volume data sets using Azure Databricks.
- Build, automate, and support data pipelines to ensure efficient data flow.
- Implement and maintain Spark-based technologies to enhance data processing.
- Collaborate with stakeholders to define data requirements and project deliverables.
- Utilize CI/CD practices to streamline deployment processes.
- Leverage tools like Docker and Kubernetes for containerization and orchestration.
Requirements
- Extensive hands-on experience with Azure Databricks.
- Recent experience working within UK Central Government.
- Strong proficiency in Python programming.
- Experience with Spark-based technologies in data engineering roles.
- Proven track record of building and supporting data pipelines.
- Familiarity with CI/CD practices, Docker, and Kubernetes.