Overview
The Data Engineer will play a critical role in a long-term project for a Government consultancy, focusing on data architecture and processing using various cloud technologies. This position is fully remote and requires expertise in data engineering tools and frameworks, ensuring strong data pipelines and solutions are developed effectively. The role involves collaboration with technical teams to drive data strategy and infrastructure improvements.
Responsibilities
- Design and implement data processing solutions using Azure, AWS, or other cloud platforms.
- Develop and maintain data pipelines with Azure Data Factory, Synapse, or Databricks.
- Utilize Apache Spark and PySpark for data transformation and analysis.
- Integrate event-streaming technologies such as Kafka into data architecture.
- Manage containerization and orchestration using Docker and Kubernetes.
- Apply Terraform or Infrastructure as Code practices for deployment automation.
- Implement CI/CD pipelines with Azure DevOps, GitHub Actions, or Jenkins.
- Optimize data storage solutions with data lakes and lakehouse architectures.
Requirements
- Proficient in Microsoft Azure and/or AWS cloud platforms.
- Experience with Azure Data Factory, Synapse, or Databricks.
- Strong background in Apache Spark and PySpark.
- Familiarity with Snowflake and Kafka or other event-streaming technologies.
- Knowledge of containerization tools like Docker and Kubernetes.
- Experience with Infrastructure as Code using Terraform.
- Proficient in CI/CD tooling including Azure DevOps, GitHub Actions, or Jenkins.
- Familiarity with database systems such as PostgreSQL, SQL Server, or Oracle.