Overview
We are seeking skilled Data Engineers to join our fast-growing consultancy firm, focusing on building and designing data systems across diverse sectors such as telecoms, finance, healthcare, and more. In this role, you will work collaboratively with teams to develop and maintain data pipelines, ensuring high data quality and integrity, while leveraging advanced technologies like Azure and SQL. Ideal candidates will be proactive problem-solvers, eager to contribute their expertise in a dynamic environment that values continuous learning and innovation.
Responsibilities
- Develop and maintain data pipelines using Azure Data Factory (ADF).
- Utilize SQL and Python to extract, transform, and load data into Azure Databricks and Azure SQL/SQL Server.
- Design and implement metadata-driven pipelines for automated data processing.
- Collaborate with cross-functional teams to understand and fulfill data requirements.
- Ensure data quality, integrity, and security throughout the data pipeline lifecycle.
- Troubleshoot and resolve issues or errors in data pipelines.
- Stay updated with the latest Azure technologies and best practices in data engineering.
Requirements
- Solid experience in Azure services, specifically Azure Databricks and Azure SQL/SQL Server.
- Proficiency in SQL and Python programming languages.
- Hands-on experience with Azure Data Factory (ADF) for data pipeline design and implementation.
- Familiarity with data formats like JSON, CSV, and Parquet.
- Understanding of IT concepts including security, IAM, and network infrastructure.
- Exposure to Apache Airflow and DBT is advantageous.
- Experience with Agile methodologies and Azure DevOps pipelines is preferred.
- Certifications in Azure or related technologies are a plus.