Overview
As a Data Engineer, you will be an integral part of a delivery team focused on leveraging the Microsoft AI Stack for a consultancy partner. The role involves building and optimizing data services and pipelines in a regulated environment, making it essential to have active SC Clearance. You will collaborate with other engineering and data professionals to ensure effective data management and service delivery.
Responsibilities
- Develop and implement data services and pipelines using Python and SQL.
- Design and optimize ETL/ELT data pipelines at scale.
- Build and maintain data platforms on Microsoft Azure technologies such as ADF and Synapse.
- Integrate data platforms with AI and ML workloads.
- Utilize Azure DevOps to automate delivery and pipeline management.
- Ensure compliance with data governance and security standards in regulated environments.
- Work effectively within a hybrid team structure to achieve project goals.
Requirements
- Strong proficiency in Data Engineering with Python and SQL, including PySpark and Spark experience.
- Proven experience in delivering data platforms on Microsoft Azure.
- Strong knowledge of data modeling and data architecture methods.
- Experience in building and optimizing ETL/ELT data pipelines.
- Familiarity with CI/CD practices in data engineering using Azure DevOps.
- Experience in regulated and sensitive environments with a focus on data governance.
- Current and active SC clearance is mandatory for this role.