Overview
We are looking for a Senior Data Engineer to join a dynamic team on a contract basis. This role involves collaboration with the Head of Data and Lead Data Scientist to help define and execute the data strategy for our client, focusing on developing high-value projects and ensuring compliance with enterprise security standards. The contractor will be responsible for implementing data engineering solutions in a fast-paced environment, utilizing skills in Python, Dagster, and Terraform to deliver end-to-end data pipelines.
Responsibilities
- Partner with the Head of Data and Lead Data Scientist to define the data lakehouse strategy and architecture.
- Translate business objectives into detailed Technical Design Documents (TDDs) that adhere to security and compliance standards.
- Establish data governance, lineage tracking, and isolation for workflows.
- Define JIRA Epics and user stories to guide implementation based on approved TDDs.
- Architect, code, and deploy robust data pipelines using Python and Dagster.
- Manage deployments and infrastructure on AWS, ensuring optimal data storage solutions.
Requirements
- 4+ years of experience in Python distributed data engineering and backend development.
- Hands-on experience with Dagster or Airflow in a Medallion or data lakehouse environment.
- Proficient in using Terraform to manage AWS resources including EKS and Aurora PostgreSQL.
- Understanding of security frameworks such as SOC2 or ISO27001.
- Familiarity with multi-agent frameworks and libraries is a plus.
- Experience in building RAG pipelines or deploying LLM inference pipelines is beneficial.