Overview
The Data Engineer will play a crucial role in managing and optimizing data production pipelines within a dynamic organization specializing in private markets technology. The contractor will collaborate with product, valuation, and client-facing teams to ensure the delivery of accurate and timely data, while leveraging technologies such as Azure Databricks and Snowflake. This position requires hands-on experience in data ingestion, transformation, and quality management, along with a focus on automation and CI/CD processes.
Responsibilities
- Own the development and maintenance of end-to-end production pipelines for data ingestion, transformation, and delivery.
- Redefine the architecture across ingestion, processing, and delivery to enhance stability and speed.
- Build and manage batch and streaming data pipelines from various sources including databases and APIs.
- Implement change data capture and handle complex data operations such as ordering and incremental loads.
- Establish data contracts, reconciliation processes, and data quality measures before publication.
- Manage GitHub workflows and CI/CD processes for data projects, including testing and deployment.
- Investigate and resolve production data failures to maintain data integrity.
- Translate requirements from diverse teams into efficient, functional data pipelines.
Requirements
- Experience with Snowflake or Databricks, including data modeling and performance tuning.
- Strong proficiency in Python and SQL for building and testing data pipelines.
- Hands-on experience with change data capture and event processing workflows.
- Proficiency with Azure services, particularly Databricks and ADLS.
- Experience in managing GitHub workflows and CI/CD practices for data workloads.
- Strong background in data quality processes, monitoring, and incident response.
- Familiarity with building secure, multi-tenant data platforms ensuring data protection and access control.
- Knowledge of additional technologies such as Apache Kafka, Airflow, or dbt is an advantage.