Overview
We are seeking a skilled Data Engineer to join a dynamic data and technology business in the private markets sector on a contract basis. The contractor will play a critical role in managing production data pipelines from source capture to delivery while ensuring data integrity and timely processing. Collaborating closely with product teams, the Data Engineer will also implement and optimize data architecture for advanced use cases, contributing to both performance and reliability of the platform.
Responsibilities
- Own the production data pipelines from source capture through to client delivery.
- Assist in redefining the architecture for data ingestion, processing, and delivery.
- Build and manage batch and streaming data pipelines from various sources.
- Implement change data capture and incremental loads for stable data operations.
- Maintain and enhance the GitHub workflows and CI/CD pipelines for data workloads.
- Investigate and resolve production data failures efficiently.
- Transform product and client requirements into robust, operational pipelines.
- Ensure data quality, reconciliation, and monitoring are in place before data publication.
Requirements
- Proficiency in Snowflake or Databricks, including data modeling and performance tuning.
- Strong skills in Python and SQL for developing and testing data pipelines.
- Experience with change data capture and event processing techniques.
- Familiarity with Azure services, particularly Databricks and ADLS.
- Knowledge of GitHub and CI/CD practices relevant to data engineering.
- Experience building secure, multi-tenant data platforms with appropriate access controls.
- Familiarity with data quality monitoring and incident response processes.
- Preferred knowledge of tools like Debezium, Kafka Connect, or Apache Airflow.