Overview
We are seeking a hands-on Data Engineer with extensive experience in building data platforms for marketing and sustainability analytics products. This role will involve constructing data pipelines from various sources, ensuring data quality, and delivering reliable datasets to internal teams and clients through visualization tools. The successful candidate will collaborate with senior engineers on architectural decisions and handle specific components of the data platform while joining an on-call rotation for production support.
Responsibilities
- Build and run data pipelines on Azure and Databricks.
- Write transformation logic using PySpark and SQL.
- Schedule, monitor, and troubleshoot data pipelines.
- Ensure data security and governance in a multi-tenant setup using Unity Catalog.
- Collaborate with product and engineering teams on new platform features.
- Participate in an on-call rotation for production pipelines.
Requirements
- 5+ years of experience as a Data Engineer with a focus on production pipelines.
- Proficiency in PySpark and SQL.
- Experience with Azure and Databricks, preferably including Unity Catalog.
- Familiarity with feeding BI tools such as Power BI or Tableau.
- Strong understanding of engineering practices like Git, CI/CD, testing, and code reviews.
- Ability to work effectively with non-technical and client-facing teams.
- Nice to have experience with Airflow, Kubernetes/AKS, Okta or Entra ID, and ESG/carbon data exposure.