Overview
The Senior Data Engineer will join a remote project team focused on transforming high-volume data into valuable products and real-time analytics for a major insurance provider. This role requires expertise in Palantir Foundry and will involve designing and implementing data pipelines, ensuring data quality, and collaborating within an Agile team to meet project goals. The ideal candidate will have a strong background in data engineering and experience in the insurance domain is a plus.
Responsibilities
- Design, build, and maintain data pipelines in Palantir Foundry.
- Write production-grade Python, PySpark, and SQL for ELT processes.
- Implement data warehousing patterns ensuring data quality and lineage.
- Translate business requirements into robust data products using Foundry Ontology.
- Diagnose and resolve data pipeline issues; document workflows for efficiency.
- Collaborate in Scrum with cross-functional teams to deliver solutions.
Requirements
- Strong experience in data engineering for large-scale distributed systems.
- Hands-on development experience with Palantir Foundry.
- Deep proficiency in PySpark and Spark SQL.
- Strong understanding of ELT processes and data warehousing concepts.
- Experience working within Agile environments as a clear communicator.
- Bonus: background in the insurance or financial services industry.