Data Engineer

Overview

We are seeking a hands-on Senior Data Engineer for a 6-week contract focusing on the creation of a secure, AI-ready data platform. In this remote position, you will collaborate with a high-impact team to design and deliver production-grade data pipelines, enhancing the capabilities of AI applications and autonomous agents. The ideal candidate will have experience in building scalable data pipelines and integrating data platforms with advanced technologies.

Responsibilities

• Build scalable distributed data pipelines.
• Develop batch and streaming ingestion processes.
• Create robust data models for both structured and unstructured data.
• Implement data lineage, provenance, and metadata management.
• Support knowledge graphs, entity resolution, and data fusion.
• Integrate data platforms with AI and agent-based applications.
• Deploy services in secure cloud and containerized environments.

Requirements

• Strong experience with Python and Spark / PySpark.
• Familiarity with Kafka, Flink, Iceberg, and Trino.
• Proficient in data modeling, lineage, and provenance.
• Experience with knowledge graphs and entity resolution.
• Knowledge of AWS or Azure cloud services.
• Experience with Docker and Kubernetes.
• Familiarity with geospatial or time-series data.
• Active SC clearance and immediate availability.