Overview
We are seeking an experienced Data Pipeline Engineer to design and build an open-source Data Transformation Layer for a national data exchange platform. This hands-on role will involve working with a government client to create reusable and reliable data pipelines, utilizing technologies such as Apache NiFi, RDF, SPARQL, Python, and Java. The contractor will collaborate with architects and stakeholders to translate high-level requirements into production-ready solutions while ensuring secure data integration and transformation.
Responsibilities
- Design and evolve the architecture of a reusable, cloud-agnostic data transformation platform.
- Build, test and maintain Apache NiFi flows including integrations with APIs, databases, and files.
- Develop pipelines to ingest, transform, validate and publish RDF triples using SPARQL for queries and updates.
- Develop supporting components and integrations using Python and Java.
- Define data mappings, interface contracts, technical specifications, and architectural decisions.
- Implement robust processing, including error handling, retries, and recovery.
- Establish automated testing, monitoring, version control, and repeatable deployment practices.
- Produce clear technical documentation and support the delivery of reusable platform capabilities.
Requirements
- Proven experience designing and delivering data integration or ETL solutions using Apache NiFi.
- Practical project experience with RDF triples, SPARQL, and semantic data modelling.
- Hands-on development experience in both Python and Java.
- Experience designing reusable pipelines across APIs, databases, files, and other data sources.
- Understanding of reliable data processing, validation, automated testing, and operational monitoring.
- Experience with version control, CI/CD, secure integrations, and repeatable deployments.
- Ability to translate high-level requirements into technical designs and production-ready solutions.
- Familiarity with Docker, Kubernetes, and event-driven data integration is a plus.