Overview
The AI Data Engineer will play a critical role in the adoption of AI within enterprise settings, focusing on the transformation of unstructured data into structured, agent-ready formats that AI systems can utilize. This senior position will work closely with data engineers and AI teams to deliver effective data solutions, ensure data quality, and contribute to the overall growth of the engineering team’s capabilities. The role involves leveraging deep technical expertise, mentoring others, and actively participating in the continuous improvement of data engineering practices.
Responsibilities
- Design and implement data pipelines that convert unstructured enterprise data into structured formats suitable for AI consumption.
- Develop and optimize chunking and parsing strategies for effective data retrieval by AI systems in production.
- Build and maintain ingestion pipelines for AWS Bedrock Knowledge Bases and related infrastructure.
- Monitor and instrument the data layer for quality, freshness, and access control metrics.
- Work collaboratively within a Kanban delivery team, managing priorities and communicating progress effectively.
- Coach and mentor data engineers transitioning to AI data engineering practices, reviewing their work for quality and adherence to standards.
- Document pipeline logic, data contracts, and transformation decisions for team knowledge sharing.
- Raise engineering standards and contribute to organizational practices surrounding data layer management in AI.
Requirements
- Proven background in data engineering with expertise in AI data consumption patterns.
- Hands-on experience with AWS services, including S3, Glue, Lambda, and AI services like Bedrock.
- Ability to process unstructured data into structured formats suitable for production AI use.
- Experience with multiple tools and companies, showcasing adaptability in problem-solving.
- Deep understanding of retrieval mechanisms and data quality engineering in AI environments.
- Proficient in building and maintaining end-to-end data retrieval pipelines.
- Comfortable working in a Kanban-based workflow and managing independent priorities.
- Familiarity with data governance principles, particularly in sensitive or regulated contexts.