Overview
We are seeking an experienced Databricks Data Engineer specializing in AI and Governance to assist in scaling a Databricks platform for supporting AI and GenAI use cases. This remote contract position focuses on hands-on delivery rather than certifications, ensuring that the platform is properly governed from inception. You will collaborate on implementing Unity Catalog as a governance layer for data, machine learning models, and AI assets while adhering to robust governance practices.
Responsibilities
- Implement and manage Unity Catalog as a governance layer for data and AI assets.
- Design and migrate Hive metastores to Unity Catalog, ensuring effective catalog and schema structure.
- Establish fine-grained access controls using row filters and attribute-based access controls.
- Monitor Databricks Lakehouse data quality and integrity expectations.
- Ensure secure and auditable AI pipelines from data ingestion to model serving.
- Collaborate on Mosaic AI Model Serving, focusing on access controls and usage tracking.
- Deploy and maintain governance policies for CI/CD with Databricks Asset Bundles.
Requirements
- Hands-on experience in Databricks engineering or architecture on Azure, AWS, or GCP.
- Proven track record in regulated environments, such as financial services or healthcare.
- Knowledge of Unity Catalog implementation, data classification, and sensitive data tagging.
- Experience with AI governance and project implementation in production settings.
- Ability to articulate governance designs and decisions made during data engineering projects.
- Understanding of governance over AI pipelines and agent frameworks, including Mosaic AI and Agent Bricks.