Overview
The Artificial Intelligence Engineer will focus on leveraging Databricks to design, develop, and optimize machine learning workflows in a remote setting for a consultancy client. This role involves collaborating with various teams to operationalize machine learning models and improve platform tooling and practices. It offers the chance to work on impactful projects that utilize advanced cloud-based systems and machine learning technologies.
Responsibilities
- Lead the design and optimization of scalable machine learning workflows using Azure Databricks.
- Build and deploy robust ML pipelines leveraging Delta Lake and MLflow.
- Collaborate with data scientists and engineers to operationalize machine learning models.
- Champion the use of Databricks-native features for improved model lifecycle management.
- Migrate legacy model workflows into unified Databricks pipelines.
- Ensure best practices in model reproducibility and security within Databricks.
- Act as a subject matter expert on Databricks ML capabilities and architectures.
- Mentor peers and junior engineers on MLOps and Databricks workflows.
Requirements
- Deep hands-on experience with Azure Databricks, especially ML solutions using Delta Lake and MLflow.
- Strong programming skills in Python, including experience with ML libraries like scikit-learn and PySpark.
- Experience orchestrating end-to-end ML pipelines from data preparation to deployment.
- Solid understanding of MLOps principles, including model versioning and CI/CD.
- Familiarity with Azure cloud services such as Azure Data Lake and Azure Machine Learning.
- Knowledge of data governance and compliance in ML workflows.
- Strong problem-solving skills for debugging distributed ML pipelines.
- Proven track record of deploying machine learning models in enterprise environments.