Overview
This role involves working as an Artificial Intelligence Engineer focused on Databricks to enhance machine learning projects for a consultancy client. The engineer will collaborate with various teams to design and optimize scalable ML pipelines, ensuring the delivery of impactful solutions in a cloud environment. This is an excellent opportunity to engage with cutting-edge technology while mentoring peers in best practices for ML engineering.
Responsibilities
- Lead the design and optimization of scalable machine learning workflows using Azure Databricks.
- Build and deploy robust ML pipelines leveraging Delta Lake, MLflow, and Databricks Jobs.
- Collaborate with data scientists and engineers to operationalize ML models at scale.
- Implement best practices for model reproducibility, governance, and monitoring within Databricks.
- Migrate legacy ML workflows to unified Databricks pipelines.
- Act as a subject matter expert on Databricks capabilities and integrations.
- Mentor junior engineers on ML practices focusing on MLOps and Databricks workflows.
- Continuously improve the machine learning tools and deployment practices.
Requirements
- Deep hands-on experience with Azure Databricks and developing ML solutions using Delta Lake and MLflow.
- Strong programming skills in Python along with experience using SQL for data analysis.
- Experience in orchestrating end-to-end ML pipelines, including data preprocessing and model deployment.
- Solid understanding of MLOps principles, including model versioning and CI/CD for ML workflows.
- Familiarity with Azure cloud services such as Azure Data Lake and Azure Machine Learning.
- Knowledge of feature engineering and automated retraining in production environments.
- Proven experience delivering machine learning models in enterprise-scale environments.
- Strong problem-solving skills with the ability to debug distributed ML pipelines.