Overview
We are seeking a skilled Data Architect with expertise in Databricks to develop a robust data architecture on AWS. This remote role involves collaborating with a team of ETL engineers new to Databricks, ensuring the effective design and governance of data pipelines and overall architecture.
Responsibilities
- Translate project requirements into technical designs that deliver high-performance, cost-effective pipelines.
- Bring together source data from files and databases into a single AWS and Databricks platform.
- Ensure data is delivered in line with predefined silver and gold models in a Medallion architecture.
- Make optimal choices regarding Databricks configurations, including storage modes and compute clusters.
- Define governance standards, including roles and responsibilities for Databricks and downstream users.
- Collaborate with other architects to establish best practices for data quality and observability.
- Mentor data engineers who are new to Databricks through hands-on guidance and training.
Requirements
- Strong AWS experience.
- Deep hands-on Databricks experience, including performance and cost optimization.
- Proven experience delivering Medallion architecture.
- Strong background in ETL processes.
- Experience in setting data governance standards.
- Ability to mentor and upskill engineering teams.
- Desirable: Unity Catalog, PySpark and Spark performance tuning, Delta Lake, and experience with data quality and observability tooling.