Overview
As a Databricks Data Engineer, you will be involved in designing and building production workloads using Databricks within a leading consulting firm. You will collaborate with cross-functional teams to develop data pipelines and leverage your expertise in data engineering, ensuring efficient data processing and storage. This role provides a unique opportunity to work mostly remotely while occasionally engaging with clients on-site.
Responsibilities
- Develop and optimize data pipelines using Databricks and Apache Spark.
- Implement ETL/ELT processes to handle large, distributed datasets.
- Perform performance tuning of data workflows to minimize costs and enhance efficiency.
- Design reliable data solutions utilizing Delta Lake and structured streaming.
- Ensure data integrity through robust deduplication and CDC handling processes.
- Collaborate with stakeholders to understand data requirements and deliver scalable solutions.
- Maintain up-to-date knowledge of Databricks features and functionalities.
Requirements
- 5-7 years of experience in data engineering with a minimum of 4 years focused on Databricks.
- Databricks Certified Data Engineer Professional certification is mandatory.
- Strong proficiency in Python, SQL, PySpark, and Pandas.
- Experience in cloud platforms like AWS, Azure, or GCP is advantageous.
- Expertise in data warehousing, lakehouse design, and dimensional modeling.
- Proficient in batch and streaming data pipeline development.
- Knowledge of performance tuning techniques and data quality management.