Overview
The contract role of Senior Data Engineer focuses on enhancing data management and utilization within the Energy Trading sector. Collaborating with the Data Team, the engineer will be instrumental in developing automated data pipelines and improving the overall data infrastructure. This position demands a comprehensive understanding of power, gas, and commodities trading as well as proficiency in Python to ensure data is effectively processed and analyzed.
Responsibilities
- Develop, construct, test and maintain data pipelines (ETL/ELT).
- Design, implement and maintain data quality and assurance tools.
- Perform data profiling, testing and monitoring.
- Understand requirements from data owners and implement data quality KPIs and monitoring pipelines.
- Operationalize data management policies and automate data governance KPIs.
- Cleansing, pre-processing and delivery of data for prescriptive and predictive modeling.
- Develop data visualization and Business Intelligence dashboards.
- Take ownership of assigned projects, including requirements gathering and solution design.
Requirements
- Established background in Data Engineering, particularly within Energy/Commodities Trading (Power, Gas, LNG, Oil, Ags, Metals).
- Experience with creating and managing data pipelines (ETL/ELT).
- Proficiency in Python and PySpark.
- Hands-on experience with SQL.
- Familiarity with Pandas and/or Spark Dataframes.
- Experience with structured, semi-structured, and unstructured data.
- Familiarity with data testing frameworks is beneficial.
- Experience with data lake technology is advantageous.