Overview
The role of a Senior Data Engineer focuses on developing and maintaining data infrastructure for a Global Energy Trading company as part of their Digitalisation Program. Collaborating with the Data Team, the contractor will automate and optimise data management processes while engaging in Python-related projects, ultimately enhancing the efficiency of the trading business.
Responsibilities
- Develop, construct, test, and maintain data pipelines (ETL/ELT).
- Design, implement, and maintain data quality and assurance tools.
- Perform data profiling, testing, and monitoring.
- Understand requirements from data owners and implement data quality KPIs and monitoring pipelines.
- Operationalize data management policies and automate data governance KPIs.
- Follow good data governance standards and help define data policies and processes.
- Cleansing, pre-processing, and delivery of data for predictive modelling.
- Collaborate with data scientists and data engineers to build scalable models.
Requirements
- Established Data Engineering background with Energy/Commodities Trading experience (Power, Gas, LNG, Oil, Ags, Metals).
- Experience creating and managing data pipelines (ETL/ELT).
- Fluent in Python and PySpark.
- Hands-on experience with SQL.
- Comfortable with Pandas and/or Spark Dataframes.
- Experience with structured, semi-structured, and unstructured data.
- Familiarity with data testing frameworks is beneficial.
- Experience working with data lake technology is advantageous.