Job Purpose

Data Engineers build and support data pipelines and datamarts built off those pipelines. Both must be scalable, repeatable and secure. The Data Engineer helps to facilitate gathering data from a variety of different sources, in the correct format, assuring that it conforms to data quality standards and assuring that downstream users can get to that data timeously. This role functions as a core member of an agile team.

Job Objectives

– Design and develop data feeds from an on-premise environment into a datalake environment in an AWS cloud environment – Design and develop programmatic transformations of the solution, by correctly partitioning, formatting and validating the data quality – Design and develop programmatic transformation, combinations and calculations to populate complex datamarts based on feed from the datalake – Provide operational support to datamart datafeeds and datamarts

Qualifications and Experience

– IT Degree/Diploma (3 years) – Business Intelligence (3-5 years) – Extract Transform and Load (ETL) processes (3-5 years) – Agile exposure, Kanban or Scrum (2+ years)

Knowledge and Skills

– Creating data feeds from on-premise to AWS Cloud (1 year) – Support data feeds in production on break fix basis (1 year) – Creating data marts using Talend or similar ETL development tool (2 years) – Manipulating data using python and pyspark (1 year) – Talend (1 year) – Python (1 year) – Business Intelligence Data modelling (3 years) – SQL (3 years)