Data Engineer

Job Purpose

The Data Engineer is responsible for data ingestion, orchestration, transformation, and storage of data, making this available for analysis and downstream consumption.

The role will develop jobs to ingest data from external data sources, configure data orchestration processes to manage data flows, manage performance for data storage and retrieval and implement continuous improvement processes to manage data quality and reliability of data management technology.

Key Responsibilities

Reporting to the Data Lead, the Data Engineer will be responsible for the following:

  • Engaging with stakeholders to understand the business requirements and assists in specifying possible solutions.
  • Analysing existing enterprise data sets and source systems to determine most appropriate data sources to feed the solution, any data gaps, and quality of data sets.
  • Create and maintain optimal data pipeline architecture,
  • Assemble large, complex data sets that meet functional / non-functional business requirements.
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS technologies.
  • Build analytic tools that utilize the data pipeline to provide actionable insights for commercial and corporate business requirements.
  • Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs.
  • Create data tools for analytics and data scientist team members that assist them in building and optimising data products.
  • Work with data analysts to improve overall functionality in LSSA’s data systems

Key Capabilities

  • Data Engineering
    • Experience in using Data Orchestration tools to manage data flows e.g Apache AirFlow
    • Understanding of Directed Acyclic Graphs (DAGs) to manage the relationship between nodes (workflow jobs) and how they relate (edges) to manage workflow orchestration.
    • Experience in using Change Data Capture (CDC) pattern to track delta changes from data sources and tools used to implement this e.g. AWS DMS.
    • Understanding of stream and batch-based ingestion technologies e.g. Kafka, Kinesis, AWS DMS
  • Client Focus
    • Utilises a wide network of relationships internally and externally to ensure data visualisation and data consumption outputs are of high quality and targeted to current and potential requirements.
    • Understanding of the industries RDS operates, ensuing analytical outputs are market orientated and commercially appropriate.
  • Data Management
    • Applies current analytical tooling including SQL, Python and Spark to generate analytical structures for data consumption.
    • Understand common data modelling formats, including Slowly Changing Dimensions, Wide Table format and Dimensional Modelling.
    • Ability to use complex data sets and being able to sense check the outcomes.
    • Analyses source data to validate requirements and ensure data quality.
  • Operational Effectiveness
    • Can foresee issues and takes ownership and follows through and does not minimize problems.
    • Reviews work and outcomes to identify ways to improve personal or team outcomes and anticipates the impacts and risks of decisions and actions.

Desirable Capabilities or Experience

  • Experience in using cloud platforms to create analytical workloads e.g. AWS, Azure.
  • Understanding of software development practices and use of Application Lifestyle Management tools e.g. Atlassian Suite, Azure DevOps
  • Understanding of agile delivery methods e.g. Kanban, Scrum.
  • Understand APIs and secure transport mechanisms to transport large data sets e.g. sFTP.
  • Understanding of Machine Learning for use in advanced analytics.

Special Conditions

  • Must be prepared to work outside normal business hours from time to time.
  • Interstate and intrastate travel may be required.
  • Satisfactory National Police Clearance Check.