Lead AWS Data Engineer

We are seeking an accomplished and detail-oriented Lead Data Engineer – AWS to join our Data & AI practice. The successful candidate will bring deep expertise in data engineering, distributed data processing, and cloud-native platforms, with a strong focus on AWS-based data ecosystems.This role is critical in designing, building, and optimising end-to-end data pipelines and platforms, enabling scalable data processing, advanced analytics, and AI-driven solutions. You will play a key role in ensuring data quality, integrity, performance, and reliability, supported by strong engineering and testing practices.

As a senior practitioner, you will collaborate with architects, engineers, and analysts to deliver secure, scalable, and high-performing data solutions, leveraging technologies such as AWS Glue, Python/PySpark, SQL, and configuration-driven frameworks (e.g., YAML). You will thrive in a collaborative, client-facing environment, with a passion for solving complex technical challenges, ensuring delivery excellence, and driving modernisation through cloud-native engineering practices.

What you'll be doing:

Client Engagement & Delivery

• Act as a senior engineer within data engineering and cloud platform initiatives, supporting delivery across complex transformation programmes
• Collaborate with architects and stakeholders to define and implement scalable AWS-based data solutions
• Contribute to solution design, estimation, and delivery planning
• Lead engineering workstreams and ensure high-quality technical delivery

​Data Engineering & Platform Development

• Design, build, and optimise scalable data pipelines and data processing frameworks on AWS
• Develop and maintain ETL/ELT pipelines using:
  • AWS Glue
  • Python / PySpark
  • SQL
  • Configuration-driven frameworks (e.g., YAML)
• Implement robust data ingestion, transformation, and processing patterns
• Build reusable data services, components, and frameworks

Data Pipeline Testing & Reliability

• Define and implement testing strategies for data pipelines, ensuring reliability and accuracy
• Validate data processing workflows using:
  • Python / PySpark transformations
  • SQL-based validation logic
  • Configuration-driven orchestration
• Develop automated testing, monitoring, and alerting solutions
• Ensure:
  • Data completeness
  • Data accuracy
  • Consistent transformation behaviour
• Drive improvements in observability and pipeline resilience

AWS Data Platforms

• Lead development on AWS services including:
  • AWS Glue
  • S3-based data lakes
  • Supporting services within the AWS data ecosystem
• Support implementation of modern data architectures, including data lakes and Lakehouse-style platforms
• Optimise pipelines and jobs for performance, scalability, and cost efficiency

Data Transformation & Modelling

• Define and implement data transformation logic aligned to business requirements
• Support data modelling approaches for analytics and platform use cases
• Ensure consistency, usability, and quality across data assets and pipelines

Collaboration & Technical Leadership

• Collaborate with:
  • Solution Architects
  • Data Engineers
  • Analysts and ML engineers
• Provide technical leadership and mentoring to engineers within the team
• Promote engineering best practices, automation, and reusable solutions
• Contribute to engineering standards, documentation, and knowledge sharing

Quality, Governance & Security

• Ensure data quality, integrity, and reliability across data platforms
• Implement and enforce secure coding and data handling practices
• Support compliance with:
  • GDPR
  • Regulated environment standards (where applicable)
• Contribute to monitoring, auditing, and operational processes

What experience you'll bring:

• Proven experience in data engineering and cloud-based platform delivery
• Strong understanding of distributed data processing and scalable system design
• Ability to lead delivery while remaining hands-on technically
• Strong analytical, problem-solving, and communication skills
• Experience working in client-facing and delivery-focused environments
• Ability to mentor and develop engineering teams

Technical Expertise

• Strong hands-on experience with:
  • AWS cloud services, especially AWS Glue
  • Python / PySpark for large-scale data processing
  • SQL for querying, transformation, and validation
  • Configuration-driven development (e.g., YAML)
• Experience building and operating:
  • Data pipelines
  • ETL/ELT workflows
  • Cloud-native data platforms
• Familiarity with:
  • Data lakes and Lakehouse concepts
  • Distributed processing frameworks (e.g., Apache Spark)
• Strong understanding of:
  • ETL vs ELT patterns
  • Performance tuning and optimisation
• Experience with:
  • Version control (Git)
  • CI/CD and DevOps practices

Who we are:

We’re a business with a global reach that empowers local teams, and we undertake hugely exciting work that is genuinely changing the world. Our advanced portfolio of consulting, applications, business process, cloud, and infrastructure services will allow you to achieve great things by working with brilliant colleagues, and clients, on exciting projects.

Our inclusive work environment prioritises mutual respect, accountability, and continuous learning for all our people. This approach fosters collaboration, well-being, growth, and agility, leading to a more diverse, innovative, and competitive organisation. We are also proud to share that we have a range of Inclusion Networks such as: the Women’s Business Network, Cultural and Ethnicity Network, LGBTQ+ & Allies Network, Neurodiversity Network and the Parent Network.

For more information on Diversity, Equity and Inclusion please click here: Creating Inclusion Together at NTT DATA UK | NTT DATA

What we'll offer you:

We offer a range of tailored benefits that support your physical, emotional, and financial wellbeing. Our Learning and Development team ensure that there are continuous growth and development opportunities for our people. We also offer the opportunity to have flexible work options.

You can find more information about NTT DATA UK & Ireland here: https://uk.nttdata.com/

We are an equal opportunities employer. We believe in the fair treatment of all our employees and commit to promoting equity and diversity in our employment practices.

We are also a proud Disability Confident Committed Employer - we are committed to creating a diverse and inclusive workforce. We actively collaborate with individuals who have disabilities and long-term health conditions which have an effect on their ability to do normal daily activities, ensuring that barriers are eliminated when it comes to employment opportunities. In line with our commitment, we guarantee an interview to applicants who declare to us, during the application process, that they have a disability and meet the minimum requirements for the role.

If you require any reasonable adjustments during the recruitment process, please let us know. Join us in building a truly diverse and empowered team.