Description
Salary: £55,403 - £61,939
Working pattern: Flexible working, Full-time, Part-time
Business area: CO - Infected Blood Compensation Authority (IBCA)
Location: Glasgow, Newcastle-upon-Tyne
Job summary
This role will lead our data engineering capability within the engineering team of the Data Operations arm of the IBCA Data Directorate.
The Data Operations team is responsible for developing and running safe and secure data solutions that provide a single source of truth for those going through their compensation journey. They are building a new data platform using Amazon Web Services (AWS) and data management and intelligence products using Databricks, Quantexa and Tableau.
We are taking a product-centric approach treating data as a product and are building squads around our products, with a focus on paying compensation to those impacted by the infected blood scandal seamlessly.
You will focus on building robust data pipelines and data management processes (including master data management), ensuring data quality, and driving a culture of data-driven decision-making.
Working at IBCA gives you a huge opportunity to make an impact on those who deserve compensation, and this role suits a candidate who can lead a team of data engineers to deliver solutions from the ground up to take from ideation to reality so that data is an enabler to everything IBCA does.
Job description
As the Lead Data Engineer, you will join a multidisciplinary team to design, build, and deliver high-impact, scalable AWS-based data solutions using Databricks and Quantexa technologies, while also driving data quality, mentoring junior colleagues, and fostering a strong data engineering community.
You will:
- provide technical guidance for the development of robust, automated data pipelines and master data management processes for our data platform and products, encompassing DevSecOps best practices
- ensure tools and techniques are scalable, secure and efficient
- be responsible for ensuring the right data engineering practices are embedded consistently and to industry standard best practices within the data platform delivery teams
- develop the data engineering capability by developing L&D routeways, shaping careers paths and recruiting talent to the organisation
- provide leadership and guidance to a team of data engineers, further developing your own data engineering and leadership skills
- work with business stakeholders and across digital service teams, understanding their needs and translating them into data development
Main Responsibilities
- Data Solution Delivery: Designing, building, and delivering high-performance, scalable, and secure data solutions in a complex data environment within AWS, Databricks and Quantexa. This includes building robust ETL/ELT pipelines, and ensuring seamless integration between platforms.
- Collaboration & Partnership: Work extensively with multidisciplinary teams across product delivery, architecture, engineering security and analytics to understand and meet data needs.
- Quality & Operational Excellence: Driving high build and data quality, ensuring stability, robustness, and resilience of products, and embedding Agile, CI/CD, and DevOps practices.
- Leadership & Community Building: Mentoring junior team members, fostering professional development, building a data engineering community, reviewing solutions, troubleshooting complex issues and proactively driving innovation and challenging existing methods.
- Stakeholder Engagement & Problem Solving: Communicating complex data solutions and championing innovative approaches to diverse stakeholders, while also anticipating and resolving intricate data engineering challenges and building consensus to align data designs with organizational goals.
Opportunities to move to the new Infected Blood Compensation Authority (IBCA)
At the moment all IBCA roles are hosted by Cabinet Office within the Civil Service. However, once IBCA becomes operational, it is anticipated that most of these roles will transfer to IBCA, which is a new Arms Length Body (ALB) separate from the Cabinet Office. Roles that transfer to IBCA from Cabinet Office will maintain the same or substantially the same terms and conditions of employment that are overall no less favourable. It is anticipated that employees who move from Cabinet Office to IBCA will be able to continue to participate in the Civil Service Pension arrangements with no break in their pensionable service. Roles commencing after IBCA has become operational will be with IBCA itself on similar terms and conditions. All successful candidates will receive full details of the terms and conditions of employment for their role with their formal job offer.
Please note that the mission of IBCA means that it is likely to be operational for a period of approximately 5 to 7 years. When IBCA’s work begins to wind down, IBCA employees will receive support and practical guidance to find a new role, whether in the Civil Service, another Arms Length Body (ALB), or an external employer.
Person specification
Essential Criteria:
- Strong experience of cloud-native data engineering in AWS and building and maintaining complex data pipelines (both ETL and ELT) in a rapid delivery setting that covers data quality & transformation processes, data matching and master data management.
- Experience working with structured and unstructured data, and data lakes to service operational and analytical business needs.
- Proficiency in writing clear parameterised code in two or more of the following – Python, Databricks, Apache Spark (Pyspark, Spark SQL), NoSQL, Scala.
- Experience of delivering through Agile/DevOps working practices in multi-disciplinary teams – CI/CD, Scrum, Automation.
- Strong problem-solving skills, including assessing and mitigating risks while identifying opportunities for innovation.
- Experience in the full end to end data lifecycle for design, build and test, with knowledge of the interactions and dependencies with data architecture, data modelling and testing engineering.
- Strong experience in managing team dynamics and facilitating difficult discussions within the team or with a diverse set of stakeholders.
- Demonstrable experience of setting up data engineering processes from scratch or of implementing large changes to existing processes within an organisation that operate in a cloud DevSecOps environment.
Desirable Criteria:
- Management of contracts and 3rd party suppliers for software and professional services.
Additional information:
A minimum 60% of your working time should be spent at your principal workplace. Although requirements to attend other locations for official business will also count towards this level of attendance.
Behaviours
We'll assess you against these behaviours during the selection process:
- Changing and Improving
- Communicating and Influencing
Technical skills
We'll assess you against these technical skills during the selection process:
- Data analysis and synthesis (Practitioner)
- Data development process (Expert)
Benefits
- Learning and development tailored to your role.
- An environment with flexible working options.
- A culture encouraging inclusion and diversity.
- A Civil Service Pension which provides an attractive pension, benefits for dependants and employer contributions of 28.97%.
- A minimum of 25 days of paid annual leave, increasing by one day per year up to a maximum of 30.
Things you need to know
Selection process details
As part of the application process you will be asked to complete a CV and personal statement. Further details around what this will entail are listed on the application form.
Please use your statement of suitability to (in no more than 750 words) to explain how you consider your personal skills, qualities and experience provide evidence of your suitability for the role, with particular reference to the essential criteria in the person specification section of the job advert.
Should a large number of applications be received, an initial sift may be conducted using statement of suitability. Candidates who pass the initial sift may be progressed to a full sift, or progressed straight to assessment/interview.
Selection process
Candidates who are successful at sift will then be invited to an interview.
At interview, you will be assessed against Behaviour and Technical criteria.
Expected timeline (subject to change)
Expected sift date – w/c 14th July 2025
Expected interview date/s – w/c 28th July 2025
Interview location - Your interview will either be conducted face to face or by video. You will be notified of the location if you are selected for interview.
Reasonable Adjustment
If a person with disabilities is put at a substantial disadvantage compared to a non-disabled person, we have a duty to make reasonable changes to our processes.
If you need a change to be made so that you can make your application, you should:
- Contact Government Recruitment Service via recruitment.grs@cabinetoffice.gov.uk as soon as possible before the closing date to discuss your needs.
- Complete the ‘Assistance required’ section in the ‘Additional requirements’ page of your application form to tell us what changes or help you might need further on in the recruitment process. For instance, you may need wheelchair access at interview, or if you’re deaf, a Language Service Professional.
Further Information
A reserve list may be held for a period of 3 months from which further appointments can be made.
Any move to IBCA from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including Tax Free Childcare; for further information visit the Childcare Choices website.
If successful and transferring from another Government Department a criminal record check may be carried out.
In order to process applications without delay, we will be sending a Criminal Record Check to Disclosure and Barring Service/Disclosure Scotland on your behalf.
However, we recognise in exceptional circumstances some candidates will want to send their completed forms direct. If you will be doing this, please advise Government Recruitment Service of your intention by emailing Pre-EmploymentChecks.grs@cabinetoffice.gov.uk stating the job reference number in the subject heading.
For further information on the Disclosure Scotland confidential checking service telephone: the Disclosure Scotland Helpline on 0870 609 6006 and ask to speak to the operations manager in confidence, or email Info@disclosurescotland.co.uk
Please note that this role requires SC clearance, which would normally need 5 years’ UK residency in the past 5 years. This is not an absolute requirement, but supplementary checks may be needed where individuals have not lived in the UK for that period. This may mean your security clearance (and therefore your appointment) will take longer or, in some cases, not be possible.
For further information on National Security Vetting please visit the Demystifying Vetting website.
New entrants are expected to join on the minimum of the pay band.
Applicants who are successful at interview will be, as part of pre-employment screening, subject to a check on the Internal Fraud Database (IFD). This check will provide information about employees who have been dismissed for fraud or dishonesty offences. This check also applies to employees who resign or otherwise leave before being dismissed for fraud or dishonesty had their employment continued. Any applicant’s details held on the IFD will be refused employment.
A candidate is not eligible to apply for a role within the Civil Service if the application is made within a 5 year period following a dismissal for carrying out internal fraud against government.
This role is full time only. Applicants who wish to work an alternative pattern are welcome to apply however your preferred working pattern may not be available and you should discuss this with the vacancy holder before applying.
Please note terms and conditions are attached. Please take time to read the document to determine how these may affect you.
If you are experiencing accessibility problems with any attachments on this advert, please contact the email address in the 'Contact point for applicants' section.
Feedback will only be provided if you attend an interview or assessment.
Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check (opens in a new window).
This job is broadly open to the following groups:
- UK nationals
- nationals of the Republic of Ireland
- nationals of Commonwealth countries who have the right to work in the UK
- nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)
- nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
- individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
- Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service
Further information on nationality requirements (opens in a new window)
The Civil Service Code (opens in a new window) sets out the standards of behaviour expected of civil servants.
We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles (opens in a new window).
We are proud supporters of Women in Data. Connect, engage and belong to the largest free female data community in the UK – visit: www.womenindata.co.uk to join our community.
Stay connected! Follow us on LinkedIn for updates on career opportunities and more.