Staff Data Engineer | Contract | Up to £1,100/day | Inside IR35 in London
Posted by WeDoTech
£750 – £1,000 a day
London · Contract
Location: Hybrid – London (3 days per week on-site)
Rate: Up to £1,100/day
Contract: Initial 6 months
IR35: Inside IR35
We're supporting a major UK financial services organisation through a significant programme of data engineering modernisation and transformation.
We're looking for a genuinely Staff-level Data Engineer who combines exceptional hands-on Python/PySpark engineering with strong architectural thinking and practical experience of AI-assisted software development.
This is not a hands-off architecture role. You need to be equally comfortable setting technical direction, challenging engineering approaches and independently writing, debugging and optimising production-quality code.
The sweet spot is someone who can operate at a strategic level while remaining an outstanding hands-on engineer.
The roleYou'll play a key role in modernising a large-scale enterprise data estate, including the migration and transformation of thousands of data pipelines, development of new data products and adoption of modern, AI-assisted engineering practices.
Responsibilities will include:
- Providing Staff-level technical leadership across complex data engineering and modernisation initiatives.
- Remaining deeply hands-on with Python, PySpark and Apache Spark, independently building and proving engineering solutions.
- Designing and optimising scalable data pipelines and processing frameworks across AWS.
- Tackling complex Spark performance challenges, including shuffles, joins, data skew, partitioning and memory optimisation.
- Designing and challenging engineering approaches across AWS Glue, Apache Airflow and associated technologies.
- Reviewing existing code, identifying performance bottlenecks and driving improvements in engineering quality.
- Establishing reusable engineering patterns, frameworks and approaches to accelerate large-scale data migrations.
- Applying AI-assisted development tools to improve engineering productivity, code quality and delivery speed.
- Challenging technical assumptions, influencing architectural decisions and raising engineering standards across multiple teams.
- Exceptional hands-on Python/PySpark and Apache Spark expertise, with the ability to independently write, debug and optimise complex production code.
- Deep understanding of Spark internals and performance optimisation, including execution plans, joins, shuffles, partitioning, memory management and data skew.
- Strong AWS data engineering, particularly AWS Glue.
- Strong Apache Airflow experience.
- Excellent software engineering fundamentals, including code quality, testing, performance and maintainability.
- Experience designing and delivering large-scale data pipeline migrations or modernisation programmes.
- Experience operating at Staff, Principal or Senior Lead Engineer level, or demonstrably performing at that standard.
- Ability to independently assess complex engineering challenges, challenge existing approaches and shape scalable solutions.
- Strong architectural and conceptual understanding, underpinned by genuine hands-on delivery experience.
- Demonstrable experience using AI-assisted development tools (such as Claude, GitHub Copilot or similar) to accelerate real-world software engineering delivery.
- Ability to critically review, validate and improve AI-generated code rather than relying on it without understanding.
- Experience influencing engineering practices and technical direction across multiple teams.
- Snowflake.
- Legacy ETL modernisation and migration.
- Building reusable migration frameworks or automation to accelerate large-scale transformation programmes.
- Financial services or other complex, highly regulated environments.
A genuinely excellent hands-on Spark engineer who can think architecturally, solve complex problems independently and use AI to accelerate engineering delivery.
You'll need to be comfortable discussing high-level technical strategy one minute and getting directly into complex PySpark code the next.
This is an opportunity to have a meaningful impact on a major enterprise transformation programme while remaining at the forefront of hands-on engineering.
Apply Now →
Application opens at the source listing. Free for jobseekers.