Senior Data Engineer

Senior Data Engineer – Monetary Analysis & Economic Data

• Location: London, UK (Hybrid: 3 days in office, 2 days remote)
• Security Clearance: SC Clearance Required (Active or eligible to undergo)
• Salary: £75,000 to £80,000 + Benefits and Bonus
• Position Type: Full-Time, Permanent
• Experience Level: 10+ Years (Senior/Lead)

About the Role

We are seeking an expert Hybrid Data Engineer to work in London 3 days in office, 2 days remote paying £75,000 to £80,000 + Benefits and Bonus to drive the development, optimization, and scaling of our cutting-edge Azure Databricks platform. This high-performance infrastructure is critical to our core mission, directly powering our Monetary Analysis, Forecasting, and Modelling frameworks.

In this role, you will lead the engineering of robust, secure data pipelines, implement complex transformation logic, and guarantee absolute data reliability for business-critical economic datasets.

Key Responsibilities

Data Pipeline Engineering & Processing

• Build & Scale: Design, develop, and maintain robust, scalable ETL/ELT pipelines ingesting data from APIs, relational databases, streaming services, and financial data providers.
• Complex Transformations: Implement advanced data processing logic for cleaning, enriching, and aggregating large-scale data using Spark (PySpark/Scala) and SQL.
• Optimization: Fine-tune data workloads for maximum performance, throughput, and cloud cost-efficiency.

Azure Databricks & Cloud Architecture

• Platform Ownership: Drive the implementation of Azure Databricks services, leveraging Unity Catalog and Delta Lake architectures.
• Polyglot Development: Develop and maintain data solutions using a diverse technical stack including Python, SQL, R, YAML, and JavaScript.

Data Quality, Governance & Security

• Data Governance: Lead hands-on implementation of Azure Purview to manage data quality, data governance, metadata, and end-to-end data lineage tracking.
• Framework Design: Establish automated data validation, quality checks, and real-time alerting processes across all production environments.

DevOps, Automation & Collaboration

• CI/CD Integration: Partner with DevOps teams to design, build, and maintain robust CI/CD pipelines for automated environmental deployments.
• Cross-Functional Partnership: Collaborate closely with economists, data scientists, and senior analysts to translate complex analytical needs into production-ready data systems.
• Mentorship & Quality: Drive engineering excellence through active participation in code reviews, architectural discussions, and knowledge-sharing sessions.

Technical Stack & Qualifications

Essential Experience

• Core Data Engineering: 10+ years of dedicated data engineering experience managing massive, complex datasets.
• Azure Databricks Expertise: 3+ years of deep, hands-on production experience with Azure Databricks, Spark (PySpark/Scala), and Delta Lake ecosystems.
• Cloud Infrastructure: Extensive experience across the Azure data suite, specifically Azure Data Factory, Azure Blob Storage, and Azure SQL Database.
• Core Languages: Strong proficiency in Python, Spark, and SQL.

Technical & Architectural Familiarity

• Governance Tools: Practical experience using Azure Purview for governance and cataloguing.
• Languages & Infrastructure: Working knowledge or exposure to R, YAML, and JavaScript within data workflows.
• Streaming & DevOps: Experience with event-driven data (e.g., Kafka, Azure Event Hubs) and modern DevOps tooling (Azure DevOps, Git, Docker, Kubernetes).
• Data Layering: Solid understanding of both SQL and NoSQL database design, data warehousing principles, and data modelling techniques.

Domain & Interpersonal Skills

• Industry Background: Experience working within financial services, central banking, or an economic data environment is highly advantageous.
• Communication: Exceptional ability to bridge the gap between technical infrastructure and economic/analytical business logic.
• Certifications (Preferred): Microsoft Certified: Azure Data Engineer Associate or Databricks certifications.