Lead Azure Databricks Platform Engineer / Architect

Lead Azure Databricks Platform Engineer / Architect
Hands-on Platform Engineering | Serverless | FinOps | POSIT/RStudio Migration

6 Month contract

Inside IR35 - 500 a day

London/Hybrid

We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level.
The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio.

• Assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.
• Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.
• Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.
• Implement compute policies, autoscaling, quotas, budget controls and operational guardrails.
• Measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.

• Define and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.
• Implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.
• Provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.
• Implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.
• Use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement and cost anomalies.

• Enhance the Databricks Discovery Zone to support migration from POSIT/RStudio
• Enable application deployment, secure API integrations, external data ingestion, LLM integration, scheduling, BI connectivity, local IDE-based development and operational reporting.
• Define reusable onboarding and migration patterns that reduce technology sprawl while improving security, supportability and delivery speed.

• Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake.
• Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls.
• Design reusable integration patterns for REST APIs, SaaS platforms, databases, files, object storage, document repositories, enterprise applications and public/external data providers.
• Implement secure authentication and credential handling for external and internal integrations.
• Build end-to-end data flows from source through governed ingestion and curated layers to BI, ML or application consumption.

Deep hands-on Azure Databricks implementation and troubleshooting
Databricks Serverless and compute/workload optimisation
Azure identity, networking, security, secrets, monitoring and private connectivity
Databricks SQL, Delta Lake and performance optimisation
Python, PySpark and SQL
Jobs/workflows, incremental processing, CDC and data quality
REST/API and external data integration patterns
FinOps, cost attribution, tagging, budgets, monitoring and operational support

• Significant experience delivering enterprise Azure Databricks platforms in production environments.
• Demonstrable ability to move between architecture, implementation, debugging and optimisation without depending entirely on specialist engineering teams.
• Strong understanding of platform security, data governance, operational support and controlled delivery in regulated or complex enterprises.
• Experience working collaboratively with data engineers, data scientists, architects, security teams, platform teams and business stakeholders.
• Clear communication skills and the ability to document standards, patterns, decisions and operational guidance.

• POSIT/RStudio migration or consolidation experience.
• Migration of analytical/data science workloads (convert and migrate R development/Libraries to Databricks).
• AI/ML, LLM integration, model lifecycle, RAG/vector retrieval or model-serving experience.
• Large-scale enterprise platform transformation and regulated-industry experience.
• Strong cost optimisation and FinOps delivery experience across Azure and Databricks.