Lead Data Engineer

Description:

KDR Talent Solutions is hiring a Lead Data Engineer on behalf of a leading London Market insurer. This is a hybrid role based in London and offers an exciting opportunity to be a part of a greenfield data platform build using the latest Azure ecosystem and Databricks technologies. The role is situated within a newly formed Data & Analytics team and is critical in shaping the organization’s data future.

As a Data Engineering Manager, you will collaborate with both technical and business teams to deliver a scalable and high-performance data platform. This role involves platform architecture design, building data pipelines, embedding data integrity measures, and ensuring CI/CD practices are applied to data workflows. You will drive innovation while enabling the business to make data-driven decisions through a solid and modern infrastructure.

Key Responsibilities:

• Design and build a high-performing Databricks Data Platform with a focus on scalability, performance, and availability.

• Develop and maintain ETL/ELT pipelines, integrating APIs, databases, and streaming data sources.

• Ensure data integrity with strong monitoring, quality control, and alerting mechanisms.

• Lead CI/CD implementation for automated deployment and workflow execution.

• Translate complex business requirements into technical solutions in collaboration with global teams.

• Future-proof the platform to accommodate new data sources, analytical models, and evolving technologies.

What You’ll Bring:

• Proven expertise in Databricks and Microsoft Azure data services such as Azure Data Factory, Azure Synapse, or Azure SQL.

• Strong skills in dimensional modelling, data warehousing, and analytics use cases.

• Proficiency in ETL/ELT development, SQL, and Python/PySpark.

• Experience with CI/CD methodologies applied to data platforms.

• Ability to work collaboratively with global business and technical stakeholders.

Why Join?

• 🚀 Greenfield project with architectural influence and ownership

• 🌍 Collaborative global environment promoting agile innovation

• 📈 Career growth opportunities in a forward-thinking organization

• 💡 Access to the latest technology, tools, and data engineering practices