Databricks Data Engineer

Description:

A leading organization at the forefront of the UK's energy transition is seeking a skilled Databricks Data Engineer for a 6-month contract engagement. The role will be hybrid, with 2 days per week onsite in London, and offers a competitive daily rate between £550 and £600, falling inside IR35.

As part of a broader strategic initiative to align with new regulatory changes, the successful candidate will play a pivotal role in shaping scalable data solutions that enhance reporting and optimize data operations. You will support data pipeline modernization, develop robust ETL/ELT workflows, and empower analytics through high-quality datasets drawn from critical infrastructure like smart meters and power generation systems.

Key Responsibilities:

• Architect and maintain scalable data lakehouse solutions using Databricks

• Design and implement ETL/ELT pipelines for large-scale data processing

• Work with structured and unstructured data to ensure performance and reliability

• Create structured streaming pipelines using Kafka

• Prepare and transform datasets for Power BI reporting

• Deploy and manage infrastructure using Terraform and CI/CD methodologies

• Collaborate with engineering teams to reduce backlog and support delivery of data products

Required Experience & Skills:

• Strong, hands-on experience using Databricks in production environments

• Expertise in working with large and complex datasets

• Proficiency in ETL/ELT pipeline development

• Deep knowledge of Azure services (including Data Factory, Functions, and Synapse)

• Advanced skills in SQL (performance tuning and optimization)

• Strong programming capability in Python

• Familiarity with big data tools like Hadoop, Spark, and Kafka

• Solid understanding of CI/CD practices and version control systems

• Experience using Terraform for Infrastructure as Code

• Understanding of cloud-based networking and distributed systems