Description:
We are looking for an experienced Data Scientist who has practical experience in LLM Ops who can help us establish a new LLM Ops capability. This is an exciting time as this is LSEG’s first LLM Ops capability, using the latest tooling and bringing the best people to form a new team that is responsible for designing and operating a new blueprint for LLM Ops in LSEG and in the industry.
We are seeking a highly skilled and proactive Data Scientist to join our dynamic data science team which will also form part of the LLM Ops team. This role is pivotal in bridging the gap between data science, AI engineering, and project delivery. You will work closely with AI engineers and cross-functional project teams to analyse production data, identify key insights, and contribute to the development and management of AI-driven solutions, including large language models (LLMs) in
WHAT YOU'LL BE DOING:
- Collaborate with AI engineers and software delivery teams to design, develop, and deploy production-grade AI systems powered by large language models.
- Analyse production data to identify trends, anomalies, and performance issues, and provide actionable insights.
- Define and monitor key metrics to ensure the health and effectiveness of AI models in production.
- Support the lifecycle management of LLMs, including fine-tuning, evaluation, deployment, and monitoring.
- Partner with project teams to understand business needs and translate them into data science solutions.
- Develop and maintain robust, scalable code for data pipelines, model training, and evaluation.
- Contribute to the continuous improvement of data science practices and tooling within the organization.
- We work part of a global team, with team members in Manilla, Thailand, Europe and Americas. At times, eg once or twice a week, you will be required to start early or finish late in order to collaborate with others in different time zones. For example this may mean starting at 7am or 8am UK time.
WHAT YOU'LL BRING:
- Experience with version control (e.g., Git), CI/CD pipelines, and containerization (e.g., Docker).
- Knowledge of data governance, privacy, and ethical AI practices.
- Proven experience as a Data Scientist in a software delivery or technology-driven environment.
- Strong data manipulation skills (pandas, SQL) and understanding of evaluation strategies for LLM-based systems.
- Strong programming skills in Python (or similar), with experience in data manipulation, machine learning, and model deployment.
- You’ll need to have software engineering so that you can help evaluate non-deterministic systems that users can trust.
- Ability to work with ambiguity and optimise non-deterministic systems through a process of experimentation and evaluation while balancing latency/cost/quality trade-offs.
- Hands-on experience working with production data and monitoring AI/ML systems in live environments.
- Familiarity with LLMs and generative AI technologies, including prompt engineering.
- Experience with cloud platforms (e.g. AWS, Azure, GCP) and ML Ops tools is a plus. Eg AWS Bedrock
- Experience with tools such as Langfuse and Sagemaker
- Strong communication skills and the ability to work effectively with cross-functional teams.