(UoM - A* STAR) Generative Models for Enhancing Deep Reinforcement Learning with Applications in Material Discovery

The project investigates Generative Models, including Large Language Models (LLM) & diffusion models, and Deep Reinforcement Learning to support multi-modal, multi-task and multi-embodiment decision-making, and humanin-the-loop learning. Specifically, Deep Reinforcement Learning (RL) effectively optimizes policies for sequential decision-making problems under the Markov Decision Processes (MDP) setting. Majority of existing policy gradient methods in this setting focuses on single-modal policy (i.e., the observation and action space have a single modality input/output), single-task configuration (i.e., the objective to optimize is defined with respective to a single fixed reward scalar) and single-embodiment environment (i.e., the policy learned is specific to one embodiment characterized by the environment transition dynamics). By contrast, many real-world decision-making problems feature in multi-modal, multi-task and multi-embodiment, and unavoidably involve the participation of human users, i.e., human-in-the-loop. These discrepancies between the existing RL methods and the real-world problems to solve calls for a rethinking of Deep RL studies and a revamp of the MDP idea. The recent advancement of Generative

Models (GMs), including Large Language Models (LLMs) such as ChatGPT/GPT4 and diffusion models such as Dalle2, allow AI to generate images/text, write code, generate synthetic data and naturally interact with human users. Importantly, these GMs directly handle multi-modality inputs and outputs in a unified manner, solve various tasks as a generalist agent, and can be easily adapted to new contexts with fine-tuning or prompt engineering. Whilst the highest profile applications of GMs have been in text and images, they can also be applied to improve on the existing policy gradient methods for Deep RL, offering a unique opportunity to develop methods that are inherently multimodal, multi-task and multi-embodiment, and readily usable for many real-world problems. This project thereby aims to decrease the discrepancies in Deep RL and investigates the effective combination of Generative Models and Reinforcement Learning for multi-modal, multi-task and multi-embodiment decision making problems. Moreover, the project considers the human-in-the-loop learning setting and leverage the Generative Models to address the value misalignment issue when human preferences are considered. The developed ideas in this project will be applied to solve complex problems in material designs. The project considers the following topics: reward models, reinforcement learning from human feedback, human intention alignment, generative models for planning, language models for task planning.

Expected Start  

October 2025 

Before you Apply  

We recommend that you contact the supervisor(s) for this project on mingfei.sun@manchester.ac.uk and qian_hangwei@cfar.a-star.edu.sg 

How to apply 

To be considered for this project, you will need to complete an application here. Please read this page carefully before starting your application.   

When you apply you will be asked to upload the following supporting documents:  

• Personal Statement  
• Final Transcript and certificates of all awarded university level qualifications  
• Interim Transcript of any university level qualifications in progress  
• CV  
• English Language certificate  
• 2 details of academic referees  

In your application, include the project title (UoM - A* STAR) Generative Models for Enhancing Deep Reinforcement Learning with Applications in Material Discovery, supervisor Dr Mingfei Sun, and funding source Split Site ASTAR and University of Manchester Select the PhD Computer Science in the programme detail section.  

Applications missing required documents won’t be considered. If documents are unavailable or for further application question please email, email FSE.doctoralacademy.admissions@manchester.ac.uk. 

Eligibility  

1. Hold (or expect to achieve) a First Class or 2:1 UK honours degree (or international equivalent to be checked with The University of Manchester admission team). 

2. Ideally hold a master’s-level qualification at merit or distinction (or international equivalent to be checked with The University of Manchester admission team). 

3. Demonstrable excellent communication skills, including in English language, a proficiency in which should be demonstrably indicated by meeting the requirements as indicated on the University's English language requirements page and in particular securing a minimum IELTS 6.5 overall with a minimum of 6.0 in writing and listening, and 5.5 in all other sub-tests OR securing a TOEFL iBT score of at least 90 with no sub-tests below 20 OR equivalent. Project supervisor teams may recommend a candidate who has excellent English language skills but otherwise has not formal certification of such. Please note that a timely demonstrable minimum English language level is a requirement of the UK Home Office for the issue of student visas to the UK. For some projects an ATAS certificate may also be required by them. 

4. Demonstrate willingness to travel to two partner institutions to complete the programme. 

5. Demonstrate to reflect the Faculty of Science and Engineering Postgraduate Researcher person specification. 

* Educational background matches research project. 

* Potential to form effective working relationships with a diverse range of people, including working inclusively and as part of a team. 

* Potential to take the initiative, lead on projects, and be proactive in prioritising a dynamic, agile and diverse workload. 

* Potential to develop understanding of complex problems, evaluate the strengths and weaknesses of a given scenario, and apply in-depth knowledge to address them. 

* Potential to develop expertise in new areas of the subject. 

* Evidence of an understanding of the proposed area of research, including knowledge of current challenges and opportunities. 

* An interest in continuous personal and professional development. 

* Potential to communicate ideas and conclusions, verbally and in writing, clearly and effectively to specialist and non-specialist audiences. 

* Preliminary knowledge of research techniques/track record of engaging with research. 

* Commitment to principles of Equality, Diversity, Inclusion, and Accessibility in teaching, research, or experience. 

Equality, diversity and inclusion are fundamental to The University of Manchester’s success. Diversity strengthens our research community, enhancing creativity, productivity, quality, and impact. We encourage applicants from diverse backgrounds, regardless of age, disability, ethnicity, gender, gender expression, sexual orientation, or transgender status. 

We also welcome those returning from a career break. Flexible study arrangements (part-time: 50%, 60%, or 80%, depending on the project/funder) may be considered. 

FSE_Singapore, FSE_dualawards,