FindAPhD Weekly PhD Newsletter | JOIN NOW FindAPhD Weekly PhD Newsletter | JOIN NOW

Real2Sim: an end-to-end pipeline for uncontrolled scene digital twin


   Department of Computer Science

This project is no longer listed on FindAPhD.com and may not be available.

Click here to search FindAPhD.com for PhD studentship opportunities
  Dr Wenbin Li  No more applications being accepted  Funded PhD Project (UK Students Only)

About the Project

The University of Bath is inviting applications for the following PhD project commencing in October 2022 and supervised by Dr Wenbin Li in the Department of Computer Science.

The goal of this project is to develop an end-to-end pipeline to pre-capture visual assets from real-world environments and turn them into high-quality virtual scenes. This is followed by the implementation of a computational solution to create diverse synthesised scenarios and precise ground truth labels.

Modern machine learning systems can learn remarkably complex tasks from a good amount of data and the associated ground truth labels. The simulation from a virtual environment is important due to the low cost and automatically generating 'perfect' annotations. Many recent machine learning research relies on high-quality simulation. The challenge with simulated training is the absence of full capture of the real-world effects. In some common cases, models trained only on synthetic data would fail to generalize to the real-world scenario. There is very likely a discrepancy between synthesised virtual scenes and real environments, in terms of important visual and physical properties. This difficulty is so-call 'Reality Gap'.

In this project, we suppose to bridge the 'Reality Gap' by creating a digital twin from real-world scenes. We would like to capture representatives of real-world effects, and digitalise them into a controlled virtual environment and the variations. These tasks would require the system to have a certain level of understanding of the real-world surrounding, and the ability to generalise virtual scenes. The potential candidate would focus on the development of a system including:

  • capture high-quality real-world properties and their geometry estimation using multiple sensor fusion e.g. light-field, events, depth and 3D LiDAR etc;
  • create 3D editable contents using geometric and semantic information we obtained from a scene;
  • synthesise various virtual scenes and allow reasonable randomisation and a combination of certain real-world features.

In particular, the outcome of the project is intended to achieve a solution to create a digital twin from any uncontrolled real-world surroundings. This should comprise sensor fusion, visual capture and 3D reconstruction for a real-world scene including a series of real-world challenges e.g. dynamic objects, illumination changes, large textureless regions etc. In addition, the achieved visual information as elements, is then turned into virtual contents and their combination for the generation of various synthesised scenes. The key challenge will lie in automatic 3D object manipulation and arrangement.

The successful candidate will work closely with experts from CAMERA and the Department of Computer Science, as well as other collaborators from University of Bristol, and our project partners.

The project will be associated with the MyWorld programme launched in April 2021. MyWorld is the flagship for the UK’s creative technology sector, and is part of a UK-wide exploration into devolved research and development funding (UKRI video). MyWorld will position the South West as an international trailblazer in screen-based media. This £46m programme is bringing together 30 partners from Bristol and Bath’s creative technologies sector and world-leading academic institutions, to create a unique cross-sector consortium. MyWorld will forge dynamic collaborations to progress technological innovation, deliver creative excellence, establish and operate state of the art facilities, offer skills training and drive inward investment, raising the region’s profile on the global stage.  For more information on MyWorld, visit https://www.myworld-creates.com/ and https://www.bristol.ac.uk/news/2020/june/myworld-ukri-funding-announcement.html.

Project keywords: SLAM, robot perception, simulation, scene understanding/synthesis.

Candidate Requirements:

Applicants should hold, or expect to receive, a first or upper-second class honours degree in computer science, electrical engineering, mechanical engineering or a closely related discipline. A master level qualification or publication history would be advantageous.

Non-UK applicants must meet our English language entry requirement.

Enquiries and Applications:

Informal enquiries are welcomed and should be directed to Dr Wenbin Li ([Email Address Removed]).

Formal applications should be made via the University of Bath’s online application form for a PhD in Computer Science.

More information about applying for a PhD at Bath may be found on our website.

NOTE: Applications may close earlier than the advertised deadline if a suitable candidate is found; therefore, early application is recommended.

Funding Eligibility:

To be eligible for funding, you must qualify as a Home student. The eligibility criteria for Home fee status are detailed and too complex to be summarised here in full; however, as a general guide, the following applicants will normally qualify subject to meeting residency requirements: UK nationals (living in the UK or EEA/Switzerland), Irish nationals (living in the UK or EEA/Switzerland), those with Indefinite Leave to Remain and EU nationals with pre-settled or settled status in the UK under the EU Settlement Scheme). This is not intended to be an exhaustive list. Additional information may be found on our fee status guidance webpage, on the GOV.UK website and on the UKCISA website.

Exceptional Overseas students (e.g. with a UK Master’s Distinction or international equivalent and relevant research experience), who are interested in this project, should contact the lead supervisor in the first instance to discuss the possibility of applying for supplementary funding.

Equality, Diversity and Inclusion:

We value a diverse research environment and aim to be an inclusive university, where difference is celebrated and respected. We welcome and encourage applications from under-represented groups.

If you have circumstances that you feel we should be aware of that have affected your educational attainment, then please feel free to tell us about it in your application form. The best way to do this is a short paragraph at the end of your personal statement.


Funding Notes

Candidates with Home fee status who apply for this project may be considered for a university studentship covering Home tuition fees, provision of a stipend (£15,609 per annum, 2021/22 rate) and research/training expenses (£1,000 per annum) for 3 years. Eligibility criteria apply - see Funding Eligibility section above.

References

[1] Wenbin Li, Sajad Saeedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang, Rui Tang, Stefan Leutenegger. InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset, British Machine Vision Conference, BMVC 2018.
[2] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew J Davison, Stefan Leutenegger. MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM. International Conference on Robotics and Automation, ICRA 2019.
[3] Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019.

How good is research at University of Bath in Computer Science and Informatics?


Research output data provided by the Research Excellence Framework (REF)

Click here to see the results for all UK universities
PhD saved successfully
View saved PhDs