Deep learning for 3D model synthesis

Realistic 3D contents are vital to construct an immersive virtual environment and demand for such 3D models is expected to increase rapidly as AR and VR technologies evolve. However, obtaining a quality 3D model from manual scanning is still a challenging and labour-intensive task. For example, conventional vision-based reconstruction pipeline used in the photogrammetry workflow in Unity generally requires multiple images from different viewing angles to triangulate an initial 3D point cloud. In addition, an albedo texture map should be prepared by fusing colour values, which will be mapped on the approximated surfaces using different image processing and 3D modelling tools.

There are better solutions with active sensors. For example, Microsoft KinectFusion can perform real-time reconstruction. However, this approach needs to track the sensor position to integrate an intermediate depth map into a single reference volume. Thus, the quality of a final shape is highly dependent on the result of vision-based tracking, and the volumetric shape representation internally used in merging multiple depth maps makes us difficult to retrieve a photorealistic texture map.

Another practical issue of the conventional scanning process is that it is often physically not feasible to scan every part of a target object when it is too tall or having complex concave geometry. As a result of this, many practical 3D scans are prone to have undesirable artefacts, such as holes on the final result (i.e. missing photometric and geometric information). In order to address this, many ideas have been suggested but most conventional approaches are focussed on solving one isolated problem (i.e. either shape recovery or texture map correction), instead of solving them as an interconnected problem.

This 3-year studentship will investigate a novel solution to tackle this problem. Particularly, we are interested in obtaining a complete 3D model of a complex non-rigid object (such as garment) using recent deep learning approach. A successful candidate will develop a range of unsupervised learning algorithms particularly developed based on the generative adversarial network. Therefore, a student will learn about how to use standard deep learning frameworks, such as Theano, TesnorFlow, or Caffe to configure new architectures and conduct experiments. Also, this research will require to populate, process, and manage 2D and 3D training data.

As an application of the proposed research, a simple interactive virtual reality application will be developed. For this application, a student can have an opportunity to experiment with garment data obtained from commercial retailers from our collaborators, and to use state-of-the art 3D scanning facility at our collaborator to populate test data if necessary.

How to apply:
We welcome applications from highly motivated prospective students who are committed to develop outstanding research outcomes. You can apply online at www.port.ac.uk/applyonline. Please quote project code CCTS4360218 in your application form.

Applications should include:
- a full CV including personal details, qualifications, educational history and, where applicable, any employment or other experience relevant to the application
- contact details for two referees able to comment on your academic performance
- a research proposal of 1,000 words outlining the main features of a research design you would propose to meet the stated objectives, identifying the challenges this project might present and discussing how the work will build on or challenge existing research in the above field.
- proof of English language proficiency (for EU and international students)

All the above must be submitted by the 11th of February 2018.