Model Based Deep Learning for Low-light Computational Imaging: Application to Robust Multimodal 3D Imaging

   School of Engineering & Physical Sciences

   Applications accepted all year round  Funded PhD Project (Students Worldwide)

About the Project

This project aims to design new computational algorithms at the interface of statistical modelling, optimization and deep learning to solve multimodal inverse problems for 3D imaging. The project is funded for 3.5 years for international students covering tuition fees and a ~£17K annual stipend.  

For cutting-edge imaging applications, the granularity extends to particle levels, encompassing photons and electrons. This precision finds its application in a gamut of systems, from advanced 3D single-photon Lidar imaging for autonomous navigation to electron energy loss spectroscopy and microscopy. Despite recent advances, current systems can still be optimized to enhance the spatial/temporal resolution and imaging quality in low particle conditions (e.g. low light imaging), hence enabling object recognition and 3D imaging fast varying environments at kHz rates.  

In this project, we will propose new algorithms for multimodal imaging to enable 3D imaging at unprecedented scales: km range, kHz frame rates and Mega-pixels. Our image processing group will work closely with system design groups in Heriot-Watt to combine multi-sensor data acquired at high frame rates using a 3D Lidar system and other sensing modalities such as a passive optical imaging system, fast event-based imaging systems or a radar. This combination aims to reduce the noise affecting the 3D images (due to imaging through fog, rain), and to improve the spatial resolution of 3D Lidar videos thus enabling targets reconstruction or higher level information extraction such as objects recognition and human pose estimation (see [1]). Current solutions developed by our group show promising results at high frame rates (acquisition and processing at 500 frames per second in [2,3]) and the student will generalize them to enhance the spatial resolution (a necessity for accurate objects recognition) and/or temporal resolution (a requirement for rapidly changing environments). A focus will be on the combination of statistical Bayesian models [4], optimization algorithms and state-of-the-art deep learning methods to solve these challenging inverse problems. In particular, the candidate will investigate the design of efficient networks (eg using unrolling [5], or plug-and-play approaches), the development of generative models for continuous 3D/4D scene representation and the use of knowledge distillation and burst imaging for multimodal data.

The candidate will closely collaborate with world-leading computational imaging groups in Heriot-Watt University (single-photon group, Quantum Optics and Computational Imaging), Edinburgh University and the wider community via the EPSRC Quantum Technology Hub in Quantum Imaging (Quantic). The research will be conducted in collaboration with industrials partners: STMicroelectronics  and Leonardo.

All applicants must have or expect to have a 1st class Master degree in electrical engineering, applied mathematics, physics, computer science, or a related discipline. Selection will be based on academic excellence and research potential, and all short-listed applicants will be interviewed (in person or via Microsoft Teams).

More information regarding the group can be accessed in:

Software Needs and Skills:

Statistical signal and Image processing, Bayesian methods, deep learning, optimization.

Matlab, Python, C/C++.

Computer Science (8) Engineering (12) Mathematics (25)

Funding Notes

The project is funded for 3.5 years for international students covering tuition fees and a ~£17K annual stipend.


[1] A. Ruget, M. Tyler, G. Mora-Martín, S. Scholes, F. Zhu, I. Gyongy, B. Hearn, S. McLaughlin, A. Halimi, J. Leach, "Pixels2Pose: Super-Resolution Time-of-Flight Imaging for 3D Pose Estimation", Science Advances, 2022
[2] S. Plosz, A. Maccarone, S. McLaughlin, G. S. Buller, A. Halimi, "Real-Time Reconstruction of 3D Videos from Single-Photon LiDaR Data in the Presence of Obscurants", IEEE-TCI, vol. 9, p 106-119, Feb. 2023.
[3] I. Gyongy, S. W. Hutchings, A. Halimi, M. Tyler, S. Chan, F. Zhu, S. McLaughlin, R. K. Henderson, and J. Leach, "High-speed 3D sensing via hybrid-mode imaging and guided upsampling," Optica, Vol. 7, Issue 10, Sept. 2020.
[4] A. Halimi, A. Maccarone, R. Lamb, G. Buller, S. McLaughlin, "Robust and Guided Bayesian Reconstruction of Single-Photon 3D Lidar Data: Application to Multispectral and Underwater Imaging," IEEE-TCI, 2021.
[5] J. Koo, A. Halimi, S. McLaughlin, "A Bayesian Based Deep Unrolling Algorithm for Single-Photon Lidar Systems", IEEE-JSTSP, 2022.

How good is research at Heriot-Watt University in Engineering?

Research output data provided by the Research Excellence Framework (REF)

Click here to see the results for all UK universities

Register your interest for this project

Search Suggestions
Search suggestions

Based on your current searches we recommend the following search filters.