Automated Recognition of Emotional Wellbeing and Personal Needs at Ulster University

This project is no longer listed on FindAPhD.com and may not be available.

Click here to search FindAPhD.com for PhD studentship opportunities

Prof B Scotney No more applications being accepted Competition Funded PhD Project (Students Worldwide)

Add
to
shortlist

Remove
from
shortlist

Belfast United Kingdom Artificial Intelligence Computer Vision

About the Project

There are many ways in which a person may express their state of emotional wellbeing and/or their immediate needs within their environment. Typically this may be through a combination of deliberate and unconscious actions. Body posture and dynamics, facial expression and gaze are all natural and important aspects of human communication, in addition to speech, that may be effected either deliberately or unconsciously, and in combination. These non-verbal aspects can account for a significant part of the information being conveyed, and they contribute significantly to the understanding (by a human observer) of the emotional state of the person and to the observer’s assessment of any needs (for assistance) that the person may have. There are, therefore, a range of visual elements (that may accompany, or may occur independently of, speech) that may signal feelings such as frustration, fear, anxiety, sadness, confusion or anger, or may indicate that the person has immediate needs such as relief from hunger, thirst, or pain. Non-verbal aspects of emotional state and communication are principally visual in nature, enabling us to focus on how the modality of machine vision can provide the central element of systems to monitor and support elderly persons to live independently at home or safely within a carehome environment in a way that fully respects their privacy and confidentiality of personal data.

This PhD project will create an opportunity for a student to work with experts in Computer Vision and Machine Learning, along with experts in human communication to take a holistic approach to the development of AI techniques for automated analysis and understanding of human non-verbal communication. Recent developments in automated video analysis, using convolutional neural networks, have enabled AI that can recognise emotional states from facial expression. The achievement of complementary capabilities for gesticulation and posture is the focus of this PhD project, providing a challenging research programme. This is a field in which promising solutions are beginning to emerge, with the potential to build systems to support independent living and caregiving.

The PhD project will contribute to knowledge in the following areas of automated content analysis in video data:

Recognition of gestures associated with emotional or physical needs;
Interpretation of body/head posture and dynamics in human communication;
Combination of gesture, body posture, and expression for understanding emotional state.

The research programme will build on the experience of the supervisory team in the areas of pose estimation and automated gesture recognition from video, including expertise in communication provided by the external supervisor, Prof M-G Busa, University of Padova. This experience includes work on automated recognition of wellbeing from video using body pose estimation models at the BT Ireland Innovation Centre (BTIIC), and collaborative research on automated gesture recognition with the Language and Communication Lab at the University of Padova and with the Hypervision Research Lab at Keio University, Tokyo. The project will also involve collaboration with healthcare professionals and carehome staff with whom we are working currently in BTIIC to develop healthcare support services for implementation and delivery by BT.

References

Naoto Ienaga, Alice Cravotta, Kei Terayama, Bryan W. Scotney, Hideo Saito, M. Grazia Busà. (June 2022). Semi-automation of Gesture Annotation by Machine Learning and Human Collaboration. Language Resources and Evaluation, 56, 673–700. https://doi.org/10.1007/s10579-022-09586-4
Marshall, F., Zhang, S. & Scotney, B.W. Automatic Assessment of the Type and Intensity of Agitated Hand Movements. Journal of Healthcare Informatics Research (Sep 2022). https://doi.org/10.1007/s41666-022-00120-3
Ienaga, N., Scotney, B.W., Saito, H., Cravotta, A., & Busà, M.G. (Aug 2018). Natural Gesture Extraction Based on Hand Trajectory. 20th Irish Machine Vision and Image Processing conference (IMVIP 2018), 29 - 31 August 2018, Belfast, pp. 81-88
Marshall, F., Zhang, S., & Scotney, B.W. (Sep 2021). Video-Based Hand Pose Estimation for Agitated Behaviour Detection. 23rd Irish Machine Vision and Image Processing conference (IMVIP 2021), 1-3 September 2021, Dublin, Ireland
Marshall, F., Zhang, S., & Scotney, B.W. (Aug 2020). Automatic Recognition of Repetitive Hand Movements. 22nd Irish Machine Vision and Image Processing conference (IMVIP 2020), 31 August – 2 September 2020, Sligo, Ireland, pp. 137-140
Marshall, F., Zhang, S., & Scotney, B.W. (Aug 2019). Comparison of Activity Recognition using 2D and 3D Skeletal Joint Data. 21st Irish Machine Vision and Image Processing conference (IMVIP 2019), 28 - 30 August 2019, Dublin, pp. 13-20 (Winner: Best Paper prize)
Camgoz, N. C., Hadfield, S., Koller, O., & Bowden, R. (2016). Using convolutional 3d neural networks for user-independent continuous gesture recognition. 23rd International Conference on Pattern Recognition, 49-54. 10.1109/ICPR.2016.7899606
Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2017). Realtime multi-person 2d pose estimation using part affinity fields. IEEE Conference on Computer Vision and Pattern Recognition, 7291-7299.
Ilyas, C., Nunes, R., Nasrollahi, K., Rehm, M. and Moeslund, T. Deep Emotion Recognition through Upper Body Movements and Facial Expression. 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5, 669-679
Madeo, R. C. B., Peres, S. M., & de Moraes Lima, C. A. (2016). Gesture phase segmentation using support vector machines. Expert Systems with Applications, 56, 100–115.
Okada, S. & Otsuka, K. (2017). Recognizing words from gestures: Discovering gesture descriptors associated with spoken utterances. 12th IEEE International Conference on Automatic Face & Gesture Recognition, 430-437. 10.1109/FG.2017.60.
Pigou, L., Van Herreweghe, M., & Dambre, J. (2017). Gesture and sign language recognition with temporal residual networks. IEEE International Conference on Computer Vision Workshops, 3086-3093. 10.1109/ICCVW.2017.365
Pouw, W., Trujillo, J.P. & Dixon, J.A. (2020). The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking. Behavior Research Methods, 52, 723–740.
Ripperda, J., Drijvers, L., & Holler, J. (2020). Speeding up the detection of non-iconic and iconic gestures (SPUDNIG): A toolkit for the automatic detection of hand movements and gestures in video data. Behavior Research Methods, 52(4), 1783-1794.
Trujillo, J. P., Vaitonyte, J., Simanova, I., & Özyürek, A. (2019). Toward the markerless and automatic analysis of kinematic features: A toolkit for gesture and movement research. Behavior Research Methods, 51(2), 769-777
Wan, J., Lin, C., Wen, L., Li, Y., Miao, Q., Escalera, S., ... & Li, S. Z. (2020). ChaLearn Looking at People: IsoGD and ConGD Large-Scale RGB-D Gesture Recognition. IEEE Transactions on Cybernetics. 10.1109/TCYB.2020.3012092