Recognising Sign Language using Radio Frequency Sensing

PROJECT REFERENCE NUMBER

Please select the project reference number SCEBE-22SF-SLRFS-Usman from the drop-down list or refer to this project reference number in your project write-up.

BACKGROUND

It is estimated that around 5% of the world’s population live with some form of hearing impairment. Depending on the level of impairment, deaf people generally rely on sign language for communication. Sign language is a mean of communication between deaf using visual gestures and signs. The idea of understanding sign language using camera-based recordings and deep learning has recently gained research attention. However, recording video of the target raises serious privacy concerns. An alternative to camera-based sign language identification is radio frequency (RF) sensing. The principle of RF sensing is based on observing the variations in the reflected signal due to hands’ movement of the target. Useful information about the movements can be derived by processing the reflected signals. Sensing with Wi-Fi and Radar are two examples of RF sensing. For Wi-Fi, specific changes in the channel state information (CSI) of the reflected signal due to hands’ movements are analysed to identify the speech of the target. On the other hand, Radar works on the principle of Doppler effect.

AIMS

This project aims to build an RF sensing-based sign language translation system with the help multi-modal data coming from omnipresent Wi-Fi signals and ultra-wide-band (UWB) radar system. The radio frequency signatures generated from Wi-Fi and radar will converted to RF images, such as spectrograms that will be sent into a Deep Convolutional Neural Network (DNN) as an input. The DNN will be trained to classify different signs and identify words and sentences in a sign language, in real time.

CANDIDATE BREAKDOWN 

The successful candidate should be able to demonstrate a solid background in at least one of the aspects: wireless communication, signal processing, machine/deep learning, and software-defined radios (SDR).

The candidate should have good experience of working with Radars and SDRs such as Ettus x300. Training will be provided to enhance hardware skills.

The successful candidate should have strong self-motivation and dedicated passion in wireless communication and RF sensing research.

The willingness of team-working in a multi-cultural team and the ability to deliver research outcome to meet the deadlines on one’s own.