Audio-visual speech enhancement and speaker separation

Significant amounts of correlation exist between the audio speech produced by a speaker and the shape of the speaker’s lips. This project will exploit this correlation to enhance audio speech that has been contaminated by either noise or speech from competing speakers. Work will begin by developing audio-visual models of speech that allow an estimate to be made of acoustic speech features from visual speech features. Using these estimated acoustic speech features various methods of speech enhancement will be investigated, such as deriving filters (e.g. Wiener, log MMSE) and reconstructing speech using a speech model driven by the acoustic speech features. The application will begin with enhancement of noisy speech and then move to speaker separation.