Learning to Play: Assessing Music Playing Skill from AudioVisual Data

For instructions on how to apply, please see: PhD Studentships: UKRI Centre for Doctoral Training in Socially Intelligent Artificial Agents.

Supervisors

• Tanaya Guha: School of Computing Science
• Subarna Tripathi: Intel Labs San Diego

Motivation & novelty:

Humans can often assess how well someone performs at a given task simply by watching (and hearing) them in action. The task of ‘skill assessment’, if automated, can potentially create assistive technology for humans to learn and practice independently, achieving eventual mastery. Although several learning apps and tools are available these days, few can offer automated feedback on the learners’ skill level.

Aims and methodology:

This project will develop a multimodal (audiovisual) AI tool to assess human skills from video streams accompanied with audio. In particular, our aim is to assess the skill of a learner playing a musical instrument using both audio and video as inputs. This is a fine-grained video understanding problem, where the input videos have similar actions while audio could be different. The project will develop new deep learning models to combine information from the two modalities that can attend to the modalities spatially and temporally with appropriate attention. A relevant database will need to be curated from YouTube and labeled in a semi-automated fashion.

Alignment with industrial interests:

Multimodal sensing and sense-making technologies are at the heart of Intel’s effort to build smart and personalized learning space. For example, Intel’s deployment of such technologies in ‘Kid Space’ showed encouraging results in terms of students’ engagement and learning effectiveness.

Timeline:

This is envisioned as a full-time PhD project involving the following activities: Literature survey, database curation, baseline model development, new model development, testing and evaluation, dissemination of results (e.g., publication, presentation) and thesis writing.

Desired skills:

Python, Machine Learning, prior experience of working with video/audio.