CEMPS funded PhD Studentship in Computer Science: Understanding Recurrent Neural Networks Dynamics

Location: Streatham Campus, University of Exeter, EX4 4QJ

Project Description:

Many dynamical systems are naturally representable as networks characterized by time-variant properties, which are related to the topology and/or features associated with vertices and edges. Of particular interest are systems that also perform a computation when driven by an external input signal. Recurrent neural networks (RNNs) constitute an important example in this direction. RNNs are universal approximators of dynamical systems with the capability to generate complex dynamics, store the history of input signals in their transient dynamics and perform inference accordingly. Favoured by their biological plausibility, RNNs are also gaining increasing attention in neuroscience.

Nonetheless, RNNs depend on several (hyper-)parameters, whose tuning highly affect the performance on the application of interest. Moreover, the recurrent nature of their architecture complicates training and hampers interpretability of internal dynamics. This poses constraints on the applicability of such models, which are usually treated as black-box and related hyper-parameters are tuned by cross-validation. The interpretability issue is particularly important, since scientific knowledge can be acquired from experiments only if such models allow for an interpretation. Cross-validation, although effective in practice, comes with shortcomings, since it requires to evaluate the performance on a validation set, hence affecting the overall computational complexity. Moreover, in real-life applications the amount of data may be limited and/or supervised information not always available.

Unsupervised learning offers an alternative framework to address such problems. Several unsupervised methods for tuning RNN (hyper-)parameters have been proposed in the literature. Of particular interest are methods aiming at identifying the so-called edge of criticality, namely the set of (hyper-)parameters for which RNNs denote a dynamical behaviour between order and chaos. Experimental results show that, once operating in such configurations, RNNs usually achieve the best performance in terms of prediction accuracy and memory capacity, which are the quantities generally monitored by supervised approaches.

The goal of this project is to take advantage of the notion of criticality to design unsupervised learning methods contextualized within a graph-theoretical framework. Graph-theoretical approaches are expected to:

1) Provide a way to interpret RNN behaviour by using the language of graph theory

2) Offer a formal framework suitable for the identification of critical behaviour in RNNs and design of related learning mechanisms.