Arbitrary sensitive transitions in recurrent neural networks

Abstract

An Excitable Network Attractor (ENA) is a forward-invariant set in phase space that can be used to explain input-driven behaviour of Recurrent Neural Networks (RNNs) trained on tasks involving switching between a discrete set of states. An ENA is composed of two or more attractors and excitable connections that allow transitions from one attractor to another under some input perturbation. The smallest such perturbation that makes a connection between two attractors is called the excitability threshold associated with that connection. The excitability threshold provides a measure of sensitivity of the connection to input perturbations. Errors in performance of such trained RNNs can be related to errors in transitions around the associated ENA. Previous work has demonstrated that ENAs of arbitrary sensitivity and structure can be realised in a RNN by suitable choice of connection weights and nonlinear activation function. In this paper we show that ENAs of arbitrary sensitivity and structure can be realised even using a suitable fixed nonlinear activation function, i.e. by suitable choice of weights only. We show that there is a choice of weights such that the probability of erroneous transitions is very small.