SYNOPSIS
Authors: Francesco Mannella, Gianluca Baldassarre
Topic and its relevance. This research thread, started recently, concerns the use of echo-state networks and the modulation of their dynamics. Echo-state networks are an important class of neural networks belonging to the family of models called ``dynamic reservoires''. Echo-state networks have very interesting and powerful computational properties that make them suitable to learn and produce complex motor behaviours relevant for both robotics and for the study of motor behavoiur produced by brain. On the robotic side, the importance of this resides in the fact that the learning and production of sophisticated discrete and rhythmic movements is a pivotal building block of autonomous robotics architectures. In this respect, our approach is usable to face problems for which the autonomous robotics litearture uses devices such as Dynamic Movement Primitives (DMPs). The advantages of our approach with respect to DMPs is expected to be in terms of sophistication and flexibility of the movements producible with echo-state networks. The work is also important for brain modelling, in particular to model cortex viewed as a dynamical system whose dynamics is regulated by basal ganglia. The importance of this resides in the fact that the basal-ganglia and cortex form segregated loops that are a fundamental building module underlying multiple brain processes, from associative sensory processing, to motor behaviour, thinking, plannig, and reasoning.
Questions and goals. From a machine-learning/robotic perspective, what are the mathematical principles to make an echo-state network dynamics stable and at the same time capable of producing a rich internal dynamics usable to produce sophisticated rhythmic and discrete movements? How is it possibele to exploit the echo-state network dynamics to produce movements that change gradually to gradually changing environmental conditions? How is it possible to change the echo-state dynamics in more substatial ways to produce different classes of movements? From a biological perspective, does the echo-state dynamics capture similar processes taking place in brain cortex, e.g. primary and premotor cortex? Do basal-ganglia modulate the dynamics of cortex to produce different beahaviours? What are the plasticity processes that characterise this system?
Methods. We study the mathematical properties of echo-state networks with respect to stability and richness of dynamics by operating in suitable ways on the internal connectivity matrix of the dynamical reservoire. We test the capability of the sytem with kinematic and dynamic robot arms and hands, having from 3 to 15 DOFs (Degrees of Freedom). We use tests where the system has to reproduce different movements, e.g. to produce a moon-shaped rhythmic movement in different positions or of different size, or various discrete movements to produce different hand-grasp postures. We use echo-state networks to mimic the dynamics of cortex, and we use re-entrant localistic neural networks to mimic basal ganglia: we then integrate the two and use the whole system to evalute if basal ganglia can modulate the dynamics of the echo-state network so as to produce gradually changing movements (e.g., moon-shaped movements in different positions) or different families of movements (e.g., moon-shaped movements vs. diamod movements). We train the reading out layer of the echo-state network with supervised learning algorithms (e.g., ridge regression, with both batch and on-line learning) and at the moment we are also testing the use of reinforcement learning algorithms to train the system in a more autonomous, bilogically plausible way.
Results. On the robotic sides, the models show to be able: (a) to produce behaviours that change gradually with the gradual change of the input (e.g., to perform a moon-shape trajectory in different positions in space); (b) to produce different families of movements by substantially changing the dynamics of the echo-state network by releasing the activation of sub-populations of neurons within it (e.g., to produce moon-shaped or diamod-shaped trajectories; or different hand postures). On the biological side, the model architecture reflects the anatomical organisation of cortico-basal ganglia circuites and exhibits behaviours that are qualitatively comparable with those of humans (e.g., movements produced by normal people, or movements produced by Parkinson patients when the system is suitably lesioned).
Conclusions. We have found ways to treat the internal connectivity of echo-state networks so as to have stable and rich dynamics and to modulate such dynamics to produce systems that exhibit different families of motor behaviours. We believe these systems have a high potential for robotics. In this respect, a critical point to develop in the future will be the development of reinforcement learning algorithms usable to train the system (now trained with supervised learning algorithms). The models also show to have a high potential to reproduce the structure and behaviour of the cortex-basal ganglia brain module which has shown to have an inherently powerful dynamical nature (e.g., at the level of motor behaviour and working memory). Moreover, the models suggest to cast the modulation exerted by basal-ganglia on cortex in terms of strong modulation of cortical dynamics rather than in terms of the classical dishibition of the cortical contents.
Key references.
- Mannella, F. & Baldassarre, G. (in press). Selection of Cortical Dynamics for Motor Behaviour by the Basal Ganglia. Biological Cybernetics.
Work 1: Echostate networks producing complex dynamical motor trajectories: hypotheses on the cortex-basal ganglia interaction
The following videos show simulations of the interactions for motor control involving an echo-state network model of the cortex and a bio-constrained model of the basal ganglia. The key element of the models is the dynamic complex behaviours that they are able to learn, and the capacity of the basal ganglia component to cause a different dynamics of the echo-state network so that it can produce very different behaviours (e.g., drawing the shape of a moon or of a diamond or the infinite symbol). The videos also show the computational power of the models that make them highly suitable to control robots and embodied systems.
A basal ganglia-cortical model learns several rhythmic motor behaviours and reproduces them when selected. The module controls a 3-degrees-of-freedom (DoF) kinematic arm moving in a 2D environment. | |
| A basal ganglia-cortical model learns several discrete motor behaviours and reproduces them when selected. The model controls a 20 DoF dynamic hand moving in the 3D space. | |
| Two basal ganglia-cortical modules as those illustrated in the previous two videos are used as components of a system-level more sophisticated and realistic model of rythmic motor control. The model controls a 3 DoF kinematic arm moving in a 2D environment. The video also shows the effects on movement of lesions of the basal-ganglia components of the model. |