LOCEN Research Focus: Bio-constrained models of affordances and brain cortico-cortical pathways

Synopsis

 

Authors: Daniele Caligiore, Anna Borghi, Domenico Parisi, Gianluca Baldassarre

Topic and its relevance. "Embodied cognition", postulating that high-level cognition relies on the same brain mechanisms subserving sensorimotor behaviour, is a frontedge flourishing research topic of cognitive psychology and cognitive neuroscience (e.g. related to mirror neurons). Here we use computational models and theoretical analysis to propose sufficient hypotheses on the architecture, functioning, and learning processes through which the dorsal and ventral cortical pathways of brain can guide on-line action control (based on affordances and motor programs) and top-down control of them (based on context, internal motivations, and goals).

Questions and goals. How are affordances and motor programs learned and encoded in the dorsal brain pathway? How are goals learned and encoded in the ventral brain pathway? How do context, motivations, and goals select actions? What is the relation between the dorsal/ventral pathways interaction and compatibility effects (i.e., reaction times being faster when object affordances suggest the same action triggered by high-level goals)? How does this system manage multiple objects, e.g. distractors, and how is it affected by diseases, e.g. Parkinson's disease?

Methods. We built various "computational embodied neuroscience" models of these processes based on firing rate-neuron neural networks, mainly learning with Hebbian associative rules, having architectures similar to those of brain subserving them (as indicated by brain imaging and also general macro-anatomy of brain). Based on these models, we also proposed theoretical views contributing to integrate and systemize the current knowledge on the topic.

Results. The models account for how affordances, encoded in the parietal cortex (within the dorsal pathway), guide on-line control of action and for how context/motivations lead to select goals, within the prefrontal cortex (ventral pathway), that bias action selection. This in line with the known constraints on the overall organisation of brain dorsal/ventral cortical pathways. The models account for a considerable number and variety of compatibility effects on the basis of the competition/cooperation between dorsal and ventral pathways biasing action selection mechanisms taking place in motor cortex. The models also explain how the cortical system (also working with basal ganglia) might process distractors and be affected by Parkinson.

Conclusions. The models specify the embodied view of cognition with specific hypotheses on the brain mechanisms supporting it, so generating a number of predictions, and general views, that might be tested in further empirical experiments. The proposed architectures and theories also contribute to develop an integrated view of the multitude of cognitive psychology experiments on the topic and the wealth of cognitive neuroscience data on them.

Example 1: TRoPICALS model and compatibility effects

The content of this example is extracted from Caligiore, D., Borghi, A. M., Parisi, D., & Baldassarre, G. (2010). TRoPICALS: a computational embodied neuroscience model of compatibility effects. Psychological Review, 117(4), 1188.

Behavioural and brain imaging evidence indicates that perceiving objects activates the representation of their affordances (Jeannerod, 1994). However, the extent to which this activation occurs in an automatic bottom-up way or it is modulated by the task itself, is subject of debate (Castiello, 1999; Borghi et al., 2007). One way of studying how internal representations of objects are related to action representations is to devise "compatibility effects experiments"(Tucker & Ellis, 2004) in which participants are asked to produce actions which are either in agreement (compatible) with the actions typically associated with the objects (e.g., a precision grip with a small object) or in contrast (incompatible) with those actions (e.g., a precision grip with a large object). If the participants employ longer reaction times (RTs) and higher error rates in incompatible trials than in compatible ones, one can infer that seeing objects automatically elicits the representations of their affordances, independently of the performance of the experimental task.

 

Figure 1. The boxes indicate the components of the model. The label inside each box indicates the type of information encoded by the component, whereas the acronym at its top-left corner indicates the brain anatomical area putatively corresponding to it. Light and dark grey arrows, respectively, indicate connections that were hardwired and connections that were updated by learning processes based on a Hebb covariance learning rule or a Kohonen learning rule. The input of the model is formed by three RGB visual neural maps (VC) and a somatosensory map (SSC). Downstream from VC and SSC, the model divides into two main neural pathways: the dorsal pathway, which implements suitable sensorimotor transformations needed to perform action on the basis of perception, and the ventral pathway, which allows flexible control of behavior thanks to the biasing effects exerted by the PFC on action selection. In turn, the dorsal pathway is formed by a pathway controlling grasping and a pathway controlling reaching. AIP: anterior intraparietal cortex; PC: parietal cortex; PFC: prefrontal cortex; PMCd: premotor cortex, dorsal division; PMCl: premotor cortex, lateral division; PRR: parietal reach region; RGB: red–green–blue; SSC: somatosensory cortex; STC: superior temporal cortex; STCi: superior temporal cortex for instructions; STCn: superior temporal cortex for names; VC: visual cortex; VOT: ventral occipitotemporal cortex.

This work proposes the TRoPICALS model to study compatibility effects suggesting an hypothesis on the brain mechanisms underlying them. TRoPICALS presents several innovative and compelling aspects. First, the account of compatibility effects is based on four general brain organisation principles incorporated by the architecture of TRoPICALS (see above): (a) the dorsal-ventral pathways organisation of the sensorimotor brain (Milner & Goodale , 2008); (b) the guidance of action selection based on prefrontal cortex (PFC) “instructions” (Miller & Cohen, 2001); (c) the selection of actions within premotor cortex (PMC) based on the competition between affordances with bias from PFC (Cisek, 2007); (d) the capability of language to trigger internal simulations of the referents of words (Barsalou et al., 2008). The acronym "TRoPICALS" summarises these principles: Two Route, Prefrontal Instruction, Competition of Affordances, Language Simulation. Within the model the dorsal pathway implements suitable sensorimotor transformations needed to perform action on the basis of perception. The ventral pathway allows flexible control of behavior thanks to the biasing effects exerted by the PFC on action selection. The model reproduces compatibility effects as an agreement or disagreement (compatibility/incompatibility) of top-down PFC bias with the available affordances of objects produces fast or slow reaction times.

Second, the model is designed within an approach that aims at developing it cumulatively so as to furnish increasingly general and comprehensive accounts of several compatibility effects involving grasping, reaching and language. The approach imposed four constraints on the model (Caligiore &
Fischer, 2012): (a) neuroscientific constraints on architecture and functioning; (b) reproduction of psychological experiments; (c) functioning within an embodied system; (d) reproduction of the learning processes. The claim on the generality of the model is supported by a critical comparison with other models that are related to the above four principles and by an analysis of how the model could be developed to account for other compatibility effects.

Third, TRoPICALS is capable of generating novel testable predictions (Caligiore et al., 2010), including some predictions on the possible outcomes of compatibility effects with Parkinson patients (Caligiore et al., 2012; the latter predictions are relevant as Parkinson patients have damaged excitatory and inhibitory neural circuits linking PFC to PMC).

Video on "Basilar affordances acquisition. On the rigth a video showing a basilar affordances acquisition with the TRoPICALS model. Before using TRoPICALS to reproduce the data of compatibility effects experiments with images and words, the system underwent a first learning phase that simulated learning to grasp objects as experienced by real participants during life. This learning phase led the model to acquire the affordance-based behavior within the dorsal stream (PPC-PMC connections) and to form the identity of objects within the ventral stream (VC-IT connections, VC not showed). The video refers to a past version of the TRoPICALS model with respect to the one showed on Figure 1 above, where we labelled AIP as PPC; PMCl as PMC and VOT as IT. The training was performed by repeatedly presenting several objects (in the video only two for simplicity) each of the eight objects to the model (the simulation steps related to the presentation of one object are here referred to as trial). On the perceptual side, at each presentation, the VC performed color based edge detection of the object image, and the PPC performed color-independent edge detection (i.e., extracted the shape of objects) by averaging the activation of RGB neurons having the same position in the three maps of the VC. PPC–PMC connection weights were updated on the basis of a motor-babbling process and a Hebb covariance learning rule. This allowed the dorsal pathway to acquire the capacity to perform the correct grasp depending on the shape of objects. During motor babbling, the ventral stream acquired the capacity to categorize objects on the basis of their appearance. In particular, VC–IT connection weights formed the categories of objects on the basis of a Kohonen learning rule.

                                                                                        (a)                                        (b)                                 (c)               (d)       

Figure 2. The columns, from left to right, report (a) the object name, appearance, and handgrip on it; (b) the activation of the neurons forming the visual cortex (three red– green– blue maps) caused by the eight objects; (c) the activation of AIP encoding the shape of objects; and (d) the activation of the VOT encoding the identity of objects. Notice the different grips for the different objects, the activations of the visual cortex caused by the variety of shapes and colors of the objects, the encoding of shape in AIP, and the abstract representation of the identity of the objects in the VOT. AIP anterior intraparietal cortex; VOT ventral occipitotemporal cortex.

                                                   (a)                                                                                                                              (b)

Figure 3. Results of the experiment with images: RTs (y-axis) versus kind of grip (x-axis). (a): Average RTs exhibited by the real participants of the experiments of Tucker and Ellis (2004). From “Action Priming by Briefly Presented Objects,” by M. Tucker and R. Ellis, 2004, Acta Psychologica, 116, p. 198.. (b): Average RTs exhibited by the simulated participants. Notice the compatibility effect: Large objects tend to speed up RTs of power grips, whereas small objects tend to speed up RTs of precision grips. RT = reaction time.

On the left (a): Activation of the premotor cortex (PMC) in an incongruent trial. The biases from the prefrontal cortex (PFC) and the parietal cortex (PC) cause two different clusters of neurons to compete until the cluster caused by the PFC suppresses the cluster caused by the PC and triggers the suitable action. b: Activation of the PMC in a congruent trial: The biases from the PFC and the PC overlap and cause the formation of only one cluster of neurons. The panels depict the activation of the PMC after 100, 300, and 1,000 ms. The video below shows the neural dynamic competition process within the TRoPICALS model (premotor cortex) leading to have an increased reaction time in the incongruent trial.

Key references.

1. Barsalou, L. W., Santos, A., Simmons, W. K., & Wilson, C. D. (2008). Language and simulation in conceptual processing. In M. De Vega, A. M. Glenberg, & A. C. Graesser (Eds.), Symbols, embodiment, and meaning (pp. 245–284). Oxford, England: Oxford University Press.
   
2.  Borghi, A. M., Bonfiglioli, C., Lugli, L., Ricciardelli, P., Rubichi, S., & Nicoletti, R. (2007). Are visual stimuli sufficient to evoke motor information? Studies with hand primes. Neuroscience Letters, 411, 17-21.
   
3. Borghi, A. M., Glenberg, A. M., & Kaschak, M. (2004). Putting words in perspective. Memory & Cognition, 32, 863–873.
   
4. Caligiore, D., Borghi, A. M., Parisi, D., & Baldassarre, G. (2010). TRoPICALS: a computational embodied neuroscience model of compatibility effects. Psychological Review, 117(4), 1188.
   
5. Caligiore, D., Borghi, A. M., Parisi, D., Ellis, R., Cangelosi, A., & Baldassarre, G. (2013). How affordances associated with a distractor object affect compatibility effects: A study with the computational model TRoPICALS. Psychological Research, 77(1), 7-19.
   
6. Caligiore, D., & Fischer, M. H. (2013). Vision, action and language unified through embodiment. Psychological research, 77(1), 1-6.
   
7. Castiello, U. (1999). Mechanisms of selection for the control of hand action. Trends in Cognitive Sciences, 3, 264–271.
   
8. Cisek, P. (2007). Cortical mechanisms of action selection: The affordance competition hypothesis, Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences, 362, 1585-1599.
   
9. Jeannerod, M. (1994). The representing brain: Neural correlates of motor intention and imagery. Behavioral and Brain Sciences, 17, 187–246.
   
10. Miller, E. K., & Cohen, J. D. (2001). An integrative theory of premotor cortex function, Annual Review of Neuroscience, 24, 167-202.
    
11. Milner, A. D., & Goodale, M. A. (2008). Two visual systems re-viewed. Neuropsychologia, 46, 774-785.
    
12. Thill, S., Caligiore, D., Borghi, A. M., Ziemke, T., & Baldassarre, G. (2013). Theories and computational models of affordance and mirror systems: an integrative review. Neuroscience & BioBehavioral Reviews, 37(3), 491-521.
    
13. Tucker, M., & Ellis, R. (2004). Action priming by briefly presented objects. Acta Psychologica, 116, 185–203.

Example 2: Enhancing the TRoPICALS model to study dorsal/ventral interplay during mental rotation

 

 

On the left a schema of the bio-inspired model of dorsal and ventral interplay controlling the iCub  behaviour during a mental rotation task. On the right the video shows the performance of the iCub robot during a mental rotation task. The bio-inspired model of dorsal and ventral interplay controls the iCub behaviour in order to accomplish the mental rotation task. If during mental rotation the robot turns its wrist in the same direction as the mental rotation the robot performs better the task. By contrast, the performance is worse if the robot turns its wrist in the opposite direction to the mental rotation.

Key references

1. Seepanomwan, K., Caligiore, D., Baldassarre, G., & Cangelosi, A. (2013). Modelling mental rotation in cognitive robots. Adaptive Behavior, 21(4), 299-312.
2. Caligiore, D., Borghi, A. M., Parisi, D., & Baldassarre, G. (2010). TRoPICALS: a computational embodied neuroscience model of compatibility effects. Psychological Review, 117(4), 1188.