Multilevel analysis of facial expressions of emotion and script: self-report (arousal and valence) and psychophysiological correlates

In the last two decades, developmental psychology has seen an increasing interest in the study of emotion comprehension. Specifically, emotional face recognition and understanding represent a primary social competence, because they contribute to social interactions and social management[1]. These competencies are related to general cognitive functions and it was shown that language was the most important predictor of nonverbal emotion recognition ability[2]. Indeed both gender and verbal skills are important predictors of children’s emotional awareness[3].

More specifically, Bullock and Russell[4] proposed a model in which children acquire a system to represent and classify emotions which is based on a limited number of wide categories. The most important of them are the two dimensional axes of the hedonic value and the arousal level. This model was tested by some empirical studies which found that firstly children interpret facial expressions in terms of pleasure-displeasure (bipolar hedonic value) and intensity (arousal level). Only successively they use more articulated and wider conceptual categories[5, 6]. To verify the type of categorization applied to the emotional domain, affective responses organized around the arousal and valence dimension include subjective experience, often measured using self-report responses to affective stimuli. At this regard, Self-Assessment Manikin (SAM) was used to test this subjective emotional correlates[7]. It was also demonstrated that age, facial expression intensity and emotion category are important for predicting accuracy on emotion-processing tasks[8].

Previous results demonstrate how task type and children’s mood influence children’s emotion processing[9]. Indeed, in order to explain this developmental process, the type of emotions children have to recognize is a first main factor related to decoding competencies[10]. More generally, in line with Russell’s model of emotional experience, emotion fundamentally varies activation in centrally organized appetitive and aversive motivational systems that have evolved to mediate the wide range of adaptive behaviors necessary for an organism struggling to survive in the physical world[11–13]. Most pleasant affects are held to be associated with the appetitive motivation system; unpleasant affects with defensive motivation[14]. Thus, a primary distinction among emotional events is whether they are appetitive or aversive, positive or negative, pleasant or unpleasant, which clearly relates to the motivational parameter of direction. Secondly, all agree that hedonically valenced events differ in the degree of to which they arouse or engage action, which is related to intensity parameter. Emotional intensity probably reflects the strength of activation in motivational systems subserving appetitive and defensive behaviors and, as such, has impact on the type of physiological response. Intensity was conceptualized as “predator imminence”, or the distance of the threatening pattern from the subject[15] or in terms of distance from an aversive or appetitive goal[16]. More generally, arousal in humans appears to reflect the degree to which a stimulus elicits appropriate appetitive or defensive behaviors. Thus, the two dimensions of pleasure and arousal explain the majority of the emotional experience and subjective judgment, and the increased perception of emotional significance of a stimulus in term of valence may increase the perception of its arousing power[17].

Secondly, it should be noted that young subjects showed to be competent in the decoding of the primary or simple emotions (e.g. happiness and anger), but they have more difficulties in processing secondary or complex emotions, such as pride and embarrassment[18–20]. To identify these emotions more time and more informational cues must be analyzed. Moreover, as regard to the secondary emotions or emotions developed only later during the developmental phases, a more accentuated difficulty in understanding causal antecedents (the events that caused the emotion expressed by face) and contextual relations (the social situations in which the emotion is produced) emerges[21]. Bormann-Kischkel et al.[10] observed a specific difficulty in understanding the emotions that present a lack of correspondence between people expectations and environment events. These emotions have an external and social origin, such as surprise, dismay, and astonishment. In parallel, Capps, Yirmiya, and Sigman[20] observed a greater impairment in recognizing and labelling the expression of those emotions that have an external locus of control and, simultaneously, that require a wide knowledge of the social scripts and of their social consequences. In line with this hypothesis, Baron-Cohen, Spitz, and Cross[22] suggested that the comprehension is more difficult for emotions that imply the activation of some cognitive functions, such as mentalization and metarepresentation. In general, previous results provide evidence for late developmental changes in emotional expression recognition with some specificity in the time course for distinct emotions[23]. Indeed it was found that some emotions, like disgust and sadness, are more often confused with other primary and early developed emotions, such as anger or fear and they are not spontaneously labelled in comparison with other emotions such as anger, fear or happiness.

Thus, it was also supposed that, through a progressive process of script generalization, a “situated” comprehension of emotions arises. The process of emotion categorization is well illustrated by the use of adequate attribution. Indeed, emotional labels constitute the final step of a developmental process that goes through a “dimensional attribution” (characterized by the presence of pleasure-displeasure correlate) to a “situational attribution” (the script representation). This should be taken into consideration in studying the development of emotional decoding, because these competencies seem to be bound not only to cognitive but, above all, to social and communicative competencies, which have an influence on emotion conceptualization. Thus, another main concern is represented by contextual and situational elements that cause emotion and that might facilitate or not the emotion processing and comprehension[24]. As Russell and Widen[25] underlined, in everyday experience children use facial expressions in order to infer emotions. On the other hand, the facial cues are always located in an interactive context. In other words, it is necessary to take into account the role of a wider socializing context. Therefore, the concept of emotional context, considered as a complex and multidimensional representation of situational events, is relevant in facial expression processing.

Thirdly, it was observed that behavioral and physiological responses to emotional pictures co-vary with system evaluation (pleasure/displeasure) and motive intensity (arousal)[26, 27]. Psychophysiological responses are not directly observable, and they include cardiovascular, electrodermal, respiratory measures, etc. It was underlined that emotion comprehension plays a critical role in adaptive behavior, since it promotes survival and guides human behavior by exerting a direct influence on brain responsiveness and psychophysiological activities. Between the others, facial action revealed by EMG measure (electromyogram), heart rate, and skin conductance were observed to variate in concomitance of pleasure and displeasure reports while viewing of emotional patterns.

More specifically, about the psychophysiological measures, facial behavior using electromyography (EMG) suggested they were sensitive to the valence dimensions, with increased corrugator activity in response to unpleasant patterns and zygomatic activity in response to pleasant patterns. Facial EMG (electromyographic) activity accompanies changes in appetitive (positive) and defensive (negative) activation[28]. Specifically, the corrugator muscle appears to be responsive of to judgment of unpleasant event compared to neutral pictures[27]. Many studies found a consistent and significant relationship between corrugator and hedonic valence, with greater corrugator activity elicited when viewing the most unpleasant stimuli[29]. Moreover, Bradley, Codisposti, Sabatinelli, and Lang[30] showed that pictures that produce disgust (for example mutilation), that were higher in arousal, prompt larger changes than other unpleasant pictures.

Other physiological measures of emotional behavior include heart rate (HR), with observed increased HR acceleration to pleasant patterns and increased HR deceleration to unpleasant patterns[30]. Moreover, investigations exploring cardiovascular activity in emotion perception assessed variations as a function of differences in stimulus intensity, as this variable was revealed critical in eliciting orienting or defense response[27, 30–32]. Low-intensity stimuli were found to relate with heart rate deceleration, whereas intense stimuli were observed to activate defense responses associated with heart rate acceleration[33–36]. Nevertheless, also contrasting results were collected, since heart initially decelerated, rather than accelerated, when people viewed pictures of unpleasant emotional events, contrary to the notions that these aversive stimuli might prompt defensive heart rate acceleration[27, 30, 37]. However, different experimental paradigms were adopted in previous research and, in some cases, no direct comparison can be conducted between them.

Moreover, it was found electrodermal activity (Skin Conductance Response, SCR) consistently varies with emotional intensity, with larger responses elicited in either unpleasant and pleasant context and that are more pronounced in those that are rated as highly arousing[27, 38, 39]. Thus, also electrodermal reactions increase with increases in defensive or appetitive activation[30, 37]. In general, it was found increased skin conductance when people view pictures rated as emotional, compared to neutral, regardless they are rated pleasant or unpleasant in hedonic valence[27]. However, when hedonic valence and emotional arousal were co-varied, skin conductance responses were largest for highly arousing stimuli, irrespective of hedonic valence[40], consistent with the notion that these reactions primarily reflect differences in emotional arousal, rather than hedonic valence per se.

About these psychophysiological variations in response to emotions and facial stimuli, an important debate regards the presence of a coherent response by psychophysiological measures in childhood, as it was observed in adult behavior. Previous research found consistent patterns of psychophysiological activation also by children in response to emotional stimuli[41, 42]. Nevertheless, to verify the coherence of these physiological measures in young people in response to facial emotional patterns, specific analysis should be conducted which included both arousal and valence parameters.

Therefore, emotional behavior manifests within multiple domains, comprehending conceptual and self-report attribution, autonomic responses (physiological systems), and the comprehension of contextual components, which all may have a significant role in this process. However, no previous study has directly analyzed the relationship between these multilevel measures, that is self-report evaluation based on valence and arousal parameters, psychophysiological behavior and contextual cue variability. The present study was finalized to explore the convergence of these different measures.

In the present research the effect of some main factors, valence modulation (emotional type) from one hand, and contextual effect (face alone vs. facial display within a script), from the other, was considered. Specifically, we explored their influence on physiological reactivity (autonomic activity) and emotional attribution (self-report attributional process), which are all relevant to the description of the emotional responses[26, 43]. Thus, the purpose of this study is to verify the attended psychophysiological and attributional responses to emotion variation, and, secondly, to show that the attributional process was related to valence and to context modulation.

The present study produced three major results, that we summarize in the following points. First, there was a clear differentiation in children’ conceptualization (in terms of arousal and valence) as a function of different emotions; besides, the psychophysiological measures were highly modulated by emotional types, and arousal and valence parameters accounted for the psychophysiological variations in relationship with different emotional patterns; finally the presence of two different types of task – a facial expression decoding and a script comprehension –induced significant differences in the subjective representations only for a limited number (mainly disgust) of emotions.

For the first time we used multimodal measures to explore the evaluation effect (based on valence and arousal) on psychophysiological behavior taking into account an ample range of emotions. Secondly we applied this multimodal approach to study the specific domain of facial expression of emotions whereas other previous research did not specifically consider this emotional domain. Thirdly we considered the facial expression of emotion with and without an emotional script context to study the contextual impact on face decoding. Therefore the situated perspective was adopted in the present research.

As hypothesized by the dimensional approach to emotion[52, 53], the representation of the emotional domain was based on a conceptual space defined by two exes, arousal and hedonic value. In particular, the emotions with a high arousal level and a negative value were better understood, if compared with other emotions. Specifically, the emotions of fear, anger and surprise were well recognized and well labeled. A significant higher arousing power was attributed to them, and these emotions were also considered as more negative. Moreover, they were better recognized than the other emotions, specifically in comparison with sadness and disgust. The positive emotion of happiness was considered as less arousing and more positively valenced and it was well represented and recognized. On the contrary, disgust appears to be more difficult to be identified, as well as sadness, and they both were considered as less arousing and less negative. It should be considered that in present research we opted to evaluate the ability of subjects in spontaneously labelling the face/script they saw. As revealed, disgust and sadness were not immediately labelled, but in many cases they were correctly described (using a semi-structured interview) only after a successive enquire. Therefore, the subjects showed a general ability in recognizing the two emotions, although this recognition was less immediate. It should be based on the increased complexity to decode these emotions, because they are learned only successively in comparison with other primary emotions (such as anger and fear).

Therefore a first main result of the present study was that the dichotomy pleasure/displeasure and high/low arousal was considered relevant by the subjects, confirming a significant role in emotion representation, as indicated by previous researches[19, 53, 54]. In fact, not only the hedonic category was systematically well represented, but it was correctly identified in terms of negativity or positivity. Moreover, arousal rating can be considered a predictive cue of the ability to classify and differentiate emotional correlates. Indeed, it was correctly used when the child was able to attribute an adequate label to the emotion, while when the child cannot conceptualize the emotion, the arousal value seems to be more ambiguous (for example for disgust) or less relevant (sadness).

As regard to more negative and arousing emotions (fear, anger and surprise) some recent study[55, 56] revealed high rates of recognition, that the researcher attributes to the central adaptive function of these negative high arousing emotions. Indeed, they has a main role for the individual safeguard, both on an ontogenetic and a phylogenetic level. They may be represented as a cue in order to detect unfavorable environmental conditions[19, 54]. Accordingly to the functional model[57, 58], the emotional expressions represent a response to a particular event, significant in terms of costs and benefits for people. Specifically, the expression of anger and fear represents the perception of a threat for the personal safeguard and, therefore, it requires a greater investment of attentional resources. The prominence of specific category of emotion (more negative and arousing) may suggest their central role in emotion acquisition in comparison with other less relevant (and less arousing) emotions in childhood.

The script condition introduces another main explicative factor, regarding the emotional representation. Indeed, the presence of a specific context generally does not affect the correctness of the emotional label attribution, but it produces a discriminant effect exclusively for one emotion, that is disgust. Indeed in presence of a specific situational context disgust was better characterized in terms of arousal (more clearly arousing) and valence (more negatively valenced). The presence of the interactional features that characterize the emotional experience seems to introduce a facilitation element for emotion comprehension, also producing a better description in the emotion labeling (more correct recognition). It was possible to state that the situational component constitutes a facilitation cue, because it allowed the subjects to activate a more complex conceptual representation, which takes into account the context in which the emotional event happens, the emotional causes, the logical order of actions and their consequences[4].

It was noticeable, however, that the script enables a wider and a more complete representation only in case of a this “secondary” emotion, which maximally has a benefit from the situated condition. It was observed that emotion recognition was allowed by the development and the generalization of an emotional script, that is, a child can recognize a specific emotion by verifying the presence of several prototypical elements that are arranged in precise causal and temporal sequences. These scripts include not only facial expressions, but also the representation of causal factors, physical and social context, several actions and their consequences, as well as the cognitive appraisal of the situation and the subjective experience[4]. Among these cues, the representation of the causal bonds, that is a set of causal events and of their behavioral consequences, has a remarkable significance, because they constitute the more explicative elements of the emotional experience[5, 45, 59].

To conclude, even if our study does not allow us to state which of the two representational modalities (facial pattern comprehension or script decoding) precedes the other, it was possible to observe that the situational correlates provide a facilitation cue for the representation of emotional correlate when a secondary emotion is represented. However, no specific facilitation effect was observable in case of “primary” emotions, which were well recognized and described also in absence of contextual cues.

A relevant main result of the present research was related to the psychophysiological measures which were shown to vary in concomitance to the type of stimuli (different emotions) and to the categorization process (the subjective ratings). In fact, subject revealed a coherent psychophysiological behavior in response to the emotions, independently from the condition (script or face). Moreover, it was shown that SCR, HR and EMG were modulated as a function of the two main axes of valence and arousal, as they were rated by the subjects.

Firstly, SCR was shown to be increased when children processed emotional faces and scripts rated as high arousing and negative (anger, fear and surprise), whereas it decreased in concomitance with stimuli rated as low arousing (mainly sadness, disgust, and neutral patterns). A similar profile was observed for HR, which showed higher values in case of more positive, more negative and arousing stimuli. These results were in line with many other studies on adults, which postulated a significant HR effect for more arousing and relevant stimuli[33–36]. Moreover, the variation in term of arousing power (high vs. low) may determine the different impact of the emotional cues, since perception of a high arousal generally induces a consistent HR increasing independently from the stimulus valence. These multiple parameters and their combination were relevant to comprehend the effect of emotions on psychophysiological data.

An important result was also observed for the facial EMG values. Indeed we found that children were highly responsive to facial stimuli and scripts, by adopting a sort of “facial feedback” modality, since they used similar facial configurations displayed by the pictures (consonant behavior)[60]. It was observed an increasing of mimic activity in case of some conditions: the different emotions evoked distinct facial EMG response patterns, with increased zygomatic muscle activity to positive patterns and increased corrugator muscle activity to negative patterns, whereas both the corrugator and the zygomatic muscle response patterns were less pronounced in sadness, disgust and neutral condition. More generally, corrugator muscle activity was increased in response to more negative and arousing stimuli, mainly for fear, anger, and surprise. In addition, as revealed by regression analysis, arousal parameter showed to explain in greater measure the corrugator modulation, whereas valence was less relevant to describe the psychophysiological activity in response to negative, highly arousing patterns. Contrarily, zygomatic muscle was modulated by both arousal and valence, with significant increasing responsiveness related to happiness.

These variations may mark a psychophysiological response in case of a high arousing situations, since relevant (with arousing power) stimuli seem to produce and reinforce a coherent psychophysiological behavior. Contrarily, subject reported a reduced arousing power for sadness and partially for disgust, fact that may explain the concomitant reduced EMG, SCR and HR values. Thus, more arousing conditions showed a perfect consonance between subjective evaluation and psychophysiological (both facial and autonomic) measures. Specifically, anger, fear, surprise and happiness were rated as more emotionally involving. In parallel, the psychophysiological behavior was responsive of this subjective self-evaluation, with an increased “positive” (zygomatic) facial expression and a higher autonomic activity (increased HR) for happiness, from one hand; an increased “negative” (corrugator) facial expression and higher arousal response (more SCR and HR) for anger, fear and surprise, from the other.

However, more generally the modulation of psychophysiological measures was mainly related to arousing power more than to valence, since independently from the valence, the stimuli rated as high arousing (anger, fear, surprise and happiness) were able to induce a more significant and coherent emotional response. Regression analysis confirmed these results: mainly arousal attribution was significant to determine the psychophysiological variations, able to explain SCR, HR and facial response modulation, since subjects “shared” the facial behavior and autonomic activity observed in both positive vs. negative conditions.

Thus, in general psychophysiological measures may be interpreted as functional mechanism of “mirroring” the emotional condition displayed by the facial stimuli, where “sharing” similar emotional responses allows a direct form of understanding and recognize emotion by a sort of simulation process. More specifically, contexts evaluated as emotionally involving and significant may ingenerate a consonant shared response by the observer, who firstly recognizes and secondly “mimic” (by face and autonomic behavior) the somatic markers related to the experienced emotions[61]. Moreover, based on these results we may suggest that the gradual development of emotional competencies proceeds from more basic and simple emotions, which are primarily acquired by children, to more complex and less prominent emotions, which might be less relevant in terms of salience. Brain correlates may support this different learning process, related to a “maturation effect” which might explain more deeply the early acquisition of the recognition abilities in response to more salient and relevant emotions in term of human safeguard and the successive acquisition for the less relevant (less threatening and primary for the safeguard) emotions.

To summarize, self-report measures were replicated by psychophysiological behavior, that showed to vary coherently in relationship with different emotions. Children revealed a consonant and adequate behavior in terms of labeling (correct recognition), evaluation (valence and arousal attribution) and psychophysiological responsiveness. However, a clear advantage was observed for some specific emotions, those rated as more arousing and negative (fear, anger and surprise). It was suggested these emotions may be central to people safeguard and they may be priority developed by children. Arousal attribution was considered as the most critical parameter to explain the emotion recognition process and the psychophysiological behavior. Contrarily, sadness and disgust were less prominent in terms of both arousal and valence, and in some cases they were also less correctly recognized. The contextual cues (script condition) may allow to perform a better attribution, mainly for the emotion of disgust. In case of more complex emotional cue, the context (script) contribution was relevant to complete the correct recognition.

However, about the main limitations of the present study, future research may explore more directly the intrinsic effect induced by facial expression of emotion taking into account also gender effect. Indeed previous research found significant differences between male/female children in response to the emotional type. Secondly, the arousal effect we found in the present study should be better considered in relationship with different emotional valence taking into account a wider range of facial expressions which may cover the ample orthogonal axes low/high arousal positive/negative valence. Thirdly, due to the limited sample we used for the present research, it is crucial to integrate the present data with an ampler sample size, in order to extend the present results to a general population.