ITS THE WAY YOU SAID IT!

AN EVALUATION OF EMOTIONAL TONE IN APROSODIA

Marie Odejar, Ji Sook Ahn, Sona Patel
Sona.Patel@shu.edu

Voice Analytics and Neuropsychology Lab, Department of Speech-Language Pathology

Seton Hall University, South Orange, NJ 07079
RESULTS
Figure 3. Confusion matrix (i.e., what was perceived
instead of the intended emotion).

According to Blake et al. (2013), speech-language pathologists in healthcare

settings require more information about patients with right hemisphere damage and
their resulting aprosodias in order to provide quality care and treatment.
Sadly, methods to evaluate and rehabilitate individuals with emotion communication
deficits are lacking.

ASSESSMENT TOOLS
Florida Affect Battery (FAB; Leon, 2005)
Florida Emotional Expressive Battery
(FEEB)

Bedside evaluation procedure (Ross, 1981)

Battery of Emotional Expression and
Comprehension (Cancelliere & Kertesz, 1990)
Aprosodia Battery (Ross et. al, 1997)

OUR STUDY
The goal of our study was to learn about
the speech problems in people with
aprosodia. In doing so, we were able to
gain insight into one assessment tool for
evaluating emotional speechthe FEEB.
Our approach was to examine how
people with some experience in hearing
disordered voices evaluate the emotional
speech of people with aprosodia.

LIMITATIONS
Some evaluate only one domain, such
as receptive ability (e.g., FAB).
Confounding factors such as sentence
context and visual information may
bias the clinicians judgement.
None of these tests give insight into
the magnitude of expressivity or how
much emotion was produced.

53%

51%

25%

Angry

Happy

Expressive
an impaired ability to change
ones voice to express
common emotions such as joy
or anger (Rosenbek et. al, 2006)

Receptive
impaired ability
to perceive
emotional
tones

Sad

Neutral

Fear was the most difficult to perceive with an average of

25% accuracy across listeners and patients.
Most speech samples were perceived as neutral or sad.

FEEBa, which had

utterances that
contained emotional
context, was easier to
identify and perceive
when compared to
scores of FEEBb,
which had neutral
context utterances.

FEEBb
30%
FEEBa
70%

FEEBa
FEEBb

Figure 1. Types of aprosodia.

METHOD

Fear

60%

Figure 5. Percentage of correctly identified statements

as related to their context (FEEBa vs. FEEBb)

Figure 4. Percent correct for each patient.

Happy_p
Sad_p

50%
40%
30%
20%
10%
0%

62%

59%
51%

p1
Angry_i

Intended Emotions

APROSODIA
a disruption in the expression or
comprehension of the changes in pitch,
loudness, rate, or rhythm that convey a
speakers emotional intent (Leon et al., 2005).

72%

Angry_p
Fear_p
Neutral_p

Happy_i

Fear_i

Sad_i

p2

Neutral_i

50%

49%

p3

p4

p5

Patient

Intended Emotions
Listeners tended to confuse the patients expressions as sad
and neutral 30% of time or more in total.
28% of happy samples were judged as sad, which is large as
only 46% of happy samples were correctly identified.
Fear, the most difficult emotion to perceive correctly (25%
correct), was misidentified as neutral 44% of the time.

FEEBa emotions
(angry, happy, sad)
were significantly
identified correctly
more often than the
same emotions in
FEEBb.
Even with neutral
context, sad
emotion was still
perceived more
accurately than the
others in both FEEBa
and FEEBb.

100%

FEEBa

Regardless of the characteristics of the neurological injury, the

speech of all patients were similarly difficult to perceive.
Patient 4 varied his/her intonation which helped in identifying the
emotion category, even though these were sometimes unnatural.

FEEBb

60%
40%
20%
0%
Angry

Happy

Sentences with a
neutral sentence
context and an
incongruent tone and
context were most
difficult to produce.

100%

80%

Sad

Intended Emotions

Listener % Accuracy

Thus, aprosodia can cause psychological problems that affects a persons quality of
life, even though the underlying issue is actually a motoric one (Sidtis & Sidtis, 2003).

66%

70%

Listener % Accuracy

The unfortunate consequence is that people with aprosodia are often perceived as
depressed or disinterested and can lead to social isolation or exclusion.

Listener % Correct

People with aprosodia often have a flat prosody or intonation pattern, meaning that
speakers sound more or less monotone. The components of prosody help others
infer a speakers attitudes and emotions during everyday interactions (Tariska, 2009).

Figure 2. Overall accuracy of each emotion.

Listener % Correct

Vocal communication of ones emotions is important for interactions with ones

spouse, family, and friends. In many neurological diseases this ability is impaired or
nearly absent and is referred to as APROSODIA (or dysprosody).

Listener Accuracy (%)

INTRODUCTION

80%
60%
40%
20%
0%

Figure 6. Percent correct for FEEBa vs

FEEBb by each emotion.

Congruent

Incongruent

Neutral

Sentence Type

SPEECH MATERIAL
238 samples selected from a database of 5 patients with expressive aprosodia,
secondary to right hemisphere stroke.
The patients were asked to express sentences (FEEB Subtest 8a and 8b) with one of
five emotional tones (happy, sad, angry, fearful, neutral) while being recorded.
Sub 8a: Congruent vs incongruent prosody/text
Sub 8b: Neutral text spoken with emotional tones

LISTENERS
10 subjects (20-27 years) with at least 20 observation hours under a certified SLP
clinician. All but two were graduate students in Speech-Language Pathology at SHU.
All listeners passed a hearing screening at 25 dB HL at octaves from 250-8000 Hz.
All participants reported that they were not on medications to treat depression, bipolar
disorder, anxiety, schizophrenia or any other mood disorder and scored below a 50 on
the Zung Self-Rated Depression Scale (Zung, 1972).

PROCEDURES
Listeners were given a brief training task to familiarize them with the experiment.
The main experiment was conducted on a computer controlled by Matlab software.
The speech samples were randomly presented over headphones, one at a time.
The emotion intended by the speaker was shown (consistent with the FEEB).
Listeners were asked to judge, in your clinical opinion, whether the speaker correctly
produced the intended emotion or if they produced another one of the five emotional tones
shown instead. After subjects selected the emotion category they were asked to rate how
well the speaker produced the intended emotion from 1 (poor job) to 7 (good job).

Listeners were instructed to make judgments based on ONLY the tone-of-voice.

Responses were saved into Excel and averaged across listeners, patients, & emotions.

DISCUSSION
Patients with expressive aprosodia have difficulty in varying their intonation, which, as our study shows, results
in the majority of their speech samples being judged as neutral (lack of emotion) or sad. Even high arousal
emotions such as anger and fear were more often judged as neutral than as one another.
EMOTIONS. Of the 5 emotions, fear was the most difficult to perceive.
This is in line with results from Rosenbek et al. (2006) who stated, Fear was more challenging to judge
since expressions of fear can vary by context. For this reason, it may also have been more difficult to
produce (which is the case even for normal speakers). Thus, when choosing emotions to treat clinically, it
may be more important to focus on anger and happiness and contentment.
Most studies that do not test fear report happy as the most difficult emotion to perceive. The production of
happy speech requires more variation in intonation, which results in the misperception.
In consequence, relationships with family members, loved ones, co-workers, and even other professionals,
will be difficult to maintain because they wont be able to engage in the conversation with the appropriate
emotion (Wymer, 2002), potentially resulting in social isolation and low quality of life for the individual.
CONTEXT. We also observed that the listeners found it easier to identify the emotions of statements from
FEEBa. These results were influenced by the congruent statements, in which patients produced emotional
tones that matched the sentence text. Listeners found these much easier to identify (80%) than the
incongruent statements, or those that didnt match the text (43%).
We conclude that using the FEEB in a clinical setting to assess emotion communication ability does not match
how everyday listeners perceive the emotions of the speaker. The results of our study can be used to develop
methods for assessing communication through voice and speech.

FUTURE WORK
Some patients with aprosodia have difficulty expressing their emotions altogether, while for others, the
intended emotion could be identified, although with abnormal intonation patterns. Current methods to
assess emotional speech do not capture this variation. Judgments made using a rating scale in which
the clinician can numerically rate how much or how well the intended emotion was produced may
address this problem.
It is likely that objective methods to evaluate the voice based on the acoustics will help to eliminate
other factors that probably influence ones judgments, such as visual cues, accustomization to the
patients voice due to regular exposure, and awareness of the intended emotion.

ACKNOWLEDGMENT
This research was supported by a grant from National
Institute of Health (NIH), Grant No. 1R03DC013883-01.

REFERENCES
Blake, M. L., Frymark, T., & Venedictov, R. (2013). An Evidence-Based Systematic Review
on Communication Treatments for Individuals With Right Hemisphere Brain Damage., 22.
Booksh, R. L. (2002). A neuropsychological perspective of aprosody: features, function,
assessment, and treatment. Applied Neuropsychology, 9(1), 3747.
Bowers, D., Blonder, L. X., Heilman, K. M., Shaw, R. J., Dong, M., Lim, K. O., & ... Alpert, M.
(1999). Florida Affect Battery. Schizophrenia Research, 37245-250.
Cancelliere, A.E.B. & Kertesz, A. (1990). Lesion localization in acquired deficits of emotional
expression and comprehension. Brain and Cognition, 13(2), 133-147.
Nakhutina, L., Borod, J. C., & Zgaljardic, D. J. (2006). Posed prosodic emotional expression
in unilateral stroke patients: Recovery, lesion location, and emotional perception.
Archives of Clinical Neuropsychology, 21, 113.
Leon, S. A., Rosenbek, J. C., Crucian, G. P., Hieber, B., Holiway, B., Rodriguez, A. D., &
Gonzalez-Rothi, L. (2005). Active treatments for aprosodia secondary to right hemisphere
stroke. Journal of Rehabilitation Research and Development, 42(1), 93102.

Nakhutina, L., Borod, J. C., & Zgaljardic, D. J. (2006). Posed prosodic emotional
expression in unilateral stroke patients: Recovery, lesion location, and emotional
perception. Archives of Clinical Neuropsychology, 21, 113.
Rosenbeck, J.C., Rodriguez, A.D., Hieber, B., Leon, S.A., Crucian, G.P., Ketterson,
T.U., Ciampitti, M., Singletary, F., Heilman, K.M., & Rothi, L.J.G. (2006) Effects of
two treatments for aprosodia secondary to acquired brain
injury. Journal of
Rehabilitation Research & Development, 43(3), 379-390.
Ross E.D. (1981). The Aprosodias: Functional-Anatomic Organization of the Affective
Components of Language in the Right Hemisphere. Archives of Neurology, 38(9),
Ross, E.D., Thompson, R.D., & Yenkosky, J. (1997). Lateralization of Affective
Prosody in Brain and the Callosal Integration of Hemispheric Language Functions.
Brain and Language, 56(1), 27-54.
Sidtis, J. J., & Van Lancker Sidtis, D. (2003). A neurobehavioral approach to
dysprosody. In Seminars in Speech and Language (Vol. 24, pp. 93106).
Takagi, S., Hiramatsu, S., Tabei, K., & Tanaka, A. (2015). Multisensory perception of
the six basic emotions is modulated by attentional instruction and unattended
modality. Frontiers in Integrative Neuroscience, 9, 110.
Tariska, P. (2009). The aprosodias: Clinical manifestation and significance of the
syndrome. Journal of the Neurological Sciences, 283(1-2), 277278.
Wymer, J. H., Lindman, L. S., & Booksh, R. L. (2002). A neuropsychological
perspective of aprosody: features, function, assessment, and treatment. Applied
Neuropsychology, 9(1), 3747.
Zung, W.K. (1972). How Normal is Depression. Psychomatics, 13, 174-178.