CHI 2005 | Late Breaking Results: Posters
NEmESys – Neural Emotion Eliciting System
Manfred Eckschlager Regina Bernhaupt
ICT&S Center, Univ. Salzburg ICT&S Center, Univ. Salzburg
Salzburg, Austria Salzburg, Austria
Manfred.Eckschlager@sbg.ac.at Regina.Bernhaupt@sbg.ac.at
ABSTRACT
This paper describes the development of a new model of
agent emotion elicitation called Nemesys. It enhances inter-
faces with emotional and social information. Nemesys is
based on an artificial neural network and is able to learn six
basic emotional states. The elicitation of emotions is based on
models drawn from the state of the art in modeling emotions
in the field of psychology. Further the described framework
includes the Five-Factor Model of Personality to represent
different agent personalities.
Nemesys (called after Nemesis, the Greek goddess of right-
eous anger) is designed to perform in various types of inter-
faces. The usage of Nemesys is presented with an application
scenario employing a commercial 3D game-engine. Addi-
tionally a critical review of the current elicitation behavior of
Nemesys is presented and discussed.
AUTHOR KEYWORDS: EMOTION, AGENT,
BELIEVABILITY, INTELLIGENT INTERACTION, GAMES,
USER EXPERIENCE.
ACM CLASSIFICATION KEYWORDS
H5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous. (alternatively H5.1 Artificial, augmented, and
virtual realities)
INTRODUCTION
Enhancing the social-emotional interaction is a major build-
ing block in future interfaces [10]. Emotions play an impor-
tant role in experiencing an interface, additionally emotions
contribute in human attention, perception, memory processes
and social interactions [6]. Thus advances in technology ena-
bling computers to recognize and compute emotions to sup-
port computer-human interaction are clearly needed. But
modeling emotions in interfaces is a challenging task as al-
most all implementations of emotion elicitation models use
complex rule sets which have to be designed properly. Neme-
sys is a new way of emotion elicitation which can be easily
used in any interface application, for instance by using an
agent. The behavior and emotions of the agent can be visual-
ized anthropomorphic or by simply using abstract visualiza-
tions to enhance the interface with social/emotional informa-
tion.
We decided to choose a game oriented usage scenario to
demonstrate the abilities of Nemesys.
Copyright is held by the author/owner(s).
CHI 2005, April 2–7, 2005, Portland, Oregon, USA.
ACM 1-59593-002-7/05/0004.
1347
April 2-7 | Portland, Oregon, USA
Manfred Tscheligi
ICT&S Center, Univ. Salzburg
Salzburg, Austria
Manfred.Tscheligi@sbg.ac.at
The use of intelligent Non Playable Characters in games like
‘The Sims’ or ‘Black and White’ made these games a big
success. What distinguishes these game characters from oth-
ers is their ability to give the impression to think, to be social
and to experience emotions – in summary to be believable.
The believability of agents seems to help find the balance for
a “flow zone” between anxiety and boredom in the users’
experience [3], extending the willingness of users to play
more, longer and making these games more attractive.
One major aspect of believability is the agents’ realistic emo-
tional behavior. To create the impression of emotional be-
lievability many research-teams used several different model-
ing techniques to create emotional behavior. Most of them
rely on the usage of rule-based emotion generation systems,
ignoring the multitude of emotional states in different appli-
cation scenarios and situations.
RELATED WORK
Today many researchers are working in the field of believable
agents and the number of models is accordingly high. Most of
them have in common that they employ rule-based ap-
proaches to elicit emotions. For instance, Elliott’s Affective
Reasoner [5], Reilly's Woggles [11], and the system intro-
duced by Bartneck [1] are implementations of the OCC
model (Ortony et al.) [9], a model employing a complex rule
set to elicit 22 emotions.
All above mentioned systems are rule-based approaches,
which have the inherent shortcoming to require a finely
woven set of rules which has to be designed properly to cre-
ate believable agent behavior. Only a few systems integrate
non-rule based approaches. One exception is the agent archi-
tecture presented by Cunningham and Mac Namee[8]. This
architecture employs a neural network to create social interac-
tions according to Eysenks two-dimensional personality
model and Lang’s mood model. In contrast Nemesys uses the
neural network to easily learn emotion elicitation in various
scenarios.
The presented system tries to overcome the editing of rules
and to better facilitate emotion elicitation by choosing a non-
rule-based approach. Further the elicited emotions are pre-
sented by displaying them as facial expressions in the game
scenario.
NEMESYS
Nemesys is a new way to enable emotion elicitation. The
system is based on an artificial neural network giving the
CHI 2005 | Late Breaking Results: Posters
opportunity to learn special emotional behavior and reaction
schemes. Even though the rule modeling for new application
scenarios is no longer needed, the characters data such as
attitudes still have to be fed into the system.
This way of emotion eliciting enhances the user experience in
the usage scenario, where Nemesys is part of non playable
characters in highly character centric games, such as adven-
tures or MMORPGs.
Following Ekman and Friesen [4], we consider the following
six emotions: Sadness, Happiness, Anger, Fear, Disgust and
Surprise. These are said to be universal in the sense that they
are associated consistently with the same facial expressions
across different cultures.
The presented agent’s emotional state is generated employing
several units, which are sketched in figure 1. Besides many
deterministic units the system engages an artificial neural
network (ANN) for eliciting the emotion.
This ANN (called Emotion Generator) receives input from
several sources. On the one hand it gets data directly from
character-intrinsic values, for instance its personality and its
internal state. On the other hand input data for the artificial
neural network is generated by two pre-processing units.
These are the appraisal unit which is handling incoming
events, and a module calculating the current workload of the
agent.
Figure 1 System Structure.
1348
April 2-7 | Portland, Oregon, USA
The agent’s internal state, represented by two values which
describe the agents’ basic drive and its ground arousal, reflect
one input to the ANN.
The current plans which are stored in the schedule of an agent
lead to another input called work-load, that is, this input
represents how much ‘stress’ the agent has to sustain. This
value also includes the importance of each task in its account
which seems realistic because if we for example see ourselves
overwhelmed with important work we will be more stressful
than if we have to accomplish even a huge amount of unim-
portant tasks.
The agents’ personality is represented by five dimensions
reflecting the Five-Factor Model of Personality [7], which is
currently the most popular approach among psychologists for
studying personality traits. During the last 20 years it became
increasingly important and is today the most used model in
describing a person’s personality. This model is said to be a
complete description of personality and comprises the dimen-
sions Openness to Experience, Conscientiousness, Extraver-
sion, Agreeableness, and Neuroticism. Each agent holds these
five values in form of five integer values which are directly
passed to the ANN.
APPRAISAL UNIT
The ANN receives major input from the appraisal unit. The
appraisal unit evaluates events with regard to their emotional
significance to the agent, that is, if the event is worth an emo-
tion or not. According to Ortony et al.[9] each agent holds a
variety of different attitudes, goals, standards and beliefs,
with which this evaluation proceeds.
The appraisal unit considers goals, whereas each goal has a
certain importance. The character’s appraisal unit evaluates
each event with regard to how much it influences a goal with
a certain importance. If there is an event circumventing a goal
to some degree, a negative value will be passed to the ANN.
If there is an event fostering a goal, a positive value is passed.
For instance, an event keeping us from going to bed when our
goal is to get some sleep, will result in a negative value. The
following equation shows the relation between an event and
the appraisal unit’s output.
goals (event ) = ∑ imp ( goal i ) * goal i (inf)
i
According to Ortony et al. [9] standards are defined as what
an agent thinks about what ought (not) to be. Therefore the
agent also retains a list of standards about events that the
agent considers as praiseworthy or blameworthy.
The model an agent has of its environment is often repre-
sented as a set of beliefs. Beliefs are interesting when they are
contradicted, for example: The agent has a goal to eat, and
finds the fridge empty, therefore the agent will probably feel
an emotion due to this contradiction.
In addition to these four tables the appraisal unit evaluates
events also with regard to a table of trust. This table repre-
sents the value of trust between our agent and the active
agent, that is, the agent which is performing the action in an
CHI 2005 | Late Breaking Results: Posters
event. If another agent causes an event, the value of trust to
this agent is looked up and is forwarded to the ANN.
Figure 2 Structure of the emotion eliciting neural net-
work.
THE EMOTION ELICITING ARTIFICIAL NEURAL
NETWORK
The emotion eliciting ANN is designed as multi-layered per-
ceptron. This type of neural network exhibits only feed-
forward connections from the input layer to the only hidden
layer and from the hidden layer to the output layer.
Figure 2 shows the network’s structure, whereas the input
neurons in the top box refer to the input values of the charac-
ters personality, the neurons in the middle box represent the
input values from the internal status. The neurons in the bot-
tom box are inputs from the appraisal unit. Experiments
showed that a hidden layer with 25 neurons (with sigmoid
transfer function) exhibited the best performance. After input
the ANN calculates the resulting emotion, if any, and the ac-
cording emotional intensity.
ACQUISITION OF TRAINING DATA
To obtain good ANN performance a huge amount of training
data had to be used to train the ANN. For this reason an ad
hoc application was written to acquire the data from 40 test
participants (23 male, 17 female).
First participants had to fill out one of the freely available
personality tests, in this case the Five Factor personality test
published by Buchanan [2]. After determining their personal-
ity factors each participant had to run a questionnaire applica-
1349
April 2-7 | Portland, Oregon, USA
tion, which presented 20 scenarios and which asked them
about their feeling in the presented situation.
This set of data was used to build an enlarged training set by
Gaussian distribution of the original values. The ANN was
then trained using these values, keeping back the original
data, to check the network’s performance.
RESULTS
The following section gives a detailed assessment of the emo-
tion outputs of Nemesys concerning variability depending on
the agent’s personality. We will present a selection of the
most interesting exemplary outputs of the ANN in the next
paragraph.
Figures 3 and 4 represent the outputs of the ANN of Nemesys
on given input vectors, which is the agents personality and the
input from the appraisal unit and internal data. Each output is
displayed in a graph, where the x-axis represents the output
neuron and the ordinate represents the output value of a neu-
ron (i. e. emotion intensity).
Figure 3 Joy at different values of neuroticism.
Figure 3 shows the ANNs response to a situation eliciting the
emotion joy (peak at neuron 6). Experimenting with the per-
sonality factors (here neuroticism) in the input showed the
expected results: if the agents’ neuroticism value is varied,
the more neurotic agent is experiencing less joy in the same
situation.
Figure 4 Anger at different values of agreeableness.
Figure 4 depicts sample outputs of the ANN considering an-
ger while varying agreeableness. It shows an interesting ef-
fect: Depending on the value of agreeableness of the agent, it
is possible that the experienced emotion shifts. The two
CHI 2005 | Late Breaking Results: Posters
curves, which have their peak at anger (No. 4) are the result
of very low and average values of agreeableness. The inter-
esting curve where a very high value of agreeableness (≥40)
is presented to the ANN shows that the peak shifts from anger
to fear (No. 3) and increases in intensity. That is, a very
agreeable agent will react with fear, for example when it is
insulted. In reality this behavior is quite reasonable. These
results are very promising as Nemesys generated the expected
emotions.
THE 3D SCENARIO
Since Nemesys only produces emotions and presents them as
a selected emotion with an intensity value, testing and evalua-
tion would be rather boring and unattractive in a user study.
The system needs a scenario and an environment to be em-
bedded in. Therefore a prototype application was designed
and implemented, which sets up an interaction scenario and
uses Nemesys as emotion eliciting engine.
In principle, the setup consists of two parts: the ArsBox1 (a
system developed at the Ars Electronica Center, running a
stereoscopic projection) and on the other hand the prototype
application using Nemesys. Both components are connected
via network, using an adapted version of Gamebots2. At the
moment the user study is set up. A first investigation in this
game-based scenario will help us to draw conclusions about
the usage of emotion elicitation in a more fun orientated inter-
face. An experimental user study shall give insight on the
accurate emotion elicitation. If the user study shows appro-
priate emotion elicitation in the gaming scenario the next
steps will be a thorough analysis in other application areas.
One of the major advantages of Nemesys is its ability to bet-
ter understand how and why different emotions are evoked by
different scenarios. This can be a significant contribution in
developing intelligent user interaction.
CONCLUSION
Nemesys is a general module to include emotional agents in
various types of interfaces. Nemesys is using an ANN for
eliciting emotions, therefore it has many advantages to other
approaches, because it is trained with real data and is able to
steadily adapt its emotional behavior by continuously learn-
ing. The system is designed to enhance social and emotional
aspects of the user experience. Nemesys is designed to be an
easy to use module to enhance any interface with the possibil-
ity of emotional agents or information about the emotional
context of a situation. Consequently the next research-step is
a comparative user test to explore the enhancements by
Nemesys in various settings.
1
http://futurelab.aec.at/arsbox
2
Http://gamebots.sourceforge.net
1350
April 2-7 | Portland, Oregon, USA
REFERENCES
1. Bartneck, C. Integrating the OCC Model of Emotions in
Embodied Characters, in Proceedings of the Workshop on
Virtual Conversational Characters: Applications,
Methods, and Research Challenges, Melbourne, (2002).
2. Buchanan, T. available at
<http://www.personalitytest.co.uk>, (2001).
3. A. Dix., J. Finlay, G. D. Abowd, B. Russell. Human-
Computer Interaction, Prentice Hall, (2004).
4. P. Ekman and W. V. Friesen. Unmasking the Face: A
Guide to Recognizing Emotions from Facial Clues.
Prentice-Hall, Englewood Cliffs, New Jersey, (1975).
5. Elliott, C. The Affective Reasoner. A process model of
emotions in a multi-agent system. PhD thesis,
Northwestern University. Institute for the Learning
Sciences. (1992).
6. M. W. Eysenk and M. T. Keane. Cognitive Psychology:
A students’ handbook. Taylor & Francis Publishers,
(1995).
7. McCrae, R.R., Costa P.T. Toward a new generation of
personality theories: Theoretical contexts for the five-
factor model, In J.S. Wiggins (Ed.). The Five-Factor
Model of Personality:Theoretical Perspectives (pp 51-
87). New York: Guilford Press, (1996).
8. Mac Namee, B., Cunningham, P. The µ-SIC System: A
Connectionist Driven Simulation of Socially Interactive
Agents, available at
<http://citeseer.ist.psu.edu/macnamee02sic.html>,
(2002).
9. Ortony, A., Clore, G., & Collins, A. The Cognitive
Structure of Emotions. Cambridge: Cambridge University
Press, (1988).
10. R. W. Picard and A. Wexelblat. Future Interfaces: Social
and Emotional. Proceedings CHI 2002, p. 698-699.
11. Reilly, W.S.N. Believable Social and Emotional Agents,
Ph.D. Thesis. Technical Report CMU-CS-96-138, School
of Computer Science, Carnegie Mellon University,
(1996).