Grounding Language in Action

HUL 281 Mind, Machine and Language

How language conveys meaning?

Two main approaches: The dominant approach - treat language as a symbol manipulation system: Meaning conveyed by using abstract, amodal, and arbitrary symbols (i.e.,
words) combined by syntactic rules.

abstract - same word to express different things amodal - same word is used when things are spoken about or written about arbitrary - word bear no relationship to the words referent

Motor Resonance and Language

language of action evokes motor resonance 2 type of evidence: neural: cognitive neuropsychology behavioral: psycholinguistics (interference effects)

Neural Evidence: Pulvermueler (2005)

Pulvermueler et al. (2001): hearing/reading
action verbs

produces somatotopic activation in the

primary motor cortex

EEG study, movement vs. lexical decision

task

kick (leg), pick (arm), lick (face)

Neural Evidence: Pulvermueler (2005)

Hauk et al. (2004): fMRI study, movement vs. passive reading

 Pulvermueler et al. (2005): TMS study, movement vs. Passive reading

Behavioral Evidence: Indexical Hypothesis of the Meaning

meaning based on action (and affordances) transformation of language to meaning in 3 steps: 1 words => perceptual symbols 2 perceptual symbols => affordances 3 affordances + syntactic construction => action: (un)doable= (mis)understood 4 grammatical form: order of simulation, how to combine affordances language is made meaningful by cognitively simulating the actions implied by sentences

Behavioral Evidence: Action-Sentence Compatibility Effect

Glenberg and Kaschak (2002): comprehending a sentence that implies action in one
direction interferes with real action in the opposite direction

Movement from and towards the body: when incongruent with the sentence
movement => reaction time is longer

Task: sensical and nonsensical sentences, button Y/N near and far from the body John gave you a pizza. You gave a pizza to John. Also with abstract transfer sentences (Glenberg et al., 2008) Mary told you a story. You told a story to Mary.

Chinese Room Argument: Harnads (1990) version of Searles (1980)

Provides a compelling intuition as to why meaning cannot be conveyed solely by the
syntactic relations among abstract symbols Subject - foreign lady with a Chinese dictionary Task to look for meaning of abstract symbols to decode the meaning of a sentence Result - No matter how many of these abstract symbols she relates to one another, she is never going to determine the meaning of the sentence Lesson learnt - abstract symbols of language must be grounded, or mapped, to the world if they are to convey meaning In other words - if one has only abstract symbols at ones disposal, determination of the correct mapping is impossible

Affordance
potential interactions between bodies and objects (Gibson, 1979; Tucker & Ellis, 1998)
Example - a chair affords - sitting - adult humans
mice or elephants - wrong sorts of bodies to sit in an ordinary chair

meaning of the situation arises from meshing the affordances of different bodies and objects
For e.g. meshing the affordances of chair and light bulb to accomplish the goal of changing the bulb.

Indexical Hypothesis (IH)

3 processes transform words and syntax into an action-based meaning:
First, words and phrases are indexed or mapped to perceptual symbols

Unlike abstract symbols, perceptual symbols are modal and non arbitrary Based on the brain states underlying the perception of the referent
Second, affordances are derived from the perceptual symbols

arbitrary symbols = arbitrary => no affordances can be derived perceptual symbols = non arbitrarily => affordances can be derived

Third, affordances are meshed under the guidance of syntactic constructions For e.g., the affordances of an upright vacuum cleaner and of a coat to accomplish the goal of hanging up the coat

affordances do not mesh to guide action => non-sensible sentence Example : Hang the coat on the upright vacuum cleaner is sensible
Hang the coat on the upright cup is not sensible

Affordances - vacuum cleaner can be used as a coat rack but not cup
Neither based on explicit previous learning nor on abstract symbols Lastly, grammatical form of the sentence directs a cognitive simulation that combines various objects affordances to convey meaning

Experiment 1
Participants presented with series of sensible/nonsense sentences asked to distinguish between both kinds one independent variable, implied sentence direction (toward/ away),
Put your finger under your nose, => action toward the body Put your finger under the faucet, => action away from the body

Participants never instructed to consider the implied direction; just sensibility

Experimental Set-Up

Specially constructed button box to record actual response direction (3 buttons) Buttons differed in distance from body near, middle and far

pressing middle button initiates visual representation of sentence

yes-is-far condition sensible sentence middle to far button away from body yes-is-near condition sensible sentence middle to near button towards body Major dependent variable - time between presentation of the sentence and release of the middle button (to move to the near or the far button)

IH Predictions
towards sentence - simulation of actions toward the body and vice versa According to IH, interference between simulation and action
toward sentence interfere with - yes-is-far condition
away sentence - interfere with - yes-is-near condition

prediction for a statistical interaction between implied sentence direction and

actual response direction.

This interaction is referred to as the actionsentence compatibility effect (ACE).

Sentence Types
Total 80 sensible sentence pairs were used Half of them i.e. 40 toward/away pairs were in the imperative The concrete transfer pairs (20 toward/away pairs) described transfer of a
physical object between you and another person.

20 abstract transfer pairs described a nonphysical transfer, such as

Liz told you the story/You told Liz the story

Experiment 1 : Results
Analyses were conducted on the proportion of correct judgments, as well as on the reading times Error due to practice effects and outliers were taken care of by taking mean of readings

Results (Contd.)
Significant interaction between response direction and implied direction i.e. ACE Stronger effect for two types of transfer sentences, as compared with imperative ones

Consistent with IH and inconsistent with abstract symbol theories of meaning.

Merely understanding a sentence can facilitate or interfere with a physical response. Read Time (imperative sentences) < Read Time (transfer sentences)

Significant interaction between sentence direction and sentence type

EXPERIMENT 2
Designed to replicate and modestly extend the major effects from
Experiment 1

Participants responded with their left hands Aim - to determine whether ACE reflects action-planning specific for the
dominant hand (right-handed subjects)

Results : Most of the results were similar to Experiment 1

Other significant finding - effect of sentence type

Analysis of error rates: Fewer errors on the toward sentences (4%) than on the away sentences (7%) Fewer errors for imperative sentences (4%) than for concrete (6.5%) or abstract (7%) transfer sentences

EXPERIMENT 3
Designed to test a spatial location alternative to the IH Participant did not move the hand.
left index finger - over the yes button (either near to or far from the body) right index finger - over the no button (either far from or near to the body).

IH Predictions: No relevant interfering action during response => No ACE

Results: little evidence for an ACE interaction => interference arises from action not location

 Trimmed mean r.t.(Towards) = 1,742 msec < Trimmed mean r.t.(Away) = 1,800 msec Significant interaction of sentence direction and sentence type No speedaccuracy tradeoffs, errors on toward sentences (6%) < away sentences (7%)

Discussion
The results from Experiments 2 and 3 accomplished three goals. 1. First, they demonstrated replicability of the ACE. 2. Second, demonstrated that ACE is unlikely to reflect detailed action
planning at the level of particular muscles.

3. Third, contrast indicates that ACE depends on action, and not solely on
spatial location of the responses.

Meshing Affordances using Grammar

Double object construction Subjectverbobject1object2

Meaning - subject transfers object2 to object1

Notion of denominal verbs Example - The woman crutched the goalie the ball is meaningful

The woman eggshell the goalie the ball is not

THANK YOU