Tension in Music : Cognition, Emotion, Brain, Movement

Carol L. Krumhansl Department of Psychology Cornell University Ithaca, NY 14853 USA Tension: Linking Cognition, Emotion, Brain, and Motion
WIPPPPP Work in Past Published in Press Present Planned

"expectation is always ahead of the music, creating a background of diffuse tension against which particular delays articulate the affective curve and create meaning. Not only does music use no linguistic signs but, on one level at least, it operates as a closed system, that is, it employs no signs or symbols referring to the non-musical world of objects, concepts, and human desires. This puzzling combination of abstractness with concrete emotional and aesthetic experience can, if understood correctly, perhaps yield useful insights into more general problems of meaning and communication. L. B. Meyer, Emotion and Meaning in Music, 1956

Tension and cognition

Mozart, Piano Sonata, K. 282 Krumhansl, Music Perception, 1996

Duration of each beat in the music as performed

Judgments of sections ends

Duration of each beat in the music as performed

Judgments of new musical ideas

Duration of each beat in the music as performed

Krumhansl, Music Perception, 1996

Lerdahl, Tonal Pitch Space, 2001

(a) octave (root) level: (b) fifth level: (c) triadic level: (d) diatonic level: (e) chromatic level:

0 (0) 0 7 (0) 0 4 7 (0) 0 2 45 7 9 11 (0) 0 1 2 3 4 5 6 7 8 9 10 11 (0)

Diatonic basic space, set to I/C (C = 0, C# = 1, B = 11).

Computing Distance from d minor (vi) chord in F major key to C major (I) chord in C major key

Region Distance
F major to C major

Chord Distance
d minor to C major

Basic Space Differences

New entries in Basic Space

Tensing

Relaxing

Slide 40

Krumhansl, Music Perception, 1996

Chopin Prelude
125

100

75

50

25

0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Event Judged Predicted

Messiaen, Quartet for the end of time

Messiaen Quartet 125 100 75

50 25 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 Event Judged Predicted

Bach Chorale Cristus, der ist mein Liebe

Bach Chorale 125 100 75 50 25 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 Event Judged Predicted

Wagner Diatonic Hierarchical Right-Branching +

Wagner Parsifal Grail Theme

Diatonic

125 100 75 50 25 0 1 2 3 4 5 Event Judged Predicted 6 7 8 9

Wagner Chromatic Shifting Diatonic Sequential to Hierarchical

125 100 75 50 25

Chromatic

0 1 2 3 4 5 Event Judged Predicted 6 7 8 9

Lerdahl & Krumhansl, Music Perception.in press

Tipping the (Fourier) balances: A geometric approach to representing pitch structure in non-tonal music Fourier Balance One FB1
FB 1
QuickTime and a TIFF (Uncompressed) decompressor are needed to see this picture.

Stable for minor second, major seventh Unstable for fourth, fifth, and tritone Stable for a diatonic scale

Fourier Balance Two FB2

FB 2

QuickTime and a TIFF (Uncompressed) decompressor are needed to see this picture.

Stable for tritone, minor second, major seventh, fourth and fifth Unstable for minor third, major sixth Stable for an octatonic scale

FB 1

FB 2

FB 3

FB 4

FB 5

FB 6

Compare Fourier Balance Model to Tonal Pitch Space Model Predicting Judged Tension with FB model
Fourier Balance Model R-squared (2,5) = .999

Predicting Judged Tension with TPS Model

R-squared (2,5) = .942
100 75 50

Y
25 0 -25 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0

Judged Tension

Predicted from FB model Event

Predicted from TPS model

Messiaen Quartet for the End of Time Predicting Judged Tension with FB Model R-squared (18,21) = .823 Predicting Judged Tension with TPS Model R-squared (2,37) = .758
100 90 80 70 60 50 40 30 20 10 0 0 10 20 30 40

Rows

Judged Tension

Predicted from FB model

Predicted from TPS model

Summary: Fourier Balance model fits judged tension better than Tonal Pitch Space model but: The two models are closely connected

International Congress on Music Perception and Cognition Bologna, August. 2006

A Proto-Music Theory from Unsupervised Learning

Carol L. Krumhansl David G. Rand

Background
Experiments demonstrate listeners are sensitive to statistically frequent patterns Statistical learning adults -- cross-cultural (Krumhansl, et al., 1999, 2000) adults -- melodies on artificial tone-sets (Oram & Cuddy, 1995) infants -- tone and syllable sequences (Saffran, et al. 1999)

Limitations
Low-order statistics for sequences P(c1), P(c2 | c1), P(c3 | c1 c2) c1 p c2 p c3 p p ck

Results show effects of acculturation only in higher-order statistics (Krumhansl et al., 1999, 2000) Automatic discrimination of musical styles only with higher-order statistics (Krumhansl, 2000) Coding of music ignores durations of tones

Questions
Can a statistical learning model distill musically interpretable patterns from a musical corpus? What does such a model show about the relevance of rhythm to melodic structure? ADIOS (Automatic DIstillation Of Structure) Zach Solan, David Horn, Eytan Ruppin (Tel-Aviv University), Shimon Edelman (Cornell University)
Two classes of syntax models Language-specific theory of syntax (generative theory) General-purpose statistical or distributional learning models ADIOS has features of both Distillation of rule-like regularities out of the acquired knowledge Knowledge acquired only from raw distributional information

Representational Data Structure (RDS)

Directed Graph Input: raw unlabeled corpus data (not tagged for part of speech,

only BEGIN and END of each sentence) Node = one constituent (word) Directed edge is inserted if transition between constituents exists in corpus

Pattern Acquisition (PA) Algorithm

A Pattern is a similarly structured sequence of constituents that recurs in the corpus. An Equivalence Class is a set of constituents from different paths that occur in the same position in a pattern This is the syntax that ADIOS is distilling

Bundles are formed when two or more paths run in parallel and dissolved when more paths leave the bundle than stay in

Criterion for judging pattern significance

For path c1 p c2 p c3 p p ck: S = e-(L/k)2 P(c1,c2,,ck) log( P(k)(path) / P(2)(path)), where: L = typical context length k = length of the candidate path P(k) (path) = P(c1)P(c2|c1)P(c3|c1 p c2)P(ck|c1 p c2 p c3 p p ck) k-gram P(2) (path) = P(c1) P(c2|c1) P(c3|c2) P(ck|ck-1) random walk

Bootstrapping
The identification of new equivalence classes is done using acquired equivalence classes - bootstrapping

Musical Corpus
Musical Themes from Classical Corpus in Themefinder 300 themes in C major, 300 themes in A minor Monotone (no harmony) C major scale tones: C D E F G A B

A minor natural scale tones: A B C D E F G

Kern format: 2aa 2ee 4dd 8cc 8b 4a 4a 2ee 2b 4a 8g 8f 4e 4e # 2=half note, 4=quarter note, 8=eighth note, etc G = G below middle C, g = G above middle C, gg= G an octave and a fifth above middle C, etc. * = begin # = end Each duration-pitch pair is treated as an independent constituent

There is NO information given to ADIOS to indicate that: 16a and 8a are the same pitch (with different durations) 16a and 16b are the same duration (with different pitches) 16G and 16g are the same pitch in different octaves (octave equivalent)

Comparison with Musicians Judgments

43 Major, Minor, Major Shuffled, Minor Shuffled, Major Scrambled, Minor Scrambled 11.1 years instruction on musical instruments, 3 or more music theory courses

QuickTime and a TIFF (LZW) decompressor are needed to see this picture.

Overall Summary
ADIOS distills interpretable structure from musical corpus
Musically interpretable Patterns and Equivalence Classes Tonal hierarchy is evident in Patterns and Equivalence Classes When trained on Major, can correctly discriminate Major vs Minor When trained on Major and Minor, can discriminate Original vs Shuffled (duration-pitch pairs moved)

ADIOS develops Proto-Music Theory

Not fully developed, but sufficient to determine: Neighbors (pitch proximity) Same pitch independent of duration Same duration independent of pitch Octave equivalent pitches Harmonically related pitches Major vs. Minor Order of duration-pitch pairs

Tension and Emotion

Krumhansl, An exploratory study of musical emotions and psychophysiology Canadian Journal of Psychology, 1997 Experimental Design Dynamic Ratings Sad Fear Happy Tension Sad Excerpts: Tomaso Albinoni, Adagio in G minor for Strings and Orchestra Samuel Barber, Adagio for Strings, Op. 11 Fear Excerpts: Gustav Holst: Mars -- the Bringer of War from The Planets Modest Mussorgsky, Night on Bare Mountain Happy Excerpts: Antonio Vivaldi, The Four Seasons, La Primavera (Spring), Danza pastorale Hugo Alfven, Midsommarvaka

Physiological Measures Recorded at 1-second Intervals 1) cardiac interbeat interval (IBI), measured in milliseconds, with shorter IBIs taken to indicate a higher level of cardiovascular arousal 2) pulse transmission time to the finger (FPTT), measured in milliseconds, with shorter pulse transmission times indicative of greater autonomic (sympathetic) activation 3) finger pulse amplitude (FPA), a measure of the amount of blood in the periphery, with reduced amplitude indicating greater vasoconstriction and associated with greater autonomic (sympathetic) activation 4) pulse transmission time to the ear (EPTT), another measure of blood flow 5) respiration intercycle interval (ICI), measuring the time between successive inspirations in milliseconds 6) respiration depth (RD), which is the point of maximum inspiration minus the point of maximum expiration 7) respiration-sinus asynchrony (RSA) 8) systolic blood pressure (SBP) 9) diastolic blood pressure (DBP) 10) mean arterial pressure (MAP) 11) skin conductance level (SCL), with increased skin conductance indicative of greater autonomic (sympathetic) activation 12) temperature on the finger (TEM) measured in degrees Fahrenheit.

Correlations between Dynamic Emotion Ratings and Dynamic Physiology Ratings Sad Ratings Interbeat-Interval .14*** Finger Pulse Transmission Time .10** Finger Pulse Amplitude -.14*** Ear Pulse Transmission Time .07* Respiration Intercycle-Interval .05 Respiration Depth .00 Respiration Sinus Asynchrony -.02 Systolic Blood Pressure .37*** Diastolic Blood Pressure .41*** Mean Arterial Pressure .37*** Skin Conductance Level -.36*** Finger Temperature -.35*** -.01 .16*** -.31*** .15*** -.12*** .00 -.11*** -.23*** -.24*** -.25*** .06 -.20*** -.15*** -.09** .24*** .03 -.16*** -.09* -.13*** -.14*** -.14*** -.10** -.08* .21*** Fear Ratings Happy Ratings

Factor analysis of correlations between physiological measures Distinct groupings

Blood Pressure

SCL Temp

Respiration Rate

Blood Flow Heart Finger Rate

Blood Flow Ear

Tension and Brain

Functional Magnetic Resonance Imaging (fMRI) The neural mechanisms of tonal and rhythmic expectations were studied in two ways: Stimulus - introducing either tonal or rhythmic violations (or both) Task -judge either the tonal structure or the rhythmic structure -(or passively listen to the melodies)

Musical Sequences: Melodies composed by Diego Vega, Cornell University (6 sec), piano timbre. Sample sequences:

Original Tonal Violations Rhythmic Violations Tonal and Rhythmic Violations

Passive Listening: Tonal and Rhythmic Violations

This analysis contrasted musical sequences containing both Tonal and Rhythmic violations with musical sequences with no violations.

Even when the subjects were in not performing a task, activations in: superior temporal cortex (especially on the right)

Active Judgments: Tonal and Rhythmic Violations

This analysis also contrasted Musical Sequences containing both Tonal and Rhythmic violations with musical sequences with no violations (as before) Judged whether a violation occurred Superior temporal activation (as for Passive Listening), in addition, when making either Tonality or Rhythm Judgments: right dorsolateral frontal right

inferior frontal (bilateral)

Tension and motion Tension and motion in dance

"The cognitive representation of an event unit involving human motion can be described as some small set of relatively stable, preparatory motions, followed by this relatively unstable, completing motion. Such a schematic structure enables the perceiver to anticipate temporal relations within an unfolding event, and to fit successive parts of the temporal sequence into this anticipated structure. M. Lasher, Cognitive Psychology, 1981. "The interaction between movement and sound is the most fundamental element of dance. The dance does not mimic the music -- there is not a particular part of the music for every gesture and step -- but the basic "kinetic feel" or "energy shape" of the music is expressed in the dance. The choreographer uses the music not only for its rhythmic pulse, but also as a source of emotional and structural ideas. Thus, elements of the music are often observed in the dance. K. Teck, Movement to Music, 1990.

Experimental Design Music W. A. Mozart, Divertimento No. 15 Bb, Minuetto Dance George Balanchine, School of American Ballet Subjects Music Lessons 9.2 years Dance Lessons 7.3 years Conditions Music Only Dance Only Both Music and Dance Section End (discrete judgment) Tension (continuous judgment ) New Idea (discrete judgment) Emotion Expressed (continuous judgment) Emotion Quality (overall judgment)

Tasks during videotape

Task after videotape

Intrinsic Relationships Between Music and Dance Rhythmic accent, meter, sounds produced by the dancers Dynamic volume of musical and choreographic Textural number of instruments/performers, homophony versus polyphony, counterpoint Structural corresponding motives or figures, phrases, structures Qualitative choreomusical parallels of tessitura, timbre, articulation, dissonance/consonance Mimetic choreography imitates a particular sound in the music

P. Hodges (1992) Relationships between score and choreography in 20th century dance. London: Mellen.

"To see Balanchine's choreography is to hear the music with ones eyes The choreography emphasizes relationships of which I had hardly been aware and the performance was like a tour of a building for which I had drawn the plans but never explored the result. I. Stravinsky, quoted in S. Jordan, Dance Chronicle, 1993

Exploratory Study of Anticipating Human Movement in Dance

Barycenter Center of sub-region including head, trunk, legs

Stop Position

Target Position Judged Position

Camurri, Krumhansl, Mazzarino, Volpe, 2004

Tension and motion in music performance

Marcello Wanderley, IRCAM thesis, 2002 Non-obvious performance gestures (Not needed to produce tone)

Normal Expression, Exaggerated Expression, Immobile

Experimental Design Continuous Judgments of Tension Continuous Judgments of Phrasing while: watching performance (no sound) hearing performance (no image) both watching and hearing performance Stravinsky, Second of Five Pieces for Solo Clarinet

Tension judgments in experiment

Phrasing judgments in experiment

Analysis of data with Functional Data Analysis J. Ramsay Audio, Visual, Audio and Visual conditions similar for judgments of phrasing But complex interactions between Visual and Audio in judgments of tension

Vines, Wanderley, Levitin, Krumhansl, Cognition, 2006

Tension in facial and body gesture

Behavior

Cognition

Emotion Brain

Motion

Music

Cognition

Emotion Brain

Motion

Tension
L. B. Meyer Emotion and Meaning in Music 1956
"expectation is always ahead of the music, creating a background of diffuse tension against which particular delays articulate the affective curve and create meaning. Not only does music use no linguistic signs but, on one level at least,