History and Practice of Digital Sound Synthesis

Julius Smith
CCRMA, Stanford University

AES-2006 Heyser Lecture

October 6, 2006

Julius Smith AES-2006 Heyser Lecture – 1 / 84

Overview
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Summary

Overview

Julius Smith AES-2006 Heyser Lecture – 2 / 84

 Outline

Overview
Digital sound synthesis approaches in approximate historical order:
• Outline
• CCRMA Perspective
• Wavetable (one period)
Early Digital Synthesis
• Subtractive
Spectral Modeling
• Additive
Physical Modeling

Summary
• Frequency Modulation (FM)
• Sampling
• Spectral Modeling
• Physical Modeling
Some connections with audio coding will be noted

Emphasis:
• Sound examples
• Block diagrams
• Historical notes

Julius Smith AES-2006 Heyser Lecture – 3 / 84

 CCRMA Perspective

Overview
• Outline
• CCRMA Perspective

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

The Knoll, Stanford University

Julius Smith AES-2006 Heyser Lecture – 4 / 84

Overview
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Summary

Early Digital Sound Synthesis

Julius Smith AES-2006 Heyser Lecture – 5 / 84

 Wavetable Synthesis in Music I-V (1957-1969)

Overview

Early Digital Synthesis

• Music V
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 6 / 84

 Music V Scripting Language (“Note Cards”)

Overview

Early Digital Synthesis

• Music V
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example

Spectral Modeling

Physical Modeling

Summary

• Essentially Supported in MPEG-4 Structured Audio Orchestra

Language (SAOL) (Music V → csound → SAOL)
• “Encoding sounds” as “instruments” is hard, in general

Julius Smith AES-2006 Heyser Lecture – 7 / 84

 Kelly-Lochbaum Vocal Tract Model

Overview

Early Digital Synthesis

• Music V
e(n) … y(n)
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
• FM Synthesis
Glottal Pulse
• FM Formula
Train or Noise 1 + k1 1 + kM
• FM Patch Speech
e(n) z− 1 2 … z− 1 2
• FM Spectra Output
• FM Examples y(n)
k1 − k1 R1 kM − kM RM
• FM Voice
• Sampling Synthesis (Unused
• Modern Example Allpass z− 1 2 … z− 1 2
Output) 1 − k1 1 − kM
Spectral Modeling
Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)
Physical Modeling

Summary

John L. Kelly and Carol Lochbaum (1962)

Julius Smith AES-2006 Heyser Lecture – 8 / 84

 Sound Example

Overview
“Bicycle Built for Two”: (WAV) (MP3)
Early Digital Synthesis
• Music V
• KL Music
• “Daisy” • Vocal part by Kelly and Lochbaum (1961)
• Additive Analysis
• Musical accompaniment by Max Mathews
• Additive Synthesis
• FM Synthesis • Computed on an IBM 704
• FM Formula
• FM Patch
• Based on Russian speech-vowel data from Gunnar Fant’s book
• FM Spectra • Probably the first digital physical-modeling synthesis sound
• FM Examples
• FM Voice example by any method
• Sampling Synthesis • Inspired Arthur C. Clarke to adapt it for “2001: A Space Odyssey”
• Modern Example
— the computer’s “first song”
Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 9 / 84

 Classic Additive-Synthesis Analysis (Heterodyne Comb)

Overview

Early Digital Synthesis

• Music V
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example

Spectral Modeling

Physical Modeling

Summary

John Grey 1975 — CCRMA Tech. Reports 1 & 2

(CCRMA “STANM” reports — available online)

Julius Smith AES-2006 Heyser Lecture – 10 / 84

 Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)

Overview

Early Digital Synthesis

• Music V
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example

Spectral Modeling

Physical Modeling

Summary

John Grey 1975 — CCRMA Tech. Reports 1 & 2

(CCRMA “STANM” reports — available online)

Julius Smith AES-2006 Heyser Lecture – 11 / 84

 Classic Additive Synthesis Diagram

Overview A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)

Early Digital Synthesis
• Music V
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example

Spectral Modeling
noise
FIR
Physical Modeling Σ
Summary

4 t
y(t) = Ai (t) sin ωi (t)dt + φi (0)
0
"# $

i=1
!

Julius Smith AES-2006 Heyser Lecture – 12 / 84

 Classic Additive-Synthesis Examples

Overview
• Bb Clarinet
Early Digital Synthesis
• Music V
• Eb Clarinet
• KL Music • Oboe
• “Daisy”
• Additive Analysis • Bassoon
• Additive Synthesis • Tenor Saxophone
• FM Synthesis
• FM Formula • Trumpet
• FM Patch
• English Horn
• FM Spectra
• FM Examples • French Horn
• FM Voice
• Sampling Synthesis
• Flute
• Modern Example

Spectral Modeling • All of the above

Physical Modeling
• Independently synthesized set
Summary

(Synthesized from original John Grey data)

Julius Smith AES-2006 Heyser Lecture – 13 / 84

 Frequency Modulation (FM) Synthesis

Overview
FM synthesis is normally used as a spectral modeling technique.
Early Digital Synthesis
• Music V • Discovered and developed (1970s) by John M. Chowning
• KL Music
• “Daisy”
(CCRMA Founding Director)
• Additive Analysis • Key paper: JAES 1973 (vol. 21, no. 7)
• Additive Synthesis
• FM Synthesis • Commercialized by Yamaha Corporation:
• FM Formula
• FM Patch ◦ DX-7 synthesizer (1983)
• FM Spectra
• FM Examples
◦ OPL chipset (SoundBlaster PC sound card)
• FM Voice ◦ Cell phone ring tones
• Sampling Synthesis
• Modern Example
• On the physical modeling front, synthesis of vibrating-string
Spectral Modeling
waveforms using finite differences started around this time:
Physical Modeling
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
Summary

Julius Smith AES-2006 Heyser Lecture – 14 / 84

 FM Formula

Overview

Early Digital Synthesis

x(t) = Ac sin[ωc t + φc + Am sin(ωm t + φm )]
• Music V
• KL Music where
• “Daisy”
• Additive Analysis (Ac , ωc , φc ) specify the carrier sinusoid
• Additive Synthesis
• FM Synthesis (Am , ωm , φm ) specify the modulator sinusoid
• FM Formula
• FM Patch Can also be called phase modulation
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 15 / 84

 Simple FM “Brass” Patch (1970–)

Jean-Claude Risset observation (1964–1969):

Overview
Brass bandwidth ∝ amplitude
Early Digital Synthesis
• Music V
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
fm = f0
• FM Synthesis
• FM Formula g
• FM Patch A F
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example fc = f0
Spectral Modeling

Physical Modeling
A F
Summary

Out

Julius Smith AES-2006 Heyser Lecture – 16 / 84

 FM Harmonic Amplitudes (Bessel Function of First Kind)

Overview Harmonic number k , FM index β :

Early Digital Synthesis
• Music V
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
• FM Synthesis
• FM Formula
1
• FM Patch
• FM Spectra
• FM Examples
• FM Voice 0.5

• Sampling Synthesis

k
J (β)
• Modern Example
0
Spectral Modeling 0

Physical Modeling 2
−0.5
Summary 4
0
5
6
10
15
20 8
25
10 Order k
30

Argument β

Julius Smith AES-2006 Heyser Lecture – 17 / 84

 Frequency Modulation (FM) Examples

Overview
All examples by John Chowning unless otherwise noted:
Early Digital Synthesis
• Music V • FM brass synthesis
• KL Music
• “Daisy” ◦ Low Brass example
• Additive Analysis
• Additive Synthesis ◦ Dexter Morril’s FM Trumpet
• FM Synthesis
• FM Formula • FM singing voice (1978)
• FM Patch
• FM Spectra
Each formant synthesized using an FM operator pair
• FM Examples (two sinusoidal oscillators)
• FM Voice
• Sampling Synthesis ◦ Chorus
• Modern Example

Spectral Modeling
◦ Voices
Physical Modeling
◦ Basso Profundo
Summary • Other early FM synthesis
◦ Clicks and Drums
◦ Big Bell
◦ String Canon

Julius Smith AES-2006 Heyser Lecture – 18 / 84

 FM Voice

Overview
FM voice synthesis can be viewed as compressed modeling of
Early Digital Synthesis
• Music V
spectral formants
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
Carrier 1 Carrier 2
• FM Synthesis
• FM Formula Carrier 3
• FM Patch
• FM Spectra
• FM Examples
• FM Voice

Magnitude
• Sampling Synthesis
• Modern Example

Spectral Modeling

Physical Modeling

Summary 0 Frequency f
Modulation Frequency (all three)

Julius Smith AES-2006 Heyser Lecture – 19 / 84

 Sampling Synthesis History

Overview
• 1979 - Fairlight Computer Music Instrument - 8-bit
Early Digital Synthesis
• Music V ◦ First commercial sampler
• KL Music
• “Daisy” ◦ Eight voices, 8 bits, 64 KB (4 sec) RAM, 16 kHz (mono)
• Additive Analysis ◦ Editing, looping, mixing
• Additive Synthesis
• FM Synthesis ◦ One could draw waveforms and additive-synthesis amplitude
• FM Formula
envelopes (for each harmonic) with a light pen
• FM Patch
• FM Spectra ◦ $25,000–$36,000!
• FM Examples
• FM Voice • 1981 - E-mu Systems Emulator
• Sampling Synthesis
• Modern Example ◦ First “affordable” sampler ($10,000)
Spectral Modeling
◦ Eight voices, 128 K RAM, 8-bit, 80 lb.
Physical Modeling

Summary • 1986 - Ensoniq Mirage

◦ Breakthrough price-point ($1695)
◦ Eight voices, 144 K RAM, 8-bit

Julius Smith AES-2006 Heyser Lecture – 20 / 84

 Modern Sampled Piano

Overview
Example:1
Early Digital Synthesis
• Music V • 40 Gigabytes on ten DVDs (three sampled pianos)
• KL Music
• “Daisy” • Every key sampled
• Additive Analysis
• Additive Synthesis
• 4–10 “velocity layers”
• FM Synthesis • Separate recordings with soft pedal down
• FM Formula
• FM Patch
• Separate “release” recordings, for multiple striking velocities
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example

Spectral Modeling

Physical Modeling

Summary

1
Synthogy Ivory, $349 (Electronic Musician, October 2006)

Julius Smith AES-2006 Heyser Lecture – 21 / 84

 Fundamental Problem with Sampling Synthesis

Overview
Piano timbre is determined by
Early Digital Synthesis
• Music V • key number (1 byte)
• KL Music
• “Daisy”
• key velocity (2 bytes more than enough)
• Additive Analysis • pedal state (1 bit [or byte] per pedal)
• Additive Synthesis
• FM Synthesis
Piano control is relatively low-dimensional:
• FM Formula
• FM Patch
• FM Spectra
• Less than six bytes of information per note played
• FM Examples • No continuous controls (typically)
• FM Voice
• Sampling Synthesis • Ratio of total sampled data to one note of control data
• Modern Example ≈ one billion
Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 22 / 84

 Now consider bowed strings

Overview
Control parameters:
Early Digital Synthesis
• Music V • Left-hand finger position(s)
• KL Music
• “Daisy” • Left-hand vibrato
• Additive Analysis • Bow velocity
• Additive Synthesis
• FM Synthesis • Bow force
• FM Formula
• Bow position
• FM Patch
• FM Spectra • Bow angle
• FM Examples
• FM Voice
• Shoulder damping
• Sampling Synthesis • Instrument orientation
• Modern Example
• Player motion (within a room)
Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 23 / 84

 Difficulty of sampling bowed strings

Overview
• Bowed-string control is infinite-dimensional in principle
Early Digital Synthesis
• Music V
• Many time-varying functions — “gestures”
• KL Music (we counted more than 10)
• “Daisy”
• Additive Analysis • Complete sampling of bowed strings on the level of pianos has
• Additive Synthesis apparently never been done
• FM Synthesis
• FM Formula • Rule-driven navigation of the most useful recorded playing
• FM Patch
regimes has worked well (e.g., Synful Orchestra)
• FM Spectra
• FM Examples • Model-based approaches greatly reduce data requirements:
• FM Voice
• Sampling Synthesis ◦ Spectral models (inspired by sound perception)
• Modern Example
◦ Physical models (model the sound source)
Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 24 / 84

Overview
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Summary

Spectral Modeling Synthesis

(Historical Summary)

Julius Smith AES-2006 Heyser Lecture – 25 / 84

 Classic Vocoder Analysis & Resynthesis (Dudley 1939)

magnitude, or
Overview magnitude and
phase extraction
Early Digital Synthesis

Spectral Modeling
A
• Vocoder
• Vocoder Examples
x0 (t) x̂0 (t)
• Sinusoidal Modeling f

• Spectral Trajectories Data Compression,

• Sines + Noise A Transmission,
x(t) x1 (t) Storage, x̂1 (t) x̂(t)
• S+N Examples
Manipulation,
• S+N FX f
Noise reduction, ...
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows A
• HF Noise Modeling
xN −1 x̂N −1
• HF Noise Band f
• S+N+T Examples
• SM Summary
• Spec Future

Physical Modeling Analysis Processing Synthesis

Summary

Julius Smith AES-2006 Heyser Lecture – 26 / 84

 Phase Vocoder Channel Model

Overview Analysis Model Synthesis Model

Early Digital Synthesis
Channel Filter ak ak(t) ωk+Δωk(t)
Spectral Modeling
• Vocoder Response
• Vocoder Examples A F
• Sinusoidal Modeling
Δωk Sine Osc
• Spectral Trajectories
• Sines + Noise 0 ωk ω
• S+N Examples Out
• S+N FX
• S+N XSynth • Early “channel vocoder” implementations (hardware) only
• Sines + Transients
• S + N + Transients measured amplitude ak (t) (Dudley 1939)
• S+N+T TSM • The “phase vocoder” (Flanagan and Golden 1966) added phase
• S+N+T Freq Map
• S+N+T Windows tracking in each channel
• HF Noise Modeling
• Portnoff (1976) developed the FFT phase vocoder,
• HF Noise Band
• S+N+T Examples which replaced the heterodyne comb in computer-music
• SM Summary
• Spec Future
additive-synthesis analysis (James A. Moorer)
Physical Modeling
• Inverse FFT synthesis (Rodet and Depalle 1992) gave faster
Summary sinusoidal oscillator banks

Julius Smith AES-2006 Heyser Lecture – 27 / 84

 Amplitude and Frequency Envelopes

Overview
ak (t)
Early Digital Synthesis

Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX t
• S+N XSynth ∆ωk (t) = φ˙k (t)
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows 0
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future
t
Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 28 / 84

 Channel Vocoder Sound Examples

Overview
• Original
Early Digital Synthesis

Spectral Modeling
• Vocoder
• 10 channels, sine carriers
• Vocoder Examples • 10 channels, narrowband-noise carriers
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• 26 channels, sine carriers
• S+N Examples
• S+N FX • 26 channels, narrowband-noise carriers
• S+N XSynth
• Sines + Transients
• 26 channels, narrowband-noise carriers, channels reversed
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map • Phase Vocoder: Identity system in absence of modifications
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band • The FFT Phase Vocoder next transitioned to the Short-Time
• S+N+T Examples
Fourier Transform (STFT) (Allen and Rabiner 1977)
• SM Summary
• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 29 / 84

 Tracking Spectral Peaks in the Short-Time Fourier Transform

Overview
Phases
Early Digital Synthesis
atan
Spectral Modeling
• Vocoder s(t)
• Vocoder Examples FFT Frequencies
• Sinusoidal Modeling
Quadratic
• Spectral Trajectories Peak
dB mag Peak
tracking Interpolation
• Sines + Noise
window w(n)
• S+N Examples
Amplitudes
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM • STFT peak tracking at CCRMA: mid-1980s (PARSHL program)
• S+N+T Freq Map • Motivated by vocoder analysis of piano tones
• S+N+T Windows
• HF Noise Modeling • Influences: STFT (Allen and Rabiner 1977),
• HF Noise Band
ADEC (1977), MAPLE (1979)
• S+N+T Examples
• SM Summary • Independently developed for speech coding by McAulay and
• Spec Future
Quatieri at Lincoln Labs
Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 30 / 84

 Example Spectral Trajectories

Overview
f
Early Digital Synthesis

Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band t
• S+N+T Examples
• SM Summary
• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 31 / 84

 Sines + Noise Synthesis (1989)
PSfrag
Overview
A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)
Early Digital Synthesis

Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows white noise
• HF Noise Modeling filter(t)
• HF Noise Band
u(t)
• S+N+T Examples
ht (τ )
%

• SM Summary
• Spec Future
4
Physical Modeling
t
y(t) = Ai (t) cos 0 ωi (t)dt + φi (0) + (ht ∗ u)(t)
Summary i=1
% &' (

Julius Smith AES-2006 Heyser Lecture – 32 / 84

 Sines + Noise Sound Examples

Overview
Xavier Serra 1989 thesis demos (Sines + Noise signal modeling)
Early Digital Synthesis

Spectral Modeling
• Piano
• Vocoder
• Vocoder Examples ◦ Original
• Sinusoidal Modeling
◦ Sinusoids alone
• Spectral Trajectories
• Sines + Noise ◦ Residual after sinusoids removed
• S+N Examples
• S+N FX
◦ Sines + noise model
• S+N XSynth
• Sines + Transients • Voice
• S + N + Transients
• S+N+T TSM ◦ Original
• S+N+T Freq Map
• S+N+T Windows
◦ Sinusoids
• HF Noise Modeling ◦ Residual
• HF Noise Band
• S+N+T Examples
◦ Synthesis
• SM Summary
• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 33 / 84

 Musical Effects with Sines+Noise Models (Serra 1989)

Overview
• Piano Effects
Early Digital Synthesis

Spectral Modeling
◦ Pitch downshift one octave
• Vocoder ◦ Pitch flattened
• Vocoder Examples
• Sinusoidal Modeling ◦ Varying partial stretching
• Spectral Trajectories
• Sines + Noise • Voice Effects
• S+N Examples
• S+N FX ◦ Frequency-scale by 0.6
• S+N XSynth
• Sines + Transients ◦ Frequency-scale by 0.4 and stretch partials
• S + N + Transients ◦ Variable time-scaling, deterministic to stochastic
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 34 / 84

 Cross-Synthesis with Sines+Noise Models (Serra 1989)

Overview
• Voice “modulator”
Early Digital Synthesis

Spectral Modeling
• Vocoder
• Creaking ship’s mast “carrier”
• Vocoder Examples • Voice-modulated creaking mast
• Sinusoidal Modeling
• Spectral Trajectories • Same with modified spectral envelopes
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 35 / 84

 Sines + Transients Sound Examples (Serra 1989)

Overview
In this technique, the sinusoidal sum is phase-matched at the
Early Digital Synthesis
cross-over point only (with no cross-fade).
Spectral Modeling
• Vocoder
• Vocoder Examples
• Marimba
• Sinusoidal Modeling
• Spectral Trajectories ◦ Original
• Sines + Noise ◦ Sinusoidal model
• S+N Examples
• S+N FX ◦ Original attack, followed by sinusoidal model
• S+N XSynth
• Sines + Transients • Piano
• S + N + Transients
• S+N+T TSM ◦ Original
• S+N+T Freq Map
• S+N+T Windows ◦ Sinusoidal model
• HF Noise Modeling ◦ Original attack, followed by sinusoidal model
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 36 / 84

 Multiresolution Sines + Noise + Transients (Levine 1998)

Overview
Why Model Transients Separately?
Early Digital Synthesis

Spectral Modeling
• Sinusoids efficiently model spectral peaks over time
• Vocoder • Filtered noise efficiently models spectral residual vs. t
• Vocoder Examples
• Sinusoidal Modeling • Neither is good for abrupt transients in the waveform
• Spectral Trajectories
• Phase-matched oscillators are expensive
• Sines + Noise
• S+N Examples • More efficient to switch to a transient model during transients
• S+N FX
• S+N XSynth
• Need sinusoidal phase matching at the switching times
• Sines + Transients
• S + N + Transients Transient models:
• S+N+T TSM
• S+N+T Freq Map • Original waveform slice (1988)
• S+N+T Windows
• HF Noise Modeling
• Wavelet expansion (Ali 1996)
• HF Noise Band • MPEG-2 AAC (with short window) (Levine 1998)
• S+N+T Examples
• SM Summary
• Frequency-domain LPC
• Spec Future (time-domain amplitude envelope) (Verma 2000)
Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 37 / 84

 Time Scale Modification of Sines + Noise + Transients Models

Overview

Early Digital Synthesis original signal sines + sines + sines +

transients
transients
noise noise noise
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise time-scaled sines + sines + sines +
• S+N Examples signal

transients
transients

noise noise noise

• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients time
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows Time-Scale Modification (TSM) becomes well defined:
• HF Noise Modeling
• HF Noise Band • Transients are translated in time
• S+N+T Examples
• SM Summary • Sinusoidal envelopes are scaled in time
• Spec Future • Noise-filter envelopes also scaled in time
Physical Modeling
• Dual of TSM is frequency scaling
Summary

Julius Smith AES-2006 Heyser Lecture – 38 / 84

 Sines + Noise + Transients Time-Frequency Map

16
Overview
14
Early Digital Synthesis
12
Spectral Modeling
• Vocoder 10
• Vocoder Examples
• Sinusoidal Modeling 8

• Spectral Trajectories 6

frequency [kHz]
• Sines + Noise
• S+N Examples 4

• S+N FX 2
• S+N XSynth
• Sines + Transients 0
0 50 100 150 200 250
• S + N + Transients 1
• S+N+T TSM
• S+N+T Freq Map 0.5

• S+N+T Windows
0
• HF Noise Modeling

amplitude
• HF Noise Band −0.5
• S+N+T Examples
−1
• SM Summary 0 50 100 150 200 250
• Spec Future time [milliseconds]

Physical Modeling (Levine 1998)

Summary

Julius Smith AES-2006 Heyser Lecture – 39 / 84

 Corresponding Analysis Windows

Overview

Early Digital Synthesis

transient
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX

middle octave high octave

• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM

low octave
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary

amplitude
• Spec Future

Physical Modeling
0 50 100 150 200 250
Summary time [milliseconds]

Julius Smith AES-2006 Heyser Lecture – 40 / 84

 Quasi-Constant-Q (Wavelet) Time-Frequency Map

16
Overview
14
Early Digital Synthesis

Spectral Modeling
12

• Vocoder 10
• Vocoder Examples
• Sinusoidal Modeling 8
• Spectral Trajectories
6

frequency [kHz]
• Sines + Noise
• S+N Examples 4
• S+N FX
2
• S+N XSynth
• Sines + Transients 0
0 4 50 100 150 200 250
• S + N + Transients x 10
• S+N+T TSM 2
• S+N+T Freq Map
1
• S+N+T Windows
0
• HF Noise Modeling

amplitude
• HF Noise Band −1
• S+N+T Examples −2
• SM Summary
0 50 100 150 200 250
• Spec Future time [milliseconds]
Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 41 / 84

 Bark-Band Noise Modeling at High Frequencies (Levine 1998)

Overview 85

Early Digital Synthesis

80
Spectral Modeling
• Vocoder 75
• Vocoder Examples
• Sinusoidal Modeling 70
• Spectral Trajectories
• Sines + Noise 65
• S+N Examples
• S+N FX 60
• S+N XSynth
• Sines + Transients 55
• S + N + Transients

magnitude [dB]
• S+N+T TSM 50
• S+N+T Freq Map
• S+N+T Windows 45
• HF Noise Modeling
• HF Noise Band 40
• S+N+T Examples
• SM Summary 35
• Spec Future

Physical Modeling 30
0 500 1000 1500 2000 2500 3000 3500 4000 4500
Summary
frequency [Hz]

Julius Smith AES-2006 Heyser Lecture – 42 / 84

 Amplitude Envelope for One Noise Band

Overview 80

Early Digital Synthesis

70
Spectral Modeling
• Vocoder
• Vocoder Examples 60
• Sinusoidal Modeling
• Spectral Trajectories 50

Original Mag. [dB]

• Sines + Noise
• S+N Examples 40
• S+N FX 0 50 100 150 200 250 300 350
• S+N XSynth
• Sines + Transients
• S + N + Transients 80

• S+N+T TSM 75
• S+N+T Freq Map 70
• S+N+T Windows
65
• HF Noise Modeling
• HF Noise Band 60
• S+N+T Examples 55

LSA Mag. [dB]

• SM Summary
50
• Spec Future
45
Physical Modeling 0 50 100 150 200 250 300 350
time [milliseconds]
Summary

For more information, see Scott Levine’s thesis.2

Julius Smith 2 AES-2006 Heyser Lecture – 43 / 84
http://ccrma.stanford.edu/~scottl/thesis.html
 Sines + Noise + Transients Sound Examples

Overview
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)
Early Digital Synthesis
(http://ccrma.stanford.edu/~scottl/thesis.html)
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise Mozart’s Le Nozze di Figaro
• S+N Examples
• S+N FX • Original
• S+N XSynth
• Sines + Transients
• Compressed using MPEG-AAC at 32 kbps
• S + N + Transients • Compressed using sines+transients+noise at 32 kbps
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows • Multiresolution sinusoids alone
• HF Noise Modeling
• HF Noise Band • Residual Bark-band noise
• S+N+T Examples
• Transform-coded transients (AAC)
• SM Summary
• Spec Future • Bark-band noise above 5 kHz
Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 44 / 84

 Rock Example

Overview
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)
Early Digital Synthesis
(http://ccrma.stanford.edu/~scottl/thesis.html)
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
“It Takes Two” by Rob Base & DJ E-Z Rock
• S+N Examples
• S+N FX • Original
• S+N XSynth • MPEG-AAC at 32 kbps
• Sines + Transients
• S + N + Transients • Sines+transients+noise at 32 kbps
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows • Multiresolution sinusoids
• HF Noise Modeling
• HF Noise Band
• Residual Bark-band noise
• S+N+T Examples • Transform-coded transients (AAC)
• SM Summary
• Spec Future • Bark-band noise above 5 kHz
Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 45 / 84

 Time Scale Modification using Sines + Noise + Transients

Overview
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)
Early Digital Synthesis
(http://ccrma.stanford.edu/~scottl/thesis.html)
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
Time-Scale Modification (pitch unchanged)
• S+N Examples
• S+N FX • S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5]
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM S+N+T Pitch Shifting (timing unchanged)
• S+N+T Freq Map
• S+N+T Windows • Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12]
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 46 / 84

 Spectral Modeling History Highlights

• Fourier’s theory (1822) Perceptual audio coding:

• Teleharmonium (1906)
• Princen-Bradley filterbank (1986)
• Hammond organ (1930s)
• K. Brandenburg thesis (1989)
• Channel Vocoder (1939)
• Auditory masking usage
• Phase Vocoder (1966)
• Dolby AC2
• “Additive Synthesis” (1969)
• Musicam
• FFT Phase Vocoder (1976)
• ASPEC
• Sinusoidal Modeling
• MPEG-I,II,IV
(1977,1979,1985)
(incl. S+N+T “parametric sounds”)
• Sines+Noise (1989)
• Sines+Transients (1989)
• Sines+Noise+Transients (1998)

Julius Smith AES-2006 Heyser Lecture – 47 / 84

 Future Prospects

Overview
Observations:
Early Digital Synthesis

Spectral Modeling
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Vocoder • Basic “auditory masking” discards ≈ 90% information
• Vocoder Examples
• Sinusoidal Modeling • Interesting neuroscience observation:
• Spectral Trajectories
• Sines + Noise “... most neurons in the primary auditory cortex A1
• S+N Examples
• S+N FX
are silent most of the time ...”
• S+N XSynth
• Sines + Transients
(from “Sparse Time-Frequency Representations”, Gardner and
• S + N + Transients Magnesco, PNAS:103(16), April 2006)
• S+N+T TSM
• S+N+T Freq Map • What is a true and correct “psychospectral model” for sound?
• S+N+T Windows
• HF Noise Modeling ◦ The cochlea of the ear is a real-time spectrum analyzer
• HF Noise Band
• S+N+T Examples
◦ How is the “ear’s spectrogram” represented at higher levels?
• SM Summary
• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 48 / 84

Overview
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Summary

Physical Modeling Synthesis

(Historical Summary)

Julius Smith AES-2006 Heyser Lecture – 49 / 84

 Kelly-Lochbaum Vocal Tract Model

Overview

Early Digital Synthesis

Spectral Modeling e(n) … y(n)

Physical Modeling
• KL Music
• Digital Waveguide
• Signal Scattering
• Plucked String Glottal Pulse
Train or Noise 1 + k1 1 + kM
• Struck String Speech
e(n) z− 1 2 … z− 1 2
• Karplus Strong Output
• EKS Algorithm y(n)
k1 − k1 R1 kM − kM RM
• Clarinet
• Wind Examples (Unused
• Bowed Strings Allpass z− 1 2 … z− 1 2
• Distortion Guitar Output) 1 − k1 1 − kM
• Acoustic Strings
Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
John L. Kelly and Carol Lochbaum (1962)
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 50 / 84
 Digital Waveguide Models (1985)

Overview
Early Digital Synthesis
Spectral Modeling
∆
Lossless digital waveguide = bidirectional delay line
at some wave impedance R

Physical Modeling
• KL Music
• Digital Waveguide
• Signal Scattering
z−N
• Plucked String
• Struck String
• Karplus Strong R
• EKS Algorithm
• Clarinet
• Wind Examples z−N
• Bowed Strings
• Distortion Guitar
• Acoustic Strings Useful for efficient models of
• Sound Examples
• Linearized Violin
• Commuted Piano
• strings
• Pulse Synthesis • bores
• Complete Piano
• Sound Examples
• plane waves
• Phy Audio Coding • conical waves
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 51 / 84
 Signal Scattering

Overview
Signal scattering is caused by a change in wave impedance R:
Early Digital Synthesis

Spectral Modeling R − R1
k1 = 2
Physical Modeling R2 + R1
• KL Music
• Digital Waveguide
• Signal Scattering 1 + k1
• Plucked String
z−N z−N
• Struck String
• Karplus Strong
• EKS Algorithm k1 − k1
• Clarinet
R1 R2
• Wind Examples
• Bowed Strings
• Distortion Guitar z−N z−N
• Acoustic Strings 1 − k1
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
If the wave impedance changes every spatial sample, the
• Complete Piano Kelly-Lochbaum vocal-tract model results.
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 52 / 84
 Ideal Plucked String (Displacement Waves)

Overview

Early Digital Synthesis

Spectral Modeling + +
y (n) y (n-N/2)
Physical Modeling
• KL Music
• Digital Waveguide “Bridge” -1 (x = Pluck Position) -1 “Nut”
• Signal Scattering
• Plucked String
• Struck String
y-(n) y-(n+N/2)
• Karplus Strong
• EKS Algorithm
• Clarinet (x = 0) (x = L)
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples • Load each delay line with half of initial string displacement
• Linearized Violin
• Commuted Piano
• Sum of upper and lower delay lines = string displacement
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 53 / 84
 Ideal Struck String (Velocity Waves)

Overview

Early Digital Synthesis

c
Spectral Modeling
v+(n) v+(n-N/2)
Physical Modeling
• KL Music
• Digital Waveguide “Bridge” -1 (x = Hammer Position) -1 “Nut”
• Signal Scattering
• Plucked String c
• Struck String v-(n) v-(n+N/2)
• Karplus Strong
• EKS Algorithm (x = 0) (x = L)
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings Hammer strike = momentum transfer = velocity step:
• Sound Examples
• Linearized Violin
• Commuted Piano mh vh (0−) = (mh + ms )vs (0+)
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 54 / 84
 Karplus-Strong (KS) Algorithm (1983)

Overview

Early Digital Synthesis

+
Output y (n)
+
y (n-N)
Spectral Modeling N samples delay
Physical Modeling
• KL Music 1/2
• Digital Waveguide
• Signal Scattering
• Plucked String
1/2
• Struck String
z -1
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings • Discovered (1978) as “self-modifying wavetable synthesis”
• Distortion Guitar
• Acoustic Strings • Wavetable is preferably initialized with random numbers
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 55 / 84
 Karplus-Strong Sound Examples

Overview
• “Vintage” 8-bit sound examples:
Early Digital Synthesis

Spectral Modeling
• Original Plucked String: (WAV) (MP3)
Physical Modeling • Drum: (WAV) (MP3)
• KL Music • Stretched Drum: (WAV) (MP3)
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 56 / 84
 EKS Algorithm (Jaffe-Smith 1983)

Overview
Hp (z) Hβ (z) z −N HL (z)
Early Digital Synthesis

Spectral Modeling

Physical Modeling Hρ (z) Hs (z) Hd (z)

• KL Music
• Digital Waveguide
• Signal Scattering
N = pitch period (2× string length) in samples
• Plucked String
• Struck String 1−p
• Karplus Strong Hp (z) = −1
= pick-direction lowpass filter
• EKS Algorithm 1 − pz
• Clarinet
• Wind Examples Hβ (z) = 1 − z −βN = pick-position comb filter, β ∈ (0, 1)
• Bowed Strings
• Distortion Guitar Hd (z) = string-damping filter (one/two poles/zeros typical)
• Acoustic Strings
• Sound Examples Hs (z) = string-stiffness allpass filter (several poles and zeros)
• Linearized Violin
• Commuted Piano ρ(N ) − z −1
• Pulse Synthesis
Hρ (z) = −1
= first-order string-tuning allpass filter
• Complete Piano
1 − ρ(N ) z
• Sound Examples 1 − RL
• Phy Audio Coding HL (z) = = dynamic-level lowpass filter
• Phy Audio Coding? 1 − RL z −1
Summary
Julius Smith AES-2006 Heyser Lecture – 57 / 84
 STK EKS Sound Examples

Overview
• Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and
Early Digital Synthesis
others — distributed by CCRMA:
Spectral Modeling
Google search: STK ToolKit
Physical Modeling
• KL Music
• Digital Waveguide STK Plucked String: (WAV) (MP3)
• Signal Scattering
• Plucked String • Plucked String 1: (WAV) (MP3)
• Struck String
• Karplus Strong • Plucked String 2: (WAV) (MP3)
• EKS Algorithm
• Clarinet
• Plucked String 3: (WAV) (MP3)
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 58 / 84
 EKS Sound Example (1988)

Overview
Bach A-Minor Concerto—Orchestra Part: (WAV) (MP3)
Early Digital Synthesis

Spectral Modeling

Physical Modeling • Executed in real time on one Motorola DSP56001

• KL Music
• Digital Waveguide
(20 MHz clock, 128K SRAM)
• Signal Scattering
• Plucked String
• Struck String • Developed for the NeXT Computer introduction at Davies
• Karplus Strong Symphony Hall, San Francisco, 1988
• EKS Algorithm
• Clarinet
• Wind Examples
• Solo violin part was played live by Dan Kobialka of the San
• Bowed Strings
• Distortion Guitar Francisco Symphony
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 59 / 84
 Digital Waveguide Single Reed, Cylindrical Bore Model (1986)

Mouth
Overview
Pressure p−b ( n )
Early Digital Synthesis pm ( n ) hm Output
Reed to Bell Delay
2 Filter
Spectral Modeling -
Reed Table
Physical Modeling ρ̂ Reflection
• KL Music Filter
• Digital Waveguide
*
h+Δ
• Signal Scattering
• Plucked String
- Bell to Reed Delay
• Struck String Embouchure -
p+b ( n )
• Karplus Strong Offset
• EKS Algorithm Reed Bore Bell
• Clarinet
• Wind Examples
• Bowed Strings
Digital waveguide clarinet
• Distortion Guitar
• Acoustic Strings • Control variable = mouth half-pressure
• Sound Examples
• Linearized Violin • Total reed cost = two subtractions, one multiply, and one table
• Commuted Piano
lookup per sample
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 60 / 84
 Digital Waveguide Wind Instrument Sound Examples

Overview
• STK Clarinet: (WAV) (MP3)
Early Digital Synthesis
Google search: STK clarinet
Spectral Modeling

Physical Modeling ◦ Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and
• KL Music
others — distributed by CCRMA:
• Digital Waveguide
• Signal Scattering Google search: STK ToolKit
• Plucked String
• Struck String • Staccato Systems Slide Flute
• Karplus Strong
• EKS Algorithm (based on STK flute, ca. 1995): (WAV) (MP3)
• Clarinet • Yamaha VL1 “Virtual Lead” synthesizer demos (1994):
• Wind Examples
• Bowed Strings
• Shakuhachi: (WAV) (MP3)
• Distortion Guitar
• Acoustic Strings • Oboe and Bassoon: (WAV) (MP3)
• Sound Examples
• Linearized Violin
• Tenor Saxophone: (WAV) (MP3)
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 61 / 84
 Digital Waveguide Bowed Strings (1986)

Bow Force
Overview Bow Velocity
v+s, l v−s, r
Early Digital Synthesis Body
Nut to Bow Delay Bow to Bridge Delay Filter
Spectral Modeling
vb - v+Δ ρ̂ Reflection
Physical Modeling -1
* Filter
• KL Music - Bow Table
• Digital Waveguide
v+s, r
• Signal Scattering Bow to Nut Delay Bridge to Bow Delay
• Plucked String v−s, l
• Struck String Bridge-
Nut String Bow String Air
• Karplus Strong Body
• EKS Algorithm
• Clarinet
• Wind Examples • Reflection filter summarizes all losses per period
• Bowed Strings
(due to bridge, bow, finger, etc.)
• Distortion Guitar
• Acoustic Strings • Bow-string junction = memoryless lookup table
• Sound Examples
• Linearized Violin
(or segmented polynomial)
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 62 / 84
 “Electric Cello” Sound Examples (Peder Larson)

Overview
• Staccato Notes: (WAV) (MP3)
Early Digital Synthesis
(short strokes of high bow pressure, as from a bouncing bow)
Spectral Modeling
• Bach’s First Suite for Unaccompanied Cello: (WAV) (MP3)
Physical Modeling
• KL Music
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 63 / 84
 Soft Clipper

Overview
− , x ≤ −1
Early Digital Synthesis
 2

Spectral Modeling f (x) = x3

x− 3 , −1 ≤ x ≤ 1



Physical Modeling
 3

2
• KL Music x≥1
3,
• Digital Waveguide



• Signal Scattering


• Plucked String 3
x=−1:0.01:1; plot([−(2/3)*ones(1,100), x−x. /3, (2/3)*ones(1,100)])
0.8
• Struck String
• Karplus Strong
0.6
• EKS Algorithm
• Clarinet 0.4

• Wind Examples
0.2
• Bowed Strings
• Distortion Guitar
0

f(x(n))
• Acoustic Strings
• Sound Examples −0.2

• Linearized Violin
−0.4
• Commuted Piano
• Pulse Synthesis −0.6
• Complete Piano
• Sound Examples −0.8
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2
x(n)
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 64 / 84
 Amplifier Distortion + Amplifier Feedback

Overview
Sullivan 1990
Early Digital Synthesis

Spectral Modeling

Physical Modeling
• KL Music Pre-distortion output level
• Digital Waveguide
String 1
• Signal Scattering
Nonlinear Distortion
• Plucked String
• Struck String
..
. Output Signal
• Karplus Strong
• EKS Algorithm Pre-distortion gain Distortion output level
• Clarinet String N Amplifier
• Wind Examples Feedback
• Bowed Strings Gain
• Distortion Guitar
• Acoustic Strings
Amplifier Feedback Delay
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
Distortion output signal often further filtered by an amplifier cabinet
• Phy Audio Coding filter, representing speaker cabinet, driver responses, etc.
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 65 / 84
 Distortion Guitar Sound Examples

Overview
(Stanford Sondius Project, ca. 1995)
Early Digital Synthesis

Spectral Modeling
• Distortion Guitar: (WAV) (MP3)
Physical Modeling • Amplifier Feedback 1: (WAV) (MP3)
• KL Music
• Amplifier Feedback 2: (WAV) (MP3)
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 66 / 84
 Commuted Synthesis of Acoustic Strings (1993)

Overview
e(t) s(t) y(t)
Early Digital Synthesis Trigger Excitation String Resonator Output
Spectral Modeling

Physical Modeling
• KL Music Schematic diagram of a stringed musical instrument.
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
Trigger Excitation Resonator String Output
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
Equivalent diagram in the linear, time-invariant case.
• Distortion Guitar
• Acoustic Strings
• Sound Examples a(t) x(t)
Aggregate
• Linearized Violin Trigger String Output
Excitation
• Commuted Piano
• Pulse Synthesis
• Complete Piano Use of an aggregate excitation given by the convolution of original
• Sound Examples
excitation with the resonator impulse response.
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 67 / 84
 Commuted Components

Overview

Early Digital Synthesis a(t) x(t)

Aggregate String
Spectral Modeling
Trigger Output
Excitation
Physical Modeling
• KL Music
“Plucked Resonator” driving a String.
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet s(t) Bridge Guitar Air Room y(t)
• Wind Examples
Coupling Body Absorption Response Output
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples Possible components of a guitar resonator.
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 68 / 84
 Sound Examples

Overview
Electric Guitar (Pick-Ups and/or Body-Model Added) (Stanford
Early Digital Synthesis
Sondius Project → Staccato Systems, Inc. → ADI, ca. 1995)
Spectral Modeling

Physical Modeling • Example 1: (WAV) (MP3)

• KL Music
• Digital Waveguide • Example 2: (WAV) (MP3)
• Signal Scattering
• Plucked String
• Example 3: (WAV) (MP3)
• Struck String • Virtual “wah-wah pedal”: (WAV) (MP3)
• Karplus Strong
• EKS Algorithm
• Clarinet
STK Mandolin
• Wind Examples
• Bowed Strings • STK Mandolin 1: (WAV) (MP3)
• Distortion Guitar
• Acoustic Strings • STK Mandolin 2: (WAV) (MP3)
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 69 / 84
 Sound Examples

Overview
More Recent Acoustic Guitar
Early Digital Synthesis

Spectral Modeling
• Bach Prelude in E Major: (WAV) (MP3)
Physical Modeling • soundexamplewavBach silenceLoure in E Major: (WAV) (MP3)
• KL Music
• Digital Waveguide
• Signal Scattering Virtual performance by Dr. Mikael Laurson, Sibelius Institute
• Plucked String
• Struck String
• Karplus Strong Virtual guitar by Helsinki Univ. of Tech., Acoustics Lab3
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding 3
• Phy Audio Coding? http://www.acoustics.hut.fi/
Summary
Julius Smith AES-2006 Heyser Lecture – 70 / 84
 Commuted Synthesis of Linearized Violin

a) Output
Overview e(n) s(n) x(n)
Amplitude(n) Impulse
Frequency(n) String Resonator
Early Digital Synthesis Train
Spectral Modeling
b) Output
Physical Modeling e(n)
Amplitude(n) Impulse a(n) x(n)
• KL Music Resonator String
Frequency(n) Train
• Digital Waveguide
• Signal Scattering
• Plucked String c) Output
a(n) x(n)
• Struck String Amplitude(n) Impulse-Response
• Karplus Strong Frequency(n) Train String
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Assumes ideal Helmholtz motion of string
• Distortion Guitar • Sound Examples (Stanford Sondius project, ca. 1995):
• Acoustic Strings ◦ Bass: (WAV) (MP3) ◦ Violin 1: (WAV) (MP3)
• Sound Examples
• Linearized Violin ◦ Cello: (WAV) (MP3) ◦ Violin 2: (WAV) (MP3)
• Commuted Piano
• Pulse Synthesis
◦ Viola 1: (WAV) (MP3) ◦ Ensemble: (WAV) (MP3)
• Complete Piano ◦ Viola 2: (WAV) (MP3)
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 71 / 84
 Commuted Piano Synthesis (1995)

Overview
Hammer-string interaction pulses (force):
Early Digital Synthesis

Spectral Modeling Force

0.5
Physical Modeling
0.4
• KL Music
0.3
• Digital Waveguide
• Signal Scattering 0.2

• Plucked String 0.1

• Struck String Time
5 10 15 20
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 72 / 84
 Synthesis of Hammer-String Interaction Pulse

Overview
Impulse Impulse Response
Early Digital Synthesis

Spectral Modeling
Lowpass
Physical Modeling Filter
• KL Music
• Digital Waveguide Time Time
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong • Faster collisions correspond to narrower pulses
• EKS Algorithm
• Clarinet (nonlinear filter )
• Wind Examples
• Bowed Strings
• Distortion Guitar • For a given velocity, filter is linear time-invariant
• Acoustic Strings
• Sound Examples
• Linearized Violin • Piano is “linearized” for each hammer velocity
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 73 / 84
 Multiple Hammer-String Interaction Pulses

Overview
Superimpose several individual pulses:
Early Digital Synthesis

Spectral Modeling

Physical Modeling
Force
• KL Music
• Digital Waveguide Impulse 1 LPF1
• Signal Scattering
δ1
String
• Plucked String
Impulse 2 LPF2
δ2 + Input
• Struck String
Impulse 3 LPF3
• Karplus Strong δ3
• EKS Algorithm 0 Time
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 74 / 84
 Multiple Hammer-String Interaction Pulses

Overview
Superimpose several individual pulses:
Early Digital Synthesis

Spectral Modeling

Physical Modeling
Force
• KL Music
• Digital Waveguide Impulse 1 LPF1
• Signal Scattering
δ1
String
• Plucked String
Impulse 2 LPF2
δ2 + Input
• Struck String
Impulse 3 LPF3
• Karplus Strong δ3
• EKS Algorithm 0 Time
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar As impulse amplitude grows (faster hammer strike), output pulses
• Acoustic Strings
• Sound Examples become taller and thinner, showing less overlap.
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 74 / 84
 Complete Piano Model

Natural Ordering:
Overview

Early Digital Synthesis

vc
Spectral Modeling
LPF1
Physical Modeling Impulse δ1 Tapped δ1
Sound Board
• KL Music Gener- Delay LPF2 + String
Trigger ator Line δ2
& Enclosure Output
• Digital Waveguide
LPF3
• Signal Scattering δ3
• Plucked String
• Struck String
• Karplus Strong Commuted Ordering:
• EKS Algorithm
• Clarinet vc
• Wind Examples
• Bowed Strings LPF1
Sound Board Tapped
• Distortion Guitar & Enclosure String
Delay LPF2 + Output
• Acoustic Strings Trigger Impulse Response Line
• Sound Examples LPF3
• Linearized Violin
• Commuted Piano
• Pulse Synthesis • Soundboard and enclosure are commuted
• Complete Piano
• Sound Examples • Only need a stored recording of their impulse response
• Phy Audio Coding
• Phy Audio Coding?
• An enormous digital filter is otherwise required
Summary
Julius Smith AES-2006 Heyser Lecture – 75 / 84
 Piano and Harpsichord Sound Examples

Overview
(Stanford Sondius Project, ca. 1995)
Early Digital Synthesis

Spectral Modeling • Piano: (WAV) (MP3)

Physical Modeling • Harpsichord 1: (WAV) (MP3)
• KL Music
• Digital Waveguide • Harpsichord 2: (WAV) (MP3)
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 76 / 84
 More Recent Harpsichord Example

Overview
• Harpsichord Soundboard Hammer-Response: (WAV) (MP3)
Early Digital Synthesis
• Musical Commuted Harpsichord Example: (WAV) (MP3)
Spectral Modeling

Physical Modeling Reference:

• KL Music
• Digital Waveguide
• Signal Scattering “Sound Synthesis of the Harpsichord Using a Computationally
• Plucked String Efficient Physical Model”,
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
by Vesa Välimäki, Henri Penttinen, Jonte Knif, Mikael Laurson,
• Wind Examples and Cumhur Erkut
• Bowed Strings
• Distortion Guitar
• Acoustic Strings JASP-2004
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
Google search: Harpsichord Sound Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 77 / 84
 Physical Modeling in Audio Coding

Spectral modeling synthesis is finding application in audio coding.

Overview
Can physical modeling synthesis be used as well?
Early Digital Synthesis

Spectral Modeling • MPEG-4/SAOL already supports essentially all sound synthesis

Physical Modeling methods
• KL Music
• Digital Waveguide • Ability to encode sounds automatically is limited
• Signal Scattering
• Plucked String ◦ Codebook-Excited Linear Prediction (CELP) is a successful
• Struck String
• Karplus Strong
source-filter model (not quite physical)
• EKS Algorithm ◦ There are many isolated examples of model-fitting to
• Clarinet
• Wind Examples recorded data
• Bowed Strings ◦ Good model-based denoising results have been obtained
• Distortion Guitar
• Acoustic Strings ◦ Coder problem much harder when many sources are mixed
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 78 / 84
 Best Known Model-Based Audio Coders

Overview
A “cover band” can put together a very convincing facsimile of
Early Digital Synthesis
popular music performance
Spectral Modeling

Physical Modeling
• KL Music
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
JOS high-school band “Bittersweet”
Summary
Julius Smith AES-2006 Heyser Lecture – 79 / 84
 Future Physical Modeling in Audio Coding?

Overview
A “Cover Band” Approach to Model-Based Audio Coding:
Early Digital Synthesis

Spectral Modeling
1. Recognize individual “audio streams” in a mix (CASA)
Physical Modeling (“I hear a trap set, electric bass, Fender Rhodes, and a strat”)
• KL Music
2. For each stream, calibrate its model heuristically
• Digital Waveguide
• Signal Scattering (“Here is what I hear the bass part doing: ...”)
• Plucked String
• Struck String
3. Fine-tune the synthetic mix to the real mix
• Karplus Strong (joint “maximum likelihood estimation”)
• EKS Algorithm
• Clarinet Features of “Cover-Band Coding” (CBC):
• Wind Examples
• Bowed Strings
• The “playing experience” of each “virtual performer” prevents
• Distortion Guitar
• Acoustic Strings artifacts — “musically unreasonable” parameters are made
• Sound Examples
• Linearized Violin
unlikely (“Bayesian priors”)
• Commuted Piano • An incorrect instrument must “imitate” its assigned stream
• Pulse Synthesis
• Complete Piano • New arrangements can be synthesized by deliberately choosing
• Sound Examples a new ensemble!
• Phy Audio Coding
• Phy Audio Coding?

Summary
Julius Smith AES-2006 Heyser Lecture – 80 / 84
Overview
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Summary

Summary

Julius Smith AES-2006 Heyser Lecture – 81 / 84

 Summary

Overview
We have reviewed a “CCRMA-centric slice” of the history of digital
Early Digital Synthesis
sound synthesis (usually starting with results from Bell Labs):
Spectral Modeling

Physical Modeling • Wavetable (one period)

Summary • Subtractive
• Additive
• FM
• Sampling
• Spectral Modeling
• Physical Modeling (more in tomorrow’s 4:30 PM masterclass)
• Connections to audio coding

Julius Smith AES-2006 Heyser Lecture – 82 / 84

Overview
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Summary
Sound Acknowledgment

Sound Acknowledgment

Julius Smith AES-2006 Heyser Lecture – 83 / 84

 Sound Acknowledgment

Overview
Thanks to Emu / Creative Labs for providing a superb-quality
Early Digital Synthesis
external D/A converter for this talk (an E-Mu 0404/USB)
Spectral Modeling

Physical Modeling

Summary

Sound Acknowledgment

Julius Smith AES-2006 Heyser Lecture – 84 / 84