Professional Documents
Culture Documents
Julius Smith
CCRMA, Stanford University
October 6, 2006
Overview
Overview
Digital sound synthesis approaches in approximate historical order:
• Outline
• CCRMA Perspective
• Wavetable (one period)
Early Digital Synthesis
• Subtractive
Spectral Modeling
• Additive
Physical Modeling
Summary
• Frequency Modulation (FM)
• Sampling
• Spectral Modeling
• Physical Modeling
Some connections with audio coding will be noted
Emphasis:
• Sound examples
• Block diagrams
• Historical notes
Overview
• Outline
• CCRMA Perspective
Spectral Modeling
Physical Modeling
Summary
Overview
Spectral Modeling
Physical Modeling
Summary
Overview
Spectral Modeling
Physical Modeling
Summary
Overview
Summary
Overview
“Bicycle Built for Two”: (WAV) (MP3)
Early Digital Synthesis
• Music V
• KL Music
• “Daisy” • Vocal part by Kelly and Lochbaum (1961)
• Additive Analysis
• Musical accompaniment by Max Mathews
• Additive Synthesis
• FM Synthesis • Computed on an IBM 704
• FM Formula
• FM Patch
• Based on Russian speech-vowel data from Gunnar Fant’s book
• FM Spectra • Probably the first digital physical-modeling synthesis sound
• FM Examples
• FM Voice example by any method
• Sampling Synthesis • Inspired Arthur C. Clarke to adapt it for “2001: A Space Odyssey”
• Modern Example
— the computer’s “first song”
Spectral Modeling
Physical Modeling
Summary
Overview
Spectral Modeling
Physical Modeling
Summary
Overview
Spectral Modeling
Physical Modeling
Summary
Spectral Modeling
noise
FIR
Physical Modeling Σ
Summary
4 t
y(t) = Ai (t) sin ωi (t)dt + φi (0)
0
"# $
i=1
!
Overview
• Bb Clarinet
Early Digital Synthesis
• Music V
• Eb Clarinet
• KL Music • Oboe
• “Daisy”
• Additive Analysis • Bassoon
• Additive Synthesis • Tenor Saxophone
• FM Synthesis
• FM Formula • Trumpet
• FM Patch
• English Horn
• FM Spectra
• FM Examples • French Horn
• FM Voice
• Sampling Synthesis
• Flute
• Modern Example
Overview
FM synthesis is normally used as a spectral modeling technique.
Early Digital Synthesis
• Music V • Discovered and developed (1970s) by John M. Chowning
• KL Music
• “Daisy”
(CCRMA Founding Director)
• Additive Analysis • Key paper: JAES 1973 (vol. 21, no. 7)
• Additive Synthesis
• FM Synthesis • Commercialized by Yamaha Corporation:
• FM Formula
• FM Patch ◦ DX-7 synthesizer (1983)
• FM Spectra
• FM Examples
◦ OPL chipset (SoundBlaster PC sound card)
• FM Voice ◦ Cell phone ring tones
• Sampling Synthesis
• Modern Example
• On the physical modeling front, synthesis of vibrating-string
Spectral Modeling
waveforms using finite differences started around this time:
Physical Modeling
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
Summary
Overview
Spectral Modeling
Physical Modeling
Summary
Physical Modeling
A F
Summary
Out
• Sampling Synthesis
k
J (β)
• Modern Example
0
Spectral Modeling 0
Physical Modeling 2
−0.5
Summary 4
0
5
6
10
15
20 8
25
10 Order k
30
Argument β
Overview
All examples by John Chowning unless otherwise noted:
Early Digital Synthesis
• Music V • FM brass synthesis
• KL Music
• “Daisy” ◦ Low Brass example
• Additive Analysis
• Additive Synthesis ◦ Dexter Morril’s FM Trumpet
• FM Synthesis
• FM Formula • FM singing voice (1978)
• FM Patch
• FM Spectra
Each formant synthesized using an FM operator pair
• FM Examples (two sinusoidal oscillators)
• FM Voice
• Sampling Synthesis ◦ Chorus
• Modern Example
Spectral Modeling
◦ Voices
Physical Modeling
◦ Basso Profundo
Summary • Other early FM synthesis
◦ Clicks and Drums
◦ Big Bell
◦ String Canon
Overview
FM voice synthesis can be viewed as compressed modeling of
Early Digital Synthesis
• Music V
spectral formants
• KL Music
• “Daisy”
• Additive Analysis
• Additive Synthesis
Carrier 1 Carrier 2
• FM Synthesis
• FM Formula Carrier 3
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Magnitude
• Sampling Synthesis
• Modern Example
Spectral Modeling
Physical Modeling
Summary 0 Frequency f
Modulation Frequency (all three)
Overview
• 1979 - Fairlight Computer Music Instrument - 8-bit
Early Digital Synthesis
• Music V ◦ First commercial sampler
• KL Music
• “Daisy” ◦ Eight voices, 8 bits, 64 KB (4 sec) RAM, 16 kHz (mono)
• Additive Analysis ◦ Editing, looping, mixing
• Additive Synthesis
• FM Synthesis ◦ One could draw waveforms and additive-synthesis amplitude
• FM Formula
envelopes (for each harmonic) with a light pen
• FM Patch
• FM Spectra ◦ $25,000–$36,000!
• FM Examples
• FM Voice • 1981 - E-mu Systems Emulator
• Sampling Synthesis
• Modern Example ◦ First “affordable” sampler ($10,000)
Spectral Modeling
◦ Eight voices, 128 K RAM, 8-bit, 80 lb.
Physical Modeling
Overview
Example:1
Early Digital Synthesis
• Music V • 40 Gigabytes on ten DVDs (three sampled pianos)
• KL Music
• “Daisy” • Every key sampled
• Additive Analysis
• Additive Synthesis
• 4–10 “velocity layers”
• FM Synthesis • Separate recordings with soft pedal down
• FM Formula
• FM Patch
• Separate “release” recordings, for multiple striking velocities
• FM Spectra
• FM Examples
• FM Voice
• Sampling Synthesis
• Modern Example
Spectral Modeling
Physical Modeling
Summary
1
Synthogy Ivory, $349 (Electronic Musician, October 2006)
Overview
Piano timbre is determined by
Early Digital Synthesis
• Music V • key number (1 byte)
• KL Music
• “Daisy”
• key velocity (2 bytes more than enough)
• Additive Analysis • pedal state (1 bit [or byte] per pedal)
• Additive Synthesis
• FM Synthesis
Piano control is relatively low-dimensional:
• FM Formula
• FM Patch
• FM Spectra
• Less than six bytes of information per note played
• FM Examples • No continuous controls (typically)
• FM Voice
• Sampling Synthesis • Ratio of total sampled data to one note of control data
• Modern Example ≈ one billion
Spectral Modeling
Physical Modeling
Summary
Overview
Control parameters:
Early Digital Synthesis
• Music V • Left-hand finger position(s)
• KL Music
• “Daisy” • Left-hand vibrato
• Additive Analysis • Bow velocity
• Additive Synthesis
• FM Synthesis • Bow force
• FM Formula
• Bow position
• FM Patch
• FM Spectra • Bow angle
• FM Examples
• FM Voice
• Shoulder damping
• Sampling Synthesis • Instrument orientation
• Modern Example
• Player motion (within a room)
Spectral Modeling
Physical Modeling
Summary
Overview
• Bowed-string control is infinite-dimensional in principle
Early Digital Synthesis
• Music V
• Many time-varying functions — “gestures”
• KL Music (we counted more than 10)
• “Daisy”
• Additive Analysis • Complete sampling of bowed strings on the level of pianos has
• Additive Synthesis apparently never been done
• FM Synthesis
• FM Formula • Rule-driven navigation of the most useful recorded playing
• FM Patch
regimes has worked well (e.g., Synful Orchestra)
• FM Spectra
• FM Examples • Model-based approaches greatly reduce data requirements:
• FM Voice
• Sampling Synthesis ◦ Spectral models (inspired by sound perception)
• Modern Example
◦ Physical models (model the sound source)
Spectral Modeling
Physical Modeling
Summary
magnitude, or
Overview magnitude and
phase extraction
Early Digital Synthesis
Spectral Modeling
A
• Vocoder
• Vocoder Examples
x0 (t) x̂0 (t)
• Sinusoidal Modeling f
Summary
Overview
ak (t)
Early Digital Synthesis
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX t
• S+N XSynth ∆ωk (t) = φ˙k (t)
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows 0
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future
t
Physical Modeling
Summary
Overview
• Original
Early Digital Synthesis
Spectral Modeling
• Vocoder
• 10 channels, sine carriers
• Vocoder Examples • 10 channels, narrowband-noise carriers
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• 26 channels, sine carriers
• S+N Examples
• S+N FX • 26 channels, narrowband-noise carriers
• S+N XSynth
• Sines + Transients
• 26 channels, narrowband-noise carriers, channels reversed
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map • Phase Vocoder: Identity system in absence of modifications
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band • The FFT Phase Vocoder next transitioned to the Short-Time
• S+N+T Examples
Fourier Transform (STFT) (Allen and Rabiner 1977)
• SM Summary
• Spec Future
Physical Modeling
Summary
Overview
Phases
Early Digital Synthesis
atan
Spectral Modeling
• Vocoder s(t)
• Vocoder Examples FFT Frequencies
• Sinusoidal Modeling
Quadratic
• Spectral Trajectories Peak
dB mag Peak
tracking Interpolation
• Sines + Noise
window w(n)
• S+N Examples
Amplitudes
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM • STFT peak tracking at CCRMA: mid-1980s (PARSHL program)
• S+N+T Freq Map • Motivated by vocoder analysis of piano tones
• S+N+T Windows
• HF Noise Modeling • Influences: STFT (Allen and Rabiner 1977),
• HF Noise Band
ADEC (1977), MAPLE (1979)
• S+N+T Examples
• SM Summary • Independently developed for speech coding by McAulay and
• Spec Future
Quatieri at Lincoln Labs
Physical Modeling
Summary
Overview
f
Early Digital Synthesis
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band t
• S+N+T Examples
• SM Summary
• Spec Future
Physical Modeling
Summary
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows white noise
• HF Noise Modeling filter(t)
• HF Noise Band
u(t)
• S+N+T Examples
ht (τ )
%
• SM Summary
• Spec Future
4
Physical Modeling
t
y(t) = Ai (t) cos 0 ωi (t)dt + φi (0) + (ht ∗ u)(t)
Summary i=1
% &' (
Overview
Xavier Serra 1989 thesis demos (Sines + Noise signal modeling)
Early Digital Synthesis
Spectral Modeling
• Piano
• Vocoder
• Vocoder Examples ◦ Original
• Sinusoidal Modeling
◦ Sinusoids alone
• Spectral Trajectories
• Sines + Noise ◦ Residual after sinusoids removed
• S+N Examples
• S+N FX
◦ Sines + noise model
• S+N XSynth
• Sines + Transients • Voice
• S + N + Transients
• S+N+T TSM ◦ Original
• S+N+T Freq Map
• S+N+T Windows
◦ Sinusoids
• HF Noise Modeling ◦ Residual
• HF Noise Band
• S+N+T Examples
◦ Synthesis
• SM Summary
• Spec Future
Physical Modeling
Summary
Overview
• Piano Effects
Early Digital Synthesis
Spectral Modeling
◦ Pitch downshift one octave
• Vocoder ◦ Pitch flattened
• Vocoder Examples
• Sinusoidal Modeling ◦ Varying partial stretching
• Spectral Trajectories
• Sines + Noise • Voice Effects
• S+N Examples
• S+N FX ◦ Frequency-scale by 0.6
• S+N XSynth
• Sines + Transients ◦ Frequency-scale by 0.4 and stretch partials
• S + N + Transients ◦ Variable time-scaling, deterministic to stochastic
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future
Physical Modeling
Summary
Overview
• Voice “modulator”
Early Digital Synthesis
Spectral Modeling
• Vocoder
• Creaking ship’s mast “carrier”
• Vocoder Examples • Voice-modulated creaking mast
• Sinusoidal Modeling
• Spectral Trajectories • Same with modified spectral envelopes
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future
Physical Modeling
Summary
Overview
In this technique, the sinusoidal sum is phase-matched at the
Early Digital Synthesis
cross-over point only (with no cross-fade).
Spectral Modeling
• Vocoder
• Vocoder Examples
• Marimba
• Sinusoidal Modeling
• Spectral Trajectories ◦ Original
• Sines + Noise ◦ Sinusoidal model
• S+N Examples
• S+N FX ◦ Original attack, followed by sinusoidal model
• S+N XSynth
• Sines + Transients • Piano
• S + N + Transients
• S+N+T TSM ◦ Original
• S+N+T Freq Map
• S+N+T Windows ◦ Sinusoidal model
• HF Noise Modeling ◦ Original attack, followed by sinusoidal model
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future
Physical Modeling
Summary
Overview
Why Model Transients Separately?
Early Digital Synthesis
Spectral Modeling
• Sinusoids efficiently model spectral peaks over time
• Vocoder • Filtered noise efficiently models spectral residual vs. t
• Vocoder Examples
• Sinusoidal Modeling • Neither is good for abrupt transients in the waveform
• Spectral Trajectories
• Phase-matched oscillators are expensive
• Sines + Noise
• S+N Examples • More efficient to switch to a transient model during transients
• S+N FX
• S+N XSynth
• Need sinusoidal phase matching at the switching times
• Sines + Transients
• S + N + Transients Transient models:
• S+N+T TSM
• S+N+T Freq Map • Original waveform slice (1988)
• S+N+T Windows
• HF Noise Modeling
• Wavelet expansion (Ali 1996)
• HF Noise Band • MPEG-2 AAC (with short window) (Levine 1998)
• S+N+T Examples
• SM Summary
• Frequency-domain LPC
• Spec Future (time-domain amplitude envelope) (Verma 2000)
Physical Modeling
Summary
Overview
transients
transients
noise noise noise
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise time-scaled sines + sines + sines +
• S+N Examples signal
transients
transients
16
Overview
14
Early Digital Synthesis
12
Spectral Modeling
• Vocoder 10
• Vocoder Examples
• Sinusoidal Modeling 8
• Spectral Trajectories 6
frequency [kHz]
• Sines + Noise
• S+N Examples 4
• S+N FX 2
• S+N XSynth
• Sines + Transients 0
0 50 100 150 200 250
• S + N + Transients 1
• S+N+T TSM
• S+N+T Freq Map 0.5
• S+N+T Windows
0
• HF Noise Modeling
amplitude
• HF Noise Band −0.5
• S+N+T Examples
−1
• SM Summary 0 50 100 150 200 250
• Spec Future time [milliseconds]
Overview
transient
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
low octave
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
amplitude
• Spec Future
Physical Modeling
0 50 100 150 200 250
Summary time [milliseconds]
16
Overview
14
Early Digital Synthesis
Spectral Modeling
12
• Vocoder 10
• Vocoder Examples
• Sinusoidal Modeling 8
• Spectral Trajectories
6
frequency [kHz]
• Sines + Noise
• S+N Examples 4
• S+N FX
2
• S+N XSynth
• Sines + Transients 0
0 4 50 100 150 200 250
• S + N + Transients x 10
• S+N+T TSM 2
• S+N+T Freq Map
1
• S+N+T Windows
0
• HF Noise Modeling
amplitude
• HF Noise Band −1
• S+N+T Examples −2
• SM Summary
0 50 100 150 200 250
• Spec Future time [milliseconds]
Physical Modeling
Summary
Overview 85
magnitude [dB]
• S+N+T TSM 50
• S+N+T Freq Map
• S+N+T Windows 45
• HF Noise Modeling
• HF Noise Band 40
• S+N+T Examples
• SM Summary 35
• Spec Future
Physical Modeling 30
0 500 1000 1500 2000 2500 3000 3500 4000 4500
Summary
frequency [Hz]
Overview 80
• S+N+T TSM 75
• S+N+T Freq Map 70
• S+N+T Windows
65
• HF Noise Modeling
• HF Noise Band 60
• S+N+T Examples 55
Overview
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)
Early Digital Synthesis
(http://ccrma.stanford.edu/~scottl/thesis.html)
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise Mozart’s Le Nozze di Figaro
• S+N Examples
• S+N FX • Original
• S+N XSynth
• Sines + Transients
• Compressed using MPEG-AAC at 32 kbps
• S + N + Transients • Compressed using sines+transients+noise at 32 kbps
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows • Multiresolution sinusoids alone
• HF Noise Modeling
• HF Noise Band • Residual Bark-band noise
• S+N+T Examples
• Transform-coded transients (AAC)
• SM Summary
• Spec Future • Bark-band noise above 5 kHz
Physical Modeling
Summary
Overview
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)
Early Digital Synthesis
(http://ccrma.stanford.edu/~scottl/thesis.html)
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
“It Takes Two” by Rob Base & DJ E-Z Rock
• S+N Examples
• S+N FX • Original
• S+N XSynth • MPEG-AAC at 32 kbps
• Sines + Transients
• S + N + Transients • Sines+transients+noise at 32 kbps
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows • Multiresolution sinusoids
• HF Noise Modeling
• HF Noise Band
• Residual Bark-band noise
• S+N+T Examples • Transform-coded transients (AAC)
• SM Summary
• Spec Future • Bark-band noise above 5 kHz
Physical Modeling
Summary
Overview
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)
Early Digital Synthesis
(http://ccrma.stanford.edu/~scottl/thesis.html)
Spectral Modeling
• Vocoder
• Vocoder Examples
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
Time-Scale Modification (pitch unchanged)
• S+N Examples
• S+N FX • S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5]
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM S+N+T Pitch Shifting (timing unchanged)
• S+N+T Freq Map
• S+N+T Windows • Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12]
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
• SM Summary
• Spec Future
Physical Modeling
Summary
Overview
Observations:
Early Digital Synthesis
Spectral Modeling
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Vocoder • Basic “auditory masking” discards ≈ 90% information
• Vocoder Examples
• Sinusoidal Modeling • Interesting neuroscience observation:
• Spectral Trajectories
• Sines + Noise “... most neurons in the primary auditory cortex A1
• S+N Examples
• S+N FX
are silent most of the time ...”
• S+N XSynth
• Sines + Transients
(from “Sparse Time-Frequency Representations”, Gardner and
• S + N + Transients Magnesco, PNAS:103(16), April 2006)
• S+N+T TSM
• S+N+T Freq Map • What is a true and correct “psychospectral model” for sound?
• S+N+T Windows
• HF Noise Modeling ◦ The cochlea of the ear is a real-time spectrum analyzer
• HF Noise Band
• S+N+T Examples
◦ How is the “ear’s spectrogram” represented at higher levels?
• SM Summary
• Spec Future
Physical Modeling
Summary
Overview
Summary
Julius Smith AES-2006 Heyser Lecture – 50 / 84
Digital Waveguide Models (1985)
∆
Overview Lossless digital waveguide = bidirectional delay line
Early Digital Synthesis
at some wave impedance R
Spectral Modeling
Physical Modeling
• KL Music
• Digital Waveguide
• Signal Scattering
z−N
• Plucked String
• Struck String
• Karplus Strong R
• EKS Algorithm
• Clarinet
• Wind Examples z−N
• Bowed Strings
• Distortion Guitar
• Acoustic Strings Useful for efficient models of
• Sound Examples
• Linearized Violin
• Commuted Piano
• strings
• Pulse Synthesis • bores
• Complete Piano
• Sound Examples
• plane waves
• Phy Audio Coding • conical waves
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 51 / 84
Signal Scattering
Overview
Signal scattering is caused by a change in wave impedance R:
Early Digital Synthesis
Spectral Modeling R − R1
k1 = 2
Physical Modeling R2 + R1
• KL Music
• Digital Waveguide
• Signal Scattering 1 + k1
• Plucked String
z−N z−N
• Struck String
• Karplus Strong
• EKS Algorithm k1 − k1
• Clarinet
R1 R2
• Wind Examples
• Bowed Strings
• Distortion Guitar z−N z−N
• Acoustic Strings 1 − k1
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
If the wave impedance changes every spatial sample, the
• Complete Piano Kelly-Lochbaum vocal-tract model results.
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 52 / 84
Ideal Plucked String (Displacement Waves)
Overview
Spectral Modeling + +
y (n) y (n-N/2)
Physical Modeling
• KL Music
• Digital Waveguide “Bridge” -1 (x = Pluck Position) -1 “Nut”
• Signal Scattering
• Plucked String
• Struck String
y-(n) y-(n+N/2)
• Karplus Strong
• EKS Algorithm
• Clarinet (x = 0) (x = L)
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples • Load each delay line with half of initial string displacement
• Linearized Violin
• Commuted Piano
• Sum of upper and lower delay lines = string displacement
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 53 / 84
Ideal Struck String (Velocity Waves)
Overview
Summary
Julius Smith AES-2006 Heyser Lecture – 54 / 84
Karplus-Strong (KS) Algorithm (1983)
Overview
Summary
Julius Smith AES-2006 Heyser Lecture – 55 / 84
Karplus-Strong Sound Examples
Overview
• “Vintage” 8-bit sound examples:
Early Digital Synthesis
Spectral Modeling
• Original Plucked String: (WAV) (MP3)
Physical Modeling • Drum: (WAV) (MP3)
• KL Music • Stretched Drum: (WAV) (MP3)
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 56 / 84
EKS Algorithm (Jaffe-Smith 1983)
Overview
Hp (z) Hβ (z) z −N HL (z)
Early Digital Synthesis
Spectral Modeling
Overview
• Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and
Early Digital Synthesis
others — distributed by CCRMA:
Spectral Modeling
Google search: STK ToolKit
Physical Modeling
• KL Music
• Digital Waveguide STK Plucked String: (WAV) (MP3)
• Signal Scattering
• Plucked String • Plucked String 1: (WAV) (MP3)
• Struck String
• Karplus Strong • Plucked String 2: (WAV) (MP3)
• EKS Algorithm
• Clarinet
• Plucked String 3: (WAV) (MP3)
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 58 / 84
EKS Sound Example (1988)
Overview
Bach A-Minor Concerto—Orchestra Part: (WAV) (MP3)
Early Digital Synthesis
Spectral Modeling
Summary
Julius Smith AES-2006 Heyser Lecture – 59 / 84
Digital Waveguide Single Reed, Cylindrical Bore Model (1986)
Mouth
Overview
Pressure p−b ( n )
Early Digital Synthesis pm ( n ) hm Output
Reed to Bell Delay
2 Filter
Spectral Modeling -
Reed Table
Physical Modeling ρ̂ Reflection
• KL Music Filter
• Digital Waveguide
*
h+Δ
• Signal Scattering
• Plucked String
- Bell to Reed Delay
• Struck String Embouchure -
p+b ( n )
• Karplus Strong Offset
• EKS Algorithm Reed Bore Bell
• Clarinet
• Wind Examples
• Bowed Strings
Digital waveguide clarinet
• Distortion Guitar
• Acoustic Strings • Control variable = mouth half-pressure
• Sound Examples
• Linearized Violin • Total reed cost = two subtractions, one multiply, and one table
• Commuted Piano
lookup per sample
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 60 / 84
Digital Waveguide Wind Instrument Sound Examples
Overview
• STK Clarinet: (WAV) (MP3)
Early Digital Synthesis
Google search: STK clarinet
Spectral Modeling
Physical Modeling ◦ Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and
• KL Music
others — distributed by CCRMA:
• Digital Waveguide
• Signal Scattering Google search: STK ToolKit
• Plucked String
• Struck String • Staccato Systems Slide Flute
• Karplus Strong
• EKS Algorithm (based on STK flute, ca. 1995): (WAV) (MP3)
• Clarinet • Yamaha VL1 “Virtual Lead” synthesizer demos (1994):
• Wind Examples
• Bowed Strings
• Shakuhachi: (WAV) (MP3)
• Distortion Guitar
• Acoustic Strings • Oboe and Bassoon: (WAV) (MP3)
• Sound Examples
• Linearized Violin
• Tenor Saxophone: (WAV) (MP3)
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 61 / 84
Digital Waveguide Bowed Strings (1986)
Bow Force
Overview Bow Velocity
v+s, l v−s, r
Early Digital Synthesis Body
Nut to Bow Delay Bow to Bridge Delay Filter
Spectral Modeling
vb - v+Δ ρ̂ Reflection
Physical Modeling -1
* Filter
• KL Music - Bow Table
• Digital Waveguide
v+s, r
• Signal Scattering Bow to Nut Delay Bridge to Bow Delay
• Plucked String v−s, l
• Struck String Bridge-
Nut String Bow String Air
• Karplus Strong Body
• EKS Algorithm
• Clarinet
• Wind Examples • Reflection filter summarizes all losses per period
• Bowed Strings
(due to bridge, bow, finger, etc.)
• Distortion Guitar
• Acoustic Strings • Bow-string junction = memoryless lookup table
• Sound Examples
• Linearized Violin
(or segmented polynomial)
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 62 / 84
“Electric Cello” Sound Examples (Peder Larson)
Overview
• Staccato Notes: (WAV) (MP3)
Early Digital Synthesis
(short strokes of high bow pressure, as from a bouncing bow)
Spectral Modeling
• Bach’s First Suite for Unaccompanied Cello: (WAV) (MP3)
Physical Modeling
• KL Music
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 63 / 84
Soft Clipper
Overview
− , x ≤ −1
Early Digital Synthesis
2
Physical Modeling
3
2
• KL Music x≥1
3,
• Digital Waveguide
• Signal Scattering
• Plucked String 3
x=−1:0.01:1; plot([−(2/3)*ones(1,100), x−x. /3, (2/3)*ones(1,100)])
0.8
• Struck String
• Karplus Strong
0.6
• EKS Algorithm
• Clarinet 0.4
• Wind Examples
0.2
• Bowed Strings
• Distortion Guitar
0
f(x(n))
• Acoustic Strings
• Sound Examples −0.2
• Linearized Violin
−0.4
• Commuted Piano
• Pulse Synthesis −0.6
• Complete Piano
• Sound Examples −0.8
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2
x(n)
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 64 / 84
Amplifier Distortion + Amplifier Feedback
Overview
Sullivan 1990
Early Digital Synthesis
Spectral Modeling
Physical Modeling
• KL Music Pre-distortion output level
• Digital Waveguide
String 1
• Signal Scattering
Nonlinear Distortion
• Plucked String
• Struck String
..
. Output Signal
• Karplus Strong
• EKS Algorithm Pre-distortion gain Distortion output level
• Clarinet String N Amplifier
• Wind Examples Feedback
• Bowed Strings Gain
• Distortion Guitar
• Acoustic Strings
Amplifier Feedback Delay
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
Distortion output signal often further filtered by an amplifier cabinet
• Phy Audio Coding filter, representing speaker cabinet, driver responses, etc.
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 65 / 84
Distortion Guitar Sound Examples
Overview
(Stanford Sondius Project, ca. 1995)
Early Digital Synthesis
Spectral Modeling
• Distortion Guitar: (WAV) (MP3)
Physical Modeling • Amplifier Feedback 1: (WAV) (MP3)
• KL Music
• Amplifier Feedback 2: (WAV) (MP3)
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 66 / 84
Commuted Synthesis of Acoustic Strings (1993)
Overview
e(t) s(t) y(t)
Early Digital Synthesis Trigger Excitation String Resonator Output
Spectral Modeling
Physical Modeling
• KL Music Schematic diagram of a stringed musical instrument.
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
Trigger Excitation Resonator String Output
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
Equivalent diagram in the linear, time-invariant case.
• Distortion Guitar
• Acoustic Strings
• Sound Examples a(t) x(t)
Aggregate
• Linearized Violin Trigger String Output
Excitation
• Commuted Piano
• Pulse Synthesis
• Complete Piano Use of an aggregate excitation given by the convolution of original
• Sound Examples
excitation with the resonator impulse response.
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 67 / 84
Commuted Components
Overview
Summary
Julius Smith AES-2006 Heyser Lecture – 68 / 84
Sound Examples
Overview
Electric Guitar (Pick-Ups and/or Body-Model Added) (Stanford
Early Digital Synthesis
Sondius Project → Staccato Systems, Inc. → ADI, ca. 1995)
Spectral Modeling
Summary
Julius Smith AES-2006 Heyser Lecture – 69 / 84
Sound Examples
Overview
More Recent Acoustic Guitar
Early Digital Synthesis
Spectral Modeling
• Bach Prelude in E Major: (WAV) (MP3)
Physical Modeling • soundexamplewavBach silenceLoure in E Major: (WAV) (MP3)
• KL Music
• Digital Waveguide
• Signal Scattering Virtual performance by Dr. Mikael Laurson, Sibelius Institute
• Plucked String
• Struck String
• Karplus Strong Virtual guitar by Helsinki Univ. of Tech., Acoustics Lab3
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding 3
• Phy Audio Coding? http://www.acoustics.hut.fi/
Summary
Julius Smith AES-2006 Heyser Lecture – 70 / 84
Commuted Synthesis of Linearized Violin
a) Output
Overview e(n) s(n) x(n)
Amplitude(n) Impulse
Frequency(n) String Resonator
Early Digital Synthesis Train
Spectral Modeling
b) Output
Physical Modeling e(n)
Amplitude(n) Impulse a(n) x(n)
• KL Music Resonator String
Frequency(n) Train
• Digital Waveguide
• Signal Scattering
• Plucked String c) Output
a(n) x(n)
• Struck String Amplitude(n) Impulse-Response
• Karplus Strong Frequency(n) Train String
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Assumes ideal Helmholtz motion of string
• Distortion Guitar • Sound Examples (Stanford Sondius project, ca. 1995):
• Acoustic Strings ◦ Bass: (WAV) (MP3) ◦ Violin 1: (WAV) (MP3)
• Sound Examples
• Linearized Violin ◦ Cello: (WAV) (MP3) ◦ Violin 2: (WAV) (MP3)
• Commuted Piano
• Pulse Synthesis
◦ Viola 1: (WAV) (MP3) ◦ Ensemble: (WAV) (MP3)
• Complete Piano ◦ Viola 2: (WAV) (MP3)
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 71 / 84
Commuted Piano Synthesis (1995)
Overview
Hammer-string interaction pulses (force):
Early Digital Synthesis
Summary
Julius Smith AES-2006 Heyser Lecture – 72 / 84
Synthesis of Hammer-String Interaction Pulse
Overview
Impulse Impulse Response
Early Digital Synthesis
Spectral Modeling
Lowpass
Physical Modeling Filter
• KL Music
• Digital Waveguide Time Time
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong • Faster collisions correspond to narrower pulses
• EKS Algorithm
• Clarinet (nonlinear filter )
• Wind Examples
• Bowed Strings
• Distortion Guitar • For a given velocity, filter is linear time-invariant
• Acoustic Strings
• Sound Examples
• Linearized Violin • Piano is “linearized” for each hammer velocity
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 73 / 84
Multiple Hammer-String Interaction Pulses
Overview
Superimpose several individual pulses:
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Force
• KL Music
• Digital Waveguide Impulse 1 LPF1
• Signal Scattering
δ1
String
• Plucked String
Impulse 2 LPF2
δ2 + Input
• Struck String
Impulse 3 LPF3
• Karplus Strong δ3
• EKS Algorithm 0 Time
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 74 / 84
Multiple Hammer-String Interaction Pulses
Overview
Superimpose several individual pulses:
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Force
• KL Music
• Digital Waveguide Impulse 1 LPF1
• Signal Scattering
δ1
String
• Plucked String
Impulse 2 LPF2
δ2 + Input
• Struck String
Impulse 3 LPF3
• Karplus Strong δ3
• EKS Algorithm 0 Time
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar As impulse amplitude grows (faster hammer strike), output pulses
• Acoustic Strings
• Sound Examples become taller and thinner, showing less overlap.
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 74 / 84
Complete Piano Model
Natural Ordering:
Overview
Overview
(Stanford Sondius Project, ca. 1995)
Early Digital Synthesis
Summary
Julius Smith AES-2006 Heyser Lecture – 76 / 84
More Recent Harpsichord Example
Overview
• Harpsichord Soundboard Hammer-Response: (WAV) (MP3)
Early Digital Synthesis
• Musical Commuted Harpsichord Example: (WAV) (MP3)
Spectral Modeling
Summary
Julius Smith AES-2006 Heyser Lecture – 77 / 84
Physical Modeling in Audio Coding
Summary
Julius Smith AES-2006 Heyser Lecture – 78 / 84
Best Known Model-Based Audio Coders
Overview
A “cover band” can put together a very convincing facsimile of
Early Digital Synthesis
popular music performance
Spectral Modeling
Physical Modeling
• KL Music
• Digital Waveguide
• Signal Scattering
• Plucked String
• Struck String
• Karplus Strong
• EKS Algorithm
• Clarinet
• Wind Examples
• Bowed Strings
• Distortion Guitar
• Acoustic Strings
• Sound Examples
• Linearized Violin
• Commuted Piano
• Pulse Synthesis
• Complete Piano
• Sound Examples
• Phy Audio Coding
• Phy Audio Coding?
JOS high-school band “Bittersweet”
Summary
Julius Smith AES-2006 Heyser Lecture – 79 / 84
Future Physical Modeling in Audio Coding?
Overview
A “Cover Band” Approach to Model-Based Audio Coding:
Early Digital Synthesis
Spectral Modeling
1. Recognize individual “audio streams” in a mix (CASA)
Physical Modeling (“I hear a trap set, electric bass, Fender Rhodes, and a strat”)
• KL Music
2. For each stream, calibrate its model heuristically
• Digital Waveguide
• Signal Scattering (“Here is what I hear the bass part doing: ...”)
• Plucked String
• Struck String
3. Fine-tune the synthetic mix to the real mix
• Karplus Strong (joint “maximum likelihood estimation”)
• EKS Algorithm
• Clarinet Features of “Cover-Band Coding” (CBC):
• Wind Examples
• Bowed Strings
• The “playing experience” of each “virtual performer” prevents
• Distortion Guitar
• Acoustic Strings artifacts — “musically unreasonable” parameters are made
• Sound Examples
• Linearized Violin
unlikely (“Bayesian priors”)
• Commuted Piano • An incorrect instrument must “imitate” its assigned stream
• Pulse Synthesis
• Complete Piano • New arrangements can be synthesized by deliberately choosing
• Sound Examples a new ensemble!
• Phy Audio Coding
• Phy Audio Coding?
Summary
Julius Smith AES-2006 Heyser Lecture – 80 / 84
Overview
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Summary
Summary
Overview
We have reviewed a “CCRMA-centric slice” of the history of digital
Early Digital Synthesis
sound synthesis (usually starting with results from Bell Labs):
Spectral Modeling
Sound Acknowledgment
Overview
Thanks to Emu / Creative Labs for providing a superb-quality
Early Digital Synthesis
external D/A converter for this talk (an E-Mu 0404/USB)
Spectral Modeling
Physical Modeling
Summary
Sound Acknowledgment