01 Motivation

Augmented Reality in Medicine
Hauptseminar SS04
Tobias Sielhorst & Joerg Traub
Computer Aided Medical Procedures

Prof. N. Navab
Hauptseminar: Augmented Reality in Medicine April 22, 2004

Joerg Traub 1/55
What could physicians do without images

Joerg Traub 2/55
What could physicians do without images
¾ Invasive – Open & See

Joerg Traub 3/55
Advanced Interventional Suites
Courtesy of BrainLAB
Joerg Traub 4/55
Advanced Interventional Suites
Telesurgery

Joerg Traub 5/55
Interventional Suites of the Future

Joerg Traub 6/55
Motivation
System
Tracking
3D
Ima
Vis
ging
u
aliz
atio
n
nce
it en
t A s s is t a
Pa
u r geon
S
se
Nur men
ts
tru
Ins

Joerg Traub 7/55
Motivation
¾ in-situ visualization
¾ augmentation

Joerg Traub 8/55
Outline
¾ What is Augmented Reality

Definition
Displays
Tracking
User Interfaces
¾ Application Domain
¾ Medical Imaging Devices
¾ Augmented Reality in Medicine
Exemplary Setup
Prototypical Applications

Joerg Traub 9/55
Outline

Definition
Displays
Tracking
User Interfaces
Exemplary Setup

Joerg Traub 10/55
Definition Augmented Reality
¾ [Azuma98]:
Overlays real and virtual
Interactive in real time
Registered in 3D

Joerg Traub 11/55
Definition Augmented Reality
¾ [Milgram94]: Reality Virtuality Continuum
Mixed Reality (MR)
Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Joerg Traub 12/55
Overlay of Real and Virtual

Joerg Traub 13/55
Centricity
Exocentric Egocentric
Augmented Reality is Mostly Supported by Egocentric Views

Joerg Traub 14/55
Multiple Discipline of AR
Courtesy of Azuma, UNC
Courtesy of GRASP Lab, UPenn
Courtesy of Andrei State, UNC

Visualization/
Perception
Tracking/Sensor
Augmented Reality
Hitachi SR8000-F1 at LRZ Muenchen
Courtesy of Andrei State, UNC
Data Models/
Scene Graphs
Computing Unit
Human Interaction
Joerg Traub 15/55
Merging Real and Virtual
¾ Visualization of Augmented Frames -> Display Systems

¾ Tracking of Objects and User’s Eyes -> Tracking Systems
¾ Calibration of Camera and Registration of Virtual Models
¾ User Interfaces for Interaction

Joerg Traub 16/55
Outline

Definition
Displays
Tracking
User Interfaces
Exemplary Setup

Joerg Traub 17/55
Displays
NVISOR SX
¾ Head-mounted
Displays Microvision, Nomad
Optical see-trough
Video see-through
Projector mounted
¾ Projector based
4D-Vision, X3D Technologies GmbH
¾ Display Panels CourtesyCourtesy

of Karitsuka
of Greg
and Welch,
Sato Osaka
UNC University
¾ Monitor based Courtesy of Martin Bauer, TUM

Courtesy of Martin Bauer, TUM
MedARPa, ZGDV Darmstadt

Joerg Traub 18/55
Outline

Definition
Displays
Tracking
User Interfaces
Exemplary Setup

Joerg Traub 19/55
Tracking Devices
¾Acoustic
¾Mechanical
¾Optical
¾Electromagnetical
¾Inertial Polaris by Northern Digital Inc.
daVinci by Intuitive Surgical Inc.

Courtesy of HyperPhysics, Georgia State University
Aurora by Northern Digital Inc.

Joerg Traub 20/55
Outline

Definition
Displays
Tracking
User Interfaces
Exemplary Setup

Joerg Traub 21/55
User Interfaces
¾ Vision Based
Interaction
Gesture Recognition
Virtual Touchpad
¾ Acoustic
Speech Recognition
¾ Tangible / Haptics
Spaceball 5000; Courtesy of 3D Connexion
3D Space Mouse
Flystick Tracked Flystick; Courtesy of A.R.T. Herrsching

Joerg Traub 22/55
Outline

Definition
Displays
Tracking
User Interfaces
Exemplary Setup

Joerg Traub 23/55
Application Domain
¾Industrial / Mechanical
¾Education / Edutainment
¾Architecture
¾Games
¾Medicine
University of South Australia
ARVIKA Project sponsored by BMBF

3D Construct Application, Studierstube, Wien
ARQuake, Wearable Lab, UNISA, Australia

Joerg Traub 24/55
Historical Augmented Reality Prototypes
KARMA:
Knowledge-based Augmented Reality for Maintenance Assistance
Steven Feiner, Blair MacIntyre, Dorée Seligmann (Columbia University, New York)

Joerg Traub 25/55
Outline

Definition
Displays
Tracking
User Interfaces
Exemplary Setup

Joerg Traub 26/55
Medical Imaging Devices
Acquire Data for Augmentation
¾ X-Ray
¾ Angiography
¾ DSA
¾ CT
¾ MRI (NMRI)
¾ Ultrasound (US)
¾ PET
¾ Intra-operative Imaging
Joerg Traub 27/55
X-Ray
¾ 1895 by Conrad Roentgen
¾ 3 month later 1st clinical trail by Cox (Montreal)
Wilhelm Conrad Roentgen was born on
March 27, 1845, in Lennep, Prussia.
Educated in the Netherlands and
Switzerland, Roentgen obtained his
doctoral degree in physics at the
University of Zürich in 1869. He
conducted research and taught at the
universities of Strasbourg, Giessen,
Würzburg, and Munich.
He received the Nobel prize for physics

in 1901. His achievement heralded the
age of modern physics and
transformed medical practice.

Joerg Traub 28/55
X-Ray

Joerg Traub 29/55
X-Ray Angiography
¾ Contrast Agents in Blood Vessels
¾ Highlights Vessels against other tissues
Liquid contrast was introduced into the stomach as
the bismuth meal and later using the less toxic
barium sulphate. The opaque meal to diagnose
ulcers and cancers of the stomach and duodenum
was developed in Vienna in 1904 by Reider and
popularised in the UK by workers such as AE
Barclay and Sebastian Gilbert Scott. Rectally
administered the colon could be filled with bismuth
or barium to assist in the diagnosis of large bowel
disease such as cancer or diverticulosis.

Joerg Traub 30/55
DSA
¾ Digitally Subtracted Angiography
¾ 1963 by Ziedses des Plantes
¾ Ability to see only vessels

Joerg Traub 31/55
Computed Tomography (CT)
CT was invented in 1972 by British

engineer Godfrey Hounsfield of EMI
Laboratories in England. Hounsfield
was later awarded the Nobel Peace
Prize and honoured with a Knighthood
for his contributions to medicine and
science. CT combined x-ray images
with a computer. If you took many x-
rays of the same area, at different
angles, a computer could put the
information from the x-rays together to
create a cross-sectional image. In CT
scanning the information from a X-ray
tube rotating around the patient is
processed to produce a cross section of
the anatomy.

Joerg Traub 32/55
Computed Tomography (CT)
CT RECONSTRUCTION
Godfrey Hounsfield made a prototype in
1971 and the following year tried it out on
a patient. The CT scanners for clinical use
were first installed in 1975. The original
systems were dedicated to head scanning
but whole body scanners with larger
patient openings became available in
1976. The first CT scanner developed by
Hounsfield in his lab at EMI took several
hours to acquire the raw data for a single
scan (slice) and took days to reconstruct a
single image from this raw data.
Joerg Traub 33/55
Magnetic Resonance Imaging (MRI)
The first clinical use of MRI took place in

Nottingham University Hospital in 1967.
The images then were of poor quality and
could not be used for clinical medicine.
Paul Lauterbur from Stony Brook applied

magnetic field gradients in all three
dimensions and computerized axial
tomography (CAT)-scan back-projection (=
projection-reconstruction) to create NMR
images.
Therefore, he received the Nobel Prize in

Physiology or Medicine in 2003.

Joerg Traub 34/55
Magnetic Resonance Imaging (MRI)
Today MRI is the imaging modality of choice for most parts of the body. The images of
the spine, musculoskeletal system, neck and mediastinal structures are of excellent
quality.

Joerg Traub 35/55
Ultrasound
Ultrasonic underwater detection systems were

developed after the Titanic sank in 1912 and for
the purpose of underwater navigation by
submarines in WW1. Between 1914 and 1918
SONAR was in great demand for the detection of
German submarines at sea.
The medical use of Ultrasound started in

Glasgow. Professor Ian Donald M.D. and his
colleagues, working at the University of Glasgow’s
Department of Midwifery were the first to apply
ultrasound as a diagnostic modality in the fields of
obstetrics and gynaecology.

Joerg Traub 36/55
Ultrasound

Joerg Traub 37/55
Doppler Ultrasound
¾ change in the frequency of the wave resulting
from the motion of the reflector
¾ detect and measure blood flow
¾ reflector is the red blood cell
2 f ⋅ v cos(α )
Df =
c
Df ⋅ c
v=
2 f cos(α ) Courtesy of Raymond Hung

Joerg Traub 38/55
PET
¾ drugs injected with p+ isotopes
¾ isotope decays, emitting p+
¾ p+ annihilates with e- from tissue
¾ forming back-to-back 511 keV photon pair.

Joerg Traub 39/55
Intra-operative Imaging
¾ Cone Beam Reconstruction
(SIEMENS SIREMOBIL Iso-C3D mobile C-arm )
true isocentric

Joerg Traub 40/55
Intra-operative Imaging
¾ MR (GE Signa SP/i 0.5T)

Joerg Traub 41/55
Outline

Definition
Displays
Tracking
User Interfaces
Exemplary Setup

Joerg Traub 42/55
¾In-Situ Visualization
¾Transparent Patient
¾Detect features an eye can not see
¾Error reducing – Higher Accuracy/Precision
¾Time optimization – Guided Surgery
¾Diagnoses during Intervention
Joerg Traub 43/55
¾In-Situ Visualization during

Interventions
Training and Simulation

Joerg Traub 44/55
¾ Detect features an eye can not see

using imaging data
Anatomical structure
Functional structure
¾ Visualize patient’s vitality functions
¾ Visualize environment
Joerg Traub 45/55
Outline

Definition
Displays
Tracking
User Interfaces
Exemplary Setup

Joerg Traub 46/55
Medivision – C-Arm for Trauma Surgery
Navigated Surgery
Courtesy of Medivision

Joerg Traub 47/55
Varioscope AR
A Head-Mounted Operating Microscope for Augmented Reality
Courtesy of Wolfgang Birkfellner & Liveoptics, Vienna

Joerg Traub 48/55
Telemedicine:
Augmenting the View of the Stereoscopic Endoscope
of the DaVinci Telemanipulator
Courtesy of Fabien Mourgues, Thierry Vieville, Volkmar Falk, and Eve Coste-Maniere
INRIA Sophia-Antipolis& Herzzentrum Leipzig
Joerg Traub 49/55
RAMP – Realtime Applications for Medical Procedures
Siemens Corporate Research, Princeton, USA
Courtesy of Ali Khamene, Frank Sauer & Sebastian Vogt

Joerg Traub 50/55
RAMP – Realtime Applications for Medical Procedures
Siemens Corporate Research, Princeton, USA
Courtesy of Ali Khamene, Frank Sauer & Sebastian Vogt

Joerg Traub 51/55
This Seminar
¾ Deliverables
¾ Course Schedule

Joerg Traub 52/55
Deliverables
¾ Presentation
¾ Moderate Discussion
¾ Slides (softcopy)
¾ Handout (hardcopy & softcopy)
¾ Attendance in weekley classes
(Thursday 4-6pm / MI 02.13.010)

Joerg Traub 53/55
Course Schedule
Introduction: Motivation and Augmented Reality

April 22, 2004 Joerg Traub
Overview
April 29, 2004 Introduction: The RAMP system and Live Demo Tobias Sielhorst
May 06, 2004 HMDs and other Display Devices Frederik Bender
May 13, 2004 HMPD Human Mounted Projection Devices ---
May 27, 2004 Human vision and perception Jonas von Beck
June 03, 2004 Electromagnetical Tracking Christian Waechter
June 17, 2004 Optical Tracking / Hybrid Tracking / Tracker Fusion Christoph Domann
June 24, 2004 Patient Registration Troels Frimor
July 01, 2004 Use Case Study: Neurosurgery Surgery Bernhard Findeiss
July 08, 2004 Registration of different imaging data ---
DA Status Report: Occlusion Heandling in Augmented
July 15, 2004 Hauke Heibel
Reality
July 22, 2004 VarioAR - An AR Operating Microscope Henning Herbers

Joerg Traub 54/55
Next Week: Live Demo of RAMP

Joerg Traub 55/55

01 Motivation

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

01 Motivation

Uploaded by

Copyright:

Available Formats

Augmented Reality in Medicine

Computer Aided Medical Procedures

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ Invasive – Open & See

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ What is Augmented Reality

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ What is Augmented Reality

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ [Milgram94]: Reality Virtuality Continuum

Mixed Reality (MR)

Real Augmented Augmented Virtual

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Augmented Reality is Mostly Supported by Egocentric Views

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Courtesy of Andrei State, UNC

¾ Visualization of Augmented Frames -> Display Systems

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ What is Augmented Reality

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ Display Panels CourtesyCourtesy

¾ Monitor based Courtesy of Martin Bauer, TUM

MedARPa, ZGDV Darmstadt

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ What is Augmented Reality

Hauptseminar: Augmented Reality in Medicine April 22, 2004

daVinci by Intuitive Surgical Inc.

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ What is Augmented Reality

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ What is Augmented Reality

Hauptseminar: Augmented Reality in Medicine April 22, 2004

University of South Australia

ARVIKA Project sponsored by BMBF

ARQuake, Wearable Lab, UNISA, Australia

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

¾ What is Augmented Reality

Hauptseminar: Augmented Reality in Medicine April 22, 2004

He received the Nobel prize for physics

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

CT was invented in 1972 by British

Hauptseminar: Augmented Reality in Medicine April 22, 2004

The first clinical use of MRI took place in

Paul Lauterbur from Stony Brook applied

Therefore, he received the Nobel Prize in

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Ultrasonic underwater detection systems were

The medical use of Ultrasound started in

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004

Hauptseminar: Augmented Reality in Medicine April 22, 2004