Hauptseminar: Augmented Reality in Medicine

by Tobias Sielhorst, Joerg Traub & Prof. N. Navab

Introduction

This seminar focus on the techniques of augmented reality (AR) and their clinical applicability. Augmented Reality is a procedure in which the real world view is superimposed by virtual object. In medical context the goal of such applications is in computer aided surgery, which provides the surgeons image overlays for guidance. Prototypic applications are already available.

In the seminar we like to tackle techniques of computer vision, augmented reality, tracking devices, display devices and their use in the medical domain. From left to right:

1. Surgeon's view: Real world view augmented by segmented imaging data for neurosurgery;
2. Visionary scenario of Augmented Reality with ultrasound;
3. Slices of Computed Tomography (CT) as a source of imaging data

Registration

• Email to Joerg Traub including a listing of two desirable topics.
• Preliminary discussion on Thursday, 29 Jan at 16.00 (MI 02.12.51)

Time and Location

• Weekly meeting - Thursday 4:00PM-6:00PM, room 02.13.010

Schedule

This is the first draw of topics for the students.

Date	Topic	by	Supervision
April 22, 2004	Introduction: Application Domain, Motivation and Augmented Reality Overview (.pdf 8.3mb)	Joerg Traub
April 29, 2004	Introduction: The RAMP system and Live Demo	Tobias Sielhorst
May 06, 2004	no class
May 13, 2004	HMDs and other Display Devices (Slides, Handout)	Frederik Bender	Tobias Sielhorst
May 27, 2004	Human vision and perception (only Slides)	Jonas von Beck	Tobias Sielhorst
June 03, 2004	no class
June 17, 2004	Electromagnetical Tracking ( Slides, Handout )	Christian Waechter	Joerg Traub
June 17, 2004	Optical Tracking / Hybrid Tracking / Tracker Fusion ( Slides, Handout )	Christoph Domann	Joerg Traub
June 24, 2004	Patient Registration ( Slides, Handout )	Troels Frimor	Joerg Traub
July 01, 2004	Use Case Study: Neurosurgery Surgery (Slides,Handout)	Bernhard Findeiss	Tobias Sielhorst
July 08, 2004	no class
July 15, 2004	Registration of different imaging data (Slides,Handout)	Boris Saydakov	Joerg Traub
July 22, 2004	VarioAR - An AR Operating Microscope (Slides,Handout)	Henning Herbers	Joerg Traub

Literature

HMDs and other Display Devices (Tobias Sielhorst)

• Ronald T. Azuma: Recent Advances in Augmented Reality, IEEE Computer Graphics and Applications,November/December 2001 (http://www.cs.unc.edu/~azuma/cga2001.pdf)
• Keller and Colucci, Perception in HMDs: what is it in HMDs that really make them all so terrible, SPIE Proceedings on "Helmet and HMDs", 1998, gives a good overview on perception issues associated with HMDs (http://www.cs.unc.edu/~keller/Papers/Aerosense98-95/HMD.pdf)
• Jannick P. Rolland, Henry Fuchs Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization in Presence, Vol. 9, No. 3, June 2000, 287–309
• Mike Blackwell, Constantinos Nikou, Anthony M. Di Gioia and Takeo Kanade An Image Overlay System for Medical Data Visualization in MICCAI'98, LNCS 1496, pp. 232-240, 1998.

HMPD - Head Mounted Projective Display(Tobias Sielhorst)

• Hong Hua, Chunyu Gao, Leonard D. Brown, Narendra Ahuja, Jannick P. Rolland A Testbed for Precise Registration, Natural Occlusion and Interaction in an Augmented Environment Using a Head-Mounted Projective Display (HMPD) in IEEE VR02
• Jannick P. Rolland, Felix G. Hamza-Lup Artificial Reality Center (ARC) Robotics and Vision Workshop 03
• Hong Hua Development of Head-Mounted Projective Display and Its Application for Collaborative Environments Robotics and Vision Workshop 03

Human vision and perception (Tobias Sielhorst)

• Webtutorial by John H. Krantz (http://psych.hanover.edu/Krantz/art/cues.html)
• Webtutorial by Peter K. Kaiser (http://www.yorku.ca/eye/thejoy.htm)
• Webtutorial by Shaun Lawson (http://www.dcs.napier.ac.uk/~shaun/ve/lectures/Week04.pdf)

Electromagnetical Tracking (Joerg Traub)

• Gary Bishop, Greg Welch, and B. Danette Allen: Tracking: Beyond 15 Minutes of Thought SIGGRAPH 2001 Course (http://www.cs.unc.edu/~tracker/ref/s2001/tracker/)
• Burdea, G., & Coiffet, P. (1994). Virtual Reality Technology (First ed.): John Wiley & Sons, Inc.
• Raab, F. H., Blood, E. B., Steiner, T. O., & Jones, H. R. (1979). Magnetic Position and Orientation Tracking System. IEEE Transactions on Aerospace and Electronic Systems, AES-15, 709-718

Optical Tracking / Hybrid Tracking / Tracker Fusion (Joerg Traub)

• selected chapters of Gary Bishop, Greg Welch, and B. Danette Allen: Tracking: Beyond 15 Minutes of Thought SIGGRAPH 2001 Course (http://www.cs.unc.edu/~tracker/ref/s2001/tracker/)

• Azuma, R. T., Hoff, B. R., Neely, H. E., III, Sarfaty, R., Daily, M. J., Bishop, G., Chi, V., Welch, G., Neumann, U., You, S., Nichols, R., & Cannon, J. (1998). Making Augmented Reality Work Outdoors Requires Hybrid Tracking, First International Workshop on Augmented Reality (pp. 219-224). San Francisco, CA, USA.
• Behringer, R. (1999). Registration for Outdoor Augmented Reality Applications Using Computer Vision Techniques and Hybrid Sensors , IEEE Virtual Reality (pp. 244-251). Houston, TX, USA.
• You, S., Neumann, U., & Azuma, R. T. (1999b). Hybrid Inertial and Vision Tracking for Augmented Reality Registration, IEEE Virtual Reality (pp. 260-267). Houston, TX USA.
• Andrei State, Gentaro Hirota, David T. Chen, William F. Garrett, Mark A. Livingston Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking (Browse through http://www.cs.unc.edu/~us/hybrid.html)

Registration of the patient (Joerg Traub)

• Joseph V. Hajnal, Derek L.G. Hill and David J. Hawkes: Medical Image Registration
• J.B.A. Maintz and M.A. Viergever: An Overview of Medical Image Registration Methods

Use Case Study: Neurosurgery (Tobias Sielhorst)

• John R. Mangiardi, MD. Clinical Professor New York University (http://www.brain-surgery.com/)
• A. P. King, P. J. Edwards, C. R. Maurer Jr., D. A. de Cunha, R. P. Gaston, M. Clarkson, D. L. G. Hill, D. J. Hawkes, M. R. Fenlon, M. J. Gleeson: Stereo Augmented Reality in the Surgical Microscope

Registration of different imaging data (Joerg Traub)

• Joseph V. Hajnal, Derek L.G. Hill and David J. Hawkes: Medical Image Registration
• J.B.A. Maintz and M.A. Viergever: An Overview of Medical Image Registration Methods

Augmented medical imaging devices (Tobias Sielhorst)

• George D. Stetten, Aaron Cois, Wilson Chang, Damion Shelton, Robert J. Tamburo, John Castellucci, Olaf von Ramm: C-Mode Real Time Tomographic Reflection for a Matrix Array Ultrasound Sonic Flashlight MICCAI 03
• George D. Stetten, Vikram Chib: Magnified Real-Time Tomographic Reflection MICCAI 01
• N. Navab and A. Bani-Hashemi Merging Visible and Invisible: Two Camera-Augmented Mobile C-arm (CAMC) Applications

VarioAR - An AR Operating Microscope (Joerg Traub)

• M. FIGL, W. BIRKFELLNER, C. EDE, J. HUMMEL, R. HANEL, F. WATZINGER, F. WANSCHITZ, R. EWERS, and H. BERGMANN, The Control Unit for a Head Mounted Operating Microscope Used for Augmented Reality Visualization in Computer Aided Sugery , in International Symposium on Mixed and Augmented Reality (ISMAR), September 2002, pp. 69–75.
• M. FIGL, W. BIRKFELLNER, J. HUMMEL, R. HANEL, P. HOMOLKA, F. WATZINGER, F. WANSCHITZ, R. EWERS, and H. BERGMANN, Current Status of the Varioscope AR, a Head-Mounted Operating Microscope for Computer-Aided Surgery, in International Symposium on Augmented Reality (ISAR), October 2001, pp. 20–29.

EMEA-, FDA- and BfArM- approval (Tobias Sielhorst)

• Bundesinstitut für Arzneimittel und Medizinprodukte (http://www.bfarm.de)
• The European Agency for the Evaluation of Medicinal Products (http://www.emea.eu.int)
• U.S. Food&Drug Administration (http://www.fda.gov/)

Schein Requirements

• active and motivated participation in all classes
• conduct one class (presentation about 45min and discussion at least 20min) in English
• structure of the presentation has to be discussed at least 2 weeks in advance with the supervisor
• power point slides have to be send to supervisor two days before the presentation
• short handout (up to 5 pages)