Virtual reality in rhinology

Ivica Klapan Klapan Medical Group University Polyclinic Ilica 191A, HR-10000 Zagreb, Croatia, EU E-Mail: telmed@mef.hr
Pero Raos Mechanical Engineering Faculty in Slavonski Brod Trg I. B. Mazuranic 2, HR-35000 Slavonski Brod, Croatia, EU Phone: +385 35 493 428 E-Mail: praos@sfsb.hr
Tomislav Galeta Mechanical Engineering Faculty in Slavonski Brod Trg I. B.Mazuranic 2, HR-35000 Slavonski Brod, Croatia, EU Phone: +385 35 493 439 E-Mail: tgaleta@sfsb.hr
Goranka Kubat Sunce Polyclinic, Radiology Division, Zagreb Trnjanska cesta 108, HR-10000 Zagreb, Croatia, EU Phone: +385-1-304-6666 E-Mail: goranka.kubat@sunce.hr
Keywords : virtual reality, CAS, Tele-CAS, rhinosurgery, navigation surgery, rapid prototyping Abstract The need of a more precise identification of the pathologic process extent as well as of the fine elements of intracranial anatomic features is often experienced in diagnostic process and during many operations in the nose, sinus, orbit and base of the skull region. In two case reports, the methods used in diagnostic work-up and surgical therapy in the nose and paranasal sinus region are described. Besides baseline x-ray, multislice computed tomography and magnetic resonance imaging scans, the techniques of operative field per viam imaging by use of rapid prototyping model, virtual endoscopy, and patient head 3D-imaging (3D-Doctor) were employed with differential coloration of all substantial head tissues (different tissues visualized in different colors), their anatomic inter-relations, and the extent of pathologic tissue within the operative field. This approach has not yet been used as a standard preoperative or intraoperative procedure in otorhinolaryngology. In this way, we tried to understand the new, visualized 'world of anatomic relations within the patient's head' by creating an impression of perception (virtual perception) of the given position of all elements in a particular anatomic region of the head, which does not exist in the real world (virtual world). This approach was aimed at upgrading diagnostic work-up and surgical therapy by ensuring a faster, safer and above all simpler operative procedure. In conclusion, every ENT specialist, i.e. any member of our surgical team, is able to provide virtual reality (VR)-support in implementing surgical procedures, with additional correct control of all risks, within the limits of surgical normal tissue, without additional trauma to the surrounding tissue in the anatomic region undergoing surgical treatment, while having an impression of the presence in virtual world, navigating through it and manipulating with virtual objects.


1. Introduction Virtual endoscopy (VE) is a new method of diagnosis using computer processing of 3D image datasets (such as 2D-multislice computed tomography (MSCT) and/or magnetic resonance imaging (MRI) scans) to provide simulated visualization of patient specific organs similar or equivalent to those produced by standard endoscopic procedures1,2. Visualization avoids the risks associated with real endoscopy, and when used prior to performing an actual endoscopic examination, it can minimize procedural difficulties3, especially for endoscopists on training, which was proved in Croatian 3D computer assisted-functional endoscopic sinus surgery (3D-CA-FESS) performed on June 3, 19944, our first Croatian Tele-3D-CA-FESS in October, 19983,4, as well as in our ongoing surgical activities (Fig. 1.).

Modern Tele-3D-computer assisted surgery (Tele-3D-CAS)4,5 should enable less experienced surgeons to perform critical surgeries using guidance and assistance from a remote, experienced surgeon. In telesurgery, more than two locations can be involved; thus less experienced surgeon can be assisted by one, two or more experienced surgeons, depending on the complexity of the surgical procedure. In our activities, 3D-CAS4 and Tele-3D-CAS6 also provide the transfer of computer data7 (images, 3D-models) in real time during the surgery and, in parallel, of the encoded live video signals4. Through this network, the two encoded live video signals from the endocamera and OR camera have to be transferred to the remote locations involved in the telesurgery/consultation procedure8,9. Comparative analysis of 3D anatomic models with intraoperative findings, in any kind of CAS and/or CA-telesurgery, shows the 3D volume rendering image to be very good, actually a visualization standard that allows imaging likewise the real intraoperative anatomy10,11. The mentioned technologies represent a basis for realistic simulations that are useful in many areas of human activity (including medicine), and can create an impression of immersion of a physician in a non-existing, virtual environment12,13 . Such an impression of immersion can be realized in any medical institution using advanced computers and computer networks that are required for interaction between a person and a remote environment, with the goal of realizing tele-presence (Fig. 2).

In order to understand the idea of virtual reality (VR), it is necessary to recognize that the perception of the surrounding world created in our brain is based on the information coming from human senses and with the help of the knowledge that is stored in our brain8. The usual, well known definition says that the impression of being present in a virtual environment, such as virtual/tele-virtual endoscopy (VE/tele VE) of the patient’s head that does not exist in reality is called virtual reality. The otorhinolaryngologist, e.g., any member of our surgical team, has an impression of the presence in the virtual world and can navigate through it and manipulate virtual objects9,13.
2. Methods 2.1 Preoperative preparation The real-time requirement means that the simulation must be able to follow the actions of the user that may be moving in the virtual environment8. The computer system must also store in its memory a 3D model of the virtual environment. In that case, a real-time virtual reality system will update the 3D graphic visualization as the user moves, so that up-to-date visualization is always shown on the computer screen. For realistic simulations it is necessary for the computer to generate at least 30-40 such images per second, which imposes strong requirements upon the computer processing power (Fig. 3)9. Virtual reality systems may be used for visualization of anatomic structures, virtual endoscopy, 3D-image-guided surgery, as well as of the pathology and/or anatomy during therapy planning14,15.

So, the basic goal is to develop methods for fast and realistic visualization of 3D objects that are in the VE. Advanced technologies of exploring 3D spatial models allow for simulation of endoscopic surgery and planning the course of the future procedure (VE) or telesurgery (Tele-VE).
2.2 Virtual endoscopy Various 64 MSCT-scanners were used for image acquisition in our VE-activities. Recordings of VE images together with appropriate CT images in three major planes during fly-through were performed in real time. In order to prepare virtual endoscopy for case studies presented in this paper, we used a specialized 3D-DOCTOR software16. MSCT and MR DICOM cross-sectional images were used as a starting point of the preparation and as an input for the software. Prior to interactive segmentation of tissues, several logical objects were prepared for corresponding tissue, e.g., for bones, flesh or pathology, but also for voids. Voids can also be rendered and presented in spatial form and so to provide additional significant information when internal anatomy is observed. Real-time 3D surface rendering done in software finally enabled full and detailed virtual endoscopy. Rendering also provided helpful capabilities to measure objects and to highlight or to hide particular regions and particular objects or tissues. Furthermore, when the 3D surface with tissues arranged by objects is obtained, it is possible to derive spatial cross-sections at selected cutting planes, thus providing additional insight into the internal regions observed. Although the software provides fully automatic segmentation divided in a selected number of objects, in order to highlight and analyze the affected region, it is necessary to perform interactive segmentation. Interactive segmentation in the affected region should be manually constrained to this region of interest and segmentation has to be carefully performed regarding thresholds in order to avoid interleaving of objects. Due to the characteristics of medical imaging technique and segmentation algorithm, some objects are harder to segment. For example, orbital fat around the eye has similar intensity as the nearby tissue, so segmentation of orbital fat requires exhaustive and systematic work of the software operator during interactive segmentation with advisable supervision of an ENT specialist.
3. Case Reports 3.1 A 46-year-old male with chronic sinusitis and shadow intensity characteristic of a mucocele in the right orbital space and ethmoids was examined as the first case. MSCT (Siemens, 64 multislice) of the sinuses in coronal, sagittal and axial sections demonstrated a disease of the ethmoidal infundibulum on the right side, with homogeneous opacification and/or retention in the region of the right anterior and posterior ethmoidal cells and the orbit, as well as frontal sinus, with the sphenoid and the maxillary sinuses of normal transparency. MSCT scanning of the orbit revealed a tumor like shadow, which partially protruded into the anterior and posterior ethmoidal cells, and partially into the orbit. The medial wall of the orbit, right medial rectus muscle, optic nerve and the eyeball were displaced laterally (exophthalmos). A huge inferior turbinate was noticed on the right. Pronounced chronic inflammatory changes with signs of ostiomeatal block were also observed in the region surrounding ethmoidal cells and frontal sinus (Fig. 4).

The patient also complained of difficult breathing, with additional headaches in the right fronto-ethmoidal region, and recurrent sinusitis with postnasal dripping (appropriate nasal discharge). Visual function was partially reduced on the right eye, with normal finding on the left eye. Postoperatively, antibiotic therapy with local corticosteroids was prescribed.
On diagnostic work-up, we used the standard 2D-MSCT-imaging sections, virtual endoscopy of the patient’s head model over a few applied travel sections through the patient's “virtual” head. Physical RP-models of the patient’s head showed clear demarcation (anatomic position) between the diseased and healthy tissues in the projection of the nose, paranasal sinuses and orbit. The patient underwent image guided CA-VE-FESS. Three months after the surgery, the patient was symptom-free. The VE-approach proved efficient not only for diagnostic localization of the radiologic soft tissue shadow in the region of anterior/posterior ethmoids and its identification within the orbital area, but also during the operation itself (Fig. 5).

3.2 A 35-year-old male with a history of chronic frontal, ethmoidal and maxillary sinusitis with strong left frontal headaches and orbital pain, and postnasal dripping was examined as the second case (Fig. 6).

On diagnostic work-up, we used the same approach as in the first patient (2D-MSCT, VE). During the surgery, we had a very clear view of the size, shape and anatomic characteristics of the pathologic process and the status of all sinuses, as well as the sinus ostium, and especially of the sinus periosteum involvement in this case (Fig. 7).

In our patient, the presence of inflammatorily changed mucosa of paranasal sinuses (left side), with “suspected protrusion of the cerebral tissue” into the left frontal sinus (as suspected/diagnosed by plain x-rays in another clinical institution), the basic diagnosis (per viam MSCT and VE) was quite easy to make, however, with highly reflective structures, especially in the borderline area toward the region of frontal sinus, skull base, and posterior ethmoidal cells. 2D-coronal and transverse sections were compared with 3D models16. At the same time, the static and dynamic 3D models and VE of the sinuses were used on surgery planning and were available to the surgeon in the operating room (OR) throughout the procedure. 3D models helped describe the expansion and the nature of the pathologic process8. The density of the “questionable substance” in the frontal sinus was detected with a high level of certainty by use of MSCT sections of the orbit, paranasal sinuses and skull base/brain tissue, in axial and coronal sections, in comparison with soft tissue of the brain itself. Although it was known that a very suspected “unknown tissue” within the frontal sinus region could be visualized by MSCT scanning, vital anatomic soft tissue structures of the brain within the frontal sinus could not be definitely determined (index of transparency, 3D models). It was the reason for creating appropriate objects for matching tissues, e.g., for bones, flesh or for pathology, but also for voids, before the interactive segmentation of tissues. When rendered, voids can be presented as three-dimensional surfaces and so to provide noteworthy information when internal anatomy is observed. The real-time 3D-surface rendering performed in the 3D-DOCTOR software enabled full and detailed VE. Rendering also provided helpful capabilities to measure objects and to highlight or to hide particular regions and particular objects or tissues. Furthermore, when the 3D-surface with tissues arranged by objects was obtained, it was possible to derive spatial cross-sections at selected cutting planes, thus providing additional insight into the internal regions observed. Thus, we were able to conclude that we had done the procedure correctly, within the limits of surgical normal tissue, without additional trauma of the surrounding tissue of the anatomic region undergoing surgical treatment (Fig. 8).

4. Discussion Using CA-surgery17 and/or tele-CA-system18, the possibility of exact preoperative, noninvasive visualization of the spatial relationships of anatomic and pathologic structures, including extremely fragile ones19,20, size and extent of pathologic process10, and to precisely predict the course of surgical procedure21, definitely allows the surgeon in any 3D-CAS or Tele-CAS procedure to achieve considerable advantage in the preoperative examination of the patient, and to reduce the risk of intraoperative complications, all this by use of VR or diagnosis (the surgeon and/or telesurgeon operates in the “virtual world”) . With the use of 3D model, the surgeon's orientation in the operative field is considerably facilitated10, including “patient location” and "location of the tele-expert consultant”22. VR has many applications in CA-surgery. Using the computer recorded co-ordinate shifts of 3D digitalizer during the telesurgery procedure, an animated image of the course of surgery can be created in the form of navigation, i.e. the real patient operative field fly-through, as it has been done from the very beginning (since 1998) in our CA-telesurgeries. Upon the completion of the CAS and/or tele-CAS-operation, the surgeon compares the preoperative and postoperative images and models of the operative field, and studies video records of the procedure itself. In otorhinolaryngology, especially in rhinology, research in the area of 2D and 3D image analysis, visualization, tissue modeling, and human-machine interfaces provides scientific expertise necessary for developing successful VR applications12. The basic requirement in rhinology resulting from the above mentioned refers to the use of a computer system for visualization of the anatomic 3D-structures and integral operative field to be operated on (Fig. 9).

5. Conclusions Different VR applications have become a routine preoperative procedure in human medicine, as we have already shown in our surgical activities in the last two decades, providing a highly useful and informative visualization of the regions of interest, thus bringing advancement in defining the geometric information on anatomic contours of 3D-human head-models by the transfer of so-called “image pixels” to “contour pixels”. The physician has an impression of the presence in the virtual world and can navigate through it and manipulate virtual objects. A 3D-CAS/VR and tele-3D-CAS system may be designed in such a way that the physician is completely immersed in the virtual environment or tele-virtual-environment. We can conclude that the basic purpose of such medical and tele-medical systems is to create a sense of physical presence at a remote location. Tele-presence is achieved by generating sensory stimulus, so that the operator has an illusion of being present at a location distant from the location of physical presence8. A tele-presence system extends the operator’s sensory-motor facilities and problem solving abilities to a remote environment. A tele-operation system enables operation at a distant remote site by providing the local operator with necessary sensory information to simulate operator’s presence at the remote location8. Realization of VR systems requires appropriate software for running VR applications in real time6,9. Simulations in real-time require powerful computers that can perform real-time computations required for generation of visual displays. VE or fly-through12,14 methods, which combine the features of endoscopic viewing and cross-sectional volumetric imaging, provided more effective and safer endoscopic procedures in the diagnosis and management of our patients, especially preoperatively, as already discussed (Fig. 10)8.
Interactive display of correlated 2D and 3D data13,14,15 in the four-window format may assist the endoscopist in performing various image guided procedures. In comparison with real endoscopy, the VE is completely noninvasive. In the nasal or sinus cavity, VE can clearly display the anatomic structure of the paranasal sinuses17,23, nasopharyngeal cavity and upper respiratory tract, revealing damage to the sinus wall caused by a bone tumor or fracture8,24, and use the corresponding cross-sectional image or multiplanar reconstructions to evaluate structures outside the sinus cavity. A major disadvantage of VE is its inability to make an impact on OR performance, as well as considerable time consumption25 to evaluate the mucosal surface, or to provide a realistic illustration26 of the various pathologic findings in cases with highly obstructive sinonasal disease. We would like to underline that ordinary, and occasionally even expert surgeons may need some additional intraoperative consultation (or VE/3D support), for example, when anatomical markers are lacking in the operative field due to trauma (war injuries)6 or massive polypous lesions/normal mucosa consumption, bleeding, etc. Now, imagine that we can substitute artificially generated sensations for the real standard daily information received by our senses8. In this case, the perception system in humans could be deceived, creating an impression of another 'external' world around the man (e.g., 3D navigation surgery). In this way, we could replace the true reality with the simulated reality that enables precise, safer and faster diagnosis, as well as surgery9. All systems of simulated reality share the ability to offer the user to move and act within the apparent worlds instead of the real world.
5. References
1. Ecke U, Klimek L, Muller W, Ziegler R, Mann W. Virtual reality: preparation and execution of sinus surgery. Comput Aided Surg. 1998;3(1):45-50. 2. Urban V, Wapler M, Neugenbauer J, Hiller A, Stallkamp J, Weisener T. Robot-assisted surgery system with kinesthetic feedback. Comput Aided Surg. 1998;3(4):205-209. 3. Knezović J, Kovač M, Klapan I, et al. Application of novel lossless compression of medical images using prediction and contextual error modeling. Coll Antropol. 20076;31(4):315-319. 4. Klapan I, Šimičić Lj, Rišavi R, et al. Tele-3D-computer assisted functional endoscopic sinus surgery: new dimension in the surgery of the nose and paranasal sinuses. Otolaryngol Head Neck Surg. 2002;127(6):549-557. 5. Robb RA. Virtual endoscopy: development and evaluation using the Visible Human Datasets. Comput Med Imaging Graph 2000;24(3):133-151. 6. Klapan I, Šimičić Lj, Bešenski N, et al. Application of 3D-computer assisted techniques to sinonasal pathology. Case report: war wounds of paranasal sinuses with metallic foreign bodies. Am J Otolaryngol. 2002;23(1):27-34. 7. Anon J. Computer-aided endoscopic sinus surgery. Laryngoscope. 1998;108(7):949-961. 8. Klapan I, Raos P, Galeta T, et al. Application of advanced virtual reality and 3D computer assisted technologies in computer assisted surgery and Tele-3D-computer assisted surgery in rhinology. In: Kim J-J, ed. Virtual Reality. Rijeka: Intech; 2011:303-336. 9. Klapan I, Šimičić Lj, Rišavi R, et al. Real time transfer of live video images in parallel with three-dimensional modeling of the surgical field in computer-assisted telesurgery. J Telemed Telecare. 2002;8(3):125-130. 10. Burtscher J, Kremser C, Seiwald M, et al. 3D-computer assisted MRI for neurosurgical planning in parasagittal and parafalcine central region tumors. Comput Aided Surg. 1998;3(1):27-32. 11. Thrall JH. Cross-sectional era yields to 3D and 4D imaging. Diagn Imaging Eur. 1999;15(4):30-31. 12. Johnson E. Surgical simulators and simulated surgeons: reconstituting medical practice and practitioners in simulations. Soc Stud Sci. 2007;37(4):585-608. 13. Klapan I, Vranješ Ž, Rišavi R, et al. Computer assisted surgery and computer-assisted telesurgery in otorhinolaryngology. Ear Nose Throat J. 2006;85(5):318-321. 14. Bisdas S, Verink M, Burmeister HP, Stieve M, Becker H. Three-dimensional visualization of the nasal cavity and paranasal sinuses. Clinical results of a standardized approach using multislice helical computed tomography. JComput Assist Tomogr. 2004;28(5):661-669. 15. Klapan I, Šimičić Lj, Rišavi R, et al. Dynamic 3D computer-assisted reconstruction of metallic retrobulbar foreign body for diagnostic and surgical purposes. Case report: orbital injury with ethmoid bone involvement. Orbit.2001;20(1):35-49. 16. 3D-DOCTOR. DFA 510K Cleared, vector-based 3D imaging, modeling and measurement software. http://www.ablesw.com/3d-doctor. Accessed January 24, 2013. 17. Klapan I, Vranješ Ž, Prgomet D, Lukinović J. Application of advanced virtual reality and 3D computer assisted technologies in tele-3D-computer assisted surgery in rhinology. Coll Antropol. 2008;32(1):217-219. 18. Sezeur A. Surgical applications of telemedicine. Ann Chir. 1998;52(5):403-411. 19. Hauser R, Westermann B, Probst R. Noninvasive 3D patient registration for image guided intranasal surgery In: Medical Robotics Comput Assisted Surgery, New York: Springer; 1997:327-336. 20. Vinas FC, Zamorano L, Buciuc R, Shamsa F. Application accuracy study of a semipermanent fiducial system for frameless stereotaxis. Comput Aided Surg. 1997;2(5):257-263. 21. Mann W, Klimek L. Indications for computer-assisted surgery in otorhinolaryngology. Comput Aided Surg. 1998;3(4):202-204. 22. Olson G, Citardi M. Image-guided functional endoscopic sinus surgery. Otolaryngol Head Neck Surg. 2000;123(3):187-194. 23. Di Rienzo L, Coen Tirelli G, Turchio P, Garaci F, Guazzaroni M. Comparison of virtual and conventional endoscopy of nose and paranasal sinuses. Ann Otol Rhinol Laryngol. 2003;112(2):139-142. 24. Tao X, Zhu F, Chen W, Zhu S. The application of virtual endoscopy with computed tomography in maxillofacial surgery. Chin Med J. 2003;116(5):679-681. 25. Caversaccio M, Eichenberger A, Hausler R. Virtual simulator as a training tool for endonasal surgery. Am J Rhinol. 2003;17(5):283-290. 26. Vibert E, Denet C, Gayet B. Major digestive surgery using a remote-controlled robot: the next revolution. Arch Surg. 2003;138(9):1002-1006.

Acknowledgment The work presented in this paper was financially supported by the Ministry of Science, Education and Sports of the Republic of Croatia through the scientific research projects No. 108-1521473-0339 (Customized Implants) and No. 152-1521473-1474 (Advanced Technologies of Direct Manufacturing of Polymeric Products).