Real time transfer of live video image in parallel with volume 3D-models of the surgical field in Tele-3D-computer assisted surgery: two different approaches

Published as: Klapan Ivica, et all., J Telemed Telecare, 2002, 8:125-130.
Medical Institutions:
University Polyclinic Klapan Medical Group, Zagreb,
Department of Otorhinolaryngology and Head & Neck Surgery, Zagreb University School of Medicine and Zagreb Clinical Hospital Center, Zagreb, Croatia,
Croatian Telemedicine Society of the Croatian Medical Association, Zagreb
Croatian Rhinologic Section of the Croatian Medical Association, Zagreb
Reference Center for Computer Aided ENT Surgery and Telesurgery, Croatian Ministry of Health, Zagreb
Department of Radiology, Zagreb Clinical Hospital Center, Zagreb, Croatia
Physics Department University of Zagreb, Croatia
Croatian Telecom, Zagreb, Croatia
Address for correspondence: Ivica Klapan, M.D., Ph.D., University Polyclinic Klapan Medical Group, HR-10000 Zagreb, Croatia (

Keywords: Telesurgery, FESS, Computer Assisted Surgery, Three-dimensional visualization
Tele-3D-C FESS project supported by an unrestricted grant provided by the Croatian Ministry of Science, No. 3-01-543/96-2001 (Dr. Klapan)
The use of minimally invasive surgical methods such as FESS and Tele-FESS has imposed the need of as reliable as possible preoperative imaging of the spatial anatomic relationship in the respective region. In the region of paranasal sinuses, such an image is of paramount importance for the surgeon or tele-expert-surgeon because of the proximity of intracranial structures and limited operative field layout hampering spatial orientation during the operative procedure. During our pilot projects, 3D-C-FESS and Tele-3D.-C-FESS, we used several standards to encode live video signals in telesurgery, such as M-JPEG, MPEG1 and MPEG2. It has been definitely concluded that MPEG2 streams, without audio, have the best picture quality for the operating field/endo camera. For conferencing/consultation cameras used between two or more connected sites during the surgery, we used JPEG and MPEG1 stream with audio. Operating rooms were connected using several computer network technologies with different bandwidths, from T1, E1 and multiple E1 to ATM-OC3 (from 1Mb/s to 155Mb/s). For computer communications using X-protocol for image/3D-models manipulations, we needed an additional 4Mb/s of bandwidth, instead of the 1Mb/s when we used our own communication tools for the transfer of surgical instrument movements. The bandwidth for the transfer in one direction of one MPEG2 stream without audio and one MPEG1 with audio was about 10MB/s.
FESS- Functional endoscopic sinus surgery
3D C-FESS -Three dimensional computer assisted functional endoscopic sinus surgery
OR - Operation room
Tele-FESS - Tele-functional endoscopic sinus surgery
Tele-3D CAS- Tele three dimensional computer assisted surgery
Tele-3D C-FESS - Tele-three dimensional computer assisted functional endoscopic sinus surgery
Tele VE- Tele virtual endoscopy
VE - Virtual endoscopy
VS - Virtual surgery
2D/3D - Two dimensional/Three dimensional
CT - Computed thomography
LCD/TFT - Liquid Crystal Display/Thin Film Transistor
PC - Personal computer
OMC - Ostio meatal complex
1st/2nd TS - First telesurgery / Second telesurgery
SGI O2 - Silicon Graphics O2
ATM OC-3 - Asynchronous Transfer Mode
LANE - ATM LAN Emulation (LANE)
TCP/IP - Transmission control protocol/Internet protocol 
MPEG - Moving Pictures Experts Group
DVD-ROM - Digital Video Disc or Digital Versatile Disc
CD-ROM - Compact Disk
AAL-5 - ATM Adaptation Layer Type 5
T1 - A T1 line (high-speed digital connection)
Technological systems of medical diagnostics are most appropriate for the use of computer technologies, because they actually are computers. All up-to-date diagnostic devices possess the software and interface for connection to computer networks by use of DICOM protocol. The introduction of computer technologies has allowed the patient's images to be simultaneously analyzed on a number of users' computers within a local computer network (operating rooms, lecture rooms, etc.).
Storage of the patient's images and diagnoses in a multimedia form in computer systems allows them to be subsequently searched and some specific cases re-examined for analysis and physician education, even via Internet (e.g., Eurorad Project)1.

Figure 1: By connecting CT devices to the computer, using DICOM protocol, images are transferred from CT to the computer in the original digital format.
The storage of data in the computer system by use of DICOM protocol (Figure 1) is very important when the image information are to be used for more complex testing and examination, e.g., for precise delineation of subtle anatomic features, of the pathologic and healthy tissue on complex spatial models (3D volume rendering)2,3, or by use of VE or VS4,5 during preoperative preparation (diagnosis, preoperative planning), conduct of operation6, or analysis of the course of operation7 in the “standard surgery” and/or telesurgery. In this way, 3D reconstruction of anatomic units becomes a routine preoperative procedure8,9 that provides a highly useful and informative visualization of the regions of interest9, thus bringing advancement in defining the geometric information on anatomical contours of 3D model by the transfer of so-called image pixel to contour pixel7,9.
With defining the procedure of the use of advanced techniques of computer analysis of image data, the 3D method of visualization represents a significant scientific contribution which has also been demonstrated to be of immediate practical value, especially in Tele-3D-C-FESS in otorhinolaryngology10,11. Subsequent computer analysis of digital CT or MRI image information, translating them from 2D into the 3D form, allows for a reliable diagnosis to make, especially on defining precise localization and spread of the pathologic process.
Telesurgery, as a specific part of telemedicine, consists of two or more ORs connected with a computer network. Through this network, two encoded live video signals from the endocamera and OR camera are transferred to other remote locations involved in the telesurgery/consultation procedure. Our telesurgery approach, Tele-3D-C-FESS, allows surgeons not only to see and to transfer video signals but also to transfer 3D computer models and surgical instrument movements with image/3D-model manipulations in real time during the surgery. We use JPEG and MPEG2 encoders and decoders, ATM communication equipment, graphic workstations, 3D digitizers and standard endoscopic surgical instruments. The new video encoders using MPEG2 standards and ATM computer networks using inverse multiplexing greatly improve the safety of surgical procedures, especially in endoscopic surgery. The best results are obtained using ATM-OC3 technologies, with the most acceptable price-performance using the inverse multiplexing method across 4-8 E1 lines.

A. Case reports:
I) A 38-year-old male with chronic sinusitis. Computed tomography of the sinuses shows disease of the ethmoidal infundibulum bilaterally with homogenous opacification in the region of the anterior and posterior ethmoids (Figure 2).

Figure 2. CT-slices of the first case report
II) A 35-year-old female with ASA triad (aspirin sensitive asthma and chronic sinusitis). Computed tomography of the sinuses in transverse and additional coronal sections; signs of chronic inflammatory changes of both maxillary sinuses, completely with signs of current inflammation in the region of the right maxillary sinus, where a horizontal level of liquid inflammatory content is observed. Signs of osteomeatal block bilaterally. A hypertrophic uncinate process on the left. Huge middle turbinates. Mucosal thickening in the region of the middle nasal turbinates, with polypoid formations inside. Deviated nasal septum. The presence of content in the region of the ethmoidal bullae bilaterally, with inflammatory alterations also in the region of the anterior and posterior ethmoidal cells and a compromised fronto-ethmoidal recess on the right. Spheniodal sinus free, septated. Intact bony borders of the orbits. Endoscopy revealed viscous secretion between polypoid tissues in the nasal cavity arising from the middle meatus bilaterally (between the middle turbinate and lateral nasal wall) (Figure 3).

Figure 3. CT-slices of the second case report
B. Video technologies
Our 1st TS used M-JPEG compression over ATM (OC-3, 155Mb/s). Also, at each of the four locations involved in the telesurgical procedure, there was a remotely controlled video switch with 8 video inputs and 8 video outputs. At the expert location, remote from the OR, there was a video processor for the acquisition of all video signals from all sites involved in the telesurgery procedure and software for the remote control of all video inputs/outputs and pan/tilt/zoom cameras of all locations. Thus, from this point in the telesurgery network, a consultant or conference moderator can view all the video signals or just the primary display. For all these possibilities, a bandwidth of at least 155Mb/s (ATM OC-3) is needed (Figure 4).

Figure 4: Our telesurgical equipment in OR. Based on Siemens, Newbridge, StarVision equipment, October 1998.

The 2nd TS procedure was transmitted over four T1 lines, each about 8Mb/s of bandwidth, and better compression standards, such as MPEG1/2, were used. There was one video stream from every site involved in the telesurgery procedure. At the OR location, a manually controlled video switch for switching between the endocamera and the conference camera was used. The full frame video signal was transmitted from every location to any other location through MPEG1/2 encoders/decoders and inverse multiplexing switches. Video signal processing was not possible because of lower bandwidth (only 8Mb/s).

C. Network technologies
In our 1st TS, ATM switches and AAL-5 were used for video transmission and native or LANE for TCP/IP computer communications. The 2nd TS procedure was built over inverse multiplexing technologies (4xT1 lines), with TCP/IP computer communications. There also was one spare ISDN line (64Kb/s) for redundant conferencing, video conferencing (Figure 5).
Figure 5: Second Tele-CAFESS in Croatia, April 1999.

D. Collaboration
In our 1st TS, there was communication between all sites using InPerson teleconferencing software and the native TCP/IP network. Consultations using computer images and 3D-models were performed using the video network; outputs from the computer were encoded into video stream and transmitted to the remote locations through video communication protocols. 
In our 2nd TS, standard T.120 collaborative tools (SGI meeting, NetMeeting) were used. Both locations involved could view images and 3D-models and manipulate them with software on the local, expert location. We used OmniPro software on top of SGI meeting tools.
The use of an endocamera allows recording of the complete surgical procedure in two different media: Video Tape and MPEG Stream. In postoperative analysis, MPEG2 stream records can be synchronized with movement records of the spatial localizer to produce animation of the computer 3D model based on the real surgical procedure.
In addition, encoded MPEG2 and MPEG1 streams from the telesurgery procedure to the SGI Media Base Server and MS Net Show Theatre were recorded. Thus, users in our local network can view the recorded procedures with standard viewers using the video on demand system.
Endoscopic, computer and communication technologies are very sophisticated and very exciting but "what if something goes wrong" during the telesurgical procedure? That is why we need to build a default tolerant system with multiple, redundant endoscopic, computer and communication equipment and paths12.
A comparison between the first and second Tele-3D C-FESS procedure is shown in Table 1.
First telesurgery
Second telesurgery
Video technology
M-JPEG, H.323
MPEG1/2, H.323
Network technology
ATM OC-3 155Mb/s
Inverse multiplexing 4xT1 8Mb/s
Collaborative computing
SGImeeting, NetMeeting, T.120
Through video
Collaborative tools T.120
Through video
Preoperative consultancy
Through video
InPerson, H.323
Collaborative computing T.120, H.323
Number of involved locations
Video acquisition/processing
Multiple In/Out (8/8)
Quad split
Manually controlled switch
Number of video signals
2 from O.R.
1 from other sites
1 from O.R.
1 from other sites
SGI IRIX, MS Windows, StarMED, OmniPro, InPerson
SGI IRIX, MS Windows, OmniPro, SGImeeting, NetMeeting, InPerson
Table 1. Comparison between our first and second Tele-3D-C-FESS procedures.
The 1st kind of our Tele 3D C-FESS took place between two locations in the city of Zagreb, 10 km apart, with interactive collaboration from a third location. A surgical team carrying out an operative procedure at the Šalata ENT Department, Zagreb University School of Medicine and Zagreb Clinical Hospital Center, received instructions, suggestions and guidance through the procedure by an expert surgeon from an expert center. The third active point was the Faculty of Electrical Engineering and Computing.   The 2nd Tele 3D C-FESS took place between two locations, two cities in Croatia (Osijek and Zagreb, 300 km apart). The surgical team carrying out an operative procedure at the ENT Department, Osijek Clinical Hospital, received instructions, suggestions and guidance through the procedure by an expert surgeon and radiologist from the Expert Center in Zagreb. This Tele 3D C-FESS surgery, performed as described above, was successfully completed in 25-30 minutes.
Taking into account the opinion of the leading world authorities in FESS surgery, we believe that each FESS operation is a demanding procedure, including those described in the two Tele-3D-C-FESS surgeries presented. Nevertheless, we would like to underline herewith that ordinary, and occasionally even expert surgeons may need some additional intraoperative consultation, for example, when anatomical markers are lacking in the operative field due to trauma (war injuries) or massive polypous lesions/normal mucosa consumption, bleeding, etc.
What do experts think about additional support per viam computer assisted reconstruction of the anatomy and pathology of the head and neck; what is the truth and level of reliability of the computer reconstruction of CT images in telesurgery transmission; the question of availability and very expensive equipment; 3D image reality; control parameters; what is the use of computer 3D image of the surgical field and isn't a real live video image much better for telesurgery?
Sinus CT scan in coronal projection is a term familiar to every radiologist dealing with CT in the world. Layer thickness, shift, gantry, and window are internationally standardized and accepted values, thus being reproducible all over the world. The method is standardized, reproducible and distinctly defined13,14, and it is by no means contradictory. But on the other side, the basic CT diagnostics has also limited possibilities, first of all because it presents summation images in a single plane (layer) but cannot present the continuity of structures. This has been solved by 3D reconstruction which is now available on PCs equipped with Pentium III or IV processors/1,4 GHz. By presenting the continuity of structures (0.5-1 mm sections), this reconstruction allows visualization of the region as a whole, avoiding the loss of images by use of the standard approach in sinus CT imaging (3-5 mm sections).
During the telesurgery, the computer with its operative field image allows the surgeon, by means of up-to-date technologies, to connect the operative instrumentarium to spatial digitalizers connected to the computer. Upon the completion of the tele-operation, the surgeon compares the preoperative and postoperative images and models of the operative field, and studies video records of the procedure itself. Using intraoperative records, animated images of the real tele-procedure performed can be designed. By means of computer records labeled coordinate shifts of 3D digitalizer during the surgery, an animated image of the course of operation in the form of journey, i.e. operative field fly-through in the real patient, can be designed. Beside otorhinolaryngology6,8,15, this has also been used in other fields16. The more so, in addition to educational applications, VS offers the possibility of preoperative planning in sinus surgery, and has become a very important segment in surgical training and planning of each individual surgical or telesurgical intervention, not only in the reigon of paranasal sinuses5,16.
The complex software systems allow tele-visualization of CT or MRI section in its natural course (the course of examination performed), or in an imaginary, arbitrary course. Particular images can be transferred, processed and deleted, or can be presented in animated images, as it was done during our 1st TS (Figure 6). Multiple series of images can be simultaneously observed in different color tables and at various magnification, with various grades of transparency, observing them as a unique 3D model system. The work with such models allows different views, shifts, cuts, separations, labeling, and animation. The series of images can be changed, or images can be generated in different projections through the volume recorded, as we have showed in our OR17.

Figure 6: Šalata ENT Department, Merkur Clinical Hospital, Croatia Telecom, InfoNET, Silicon Master (May 1996).

Before the development of 3D spatial model, each individual image or the whole series of images have to be segmented, in order to single out the image parts of interest. Thus, separate models of bones, healthy tissue, affected tissue, and all significant anatomic entities of the operative field are developed18. The complete tele-procedure planned can be developed on computer models and a series of animations describing the procedure can be produced. It has become a routine mode of action in a number of centers worldwide when necessary.
Comparative analysis of 3D anatomic models with intraoperative finding, in any kind of telesurgery, shows the 3D volume rendering image to be very good, actually a visualization standard that allows imaging likewise the real intraoperative anatomy2,4,9 (Figure 7).

Figure 7. Transmission of 3D-models as virtual endoscopy of the human head,
realized during our second Croatian Tele-3D C-FESS in April 1999
Computer communications and collaboration in our Tele-3D-C-FESS based on CT images and 3D models between two locations worked well, but the video image quality was inadequate for telesurgery procedures because images of 320x240 pixels with only 5 frames per second were transferred. The basic video image as a standard record of the operation was only used as a standard record of the course of operation.

Figure 8: Our Telesurgical equipment in ATM Network (OC-3, 155Mb/s).
Using routable shared and switched Ethernet connections (Figure 8), 25 frames were transferred per second in full PAL resolution but with 20% of dropped frames. After some tests, it was found that the routing protocol between two or more sites could not offer a constant frame rate. Packages sent from the source travel to the destination using different network paths, thus some packages may be lost during communication or some may reach the destination with unacceptable delays.

For all these reasons, experimental links were introduced using ATM (OC-3) protocol with AAL-5 for video transmission and LANE or native TCP/IP data communication.

Another problem we faced with video signals was how to transfer multiple video signals to remote locations. The native, uncompressed video required a bandwidth of about 34Mb/s, thus the video signals had to be compressed for the transfer of multiple video streams to remote locations using 155Mb/s.
The video image is critical in teleendoscopic surgery and must be of the highest quality. Using software and hardware M-JPEG compression, it was found that one video stream from the endocamera in full PAL resolution and with audio required a bandwidth of about 20-30Mb/s. Our M-JPEG encoders were upgraded with MPEG1 and later with MPEG2 encoders, because we had a bandwidth of only 155Mb/s for data, video, audio and control communication. MPEG1 seems very good for conferencing; however, the endoscope video signal of the operating field required better image quality. The encoded MPEG1 video stream with audio was transferred to the remote location in full frame resolution using a bandwidth of about 6Mb/s (multiple T1 lines). When our encoder was upgraded to the MPEG2 standard, the quality of the video image was adequate for the operating field endocamera. A bandwidth of 8Mb/s produced a high quality video stream at the remote site. Using MPEG streams, the video signal could be transferred from the endocamera to the remote location for consultancy or education.
As known in the circles of telemedicine/telesurgery, the basic cost of the presented system is known. It includes standard telemedicine equipment which should be mounted at any institution for the institution to be connected to the telemedicine network. This equipment allows for transfer of live video image of the operative field, CT images, commentaries, surgery guidance, etc. In addition, a computer with appropriate volume rendering software support should be introduced in the operating theater. All these devices are now standard equipment available on the market at quite reasonable prices. For comparison, the overall price of these devices is by far lower than the price of a system for image guided surgery, currently mounted in many hospital all over the world. Once installed and tested, the whole system can be used without any special assistance of technical personnel (computer experts and/or network specialties). Clinical institutions (e.g., Zagreb University Hospital Center), which have expert clinical work places, employ properly educated and trained technical personnel who can be readily included in the preparation and performance of tele-3D-C-FESS surgery as well as in the storage of the procedure itself and of intraoperatively generated computer 3D animations.

Figure 9 . Virtual endoscopy (VE), realized during our second Croatian Tele-3D C-FESS in April 1999.
It should be made clear that the main message of our tele-3D-C-FESS surgery, as differentiated from the standard telesurgeries worldwide, is the use of the 3D-model operative field, and thus of VS (Figure 9). This is of paramount importance for emergency surgical interventions. We would like to underline that no Tele-3D-C-FESS procedure was performed by a “novice surgeon” who had not yet mastered the basic FESS techniques, nor is this type of tele-surgery intended for them. We do not advocate that “surgeons beginners” operate on patients, not even with “guidance of a remote surgeon”.
Considering the specificities and basic features of Tele-3D C-FESS, we believe that this type of surgery would be acceptable to many surgeons all over the world.
1.       Caramella D, Sigal R. Radiology unlocs vast potential of the Internet. Diagn Imaging Eur 1999; 15(4):13-17.
2.       Burtscher J, Kremser C, Seiwald M, Obwegeser A, Wagner M, Aichner F, Twerdy K, Felber S. 3D-computer assisted MRI for neurosurgical planning in parasagittal and parafalcine central region tumors. Comput Aided Surg 1998; 3:27-32.
3.       Vannier MW, Marsh JL. 3D imaging, surgical planning and image guided therapy. Radiol Clin North Am 1996; 34:545-563.
4.       Holtel MR, Burgess LP, Jones SB. Virtual realiry and technologic solutions in Otolaryngology. Otolaryngol Head Neck Surg 1999; 121(2):181.
5.       Keeve E, Girod S, Kikins R, Girod B. Deformable modeling of facial tissue for craniofacial surgery simulation. Comput Aided Surg 1998; 3:228-238.
6.       Klimek L, Mosges M, Schlondorff G, Mann W. Development of computer-aided surgery for otorhinolaryngology. Comput Aided Surg 1998; 3:194-201.
7.       Hamadeh A, Lavallee S, Cinquin P. Automated 3D CT and fluoroscopic image registration. Comput Aided Surg 1998; 3:11-19.
8.       Ecke U, Klimek L, Muller W, Ziegler R, Mann W. Virtual reality: preparation and execution of sinus surgery. Comput Aided Surg 1998; 3:45-50.
9.       Thrall JH. Cross-sectional era yields to 3D and 4D imaging. Diagn Imaging Eur 1999; 15(4):30-31.
10.   Klapan I, Rišavi R, Šimičić Lj et al. Tele-3D-C-FESS approach with high-quality video transmission. Otolaryngol Head Neck Surg 1999; 121:187-188.
11.   Klapan I, Šimičić Lj, Rišavi R et al. Tele-3D-Computer Assisted Functional Endoscopic Sinus Surgery (Tele-3D-C-FESS). In: Lemke HU, Vannier MW, Inamura K, Farman AG, eds. Proceedings of the 13th CARS'99, Elsevier, Amsterdam, NewYork, Oxford, Shannon, Singapore, Tokyo, 1999, 784-789.
12.   Schulam PG, Docimo SG, Saleh W et al.. Telesurgical mentoring - initial clinical experience. Surgical Endoscopy-Ultrasound & Interventional Techniques 1997; 11:1001-1005.
13.   Stewart MG, Donovan D, Parke RB, Bautista M. Does the sinus CT scan severity predict outcome in chronic sinusitis? Otolaryngol Head Neck Surg 1999; 121(2):110.
14.   Kenny T, Yildirim A, Duncavage JA, Bracikowski J, Murray JJ, Tanner SB. Prospective analysis of sinus symptoms and correlation with CT scan. Otolaryngol Head Neck Surg 1999; 121(2):111.
15.   Mann W, Klimek L. Indications for computer-assisted surgery in otorhinolaryngology. Comp Aided Surg 1998; 3:202-204.
16.   Hassfeld S, Muhling J. Navigation in maxillofacial and craniofacial surgery. Comput Aided Surg 1998; 3:183-187.
17.   Klapan I, Barbir A, Šimičić Lj, Rišavi R, Bešenski N, Bumber Ž, Štiglmayer N, Antolić S,
Janjanin S, Bilić M. Dynamic 3D computer-assisted reconstruction of a metallic retrobulbar
foreign body for diagnostic and surgical purposes. Case report: orbital injury with ethmoid
bone involvement. Orbit, 20(1):35-49, 2001.
18.   Klapan I, Šimičić Lj, Bešenski N, Bumber Ž, Janjanin S, Sruk V, Mihajlović Ž, Rišavi R, Mladina R. Application of 3D-computer Assisted Techniques to Sinonasal Pathology. Case Report: War Wounds of Paranasal Sinuses Caused by Metallic Foreign Bodies. Am J Otolaryngol, 23(1):27-34, 2002.