What is Artificial Intelligence Surgery?

Artificial Intelligence, machine learning, deep learning, and natural language processing

The term artificial intelligence (AI) was first coined by John McCarthy in his Dartmouth Summer Research Project in 1956, but the foundations for the concept date back to the famed Alan Turing^[2]. AI is the marriage of numerical calculations done with the aid of a computer to create a form of intelligence. Some authors like to think that AI creates simulations, again done on a computer, with three main objectives: to analyze, to understand, and to predict^[3]. Another definition has been proposed that describes AI as machines that function “appropriately” and with “foresight”^[4]_. When these definitions are taken as a whole, it becomes fair to say that AI involves the use of a computer to interpret a situation and/or help accomplish a task, and in short to make our lives easier and better. Two branches of AI in medicine that are often discussed are machine learning and deep learning^[5]. Machine learning is a field analyzing how computer algorithms and statistics can be used to autonomously improve through trial and error with pattern recognition. Deep learning, or deep structured learning, is a vaster application of machine learning concepts whose basis is centered-around the formation and utilization of artificial “learning” or “neural” networks (ALNs and ANNs) that are based on the structure of biological brains, such as being multi-layered. Natural language processing deals with pattern recognition of data that comes in unstructured formats^[6].

Machine learning, deep learning, and natural language processing have enormous potential to improve surgical decision making pre-operatively and post-operatively. By analyzing big data, these branches of AI have enormous potential to improve decision making on even a global public health level, which has been laid bare during the COVID-19 pandemic^[7]. Unfortunately, surgeons already have limitations in their understanding of regressions such as relative risks and odds ratios, and even in the interpretation of P-values^[8,9]. Currently, there are many limitations to AI, as current regression models are imperfect, making it increasingly important for modern surgeons to know and understand certain fundamentals of statistics. The hope is that AI can interpret regression analysis for us via machine learning and deep learning, thus, bypassing the need for surgeons to understand regression models that are clearly hard to grasp for the everyday practicing clinician^[5].

Curiously, surgeons are reluctant to accept anything less than totally autonomous surgery as AIS and researchers have even coined a term to describe this reluctance, “the trough of disillusionment.” Part of the Gartner Hype cycle, described by a technology firm of the same name, the AI “hype cycle” goes through several general phases before true innovation can occur. It broadly includes: (1) a peak of inflated expectations; (2) machine learning, deep learning; (3) natural language processing; (4) computer vision; (5) a trough of disillusionment; and (6) a slope of enlightenment followed by a long plateau phase where the real work begins known as the “plateau of productivity” [Figure 1]^[10]. By modifying surgeon’s expectations, it is believed that the phase of enlightenment can be entered and the slope improved, which is why every device or innovation that provides any form of automation and improves patient care must be celebrated, studied, and if appropriate embraced.

Figure 1. Artificial Intelligence Surgery Gartner Hype Cycle: NOTES = Natural Orifice Transluminal Endoscopic Surgery. Adapted from Gartner Hype Cycle for Artificial Intelligence, 2019 gartner.com/smarterwithgartner and modifications by Oosterhoff et al^[10].

Attitudes and ethics of AI

In addition to overly inflated expectations, AIS suffers from the fear that surgeons have that robotics are being developed to replace them^[11]. This fear of being replaced in the workforce is enhanced by the growing fear of the technological singularity (TS), or when AI exceeds mankind’s intelligence. Although this fear seems to originate from watching one too many science fiction films, AI thought leaders and researchers have considered this early in the development of AI in medicine. In fact, the European Group on Ethics (EGE) in Science and New Technologies published a list of 23 fundamental tenets in their treatise the “Asilomar AI Principles.” The general principles are that AI should enhance social justice globally, and that all of mankind should benefit from the promise of AI^[12]. Interestingly, they separate AI, robotics, and autonomous systems in their document; highlighting the importance of surgeons in these discussions because for us a combination of AI and robotics is how we will ultimately develop autonomous surgical systems.

The newer generation of surgeons is embracing AI even though there is a healthy fear of the privacy implications because of the shear amount of patient data that is needed to create these algorithms. An article looking at attitudes of neurosurgical teams found that 66% of them were open to autonomous surgery^[13]. Perhaps the simplest argument for increased AI in surgery comes from breast surgery. Breast specialists lose enormous amounts of time analyzing breast imaging for screenings and diagnosis, and as a result, have very little time to actually counsel and talk with their patients. The hope is that AI can liberate breast surgeons from these time-consuming tasks, which can then be used for actual doctor-patient interactions^[6]. Ultimately, it is hoped that AI can liberate the surgeon to be the doctor that they always dreamed of being, and at the same time decrease physician “burnout,” a scourge that has only gotten worse during the COVID-19 pandemic.

In the era of heightened awareness of global warming, it is inspiring to know that AI thought leaders have placed “sustainability” as a priority in future AI initiatives. Other authors believe that 3 main domains of ethical issues exist in this burgeoning field: (1) data; (2) algorithms; and (3) healthcare practices. Specific examples where ethical issues become problematic include: the need for the protection of data, the ability for algorithms to be manipulated/erroneous and the fear that AI will destroy the doctor-patient relationship^[6]. It is feared that blind reliance on AI without quality data may lead to unintended biases and loss of empathy, which is fundamental to healing. Perhaps the clearest instance of the pitfalls of AI lies in facial recognition software, which has been shown to be susceptible to racial discrimination and could be particularly dangerous in plastic surgery^[6]. For example, if we accept that racial profiling indeed exists, what is to prevent certain populations from trying to change their features to avoid facial recognition software and attempt to better fit the general society’s standards of beauty? The urgent need for surgeons to be at the vanguard of this technology is further elucidated by the fact that the American Artificial Intelligence Initiative, which was enacted in 2019, involves an investment of over 142 million US dollars for AI research in 2021 alone^[6].

Diagnosis: social media and radiomics

As mentioned above, the COVID-19 pandemic has exposed the need for AI in the medical field, both to deal with radically changing public healthcare emergencies and to help quell the rampant physician burnout that is becoming a global problem. Fortuitously, the technology now exists to develop new forms of healthcare delivery. Telemedicine long hailed as the future of healthcare did not truly become realized until 2020. Researchers even used social media and the Patient-Reported Information Multidimensional Exploration (PRIME) framework to track the well-being of cancer patients during the lockdown^[14]. They identified areas of anxiety in large populations and were able to plan and provide for interventions of emotional support^[14]. Even Instagram images have been able to be used to identify signs of depression, revealing that AI has clearly entered the phase of computer vision in the Hype Cycle [Figure 1]^[15]. It is in the cauldron of desperate times that some of the most innovative discoveries have taken place.

Researchers from Switzerland were even able to use radiomics, a form of computer vision, to develop methods to instantaneously diagnose COVID-19 patients using CT imaging alone, this finding which was published in December of 2020 less than 12 months after the start of the pandemic, elucidates the fundamental role that AI can hold in healthcare and public health in the future^[16]. Radiomics uses imaging and algorithms to enhance diagnostic accuracy via interpretation of vast amounts of imaging data. It is perhaps best known for its use in oncology. Recent studies using a radiomics risk score and an associated nomogram for lung cancer have been able to predict whether or not adjuvant chemotherapy would have benefit in patients who had undergone surgical resection for early stage non-small cell lung cancer^[17].

Radiomics is not limited to thoracic tumors and has been used in essentially all types of gastrointestinal cancers to predict response to treatment, but also to help predict presence of malignancy^[18,19]. As noted from the study above that was able to predict COVID-19 infection, radiomics has the potential to use imaging and machine learning via utilization of algorithms to achieve more powerful diagnostic accuracy and vastly improved therapeutic decisions based on big data analysis and not simply instinct. This is perhaps most clearly seen in the finding that AI and machine learning can predict fractures of the spine, ankle, and upper extremity better than orthopedic surgeons and that as of 2020, over 3,300 articles on AI in orthopedics alone have been published^[20,21].

Surgical decisions

In addition to improvements in diagnosis, ascertainment of efficacy of treatment and autonomous actions, AI has the potential to improve surgeons’ ability to better decide if acute surgery is indicated or not. Bayesian networks or decision networks have been used to help determine if patients with arterial injuries due to limb trauma would benefit from revascularization^[22]. Bayesian networks are directed acyclic graphs that are utilized to understand the probability of something occurring based on multiple variables. Directed acyclic graphs are essentially decision trees that when powered with computerized algorithms can help quickly predict outcomes. Using a 10-Predictor Bayesian network, researchers from the United States and United Kingdom created a website (https://www.traumamodels.com), accessible to all, that can help surgeons more accurately predict the probability of success of lower limb revascularization upon initial evaluation when compared to clinical acumen alone^[22].

Ophthalmologic surgeons have shown that machine learning can be used to better diagnose patients with certain ophthalmologic pathologies such as keratoconus, and that AI may have the ability to screen for macular degeneration, cataract, diabetic retinopathy, and glaucoma^[23]. AI has been used to predict which patients would benefit from small incision lenticular extraction and laser-assisted epithelial keratomileusis (LASIK) surgeries with an accuracy greater than 93%. Additionally, the risk of certain surgical complications such as post-surgical ectasia after LASIK surgery can be predicted with machine learning and deep learning algorithms^[23]. The benefits are obvious, surgeons can have more confidence when they propose surgical intervention to their patients and patients can have more accurate information when making decisions regarding their health. As opposed to separating surgeons from their patients, AI can actually bring them closer.

Pre-operative planning/practice

Simulators, or virtual reality, have been available in surgery for decades; however, the simulators with the complete surgical systems are perhaps the most interesting. Unlike laparoscopic simulators that are placed into box trainers, robotic simulators are attached to the actual robotic console and more accurately represent what the surgeon will feel and experience in the operating room. Regardless of the type of simulator used, simulation is a computerized method of training surgeons and must be acknowledged as a form of AI training. Unlike previous generations of surgeons that were limited to practicing on cadavers or live animals, the AI generation has potentially unrestricted, or at least greatly increased, access to the simulator and can repeat practice sessions over and over at no additional cost. Virtual reality training was compared to traditional apprenticeship training in a meta-analysis published in 2020, and the authors found that virtual reality training improved trainee efficiency, improved tissue handling, and reduced errors when compared to the traditional apprenticeship method of surgical training^[24].

Another benefit of AI in surgical training is the development of 3-D printing. Researchers have begun creating 3-D printed vascular “phantoms” of aortic aneurysms and then using them to simulate endovascular aneurysm repair with stents pre-operatively^[25]. As opposed to having to send images out of institutions to specialized centers for 3-D printing, phantoms can now be printed on-site, thus, enhancing the use and diffusion of this technology. These phantoms can also be shown to patients so that they can better understand their pathology, the risks of the proposed intervention and conversely the risks of choosing not to undergo endovascular aneurysm repair.

The potential benefits in creating a 3-D image that patients can hold and see cannot be over-emphasized as it must be remembered that patients do not have the same experience as doctors at looking at and understanding anatomic images on a flat screen. Furthermore, by unifying simulator technology and 3-D imaging, it is hoped that hospitals will be able to scan patients, develop 3-D images and reprogram simulators so that anatomically accurate surgeries of actual patients can be simulated prior to doing the surgery on actual patients. Although time constraints will probably limit this to only the more complex cases, its potential for training residents and younger surgeons who will not be able to do the actual procedure is incalculable.

Intra-operative aids: augmented reality

Unlike hybrid operating rooms that combine image guided interventional suites with traditional operating rooms^[1], AI operating rooms also use machine learning and deep learning to improve the workflow and resource utilization in the operating theatre, pre-operative area and post-anesthesia care unit^[26]. Although the initial and most obvious benefit of operating room resource optimization is economic, increased economic resources translates into increased funds for more advanced AI platforms. Additionally, time saved in-between cases and during procedures can reduce burnout and increase time for doctor-patient interactions.

Unlike simulators, augmented reality (AR) can superimpose images onto structures during operations either during open or minimally invasive procedures. AR has often been used in liver surgery during hepatectomy for liver tumors. These AR systems are essentially guidance systems that enable the surgeon to see the tumor and its relationship to major intra-parenchymal vascular structures in real time^[27]. Although initially 2-D, images can now be created for open and laparoscopic liver resection using stereoscopic surface reconstruction and semi-automatic registration in combination with deep learning to create 3-D AR intra-operative images (SmartLiver)^[28]. Other teams have confirmed this concept, however, AR still remains in the domain of feasibility as confirmation of the AR image still need to be done by the surgeon via standard intra-operative ultrasound and via visual confirmation using pre-operative cross-sectional imaging^[29].

Solid organs are currently the best intra-cavitary organs to integrate with AR, similarly, the spine has also had good success with AR. Initially surgeons had to use glasses to reap the benefits of AR in spine surgery; however, this practice is cumbersome and has limited AR utilization. Because of this other systems that enhance the intra-operative placement of spinal screws has been developed that projects the image directly onto the patient, thus, eliminating the need to wear special glasses^[30]. However, similarly to the Cyber Knife, fiducial markers must be placed to calibrate the images. Multiple companies including robotic technologies have been developed to help with the correct placement of spinal screws; however, technology using AR in the placement of spinal screws is still in the simulation phase and has mainly been done on phantoms^[30,31]. In addition to improving accuracy of screw placement, this technology can reduce fluoroscopy times, which is beneficial to the patient, surgeon, and operating room staff. Additionally the ease of use of these technologies can shorten the learning curve and help promote the use of minimally invasive techniques, which has been shown to have clear benefits to patients in terms of decreased post-operative pain scores, loss of work days, and narcotic usage^[31].

Why robotics is AI

Robot is originally a Slavic word and means slave^[32]. In its current form, robotic surgery uses a tele-manipulator to move robotic arms enabling surgeons to perform minimally invasive surgery, specifically laparoscopy. The robotic arms enable surgeons to do minimally invasive surgery procedures with 4 main additional advantages, 3-D imaging, better ergonomics for the surgeon, the ability to control a third arm and 7 degrees of articulation; but at the cost of loss of haptics and the inability of the surgeon to be in constant contact with the actual patient. Currently, robotic surgery seems like a form of minimally invasive surgery or simply mechatronics, the fusion of mechanics and electronics, however, this is an oversimplification of current robotic surgical systems because the true power of robotic surgery exists in its potential to create autonomous actions. The current da Vinci complete surgical system (Intuitive Surgical, Sunnyvale, CA) already has an operating room table that is linked with the robotic arms and laparoscope, when the surgeon moves the table/patient the entire robot also moves in conjunction, in effect, autonomously. Furthermore, automatic stapling devices exist that have sensors and that can adjust the rate of stapling based on the thickness of tissue to be stapled (iDrive, Medtronic, Dublin, Ireland). Notably, other so-called “hand-held” robotic devices such as the robotically-controlled laparoscope holder, VideoendosKopY (ViKY, Endocontrol, Grenoble, France)^[33].

Although some automation is occurring, the machine learning and deep learning is not yet ready for fully autonomous surgical maneuvers. However, this does not diminish the advances that have been made and only emphasizes the need to understand the steps that are needed to get to automated surgery. A recently published article looking at fellowship-trained minimally invasive surgeons found that use of the complete robotic surgical system can shorten the learning curve for sleeve gastrectomy done by newly minted surgeons as early as the first year of independent practice when compared to more experienced surgeons^[34]. For years, the proponents of the robot have praised the potential benefits of robotics in minimally invasive surgery, however, cost constraints and lack of clear advantages hindered wider implementation. Currently, the newer models of complete surgical systems demand a re-appraisal of the true potential of the robot and behooves a more active embracing of this technology because many signs indicate that the next revolution in surgery is here and it involves computer-assisted technology/AI. This is perhaps best appreciated by the observation that even open surgical procedures such as breast surgery are being done with robotic assistance^[35].

At first glance it is not surprising that the utilization of the robot for mastectomy is controversial; however, this attitude is short-sighted because it is the use of robotics that is the bridge to autonomous surgery. As mentioned throughout this manuscript, AI has the potential to provide better patient care pre-operatively, intra-operatively, and post-operatively as will be seen below, and to improve the doctor-patient interaction and quality of life for healthcare workers including surgeons. To that end the emergence of AI in the evaluation of the critical view of safety is particularly interesting. IRCAD-France has recently published articles showing the reproducible ability of video capture in the real-time evaluation of the critical view of safety^[36,37]. Interestingly, applications can already be used on smartphones that can help identify the critical view of safety (https://www.bestsurgicaleducation.com/choleai/). It is the marriage of robotics and this type of “computer vision” enabling anatomical identification that will be the next step towards autonomous robotic gestures in surgery [Figure 1].

One aspect of robotics that has hindered many surgeons from embracing robotics in its present form must be considered, the lack of haptics or the sensation of touch in modern day robotics^[38,39]. The reality is that most robotic surgeons that use the complete robotic surgical system describe the existence of visual cues that make haptics unnecessary. Furthermore, the dawn of AI in surgery has turned the assumption that haptics is necessary in robotics on its head. If it is possible that surgical dissection can one day be done robotically and autonomously, perhaps an insistence on haptics is unfounded. Possibly, researchers need to focus on improving how robots can respond to resistance, and less on how to transmit this sensation to humans.

AI in the post-operative period

Interestingly, healthcare workers may have more resistance to AI in the post-operative phase of surgical care with only 49% of healthcare workers in neurosurgery having a favorable opinion of AI during this case when compared to earlier phases. This reluctance probably stems from the fact that the post-operative period of surgical care is certainly the most intimate one. Patients are in pain, tired, and sleeping in an unfamiliar and sterile environment, and surgical teams are in the constant search for post-operative complications while at the same time seeing new consultations and doing other surgeries. Personal communication devices (PCD) or smartphones may also play a strong role in this negative perception because although they enable the rapid transfer of information, there are negative side-effects of their use such as increased burnout and the observation that people are more impolite and brusquer when communicating via PCDs. The double-edged sword that PCDs have created has been termed the empowerment/enslavement paradox^[40]. Without proper regulation of PCDs, the constant access that people have to surgeons may lead to fewer people choosing a career in surgery and to more surgeons leaving the field altogether.

Ever since the Health Information Technology for Economic and Clinical Health Act of 2008, electronic health record use has exploded in the United States and worldwide, similarly resulting in a concomitant rise in burnout. It is unclear if the PCD or the electronic health record is truly to blame, regardless, the potential benefits of the electronic health record cannot be over emphasized. Aside from better and more accurate transfer of information, natural language processing of electronic health records has enabled a more accurate diagnosis of surgical site infections when compared to clinical observation alone^[41]. Although this technology is portable and automated, it is not currently attached to PCDs and highlights the potential importance of regulating new AI technologies in the future. Again, if surgeons are not making the decisions regarding new AI technologies, the ramifications are potentially disastrous: increased stress, absenteeism, disengagement, and burnout^[40].

Final thoughts

AI in surgery is not limited to machine learning, deep learning, natural language processing, and computer vision. The dream of autonomous actions in surgery is already here, albeit, in limited ways. Surgeons must understand the basics of AI and learn to better understand its potential benefits instead of insisting on resisting innovation. Robotic surgery in its present form is a form of AIS and a necessary step towards fully autonomous actions in surgery. AIS has the potential to improve surgical outcomes, doctor-patient relationships, and job satisfaction for surgeons in all phases of surgical care. Acceptance and embracing AIS is a prerequisite for surgeons to understand that we have already entered the “slope of enlightenment” as described in the Gartner Hype Cycle, and that the future is very bright indeed.