‘Black box’ founders use AI to improve surgical safety

By Kelly O’Brien

The team behind the operating room black box is working to incorporate artificial intelligence into its reporting on surgical safety.
Dr. Teodor Grantcharov, a surgeon-scientist at St. Michael’s Hospital in Toronto who specializes in advanced minimally invasive procedures, such as gastric bypasses, is the leader of the OR black box research project.

St. Michael’s has one black box, used by Dr. Grantcharov. A second black box is at the Academic Medical Centre in Amsterdam and four more are to be added in Toronto and New York by the end of the year.

What continues to be the key to the success of the black box project is the post-surgery data analysis. The research team receives all surgical data from the two boxes and produces a weekly report summarizing each of the hospitals’ procedures.

“The problems a black box detects in Amsterdam are going to be different from those problems detected in Toronto,” said Dr. Grantcharov. “We have the ability to tailor it to the individual, to the specific team, the institution, the province, the country. That’s the advantage of expanding the project. We can make educational interventions benefitting a wide range of people.”

The black box team consists of 15 people, including eight data analysts who comb each case for errors or adverse events. It takes approximately two to three hours to rate, or log all the errors and events in each case. With interest in the black box increasing, the team has begun using computer vision to improve the efficiency of its reporting and meet the growing demand.

Computer vision, a form of artificial intelligence, is a science concerned with the automatic extraction, analysis and understanding of useful information from a single image or a sequence of images.

Dr. Grantcharov said the team was “teaching” the computer to identify an adverse event by feeding it existing clips of events from procedures that a human has already analyzed.

“Take bleeding for an example,” he said. “Our library has thousands of clips of different types of bleeds. The human analysts are able to pinpoint what each bleed looks like, whether it’s an inconsequential bleed, or it’s perfusing a bit more, making it more significant. Then we create what’s called an event matrix, which allows us to identify and map out every different bleed.”

“So we take that data and we enter it into the computer, which runs its own analysis, and then we’re able to compare the human analysis to the computer analysis and see how accurate it is.”

Karthik Raj, Dr. Grantcharov’s research program manager, said each time the process is repeated, the accuracy of the computer report improves. The team has created a computer-aided framework for analyzing all the audio in a case with 62 percent accuracy.

“The more parameters we bring in, such as error detection, bleeding detection, all of those things added together, the more accurate the computer analysis will be,” he said. “When we reach 99 percent accuracy, we could actually allow the computer to help us.”

Dr. Grantcharov was recently appointed the Keenan Chair in Surgery at St. Michael’s for his research on surgical education and patient safety with focus on curriculum design, assessment of competence and impact of surgical performance on clinical outcomes-specifically his work with the OR black box.

Similar to the black boxes used in the airline industry, the OR black box is about the size of a PlayStation. It can record nearly everything that happens in the OR, including video of the surgical procedure, conversations among healthcare professionals, room temperature and decibel levels.

It works only for laparoscopic, or minimally invasive surgeries that insert video cameras in thin plastic tubes into small incisions in the body, allowing the surgeon to see what’s going on inside the patient.

Until earlier this year, the team focused on improving safety by reporting on performance issues, communication and team dynamics. But in May, Dr. Grantcharov and his team hosted a retreat for 17 principal investigators from across Toronto, with areas of expertise ranging from art and design to engineering, to begin examining all the ways in which the data they collect can be used to improve surgical safety.

“We wanted to find out whether other researchers were motivated by the data, as well as gain a better understanding of all the things we could do with this type of data,” said Dr. Grantcharov.

By bringing in researchers with different backgrounds, the ways in which Dr. Grantcharov’s team will be able to improve surgical safety will expand beyond the procedure and the actions of the surgical team.

“Take OR design as an example. Our team doesn’t have that knowledge or background, so we can’t offer any insight into how design could make surgery safer,” he said.

“When we have researchers come in with an expertise in design, they are able to see the data we’ve collected in a whole new light.”

Dr. Grantcharov said he hoped that by taking full advantage of the data from the black box, he and his team would find new ways to improve surgical safety.

“As long as there are humans, there will be mistakes. But by learning from these mistakes, and studying how we can do better, we will continue to make surgery safer.”

Written by Editor

1 Comment responses

  1. Avatar
    February 05, 2018

    ” The black box records a plane’s vital signs and cockpit recordings. There are no cameras in the cockpit. A black box is triggered during tragedy, such as a plane crash. Black boxes are not routinely opened up for analysis. Laws enabling access to video and audio on the operating room as if it was part of the medical record would be a formula ripe for misinterpretation and abuse. There are already many devices in the OR recording any number of variables. I can imagine how such a video would be used for fishing expeditions. ” Medical Justice 12/14/17

    There is already too much stress in the medical community from different areas, such as superfluous documentation, budget cuts which extend to critical staffing cuts and trickles down to all areas of patient care. A prime example is the VA.

    High stress areas in hospitals such as emergency rooms and operating rooms do not need Big Brother watching their staff. If cameras are implemented in the operating rooms, “fishing expeditions” will result and be used by ambulance chasing attorneys adding to the stress of the staff performing their duties and will not end well. It is predicted by 2030 the USA will have a shortage of over 100,000 doctors. Possibly more, if more stress will be placed upon them.

    I followed the case of the woman who died in Florida when she had her breast augmentation done by a dentist at his clinic. Unfortunately it ended tragically for her and the family she left behind. From what little I understand about the law, clinics follow different regulations. Since more clinics are performing cosmetic surgeries, maybe the idea for cameras should apply to them, or change the laws in cosmetic clinics where they are up to par with patient safety.


Leave a comment