Computer outage disrupts Island Health

VICTORIA – Island Health is looking into what caused a system-wide blackout of its hospital electronic records system last week. The problem lasted more than an hour, shutting physicians out of patient histories and lab-test results.

The health authority said no patients were affected by the system shutdown, which originated in a data centre hosted by Oracle Health/Cerner in eastern Canada, according to a report in the Times-Colonist.

It affected all hospitals in Island Health, including the main hospitals in Victoria and Nanaimo which have fully integrated electronic systems and computerized provider order entry. These systems allow doctors and nurses to electronically order medications and lab tests and make radiology requests.

Dr. Mary-Lyn Fyfe, chief medical information officer for Island Health, said the outage occurred about 2:14 p.m. on Tuesday of last week and affected the entire Island Health Cerner system.

Fyfe called “unplanned downtimes” uncommon, saying they occur two to three times a year, or about nine hours a year.

She said the health authority is working with Oracle to determine the cause. “They are sending us a full incident review report, as per our standard, and then we also do an incident report from our side and then come together to make sure that we mitigate this, hopefully, going forward.”

Oracle Health could not immediately be reached for comment.

The outage did not affect other healthcare facilities in the province — Vancouver Coastal, Provincial Health Services Authority, Northern Health and B.C. Cancer — that are also served by the U.S.-based Oracle Health-Cerner company.

“We checked in immediately with our colleagues in B.C., and they were still up and running,” said Fyfe. “That’s the nuance of trying to figure out, why did it affect one and not the other, which other clients across Canada were affected, and did they all have the same effect, or were some less affected than we were?

“That’s part of this evaluation and this investigation that’s occurring now.”

Oracle Health closed a $28-billion deal to acquire electronic health record giant Cerner in December 2021. The Cerner system originally specialized in electronic billing systems. Island Health has had a contract with Oracle Health-Cerner for 24 years and is currently in the second year of a renewed 10-year contract.

One physician, who spoke on condition of anonymity, said the immediate impact of an unplanned shutdown is “huge” and better mitigation strategies are needed.

During an electronic record system shutdown, the internet is still available but everything on the Oracle Health-Cerner system disappears — documents, patient histories, lab and diagnostic test results.

A Code Grey was called over the intercom, instructing everyone to revert to “downtime procedures.” The physician likened it to a mobile phone instantly losing all its applications.

For the hospitals that are fully electronically integrated — Victoria General Hospital, Royal Jubilee Hospital, Gorge Road Health Centre, South Island Surgical and Nanaimo Regional District Hospital — there are supposed to be no paper records in use.

“There’s nothing, you have absolutely nothing that you can get. You can’t pull an old medical record,” the doctor said. “It’s a huge problem, particularly when you are in the emergency department.”

The physician said some degree of daily paper records would be helpful in areas such as intensive-care units or emergency departments.

A back-up system should be accessible at the click of a button, the physician said, so no one has to “scramble for an hour and a half” trying to figure out how to retrieve all the needed information, by which time the system is back up and running.

The physician noted that retrieving information from provincial or patient-portal systems is time-consuming and not as easy as some administrators may make it out to be.

Fyfe said there are backups in Island Health’s systems, starting with a “read only” copy of the system up to the point when it went down on every unit. There are also two “downtime” computers on every unit, said Fyfe. “That’s the system we go to first.

“If that system is not providing us with what we need or is also having a technical challenge, then we can use our backup to our information … the provincial system, which is called CareConnect,” said Fyfe, noting it’s available on every hospital computer.

CareConnect is B.C.’s secure, view-only electronic health record, which gives authorized health-care providers 24-7 access to patient information, including lab results, diagnostic imaging, provincial immunization records and admission and discharge summaries.

(CareConnect is not to be confused with the province’s Health Gateway, which is a patient portal.)

For those entering the hospital by ambulance, a pre-arrival report from the paramedics can be printed.

In the event of a major disaster such as a destructive earthquake, Cerner/Oracle Health has a “failover,” an automatic backup system for when a primary system fails to restore lost data, said Fyfe.

Some clinicians argue the server-based system is more prone to security issues, and a cloud-based backup is needed for timely access to information during an outage, and for enhanced security.

“Everything is moving towards cloud-based, but we’re not there completely yet,” said Fyfe. “We’re trying to complete most of that work this year. We have some training domains that are not yet moved over, so there’s still some work to do.”

The province also continues to look at how all the health authorities in B.C. can be fully integrated on the same Oracle Health server system. Currently, they must use the CareConnect system to access general patient information in other health authorities.