Computers at MUHC back up after breakdown

MONTREAL – The massive computer crash at the McGill University Health Centre earlier this month was caused by two malfunctions in quick succession, rendering the back-up system powerless and resulting in the cancellation of a dozen surgeries as well as medical procedures involving nearly 40 other patients.

In the aftermath of the shutdown, MUHC officials have been scrambling to ensure that another computer crash of this magnitude does not happen again. The hospital network has reached out for assistance to its information-technology supplier and another firm, the Montreal Gazette reported.

MUHC officials, however, have been reluctant to provide more details of exactly what caused the shutdown that paralyzed the MUHC superhospital, the Montreal General, Montreal Neurological and Lachine hospitals – a computer breakdown without precedent in Quebec.

The incident follows multiple technical problems – including a blackout in 2018 during which the back-up generators also failed – that have plagued the superhospital since it opened in April 2015.

“We want to be as transparent as possible when divulging information about the network failure of Sept. 30,” MUHC spokesperson Gilda Salomone said by email. “However, we must limit the disclosure of details of our investigation in order to protect the establishment from any outside threats.”

The preliminary investigation has concluded that the MUHC was not hit with a cyber ransomware attack, as has been the case with hospitals in Europe and most recently in Ontario.

Dr. Ewa Sidorowicz, the MUHC’s director of professional services, told reporters on Monday that “there would have been an equipment problem,” and “we want to know what went wrong with the server room,” – a state-of-the-art centre in the sub-basement of the $1.3-billion superhospital. That server room controls clinical informatics at all MUHC sites.

Later, a senior MUHC official clarified that “all indicators point to … a problem with a server.”

Pressed for more details, the MUHC declared on Wednesday that “the event was caused by faulty equipment, which in turn caused the malfunction of the redundancy” – a reference to the back-up system.

“The event resulted in the following cancellations in all our hospitals: 12 surgeries, 16 cystoscopies (a procedure to examine the lining of the bladder) and testing for four patients was cancelled; plus 18 cases in medical imaging. The impact on our transplant, medical and surgical clinics was minimal, with only one exception.”

Alan Grnja, president of Healthcare IT Specialists, a company based in Florida with no ties to the MUHC, suggested the failure of the back-up system points to a deeper problem with the MUHC’s computer network.

“It raises in my mind one single point of failure,” Grnja explained. “You usually see that in smaller-scale organizations. It is very fishy that an organization that large … can run everything from a single area. In other words, the hospital said, ‘Let’s build up our own data centre. We’re going to make our own in-house data centre and we’re going to house all our equipment in the basement.’ But that truly does not give you any redundancy.”

“To compartmentalize your systems from failures,” Grnja added, “you would have to have a true, multiple-cloud system, so everything is running in a true cloud environment, so that nothing at one place could affect another.”

The Montreal Gazette has learned the MUHC board of directors rejected an offer by an external firm in February to monitor its system for malware intrusions. The board reached its decision for three reasons: the monitoring would have cost $20,000 a month, previous audits of the computer network revealed it performed well against potential malware threats, and the MUHC’s internal IT department already has anti-virus programs.

Even if the MUHC had signed up for the monitoring, the additional protection would likely not have prevented Monday’s shutdown, Grnja conceded. The crash was the result of hardware malfunctions, not infected software.

The last time a Montreal hospital was hit with a major computer breakdown was in 2015. In February that year, the Jewish General Hospital urged patients to stay away as it fixed damage to its computer system caused by a network surge. Two years earlier, the Jewish General turned away ambulances after a server malfunctioned.

For its part, the MUHC said on October 4 that the computer outage of September 30th had been repaired, and that all clinical systems were fully functional.

According to the hospital, “An analysis of the event has confirmed that it was not caused by a cyberattack, but rather by faulty equipment, which in turn caused the malfunction of our redundancy.”

It added that, “The MUHC network currently has redundant data centres across two sites in a private cloud. Due to the nature of the crash on Monday, redundant systems did not automatically initiate across the other systems. This impacted both internal and external access, denying our ability to access existing cloud-based services as well.”

“Based on this analysis, we are presently working on an action plan to minimize the risks of such incidents reoccurring in the future. We are presently in the process of reviewing the cause of this system failure with our manufacturer and implementing additional layers of proactive monitoring.”