Manitoba suffers outage of health IT systems

WINNIPEG – A massive, province-wide online health system crash occurred in Manitoba the first week of September, incapacitating computers for about 36 hours. The outage was triggered by one server going down, Manitoba’s health minister says, followed by a chain of failures that brought down more than 60 systems in every regional health authority in the province.

The system failure caused delays in hospitals, and affected computer systems ranging from emergency room patient records to admissions and discharge tracking in hospitals.

Krista Williams (pictured), chief health operations officer of the Winnipeg Regional Health Authority, said the outage forced healthcare staff to do tasks manually that would ordinarily be done by computer, including charting and, for ambulance drivers, figuring out where to bring patients.

“The EMS makes the best decision that they can based on the needs of the patient and trying to also level loads,” she said. “We recognize, though, with the system outage, it did impact the dashboard that helps us understand the volumes within the [emergency departments], so it was a bit challenging for everyone, including EMS.”

A spokesperson with the Winnipeg Regional Health Authority confirmed some patients in ambulances were redirected during the outage, although he said all patients were taken to appropriate facilities.

The outage began on Thursday September 6, but by Friday, officials said all systems were back up and running.

“It was just a series of cascading failures in our IT backbone that supports all our systems,” said Manitoba Health Minister Cameron Friesen. “At the end of that, we had multiple enterprise systems go offline,” he said.

“One of our central servers where data is stored about scheduling and about patient records … went down, and there is a process built in where the next one will essentially activate. And in this case, even the redundancy failed.”

Additional staff were brought in to help cope with the confusion and delays, but it’s not clear yet how many extra people came in.

Friesen said it doesn’t appear the outage was the result of hacking or human error.

“This was simply the case that … the software failed to perform as it usually does,” the health minister said in a conference call with media.

“We have challenges, like all governments have challenges, in respect [to] the age of some of our IT infrastructure,” he said, adding his government recently made investments to move scheduling services online.

Friesen said the outage is not without precedent in the province. While systems were down for less than 36 hours on Thursday, he pointed to an outage in 2013, when ER systems were down for 50 hours.

But Williams said the scope of the outage was unlike anything she’s seen in her time with the WRHA.

“In my experience working in the system, I haven’t seen something this widespread,” she said. “This was not just the emergency departments. It is all of the hospitals, all of the departments, all of the services, and also all of community [care].”

During the course of the outage, all sites implemented “downtime protocols” to continue patient care as the technical support teams and the service provider worked to get systems back online one by one.

Williams and Friesen both tipped their hats to healthcare staff who worked to keep care going during the outage.

“I believe that the efforts that staff and physicians took were remarkable to manage this situation, along with our leaders. They went above and beyond,” Williams said.

The province is now investigating why the failover software didn’t kick in, and how to prevent such outages from happening again in the future.