UHN practices response to Code Grey, when IT fails

TORONTO – The University Health Network recently practiced what to do if the hospitals were to be hit by a Code Grey – the failure of information technology and communication systems. As part of the exercise, the group simulated the situation that would happen if clinical staff across all sites were unable to access Epic, UHN’s year-old health information system.

For example, it’s 11:00 am on a typical, busy day at UHN.

Toronto General and Toronto Western hospitals, and Toronto Rehab are all at Capacity Management Level 4, which means they are at full capacity for admitted patients. The majority of clinics are running full schedules and the operating rooms are fully booked. The emergency departments are seeing high volumes, with multiple patients waiting for inpatient beds.

The simulated incident was presented to nearly 50 leaders across UHN’s clinical, corporate and digital teams to identify potential emergency responses and share learnings.

“This mock exercise enables TeamUHN to practice resiliency techniques in an interactive and immersive way before the stakes get high,” said Jackie Kwong Leung (pictured), senior director, UHN Digital Project Management Office, and co-chair of the IT Downtime Committee.

The objective of this exercise was to test changes made to UHN’s Code Grey Incident Response, as a result of an actual Code Grey, which occurred on Jan. 9, 2023 due to a system switch failure.

“By practicing, we can collectively mature our understanding of downtime processes, identify what is working and where there may be gaps or opportunities to improve, in a risk-free environment,” Kwong Leung said.

As UHN’s technology landscape continues to evolve, and the organization becomes more reliant on electronic systems, it is imperative to prepare for incidents so TeamUHN feels confident in how to proceed through a Code Grey.

“The tabletop exercise was engaging and invaluable to get critical feedback on our response to a potential future event,” said Dr. Anil Chopra, UHN’s vice president, Medical Affairs, who participated in the tabletop. “It helps to gather all inter-professional leaders together to simulate a real-life emergency situation in order to think of pragmatic solutions.”

Rachel Boissonneault, clinical director, Toronto Western Hospital, had similar feedback to share.

“The case-based exercise, and the ensuing rich conversation, enabled us to thoughtfully consider the strengths and gaps in our current knowledge and downtime processes in a Code Grey situation,” Rachel says. “This will result in further productive discussions at the team and department level in continuing to review and strengthen our current procedures.”

The goal of the tabletop exercise is to test and evaluate the organization’s plans and procedures. Following the exercise, the participants identified three key learnings:

• Calling a Code Grey early when a widespread outage is identified with significant clinical impact is important. This ensures staff are aware of what is going on and that the appropriate teams are investigating the issue.
• Timely and regular communication internally and externally is critical during an unplanned IT downtime.
• Ensuring teams are following the proper decision-making channels through the Incident Management System (IMS) structure when enacted is key to ensuring a coordinated response.

“Being ready for a Code Grey – Information Systems isn’t something that can be studied,” says Sarah Muttitt, UHN vice president and chief information officer. “Working through a simulation, where the situation is dynamic and the pressures feel very real helps us, as an organization, think through what we would do and how we would respond without overreacting.

“I want to thank the IT Downtime Committee, and the Clinical Informatics and Emergency Preparedness teams for organizing and assembling such a well-planned and immersive activity.”