Rehab app uses host of sensors in new BlackBerry to assess movement

One of the challenges for physical rehabilitation is efficiently collecting real-time information about patient movements and body position, within and outside the clinic, to determine the most appropriate diagnosis and treatment. While research into computing and mobility technologies can involve esoteric and specialized hardware, we’ve recently developed software using a ubiquitous technology: smartphones.

Purpose-built mobility tracking devices have been developed that patients can wear in the clinic or at home. These combine a cluster of technologies including accelerometers, gyroscopes, GPS sensors, light intensity detectors and temperature and humidity readers. However, most of these devices have not made the transition from the laboratory to clinical practice due to cost, implementation and maintenance factors, training requirements and wearability issues such as bulkiness and cabling.

Many of these sensors are already built into smartphones. This means rather than designing our own hardware, we can take advantage of the sensors, computing power, user interfaces, multimedia displays, and connectivity of a smartphone. This moves the burden of basic data collection to industry partners and frees us to focus on patient-focused applications.

Additionally, a smartphone app can be used at the person’s current location, which is important since movement assessment in the clinic does not necessarily reflect how a person moves in the real world.

To better understand how people move in their daily lives, we are developing a Wearable Mobility Monitoring (WMMS) app for BlackBerry 10 smartphones. It collects a wealth of quantitative information about a person’s movement, including the context. The person wears the smartphone on a belt clip and goes about their normal activities.

The mobile app records real-time data using the smartphone’s built-in sensors, including the magnetometer, accelerometer, gyroscopes and video camera. These data are used to drive our mobility classification software that identifies the type of movement, such as active, inactive, walking, sitting, lying, etc.

With the addition of video clip analysis, where three-second video clips are captured at each change of activity, more detailed classifications can be made (for example, riding elevators, walking on uneven ground, or working in a kitchen). This information can be used to make decisions on treatment related to a physical disability or as an indicator of changes in mobility status for chronic diseases.

To capture critical information, all the sensors and tools need to be in sync. If the camera lags behind the software’s signal processing by just a few seconds, an important activity transition may pass without a video record. In this way, the speed and multitasking capabilities of the BlackBerry 10 operating system meet an important threshold. The internal sensors can record data at about 50 times per second, which allows us to make decisions based on real-time information.

For example, if the sensors detect an incline, such as a ramp or hill, the camera could record video that can be reviewed later to get more insight into the person’s movement. Was she walking on a grassy, uneven hill? How did she adjust her gait and posture? This context provides invaluable information about movement in a real environment, and helps us track recovery and adjust treatment.

Data richness is only one of WMMS’s advantages. A smartphone approach benefits the end-user by providing a convenient device that is less intrusive than a more unwieldy, specialized device and minimizes instruction and teaching time since the person may already be using a smartphone. In these ways, a smartphone app overcomes issues such as access to training and technology. In addition, WMMS allows us to download a full activity report before a clinic visit, making the appointment more efficient.

Our WMMS project has resulted in two other related apps: a data logger that captures sensor data and saves it to a file and an app that timestamps a video when the user touches the screen.

Software development is done primarily by co-op and graduate students. One student can typically complete an application in a four-month work term, including learning time and product testing. The program is co-funded by the Natural Sciences and Engineering Research Council (NSERC) and BlackBerry, which also provides the devices and development support.

While WMMS evaluates mobility, we are also developing apps that can be used for real-time biomechanical measurement. The Biomechanics Augmented Reality (BAR) app uses the smartphone’s accelerometer and camera to superimpose a gravity reference grid and line (i.e., a plumb line) over the camera’s live video.

Since the grid is always oriented to gravity, the clinician has a repeatable reference frame when visually assessing posture and body orientation, such as during a wheelchair seating assessment or standing posture evaluation. By adding additional lines over the video that represent the phone’s orientation, the app can display live angles between the phone and the gravity line. This provides a quick tool for body angle measurement at the point of patient contact.

The BlackBerry 10 platform has allowed us to share the app globally without investing in distribution ourselves. BAR is available for free in BlackBerry World and has been downloaded 660 times from 74 countries.

The project’s success can be measured a few ways. Qualitatively, the smartphone app could unobtrusively provide information to inform clinical decision-making and evaluate new rehabilitation interventions. Quantitatively, we can measure success by its ability to answer questions about human movement.

Edward Lemaire, PhD is a Research Associate, Centre for Rehabilitation Research and Development, The Ottawa Hospital Rehabilitation Centre. He is also an Associate Professor, University of Ottawa, Faculty of Medicine.