Mobile Health Technology Evaluation: Findings and Recommendations

A new article on mobile health technology evaluation presents the findings and recommendations of a group of experts gathered together by the National Institutes of Health in the mHealth Evidence Workshop (disclosure: I was one of the participants and authors).

Although the discussions at the meeting were cross-cutting, the areas we covered fell into three areas: (1) evaluating assessments; (2) evaluating interventions; and (3) reshaping evidence generation using mHealth. The article goes into detail on each of these issues. I'll highlight just a few of the ideas here.

First, to acknowledge what is the obvious to many of us in mHealth. "Creative use of new mobile health information and sensing technologies (mHealth) has the potential to reduce the cost of health care and improve health research and outcomes. These technologies can support continuous health monitoring at both the individual and population level, encourage healthy behaviors to prevent or reduce health problems, support chronic disease self-management, enhance provider knowledge, reduce the number of healthcare visits, and provide personalized, localized, and on-demand interventions in ways previously unimaginable." The continuum of mHealth tools graphic effectively pulls these ideas together. [Note: you can click on any of these images to enlarge them.]

Evaluating mHealth assessments require us to consider the reliability and validity of the measurement tools we utilize and the quality of the data they provide. The next table illustrates some of the reliability and validity issues we touched on in the workshop. Especially when there is the rapid evolution of mHealth device technologies, reliability and validity can as likely degrade as improve as new versions appear. We particularly pointed out that research is needed to better understand the many possible confounding and intervening variables that can affect collecting time-intensive data in real-world settings, from self-quantifiers to patients being monitored at home for signs of congestive heart failure or to assure their general safety and well-being (as in aging-in-place studies). mHealth devices are frequently used by individuals with little training, or in situations where comfort and convenience are paramount. What are the trade-offs, and which ones are justifiable from patient, provider, researcher and ethical perspectives?

The participants noted that there are many unanswered questions about the appropriate use of research designs in mHealth and, alternatively, how mHealth might improve or lead to new research methods. For example, can obtaining multiple repeated measures on a few participants, rather than a few measures on many participants, reduce the size of clinical trials and make conventional research designs more efficient (i.e, quicker, cheaper, and more viable for the rapidly moving technology-based mHealth in terms of time and recruitment)?

As shown in this third table, there are many different research designs that can be employed in mHealth, the randomized clinical trial (RCT) being only one, and sometimes perhaps not the best, option. Among the alternatives suggested by participants are the continuous evaluation of evolving interventions (CEEI) for testing mHealth interventions; the multiphase optimization strategy (MOST) to assess tailoring and intervention optimization in treatment research; and the sequential multiple assignment randomized trial (SMART) might be used where individuals are randomly assigned to various intervention choices over time, rather than staying on one protocol.

We also looked at how mHealth is already reshaping evidence generation. For example, mobile technologies can provide data at very high sampling rates (e.g., 10–500 times per second) that support the quantification of phenomena (e.g., physical activity, physiological changes) that previously were poorly understood because of intermittent and limited measurement. Moving on from data collection to data capture, the data density created by these high sampling rates requires data processing methods not commonly used in health research. Finally, data analysis can be conducted more quickly, sometimes in real time, given that direct contact with participants is no longer necessary. mHealth can facilitate remote research recruitment and potentially reduce the frequency, and consequently, the burden and costs of face-to-face interactions.

A common vision of the workshop participants was using mHealth technologies to generate comprehensive data sets and information fusion. mHealth provides an opportunity to gather data from multiple sensors and modalities including divergent physiologic, behavioral, environmental, biological, and self-reported factors that can be simultaneously linked to other indicators of social and environmental context. In addition, they can be linked to healthcare system and payer data at either the individual or population level.

In the concluding section, we note that "Although these methodologic challenges present exciting new opportunities for scientific innovation, the marketplace and consumers are not waiting for scientific validation. This workshop endorsed the need for timely and increased efforts in mHealth research and for a new transdisciplinary scientific discipline incorporating medicine, engineering, psychology, public health, social science, and computer science."

So What?

Are we developing reliable and valid methods to do what we aspire to and promote for mHealth? Demonstrating their relative value through rigorous testing of methods, processes and outcomes? Showing that they improve the effectiveness, quality, equity and efficiency of promoting and improving health and health care - especially when compared with some of the alternatives? These are the major questions we should be asking of mHealth. Not - is it cool...have they done a TED talk about it...is the guy rich...can I have one too? (OK, may be I'm simplifying that...a little).

You can find the entire article by clicking on the title in the reference.

Reference
Kumar, S., Nilsen, W., Abernethy, A., Atienza, A., Patrick, K., Pavel, M., Riley, W.T., Shar, A., Spring, B., Spruijt-Metz, D., Hedeker, D., Honavar, V., Kravitz, R.L., Lefebvre, R.C., Mohr, D.C., Murphy, S.A., Quinn, C., Shusterman, V. & Swendeman, D. (2013). Exploring innovative methods to evaluate the efficacy and safety of mobile health. American Journal of Preventive Medicine; 45:228-236.

For further reading on mHealth topics, you might also look at The Best mHealth Papers of 2012