The Problem With Fitness Studies Based on Activity Apps

Every year, 5 million people die from causes associated with one of the most mundane scourges of the modern era: sitting around. That’s like losing one Norway-sized country every 365 days to the likes of heart disease, diabetes, and bowel cancer—illnesses linked to a lack of exercise. Norwegians though, aren’t falling victim to inactivity nearly as much as elsewhere in the world. At least according to the largest human movement study ever undertaken, brought to science by the ubiquitous smartphone.

In a paper published Monday in Nature, researchers at Stanford analyzed the minute-by-minute habits of 717,527 people from 111 countries to understand how things like activity levels, gender, and location impact their weight. By dissecting data from a physical activity-tracking app, the researchers found that in countries with low obesity rates, people walked a similar amount each day. The bigger the gap between those who took steps and those who didn’t, the fatter the country—a phenomenon they call "activity inequality."

“Up until now we’ve had a very limited picture of how active people are,” said Tim Althoff, a doctoral candidate in computer science and first author on the paper. “Smartphones give us this unprecedented opportunity to better understand what people are doing all day and how that relates to their health and wellbeing.” That’s the same promise digital health devotees at Stanford and elsewhere have been making ever since the iPhone debuted. But using smartphones to study public health requires reliable data—and researchers, even at well-connected universities like Stanford, still have a hard time getting their hands on the truly good stuff.

Yesterday's study came out of Stanford’s Mobilize Center—an institution dedicated to translating America's oodles of smartphone and wearable data. It was made possible by a $12 million grant from the National Institutes of Health, as part of a 2014 initiative to form 12 top data-crunching centers around the country. Althoff and his collaborators there started with data donated by Palo Alto-based Azumio, makers of the Argus app. The company anonymized the step-counting data but provided a few key demographics: age, gender, height, and weight. The last two enabled the researchers to calculate each user’s body mass index, and from there they correlated activity levels with obesity rates.

They found some interesting results. Take the US and Mexico, for example. Americans and Mexicans take roughly the same number of steps each day—about 4,500. But in the US, those steps are distributed much more widely across the population. And that gap between the activity-rich and the activity-poor corresponds with a much higher rate of obesity. “It’s not just about individuals," says Abby King, a public health researcher at Stanford who contributed to the study. "It’s about where they live.”

King leads the center’s efforts to help people manage weight via mobile health apps, and she sees a huge opportunity to use that kind of continuous data to provide more targeted, dynamic interventions to people who are headed down a wellness dead end. “We can catch people on their way toward obesity, and provide them feedback through smartphone apps, so they can actually do something about it in the moment.”

For now, though, using smartphone-based data to build public health research and guidance is still problematic. Reason number one: Step-tracking data is actually pretty unreliable.

“In particular, steps that come out of commercial devices like the Apple built-in step counters are not very accurate,” says Bruce Schatz, head of Medical Information Science at the University of Illinois-Urbana Champaign. “They’re tuned for making physically active people feel good.” The issue, he says, isn’t with the measurement device. Smartphones are equipped with accelerometers that measure tiny variations in location, and they do it well.

But the handful of algorithms that Apple and other phone manufacturers and app developers employ to package that raw data into easy-to-use step counts can't accurately capture the huge variety in people's walking mechanics. They don’t have enough flexibility to account for, say, old people who shuffle instead of stride. And not all steps are created equal. Strolling in the park burns fewer calories than sprinting up stairs. Which matters for people trying to manage their weight (though not as much as what people eat). Detecting those distinctions requires raw, not pre-packaged accelerometer data. That's why Schatz, who has worked with the NIH and NSF on their population-scale mobile health initiatives, says raw is the way to go if data is going to be used for health interventions.

The downside is it’s a lot harder to work with. Most app developers don’t keep raw data themselves because the storage costs would be huge. And constantly pulling that data from your phone (think 60 times every second instead of 60 times every hour) would knock out its battery in about an hour or two. Algorithms that store inferences about what you’re doing—walking, biking, sitting—cut down all that data and save battery power. That’s the kind of information Althoff and his Stanford collaborators got from Azumio: 1,440 data points per person per day as opposed to 5 million.

That data was constrained in a less technical way, too. By only looking at the steps of people who bought iPhones and downloaded Azumio's app, the researchers limited themselves to a self-selected group—more likely to be wealthier and healthier than average. Azumio doesn't collect data on things like income and race, and while some app users do keep track of daily food logs and calorie intakes, the company didn't share those for this study. So researchers couldn't test any other hypotheses about lifestyle variations that could impact obesity other than steps. Building accurate models with which to detect, monitor, and predict obesity will require more information than most smartphones readily give up.

Getting population-scale raw accelerometer data from phone manufacturers like Apple and Google isn't impossible. It's just wildly impractical. Researchers who wanted to do it would need to either partner with a developer or build an app themselves, then get loads of people to download it despite the battery drain. Neither Apple nor Google are just giving away data pulls on the billions of phones they have circulating the globe because of its value to paying customers, like online advertisers. And that makes the best information for building accurate predictive models for public health issues like obesity, for all intents and purposes, beyond the reach of most scientists.

“Mobile data really is good enough now to be actionable,” says Schatz. “But nobody has done it except for targeted ads.” Which means that for smartphone data to be able to tackle public health problems, it may first have to become a public good.