Health on the move—can we keep up? Activity tracker performance test to measure data and strategic skills

Abstract

This study examined data and strategic skill-related problems that activity tracker users experience and the extent to which these problems vary by gender, age and educational attainment. A performance test (N = 100) was conducted to study problems experienced during actual use of activity trackers. Video data of participants’ screen actions were analyzed by coding for skill-related problems regarding data retrieval and interpretation, and goal setting and decision making. The results revealed that both data and strategic skill-related problems are experienced by all users, but are particularly prominent amongst elderly and less educated users. Problems were mostly related to retrieving the correct data. Additionally, substantial problems were experienced in every facet of strategic use. Altogether, data and strategic skills are underdeveloped for the beneficial use of activity trackers. Moreover, the differences in the problems experienced among users cause widening digital inequalities.

1. Introduction

The Internet of Things is a system in which ubiquitous everyday “smart” devices gather and analyze data about their environment, share these data with both users and other devices, and make autonomous decisions based on algorithms (Van Deursen & Mossberger, 2018). An important application domain is health, wherein smart devices provide users with insights into current health conditions and support health-related decision making (Islam et al., 2015; Miorandi et al., 2012). A common smart device in the health domain is the activity tracker. Activity trackers continuously gather data on user activity (e.g., by measuring the number of steps, distance covered, and stairs climbed), on the intensity of activity (e.g., by measuring heartrate and calculating the number of calories burned), and on recovery from activity (e.g., by measuring sleep duration and phases). Collected data are used for setting or adjusting goals related to behavior change (e.g., improving one’s physical condition), or for improved self-understanding motivated by curiosity or fascination with numbers (Rooksby et al., 2014).

Most of the studies on activity trackers focus on particular motivations and uses. These motivations, which are mainly shaped by attitudes towards the technology, influence how users will use the activity tracker and integrate it into their daily lives. This causes not everyone to benefit to the same extent (Lutz, 2019). Research on personal informatics and everyday self-tracking behavior—including activity trackers—often distinguishes between different stages in interaction between humans and tracking devices, showing how people transition between preparation, collection, integration, reflection, and action (i.e., the five-stage model by Li et al., 2010). Studies in this field have informed designers about different user applications and improved the usability of activity trackers (Shin et al., 2019). Yet, there is far less attention for the skills that are required to use activity trackers for personal benefits. As there is an increasing concern that activity trackers are failing to inspire long-term adoption (Clawson et al., 2015), a better insight in the required skills and actual performances is necessary. This insight could help to identify vulnerable groups in society that lack the required skills to integrate activity trackers into their lives and benefit from its use. In the current contribution, we focus on two types of user skills: data skills and strategic skills (De Boer, Van Deursen & Rompay, 2020).

Data skills relate to the ability to deal with the activity tracker’s continuous data stream (Van Deursen & Mossberger, 2018). For instance, users need to be able to retrieve, extract meaning, and gain insights from data regarding their activity and sleep patterns. Only then can data be transformed into actionable knowledge (Barnaghi et al., 2012; Van Deursen & Mossberger, 2018). Consider, for example, the use of activity trackers during running sessions; when users can retrieve, read, understand, and critically evaluate data visualizations (e.g., of steps, distance, time elapsed, and heart rate), they can use the data gathered to adapt and optimize their individual training schedules (e.g., by running on heart rate zone to improve stamina, instead of using a predetermined schedule). They can even choose to follow personalized training schedules constructed by the activity tracker’s algorithms.

In addition to coping with the continuous data stream, users need strategic skills to evaluate decisions autonomously made by the algorithms of the activity tracker (Van Deursen & Mossberger, 2018). As users often do not understand how trackers work or what collected data means (Yang et al., 2015), this is a difficult undertaking. Consider a personalized training schedule. The algorithms that create this schedule follow rules such as if the user’s heart rate was too high during the previous running session, then the resting periods in between the intervals will be extended. Although users of personalized training schedules do not have to shape their own training sessions, they still need to be able to evaluate whether a proposed schedule is achievable (e.g., by evaluating whether the extended resting periods intervene with other obligations or whether this schedule is appropriate when considering physical health/injuries) and whether it is based on accurate data (e.g., by checking on missing data points).

In the current investigation we study the specific data and strategic skills of activity tracker users. Our first research question is: What data and strategic skill-related problems do users experience when using activity trackers, and how often do these problems occur? As digital skills are an important cause for some users to benefit more from technology than others (e.g., Blank & Lutz, 2018; Scheerder et al., 2017), we also aim at understanding who is most prone to experiencing skill-related problems when using activity trackers. Prior investigations suggest that gender, age, and education—the most prominent indicators in digital inequality research (Scheerder et al., 2017)—play a role in using activity trackers. We therefore ask: Does the occurrence of data and strategic skill-related problems when using activity trackers vary with gender, age, or educational attainment? To answer the research questions and extend previous research, we will conduct skill-based performance tests in which users are asked to perform tasks while using activity trackers. During the tests, individual skill-related problems that users have when they use activity trackers are investigated.

2. Theoretical background

2.1. Skills to use activity trackers

We know from previous research that the emphasis on specific skills has changed with the development of the web (Van Deursen & Mossberger, 2018). By introducing smart devices such as activity trackers to the network, information skills required for the traditional internet are shifting towards data skills to make use of data collected by smart devices (Prado & Marzal, 2013). Data skills comprise the skills to retrieve and interpret these data by (1) accessing the gathered data, (2) selecting relevant data, (3) presenting data in an understandable format (e.g., visualization), (4) extracting meaning from the data and comprehending the context in which they were gathered, and (5) evaluating the quality (accuracy and completeness) of data (Authors, 2020).

Accessing data requires an appropriate platform, i.e., an app or website or a tracker’s screen. Selecting relevant data requires users to have a clear idea of what type of data they are looking for, as identifying the appropriate data is much more complicated in a system that is continuously being supplemented with new data (Atzori et al., 2010). Presenting data in an understandable format is required even while raw data are often cleaned by the system itself. Users still need to decide on the specific data to be inspected and manner of data visualization (e.g., textually or in a graph). From these data visualizations, users have to be able to extract meaning, and they need to recognize in what contexts or situations the data were gathered (Barnaghi et al., 2012). For example, when activity tracker users want to know their maximum heart rate during a running session, they need to realize that it is most appropriate to consult the data visualization of that particular run instead of using a general overview of their heart rate that day. Finally, users have to be able to critically evaluate the data (sources) by checking for inaccuracies and missing data (Prado & Marzal, 2013). Users often incorrectly interpret the inaccuracy in tracking data (Yang et al., 2015). Ideally, an evaluation is conducted by comparing the data to previous measurements and to personal experience (Karvalics, 2014; Van Deursen & Mossberger, 2018). For instance, to draw conclusions on data accuracy, users have to be able to compare their heart rate to the run’s intensity and to determine whether it is comparable to data from similar running experiences.

In addition to data skills, strategic skills are necessary to gain positive outcomes from using activity trackers. Like with “traditional” internet skills, these skills broadly follow four steps of decision making (Van Dijk & Van Deursen, 2014) and can be defined as follows: Strategic skills are skills to benefit from activity trackers by (1) becoming oriented towards a specific goal (including understanding the relevant functions and data), (2) combining data with prior knowledge (including previous measurements), experience and other information sources to assess and refine goal setting, making data-driven decisions regarding the goal based on comparisons and actions proposed by the device, and (4) gaining goal-related benefits by taking actions towards the goal and reflecting on whether autonomous device actions fit the goal (De Boer, Van Deursen & Rompay, 2020).

Goal orientation requires users to be aware of the opportunities on offer. When using an activity tracker, a user should have a general idea of what data and functions can help to become more active and needs to translate these functions into specific goals (Consolvo et al., 2009; Karvalics, 2014; Van Deursen & Mossberger, 2018). For example, users can decide to become more active by setting the goal of increasing their daily steps. Relevant functions for example would include tracking the number of steps, stationary periods, and walking activities. The second step involves the ability to combine data with prior knowledge to assess and refine goal setting. Users need to compare data from different moments in time with personal experience and other (online or offline) information sources to draw meaningful conclusions on how a goal can be reached, what progress has been made thus far, and whether the goal is or remains feasible (Karvalics, 2014). When the goal is to increase the number of steps, data on stationary periods can be used to determine the most feasible moment to take more steps. The third step involves using these comparisons to make data-based decisions towards the goal. For example, if users recognize that their longest stationary periods are during office hours, they can decide to go for a walk during lunch time. In addition to making data-based decisions independently, users also need to be able to evaluate whether the actions proposed by algorithms fit their goals. Activity trackers, for instance, notify users who have been stationary for too long. It is then up to the users to decide whether or not to react to these notifications. The actual execution of decisions (e.g., taking a walk during lunch or reacting to the activity tracker’s notifications) is the fourth step of the decision-making process. In this step, users try to gain benefits from the decisions by acting towards the goal. This final step also concerns reflecting on the actions autonomously undertaken (Van Deursen & Mossberger, 2018). An activity tracker can, for instance, autonomously adjust the stepping goal when the goal is repeatedly met. The user must then evaluate the feasibility of the new goal and react to this change by repeating the previous steps of decision making (e.g., by increasing the walking distance or, when doing so is unfeasible, by readjusting the initial goal on the activity tracker’s platform). In other words, decision making is an iterative process. As data are continuously gathered, more extensive and more detailed insights appear as users progress towards their goals and whether the decisions—made either by the user or the activity tracker—have the desired effect. Based on such insights, new goals are set or existing goals may be revised (Holmes et al., 2015).

2.2. Differences in skills to use activity trackers

Potentially, activity trackers can empower users through data to help them make better decisions regarding their health. However, as digital inequality research indicates, the possession of these skills is likely to differ among users, causing inequalities in use and outcomes. The continuous and autonomous gathering of (complex) data makes activity trackers deceptively easy to use (Van Deursen & Mossberger, 2018). Users who lack data and strategic skills are expected to experience difficulties dealing with the vast amount of data and understanding the contexts in which the data are gathered. They may have limited understanding of the algorithms underlying decisions. In the current investigation, we focus on gender, age and education to study personal characteristics that are expected to influence skill-related problems experienced. Gender, age and education are the most commonly observed determinants of skills in internet environments (Scheerder et al., 2017). However, they have rarely or not at all been studied in relation to the skills required for activity trackers or other smart devices.

In internet research, findings regarding gender and digital skills have been inconsistent. Self-evaluations typically indicate that men possess higher skill levels when using the internet and the IoT (e.g., Van Deursen et al., 2021; Van Deursen & Van Dijk, 2015; Wasserman & Richmond-Abbott, 2005). However, these gender differences could be the result of women underestimating their digital skills as compared to men (Hargittai & Shafer, 2006). Studies using performance tests in which people actual make use of technology—as is the case in the current contribution—have revealed that men and women do not differ in internet skills (e.g., Van Deursen & Van Dijk, 2009) or in the skills to use activity trackers (Authors, 2020). In internet use, the elderly are known to experience more problems than younger users (Hunsaker & Hargittai, 2018) as they did not have the opportunity to acquaint themselves with the technology from an early age (Van Dijk & Van Deursen, 2014). Furthermore, elderly users of internet technologies, generally have less access to support and are often impeded by mental and physical conditions (Brown et al., 2013). Education has been an important determinant of initial attitudes and uptake of internet technologies, including the IoT (Scheerder et al., 2017; Van Deursen et al., 2021). Due to more positive attitudes, those who attained higher education are more likely to be the first to adopt technologies such as activity trackers and to develop the skills necessary to use them optimally (Van Deursen et al., 2021). Meanwhile, less educated users struggle to stay up to date with the advancements of internet technologies. Additionally, the greater cognitive capacity of higher educated users causes them to better utilize these technologies, thereby resulting in a widening digital inequality (Correa, 2016; Goldin & Katz, 2008). Research on smart device usage confirms this gap by showing that highly educated users exhibit more advanced skills and hence are more successful when it comes to optimal usage of smart devices and the services they offer (Van der Zeeuw et al., 2020). We hypothesize that:

H1: There are no gender differences in the experience of data and strategic skill-related problems when using activity trackers.

H2: With age an increasing number of data and strategic skill-related problems when using activity trackers is experienced

H3: With educational level an increasing number of data and strategic skill-related problems when using activity trackers is experienced

3. Methods

3.1. Sample

From April 1, 2019, through December 12, 2019, we conducted performance tests using activity trackers. In total, 100 participants were recruited by distributing (digital) flyers on social media and by door-to-door canvassing. Through the flyers, potential participants were referred to a website containing detailed information about the study and instructions for enrolling. The participants were selected based on the criteria of possessing a smartphone and having no prior experience using an activity tracker. Furthermore, to generalize the findings, quota sampling was applied for gender, age and level of educational attainment. After selection, the participants were phoned and invited to participate in this study, and appointments were planned. After planning, a confirmation email was sent. Table 1 provides an overview of the distribution of the participants by gender, age and education.

Table 1. Distribution of the participants by gender, age, and education.

	N
Gender
Male	48
Female	52
Age
18–29	24
30–39	26
40–54	24
55–80	26
Education
Low	33
Middle	34
High	33

3.2. Procedure

As part of the online sign-up, a 5-min questionnaire was administered to acquire personal information about the participants’ gender, age, educational attainment level, smartphone possession, and experience with activity trackers. Data and strategic skills were tested in two research sessions, one week apart. To ensure comparability, during and between these sessions, all the participants used the same type of activity tracker, namely the Fitbit Charge 3. The choice of this device was based on the commonality and a wide range of features for tracking activity and sleep, and its popularity among the general public. The participants collected data on their activity (e.g., steps, distance, floors, training, calories, and heart rate) and sleeping habits (e.g., sleep schedule, sleep duration, and sleep phases). After completing both test sessions, which lasted approximately one-and-a-half hours each, each participant was rewarded with €50.

During session 1, the participants received the activity trackers and installed the complementary app on their smartphones. By letting the participants use their own smartphones, we ensured that each participant was familiar with the operating system (iOS, Android, Windows). After installing the app, each participant received an explanation of both the activity tracker and the app. Regarding the tracker itself, the experimenter showed how to retrieve a general overview of the data gathered and how to start tracking a physical activity. The app was explained by showing the dashboard, an overview of the data categories. This was followed by encouraging the participants to click on the data categories (e.g., active hours) to retrieve a more detailed representation of the data (e.g., infographics showing hourly activity and longest stationary periods). The session ended by assigning the participants the task of integrating use of the activity tracker and its app into their daily lives for the upcoming week. To check whether the participants executed the task, we monitored the data to ensure that the activity tracker was regularly synchronized and that there were no substantive gaps in the continuous measurements.

In session 2, after one week of using the activity tracker, the participants received ten written assignments (discussed below), one at a time, to identify the tracker skill-related problems they experienced. The answers to the assignments could all be found in the activity tracker’s app. The screen actions performed by the participants were recorded using a camera pointed at the screen of their phone (as not every participant’s phone could be equipped with screen recording software). To assure comprehensibility and applicability, all the assignments were pilot tested with six individuals of different ages and educational levels. Moreover, time limits were set for each assignment based on these pilot tests.

In both sessions, participants were asked for permission to record the assignment completion using a GoPro Hero 7 (for audio-visual data).

3.3. Performance test assignments

Nine assignments were used to identify data skill-related problems. An overview can be found in Supplemental Appendix A. For each of these assignments, the participants had to retrieve and interpret data from the activity tracker’s app which reflected different activity tracker data components, including steps/distance, heart rate, and sleep. For two of the assignments (8: sufficient exercise and 9: good night’s sleep), the participants also had to compare the data to nine general health guidelines regarding exercising and sleeping habits (Birch, 2018; Health Council of the Netherlands, 2017). Each assignment consisted of multiple tasks. To successfully complete an assignment, the participants needed to execute all tasks correctly. Specific terms used in the activity tracker software and its app were also used in the assignments to avoid confusion, and explanations were provided for brand-specific terminology. The participants themselves decided when they had finished or wanted to move on to the next assignment. However, when the time limit for the assignment was reached, they were requested to pass on to the next assignment. The time limit (determined in pilot tests) was set to 8 min for the retrieval and interpretation assignments (1–7) and 15 min for the assignments that included comparing data (8 and 9). The difficulty of the assignments could be rated as either moderate (assignments 1–5, and 7) or advanced (assignments 6, 8 and 9) as a function of complexity of features, potential problem occurrences, and user activities required (e.g., comparing data). All assignments were completed in the same order.

One assignment was used to identify strategic skill-related problems. This assignment focused on the construction of an attainable action plan based on the data compared in assignments 8 (Sufficient exercise) and 9 (Good night’s sleep). The participants were instructed to set a goal for every general health guideline to which they did not conform and to describe how they were planning to reach these goals. This was done by following the instructions regarding the action plan construction included in Supplemental Appendix A.

3.4. Coding scheme

To analyze the video data and the created action plans, a systematic approach for coding was required. Initially, a coding scheme was derived based on the skill definitions. This coding scheme was further extended as additional skill-related problems were identified during video analysis. In total, 13 data skill-related problems and five strategic skill-related problems were coded. Data skill problems were navigation- or content-related. An observed lack of skills pertaining to searching and selecting relevant data could be translated into multiple navigation-related problems, for example, selecting the wrong (sub)category (e.g., selecting the “active minutes” category when looking for active hours), navigating too deep or not deep enough in the app’s structure (e.g., too deep refers to selecting a specific training activity when asked for the total number of training activities during that week; not deep enough refers to not going past the general overview of the heart rate category when searching for time in heart rate zones during a particular day), and navigating to the wrong time period (e.g., navigating to yesterday’s sleep phase overview when searching for today’s sleeping phases). Another coded problem was not looking up the data. Instead of searching for the data multiple participants made estimations based on personal experience. Coded content-related problems were based managing data visualizations and extracting meaning from these visualizations. Codes included misinterpreting textual or graphical data (e.g., misinterpreting the threshold heart rate of the zone “fat burn”), making rough estimations leading to broad unspecific answers (e.g., rounding of the number of steps taken during a time interval, instead of using the exact number), and using the wrong data in visualization (e.g., using the data regarding deep sleep when asked for data regarding REM sleep). An overview of the coding scheme regarding data skill-related problems can be found in Supplemental Appendix B.

The codes for strategic skill-related problems concerned goal-related and action-related problems. Goal-related problems comprised inability to recognize points of improvement and inability to construct a specific, quantifiable goal. The action-related problems involved not comprehending what actions could result in progress towards the goal, not planning when actions should take place, and not quantifying the action using data. An overview of the coding scheme regarding strategic skill-related problems can be found in Supplemental Appendix B.

3.5. Data analysis

To answer the first research question, we analyzed the data skill-related problems that were experienced by the participants when unsuccessfully completing an assignment. We mainly considered unsuccessful assignment completions, as these were the problems constraining the potential to benefit from activity trackers. When an assignment was completed unsuccessfully, each type of navigation- and content-related problem that was experienced during that assignment was rated as 1. When a participant did not experience a type of problem, it was set to 0. Strategic skill-related problems were analyzed by determining for each guideline whether the participant experienced the types of goal- and action-related problems (each problem type experienced was rated as 1). In turn, the total scores of the navigation-, content-, goal- and action-related problems were calculated by summarizing the corresponding problems experienced.

To address the second research question, we conducted one-way ANOVAs with the total scores of navigation-, content-, goal- and action-related problems as the dependent variables and gender (M/F), age (18–29/30–39/40–54/55–80) and educational attainment level (low/middle/high) as independent variables. Additionally, a Bonferroni post hoc test was conducted to determine how these skill-related problems differed across gender, age and educational attainment.

4. Results

4.1. Data skill-related problems

Table 2 shows the completion of the nine assignments constructed to measure data skills. Most problematic were assignments 6, 8 and 9. Table 3 shows that on average, each participant experienced 20% of all possible navigation-related problems. They most frequently experienced problems regarding selecting the correct category (40%) and navigating deeply enough into the correct category (27%) when searching for data. Only one of the participants did not experience any content-related problems. On average, each participant encountered 10% of the possible content-related problems. They most often overlooked data (16%) and failed to interpret text and infographics (15% and 12%, respectively).

Table 2. Successful data skill assignments completion (%).

Assignment	1	2	3	4	5	6	7	8	9
Completion	25	45	57	86	36	5	58	6	6

Table 3. Overview of encountered data skill-related related problems

Data skill-related problems (max. # of possible encounters)	M (SD; min-max)	%
Navigation-related problems (53)	10.47 (3.72; 1–20)	20
Category incorrect (9)	3.56 (1.60; 0–7)	40
Category inefficient (6)	1.03 (.72; 0–2)	17
Navigation deep (6)	0.50 (.63; 0–2)	8
Navigation superficial (9)	2.40 (1.13; 0–6)	27
Navigation time (6)	0.87 (.85; 0–3)	15
Navigation none (9)	0.29 (.64; 0–3)	3
Subcategory incorrect (8)	1.82 (1.14; 0–5)	23
Content-related problems (43)	4.46 (1.96; 0–12)	10
Data broad (8)	0.38 (.65; 0–3)	5
Data infographic misinterpretation (7)	0.84 (.94; 0–4)	12
Data text misinterpretation (8)	1.23 (.85; 0–3)	15
Data overlooked (9)	1.42 (1.00; 0–4)	16
Data time (3)	0.06 (.24; 0–1)	2
Data incorrect (8)	0.53 (.67; 0–3)	7

As shown in Table 4, between 57% and 100% of the unsuccessful participants experienced navigation-related problems when executing the assignments. For content-related problems, the percentage of unsuccessful participants experiencing problems varied from 8% to 87% when executing the assignments. An overview of all types of navigation- and content-related problems experienced can be found in Supplemental Appendix C.

Table 4. Participants experiencing data navigation and content skill-related problems at least once (when unsuccessfully completing the assignment) (%).

Assignment skill	1	2	3	4	5	6	7	8	9
Navigation	95	98	74	57	100	98	60	88	95
Content	73	29	47	50	8	60	43	73	87

During assignment 1 (Get moving), 95% of the unsuccessful participants experienced navigation-related problems. In this assignment, 76% selected incorrect categories when attempting to retrieve the data. For instance, many participants searched for their longest stationary period under the category “steps,” reasoning that this period resembled the time they did not take any steps. However, it could only be found by selecting the category of “active hours,” as both active hours and stationary periods were determined by the time in which at least 250 steps were taken. Content-related problems were experienced by 73% of the unsuccessful participants. Overlooking data was the most reoccurring problem (25%), as many participants did not notice the stationary period displayed underneath the number of active hours.

Assignment 2 (Take a walk) involved retrieving the number of steps and distance covered from infographics. During this assignment, 98% of the unsuccessful participants experienced navigation-related problems. Again, they mostly struggled to determine to which category the data belonged (75%). For example, numerous participants expected to find the data regarding distance covered in the same menu as the number of steps. This resulted in multiple participants calculating the distance covered through the number of steps by estimating the length of one step. Other participants thought the number of meters was equivalent to the number of steps. Additionally, 29% of the unsuccessful participants experienced content-related problems. Instead of reading the exact numbers from the infographic, many participants used it to make rough estimates of the number of steps and distance covered (19%).

For assignment 3 (Sleep), navigation-related problems were experienced by 74% of the unsuccessful participants. Almost half of them did not navigate deeply enough into the app’s structure (49%). When asked to compare sleep data, many participants stopped searching after reaching that day’s percentage of deep sleep, not considering the option of comparing the percentage to other (wo)men their age. Content-related problems were experienced by 47% of the participants. They especially had trouble interpreting the infographic in which their data were visualized and plotted against other users’ data (30%).

Over half (57%) of the unsuccessful participants experienced navigation-related problems during assignment 4 (Calories). In the attempt to retrieve the number of burned calories, the most frequently experienced problem was selecting the incorrect category (29%). Multiple participants expected to find the number of burned calories in the food diary or in the menu option regarding their weight, whereas the (exact) number of burned calories could only be found by selecting the “calories” category. Content-related problems were experienced by half of the participants (50%). Most commonly experienced were misinterpretations of textual data (21%) and overlooking data (21%).

All (100%) of the unsuccessful participants experienced navigation-related problems during assignment 5 (Cardio fitness), which was the assignment in which they had to retrieve data regarding their cardio fitness. A commonly made mistake was selecting an incorrect category (97%). Many participants searched for their cardio fitness score and level under the “training” category. However, although training activities can influence cardio fitness, these scores and levels could only be found by selecting the category “heartrate,” as these values are calculated by combining heart rate during training activities, resting heart rate and other personal information (e.g., gender, age and weight). Unlike navigation-related problems, almost no content-related problems were experienced in assignment 5 (8%).

During assignment 6 (Heart rate), 98% experienced navigation-related problems. In this assignment, the participants had to compare their time in the heart rate zones of a particular day to their active minutes during that same day. Again, choosing the correct category proved most difficult (88%). Numerous participants searched for the time during which their heart rate corresponded to the heart rate zones by selecting training activities. In this way, they were able to retrieve how much time they had spent in the heart rate zones during a particular training activity; however, they did not include the time in heart rate zones during the rest of the day. Content-related problems were experienced by 60% of the participants. Again, overlooking the data was the most frequently reoccurring problem (34%). The total time spent in the heart rate zones was often unnoticed.

For assignment 7 (Training), navigation-related problems were experienced by 60% of the unsuccessful participants. The most commonly made mistake was selecting the incorrect category (26%). The participants who did not know where to find the tracked training activities often returned to the number of steps, as they primarily perceived the activity tracker as a pedometer. Furthermore, content-related problems were experienced by 43% of the unsuccessful participants. 40% of these individuals misinterpreted the textual data, for instance, by mistaking the activity named “sports” for the activity of football because they did not read the activity name but instead relied solely on the emblem supporting the text.

Of the participants who failed to successfully complete assignment 8 (Sufficient exercise), 88% experienced navigation-related problems. During this assignment, the participants were required to compare their own data to general health guidelines regarding exercise. However, many of them were unable to make these comparisons, as they failed to select the correct category (80%). Participants often mistook active minutes for active hours. Despite the similarity between the names, these categories hold different data. The category “active hours” involves the number of hours in which a participant took at least 250 steps, whereas “active minutes” shows the number of minutes when activity level was high. Additionally, 73% of the unsuccessful participants experienced content-related problems. Many of the participants struggled to interpret the textual data (44%). Numerous participants had, for instance, counted the number of days in which activities were tracked instead of the number of activities.

When unsuccessfully completing assignment 9 (Good night’s sleep), participants experienced a variety of problems. Over half of them selected the wrong subcategory (64%; e.g., by selecting sleep phases when looking for sleep schedule), failed to navigate deeply enough to retrieve the data (59%; e.g., by stopping to search for the average percentage of REM sleep after reaching that day’s sleep phase overview), and/or selected a cumbersome category when trying to complete the assignment (56%; e.g., by selecting every day separately instead of using the general overview to compare bedtimes over the week). Content-related problems were experienced by 87% of the unsuccessful participants. Again, interpreting textual data proved to be difficult for the participants. For example, they often misinterpreted whether the time they had fallen asleep fitted within a predetermined time frame.

4.2. Strategic skill-related problems

All participants experienced skill-related problems in the construction of an attainable action plan based on provided guidelines and collected data in assignments 8 (Sufficient exercise) and 9 (Good night’s sleep). As shown in Table 5, on average each participant experienced 43% of the possible goal-related problems, indicating that strategic use of an activity tracker already falls short at its beginning stage: goal orientation. They had the most trouble quantifying the goal (50%), which led to unspecific goals, such as “becoming more active,” instead of “becoming active for 8 hours a day.” None of the participants were able to construct an action plan without experiencing action-related problems. On average, each participant experienced 62% of all possible action-related problems. The greatest problem was the inability to quantify the action (e.g., by stating the minimal number of steps they wanted to take during intended walks to reach their stepping goal).

Table 5. Overview of encountered strategic skill-related related problems.

Strategic skill-related problems (max. # of possible encounters)	M (SD; min-max)	%
Goal-related problems (18)	7.66 (3.56; 1–18)	43
Overlooking points of improvement (9)	3.14 (2.03; 0–9)	35
Goal unquantified (9)	4.52 (1.81; 1–9)	50
Action-related problems (27)	16.69 (4.85; 5–27)	62
Action incomprehension (9)	4.74 (2.25; 0–9)	53
Lacking action planning (9)	5.58 (1.92; 1–9)	62
Action unquantified (9)	6.37 (1.18; 4–9)	71

Table 6 reveals the problems experienced for each health guideline. Making a personal action plan for the first two guidelines (move/exercise at least twice a week and be active for at least 150 min a week) was found to be relatively easy, as many of the participants already conformed to these guidelines and did not need to develop an adjusted action plan. However, constructing an action plan accounting for the other guidelines proved to be much more difficult (goal-related problems: 41–87%, action-related problems: 34–99%). Most goal-related problems were experienced when having to construct specific goals involving setting a bedtime (82%) and a time to get up (87%)—guidelines 6 and 7, respectively. Approximately half of the participants did not recognize their irregular sleep schedules as an opportunity for improvement when looking at the (discussed) data regarding the sleep guidelines. When they did recognize it as an opportunity for improvement, the participants’ goals were often limited to “going to bed earlier” and “getting up earlier,” which do not sufficiently address the corresponding guidelines involving falling asleep and waking up at a fixed time, as these goals are too unspecific. See Supplemental Appendix D for an overview of the different goal-related problems experienced.

Table 6. Participants experiencing strategic goal and action related skill-related problems at least once per guideline (%).

Guideline skill	1^a	2^a	3^a	4^a	5^a	6^b	7^b	8^b	9^b
Goal	11	15	45	56	66	82	87	59	41
Action	19	25	90	85	97	99	96	95	34

^aExercise guideline; ^bSleep guideline.

As shown in Table 5, the biggest issue in terms of action-related problems was that participants did not quantify the action taken to work towards their goal involving active hours. For example, a participant whose current hourly step count was approximately 150 steps could have set the goal to take 100 extra steps an hour to reach the goal regarding active hours. Additionally, substantial action-related problems were experienced for three out of four of the sleep guidelines; namely, guidelines 6–8 (see Supplemental Appendix A assignment 9 for an overview). Almost all the participants failed to specify their actions towards the goals regarding their current sleep schedules (guideline 6: 99%, guideline 7: 96%) and durations (guideline 8: 95%). For example, a participant intending to sleep longer could have said “by aiming to sleep at 11 pm, I get an hour of extra sleep before the alarm goes off.” However, action plan specifications were often limited to, for instance, “going to bed earlier,” referring back to the goal of guideline 6. See Supplemental Appendix D for an overview of the different action-related problems experienced.

4.3. Skill-related problems over gender, age and education

Table 7 shows the occurrence data and strategic skill-related problems distributed by gender, age and education. Focusing on data retrieval, significant differences in navigation-related problems were found between the age groups (F(3, 96) = 9.20, p < 0.001) and educational attainment levels (F(2,97) = 4.93, p = 0.01). A Bonferroni post hoc test showed that the eldest participants (55–80) experienced more problems than the participants from the two youngest age groups (18–29 and 30–39). In all age groups, participants experienced most problems with selecting the correct category. The elderly experienced this problem most often. No significant difference was found in the navigation-related problems experienced when comparing the age group of 40–54 to the other age groups. Regarding education, less educated participants experienced significantly more navigation-related problems compared to their higher educated counterparts. Selecting an incorrect category was the most common problem regardless of the educational attainment level. However, less educated participants selected an incorrect category more often. An overview of the Bonferroni post hoc tests is shown in Table 7 No significant differences were found for gender (F(1, 98) = 0.21, p = 0.65). When it comes to data interpretation, no significant differences regarding content-related problems were found for gender (F(1, 98) = 0.01, p = 0.91), age (F(3, 96) = 1.46 p = 0.23) or educational attainment level (F(2,97) = 0.12, p = 0.89).

Table 7. Post hoc tests (Bonferroni with 5% significance level) for data skill-related problems experienced over gender, age and education (M(SD)).

	Gender		Age				Education
	Male	Female	18–29	30–39	40–54	55–80	Low	Middle	High
Navigation content	10.29 (3.29)^a	10.63 (4.10)^a	8.50 (3.51)^a	9.12 (3.57)^a	11.33 (3.29)^a,b	12.85 (2.91)^b	12.00 (3.53)^a	10.12 (3.77)^a,b	9.30 (3.42)^b
	4.44 (1.96)^a	4.48 (1.99)^a	4.13 (1.80)^a	4.08 (1.72)^a	4.54 (1.61)^a	5.08 (2.50)^a	4.36 (1.78)^a	4.59 (2.00)^a	4.42 (2.14)^a

Note: For each skill, within each row, means with noncommon superscripts are significantly different.

When using an activity tracker strategically, significant differences were found in the occurrence of goal-related problems between educational attainment levels (F(2,97) = 8.82, p < 0.001). A Bonferroni post hoc test showed that the less educated participants experienced significantly more problems than the middle and higher educated participants. They especially had more difficulties recognizing opportunities of improvement when looking at their data. The problems experienced by the middle and higher educated participants did not differ significantly. An overview of the Bonferroni post hoc tests is shown in Table 8. No significant differences were found for gender (F(1, 98) = 0.59, p = 0.44) and age (F(3, 96) = 2.39, p = 0.07).

Table 8. Post hoc tests (Bonferroni with 5% significance level) for strategic skill-related problems experienced over gender, age and education (M(SD)).

	Gender		Age				Education
	Male	Female	18–29	30–39	40–54	55–80	Low	Middle	High
Goal	7.38 (3.30)^a	7.92 (3.79)^a	6.42 (3.36)^a	7.08 (3.16)^a	8.67 (3.56)^a	8.46 (3.82)^a	9.61 (2.99)^a	7.00 (3.13)^b	6.39 (3.75)^b
Action	16.40 (4.92)^a	16.96 (4.81)^a	13.83 (3.82)^a	15.54 (5.35)^a,b	18.71 (4.32)^b	18.62 (4.10)^b	18.36 (3.90)^a	16.35 (4.51)^a,b	15.36 (5.62)^b

Note: For each skill, within each row, means with noncommon superscripts are significantly different.

Regarding action-related problems, significant differences were found between the age groups (F(3, 96) = 7.15, p < 0.001) and educational attainment levels (F(2, 97) = 3.45, p = 0.04). A Bonferroni post hoc test showed that the two eldest participants (40–54 and 55–80) scored significantly lower compared to the youngest age group (18–29). In particular, these participants did experience more problems in regard to understanding what actions could contribute to reaching their goals. No significant difference was found in the action-related problems experienced when comparing the age group of 30–39 to the other age groups. Regarding educational attainment level, less educated participants experienced significantly more action-related problems compared to their higher educated peers. They had more difficulty understanding what actions could contribute to reaching their goals. An overview of the Bonferroni post hoc tests is shown in Table 8. No significant differences were found between genders (F(1, 98) = 0.34, p = 0.56

5. Discussion

5.1. Main findings

Although the potential of activity trackers for enhancing wellbeing and quality of life is undisputed, the findings of the current contribution show that the required skills to live up to this promise fall short. Both data and strategic skills of users of activity trackers reveal much room for improvement. The most frequently encountered problems involving data skills were related to navigating to the correct data representations. Substantial strategic skill-related problems were experienced when setting goals and translating these into actions.

The number of navigation-related problems implies that the use of activity trackers is already problematic in the initial stages. Consequently, activity tracker users often do not progress beyond the device’s basic functions, thus providing them with an incomplete picture of their current (physical) activity levels and health condition in general. For instance, most users were able to retrieve their daily number of steps but experienced problems in pinpointing the periods during which they took most steps or when searching for their longest stationary periods. This suggests that users who meet the recommended daily number of steps might consider themselves active, even if they sit still for multiple consecutive hours a day, a type of inactivity that poses severe health risks (Healy et al., 2011; Matthews et al., 2012). Such skill-related problems can, to a certain extent, be ascribed to poor design choices (e.g., using an ambiguous emblem of a football player to represent data categories such as “steps” and “active hours”) as poor interface design has been found to affect users’ ability to operate and navigate through internet technologies (Van Dijk & Van Deursen, 2014). However, users’ ability to interpret data and make strategic decisions is less likely to be affected by poor design choices (Van Dijk & Van Deursen, 2014). Despite this, the experienced navigation-related problems may have prevented users to access the data they needed to interpret in this study, or to perform on content-related skills. Similarly, understanding collected data is required to make the right decisions (Authors, 2020). Therefore, improvements in activity tracker design can be derived from a navigation-related skills perspective and policies aimed at skill improvements should focus on the various data skills simultaneously.

Many problems were experienced when using the activity tracker strategically. While the participants encountered few problems identifying their healthy exercising behaviors, they struggled to recognize opportunities for improvement from the data, let alone to formulate specific goals to improve their current health. This, together with the persistent lack of specificity when planning their actions, is likely to prevent users from becoming more active, as specificity is vital in promoting motivation, self-efficacy, and, eventually, behavioral change (Shilts et al., 2004). The inability to construct a detailed, personalized action plan may also lead users to neglect other health-promoting functions of activity trackers. In line with this statement, higher levels of goal personalization, have been proven to enhance users’ physical activity levels (Gouveia et al., 2015). By using device algorithms to set adaptive, attainable goals, activity levels could be increased even further (Poirier et al., 2016).

The results suggest that large parts of the population will be excluded from effective activity tracker use. In agreement with our hypotheses, this seems to go even more for older and less educated populations. This is worrisome, as they could potentially benefit most from improved health as a result of using activity trackers (Authors, 2020). This suggests that existing inequalities are widening, as elderly and less educated users are unable to take advantage of the data at their disposal and of the decisions proposed or made by activity trackers (Van Deursen & Mossberger, 2018). There is a fair chance that they will be excluded from benefits such as early diagnosis and treatment of health issues and from savings on health insurance (Van Deursen & Mossberger, 2018). Additionally, existing biases will be reinforced, as collected data only include data from individuals using the devices correctly (Bowler et al., 2017; O’Neil, 2016). However, the results also suggest that older users will be just as capable to use activity trackers strategically when they were not hindered by navigation-related problems. To diminish these inequalities caused by skill-related problems, interventions should be introduced to provide education or public support regarding activity tracker use, with special attention to navigation-related skills. We have provided a first insight in the specific issues and skillsets involved and provided starting points for how to enhance skills levels amongst the general population in general, and those who fall behind in particular.

5.2. Limitations and future research

It should be noted that the participants of this study did not have prior experience with using activity trackers. This enabled a fair comparison of skill-related problems and a first insight in how these problems are distributed among different populations. However, for some people, 1 week of using an unfamiliar activity tracker might be too insufficient to get fully acquainted with all the subtleties of its features and embed its use into their lifestyle. Therefore, we recommend future research include experienced users and consider adding prior experience as a control variable. Furthermore, the use of one brand of activity tracker might result in some navigation-related problems to be caused by poor design choices. Therefore, instead of selecting users of a specific brand of activity trackers, future research could select a sample of users based on the activity tracker’s features and specifications.

In this study, we did not cover all the steps of strategic decision making. In addition to measuring goal- and action-related problems, future research should devote attention to problems regarding the actual implementation of the decisions made towards a goal. Additionally, future research could include skills related to data sharing. These are of crucial importance in regard to privacy concerns, as collected data hold much information about users’ daily lives. Without the skills to protect this personal information, users run the risk that actors with malicious intentions may abuse their data (Kumari & Hook, 2017; Van Deursen & Mossberger, 2018). For example, when users share their data on outdoor running sessions, they might unintentionally inform burglars about opportunities for burglary, as the data reveal when nobody is home.

Additionally, future research should address awareness and knowledge regarding algorithms operating in activity trackers. Users need to be able to critically reflect on the autonomous recommendations of their devices when making strategic decisions. For example, an activity tracker’s function to autonomously adjust the stepping goal based on user activity data may be helpful when trying to gradually increase one’s daily steps. However, an autonomous downscaling of recommended activity levels after repeated failures (e.g., when users fail to reach their step goal for three consecutive days) could reinforce inactivity. Therefore, users need to be aware of changes regarding goal setting; what causes these changes and how to intervene when these adjusted goals do not align with their intentions or their current situation.

Ethical approval

This study has been approved by the Ethics Committee Behavioural Science of the University of Twente (ref nr BCE16372).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Pia S. de Boer

Pia de Boer is a PhD candidate at the University of Twente at the department of Communication Science. She is currently involved in a project on Internet of Things skills. She measures these skills in performance tests in which people actually use IoT devices.

Alexander J. A. M. van Deursen

Alexander Van Deursen is Professor of Communication Science and director of the Centre for Digital Inclusion at the University of Twente in the Netherlands. His research theme is digital inequality. Alexander holds Visiting Scholar positions at the London School of Economic and Political Science and Arizona State University.

Thomas J. L. van Rompay

Thomas van Rompay is an associate professor at the Department of Communication Science of the University of Twente. He has a background in cognitive psychology. He studies design experience from an embodied cognition perspective, investigating how design communicates meaning and affect.