“Functioning better is doing better”: older adults’ priorities for the evaluation of assistive technology

ABSTRACT

Despite the benefits of assistive technology (AT), barriers to technology adoption still exist and are uniquely affecting older populations. Improving technology adoption can be achieved by involving end-users in the development and evaluation process. However, existing AT evaluation tools rarely take into account older adults’ experiences. The goal of this study was to fill this gap by determining which AT evaluation criteria are important for older adults. We conducted 4 nominal group meetings with 21 participants aged 50+ in Vancouver, Canada. In the meetings, participants generated AT evaluation criteria and organized them in the order of importance. The content from the meetings was analyzed using qualitative content analysis. Final rankings were collated to reveal which criteria were the most important across the groups. We found that promotion of independence, affordability, ease of use and ethics are the most important AT evaluation criteria for older adults. Some aspects of ATs that older adults value, such as reliability, are not featured in AT evaluation tools. This study provides insight into older adults’ priorities for AT evaluation criteria, and concerns that older adults have about AT use. The findings are supplemented with a comprehensive analysis of the group discussions that contextualizes the criteria.

KEYWORDS:

• aging
• technology
• technology evaluation
• user-centered research

Introduction

Living a longer, healthy life means having more time to pursue personal goals and actively contribute to society. While there are many benefits of living longer, aging causes bodily changes that often negatively influence older individuals’ quality of life. Therefore, products that help with everyday tasks are especially important and empowering for the aging population. Assistive technologies (ATs) are devices that enable participation and help maintain or improve a person’s functioning and wellbeing (WHO | Assistive Devices and Technologies, n.d.). Novel technologies that address the aging population’s specific needs are promising in facilitating aging in place (Graham et al., 2020), supporting physical and emotional wellbeing (Fomiatti et al., 2013; Pu et al., 2019) and reducing caregiver burden (Marasinghe, 2016).

According to the 2019 American Community Survey, the most prevalent disabilities among older adults include ambulatory, hearing and cognitive difficulties. In Canada, pain-related, flexibility and mobility disabilities are the most prevalent (Government of Canada, 2018). Thus, devices such as canes, walkers, wheelchairs and hearing aids are among the most frequently used ATs. Emerging ATs include exoskeletons to support mobility (Jung & Ludden, 2019), social robotics to improve mental health outcomes (Bennett et al., 2017), and smart home technology for health and safety monitoring (Alberdi Aramendi et al., 2018).

Despite many innovative ATs in development, many of them are never widely adopted, used for a short term or abandoned (Sugawara et al., 2018; Yusif et al., 2016). Barriers to technology adoption include design and functional issues, lack of technology awareness, psychological and social factors (Howard et al., 2020). A systematic review on factors influencing acceptance of technology by older adults identified additional contributors such as high cost, privacy issues; and low self-efficacy (Abu Seman et al., 2020).

Effective decision-making regarding the purchase of an assistive product requires access to relevant information about the product’s quality. Martin et al. (2011) found a positive association between people’s perception of being informed prior to purchase and satisfaction with the AT. One common approach to testing the usability and usefulness dimensions of ATs is using evaluation tools. A recent scoping review found 159 distinct evaluation tools for ATs (Tao et al., 2020). This high number makes it difficult to select an appropriate evaluation tool. AT evaluation research focuses largely on determining how well the AT performs specific tasks, rather than what its effects are on end-users’ independence and participation (Desmond et al., 2018). It could stem from the fact that only about a third of the evaluation tools identified in Tao et al. (2020) review included user feedback. Since user experience plays an important role in technology acceptance, the lack of recognition for user feedback is concerning. Therefore, there is a need for creating an integrated technology evaluation tool based on user priorities that could be applied across different AT types.

To begin the process of development of a user-led, universal AT evaluation tool, we began by engaging current AT users. In the present study, we set out to answer the following research questions:

1. What criteria are important for older adults when they select and use AT?

2. What is the relative importance of the criteria generated by older adults?

Methods

Participants

We used three recruitment strategies: 1) contacting potential participants who previously indicated interest in research participation via e-mail, 2) advertising of the research study on social media, and 3) posters in a local rehabilitation center. Criteria for participation included being: 50 years old or older, an active AT user, and able to speak and write in English. There were no exclusion criteria. No participants required additional accommodations. All participants provided written informed consent at the beginning of the meeting. As compensation, each person received a gift card to a major retailer and their parking or transit expenses were reimbursed.

Study design

To answer our research questions, we used a modified nominal group method, a technique developed specifically to elicit relative importance of concepts in a group of people (Harvey & Holmes, 2012; Van de Ven & Delbecq, 1972). We conducted four 90-minute meetings in the summer and fall of 2019. Meetings were conducted in the afternoons in a local rehabilitation center room that was accessible for all participants. Food and refreshments were available for the attendees. For our study design, we modified the original method, as we felt it would be beneficial for participants to be aware of the types of common criteria that are employed in AT evaluation tools, therefore we included a presentation of already existing criteria. The criteria presentation was based on a literature review of evaluation tools (Tao et al., 2020). A visual aid (a Power Point presentation) was shown in the background to help manage the flow of discussion. Figure 1 summarizes the nominal group process applied in this study. The group meetings were facilitated by two trained student researchers and audio recorded. To ensure audio quality, the meetings were captured on two independent recording devices. The clearer of the two recordings was used for transcription.

Figure 1. Nominal focus group process applied in this study.

Analysis

Transcript analysis

The audio recordings from nominal group meetings were transcribed verbatim. We used thematic analysis, an analytic approach in which the researchers explore the meaning of the data by identifying dominant themes (Braun & Clarke, 2006), on the resulting transcripts. The analysis is described in detail in Supplementary Material S2. We used MAXQDA software throughout the process (www.maxqda.com).

Ranking analysis

During the sessions, participants were asked to rank their top criteria, as described in the nominal group method (Harvey & Holmes, 2012; Van de Ven & Delbecq, 1972). To establish the relative importance of criteria across the four groups, we calculated an importance score, based on an established method (Manera et al., 2019). For each criterion that was included in the final group of ranked criteria, the reciprocal ranking was computed. A reciprocal is, in other words, an inverse of a number. These ranking reciprocals were added and divided by the number of meetings.

The resulting importance scores assigned to each ranked criterion ranged from 1 (most important) to 0 for criteria that did not appear in the final ranking.

Results

Participants

A total of 21 participants (8 men, 13 women) took part in the nominal group meetings in a local rehabilitation center. The number of participants per group ranged from 3 to 9. The participants had experience using ATs such as mobility devices, hearing aids, glasses, CPAP machines, computers, tablets, smartphones, and smartwatches.

Qualitative results

A summary of major themes related to motivation for AT use and AT selection criteria that were discussed across the meetings is depicted in Figure 2. Full breakdown of individual codes and their frequencies are available in Supplementary Material S2. The frequency at which the themes occurred during the meetings and their relative rankings (which were determined at the end of each meeting) do not necessarily match. For simplicity, here we start by reporting on some general motivations for AT use before reporting on criteria in order of frequency, with some criteria grouped for brevity. The quantitative rankings are reported in the last section of the results.

Figure 2. Major themes generated in the qualitative analysis of nominal group recordings.

Motivation for AT use

The discussions about criteria for AT selection were framed around the users’ motivations for AT use and purpose. A prominent theme raised in all meetings was promotion of independence. Participants highlighted the importance of AT in completing their daily tasks and being included in activities:

In the rehabilitation community we have been sold a bill – I call it a bill of goods that says that using fewer assistive devices is doing better. But I say no, functioning better is doing better.

This quote challenges the idea that needing fewer technology aids is a sign of good health.

Participants also discussed improving the quality of life as an important motivation for AT use: “That’s why you want a device because it makes your life better.” Although promoting independence and quality of life were accepted as important motivations across the four focus groups, in one of the meetings necessity was raised as a motivation for AT use: “And if [AT developers] don’t listen [to feedback], are we going to stop buying them? We can’t.” This idea is closely linked to lack of consumer choice. Social involvement was another reason for using ATs. Participants illustrated this theme by sharing that using technology enabled them to “participate in the world” and that “it prevents isolation.”

Reliability and maintenance

Within the themes of reliability and maintenance, participants discussed AT durability, whether it works as advertised, requires updates and replacements of parts. Older adults across meetings agreed that performance consistency is key: ‘To me works well means it works well like every time. (…) Not like “oh but it doesn’t work on Saturdays.” That doesn’t do me any good.’ Participants explained that it is important to them that AT has a proven track record and thus is more reliable than new products entering the market: “I should’ve known better than to purchase an [AT] the first year it was out.” One participant also expressed difficulty keeping up with emerging technology:”I would like things to stay the same forever but they don’t, so then you are constantly having to learn about the next technology.” Participants discussed dissatisfaction with devices that need to be frequently exchanged.

Accessibility and affordability

The cost of ATs was discussed extensively. Participants had differing opinions about the relative importance of cost in AT selection. While some prioritized the usefulness and helpfulness of an AT over its cost, for others affordability was crucial:”If you need a $7000 wheelchair and you are a senior person on old age pension, price becomes very important.” Participants commented on medical ATs being more expensive than other kinds of technology: “there are recreational vehicles that are built similarly (…) they are a lot cheaper than wheelchair. In many ways better, they do more for you, they would help me more.”

The discussion relating to AT access included topics such as stigma of AT use, availability, awareness, and structural barriers. In relation to stigma, the attendees expressed that many ATs are associated with ageist assumptions which dissuades potential users. Examples provided by the participants usually related to others who were reluctant to adopt ATs, rather than their own experience. For instance, one person described their friend who did not want to use a mobility aid due to stigma associated with it: “if I use it outside of the apartment people will think I’m getting old.”

Participants expressed frustration with the process of obtaining ATs necessary for their daily life: “You can’t rely on the system when the system doesn’t hear you.” This quote speaks to the structural barriers to AT access. They were discussed in the context of unclear insurance policies and difficulty fulfilling formal requirements to obtain funding for ATs. One example was a policy that covered the cost of a wheelchair once every five years, which does not account for the possibility of the product being faulty, breaking or not suiting user’s specific needs. The difficulty in accessing personalized ATs can be summarized by an exchange between two participants: P1 commented “I just wish I wasn’t so [expletive] special.” P2 replied: “Sometimes being special sucks.”

Qualities of the seller

The discussion about sellers spanned topics such as user support, whether the seller is profit-oriented, responsibilities of the seller, transparency and focus on users. Good warranty and support were considered to be a minimum requirement: “24-hour helpline is the minimum (…).The price you pay for these, you should get that.” The theme of sellers being profit-oriented emerged and was extensively discussed in one of the meetings. The participants pointed out the power disparity between consumers and producers of technology that participants rely on daily: “The guy who is making the decisions doesn’t care about affordability. He’s got control anyway.”

The participants commented on the fact that some ATs have not been adequately evaluated by intended users: “it was great intentions on their [developers] part but I don’t like it.”

Ease of use

The ability to learn and use a technology were perceived as necessary conditions to adopt AT. One participant raised the issue of ATs being difficult to use by older adults because of age-related changes in functioning, such as lowered dexterity. The attendees recognized that learning to use a technology takes time and often requires developing new skills, but expressed that the complexity should be minimized: “[ATs should be as] easy to learn as possible. I don’t want 10 months to get comfortable with it.”

Customizability

Participants noted that health conditions can change and being able to customize the product is important. An example of an AT adjustment is being able to change settings or set one’s own preferences. The need to be able to adjust the technology to improve usability was expressed: “I think any device should have options, none of this one-size-fits-all philosophy. (…) Let me have a choice, makes me feel like I can do my own stuff.”

User control

User control, defined as being in control of various aspects of technology use, was related to feelings of independence and comfort: “I think the more control somebody has over their technology, the more comforting it is for them.” In case when the AT user is unable to have control over aspects of AT, it was noted that it is important for the caregivers to retain user control. On the other hand, advanced AT functionalities that bypass the user allow for remote intervention in the device when necessary. An example of a CPAP machine was given: “[the AT] sends the information to the cloud and the people where I got it from can go onto their computer, see (…) what my oxygenation is overnight and they can reset my machine from the office. It just blows me away completely.”

Security and safety

Participants mentioned AT features that help protect their personal safety, such as lights for visibility at nighttime or safeguards to keep a mobility device from tilting. It was recognized that with technology being sophisticated and collecting more data, there is increased risk of data breaches and other digital security threats: “the more assistive technologies do have a way of sharing their information [the more] somebody could then go and possibly get. So we are trading off when we decide to use that.”

Other criteria

The remaining usability-related themes included comfort and aesthetics. Participants agreed that ATs should fit the user’s body well, as it is crucial for continued use.The theme of aesthetics emerged in relation to the stigma of AT use, as another factor that encourages AT use. One participant spoke about the “cool factor” as a contributor to using a mobility device.

Participants agreed that ethical considerations should play a fundamental role in AT development. Examples of AT companies’ unethical conduct mentioned in the meetings include releasing new, more expensive versions of the product with only cosmetic improvements, overcharging for products, ignoring user feedback and potential privacy issues. Participants noted that ethical AT development is often only recognized once unethical conduct is discovered. The relevant participant quotes are presented in Table 1.

Table 1. Criteria with relevant quotes.

Criterion	Participant quotes
Comfort	“It has to be comfortable, it has to be easy, it has to be efficient or you’ll just leave it in the drawer.”
Cool factor	“I think it’s not just appearance, it might be function that makes [AT] cool. (…) it’s cool to be able to go and ride a wheelchair on a logging road.”
Aesthetics	“And you need [the AT] to look nice as well because otherwise there is a stigma of old people (…) a good design can reduce those stigmas.”
Ethical considerations	“Think about [developers] that were not upfront in Facebook and all the backtracking that they needed to do. All the major ethical issues that have gone about the side and so we can’t assume anything.”

Criteria ranking results

After discussion of the criteria, participants were asked to organize the criteria from most to least important. The final ranking during each of the focus group meetings was achieved through consensus or majority vote. The criteria together with importance scores can be found in Table 2.

Table 2. AT evaluation criteria identified by older adults with their importance score across groups.

Rank	Criterion	Importance score
1	Promotes independence	0.625
2	Affordable	0.375
3	Easy to use	0.258
4	Ethics	0.250
5	Reliable	0.167
6	Easy to learn	0.126
7	Prevents isolation	0.125
7	User control	0.125
7	Works well	0.125
8	Comfortable	0.094
9	Customizable	0.083
10	Low maintenance	0.077
11	Increases productivity	0.050
11	Technology and user support	0.050
11	Privacy and security	0.050
12	Efficient	0.036
12	Convenient	0.036
13	Easy to store/carry	0.031
13	Brings joy	0.031
14	Appearance	0.025
15	Easy accessibility for caregivers	0.023

Discussion

Principal findings

This study aimed to determine what criteria older adults use when selecting ATs and their relative importance. These findings supplement the existing AT evaluation frameworks with insight into the priorities and values of end-users when selecting ATs. Older adult perspectives on criteria and their relative importance will contribute to the development of an AT evaluation tool and strengthen the understanding of factors contributing to AT adoption.

Criteria and their relative importance

As expected, the most frequently discussed criteria do not fully reflect the final ranking. Several topics that were controversial or sparked longer discussions were not considered as important as other criteria for AT selection easily agreed upon by participants. To provide a complete representation of which criteria are important for older adults when selecting ATs, here we combine the criteria rankings with the findings of the nominal group content analysis.

Within the criteria that were ranked as most important across all group discussions, promotes independence (rank 1), prevents isolation (rank 7) and increases productivity (rank 11) all relate to the desired impact the of AT on one’s life. These findings suggest that the impact is an important metric for end-users, however, it is not reflected in the existing AT evaluation research (Desmond et al., 2018). This view is supported by a review of AT evaluation tools which showed that participation is a construct evaluated in only 8 out of the 159 AT evaluation tools identified (Tao et al., 2020).

Affordability was ranked as second most important criterion for AT selection, despite disagreements. While some older adults may have limited budgets for purchase and maintenance of AT, others expressed that the improved quality of life is worth any price. It was suggested that manufacturers seem to exploit the fact that AT users often cannot function without technology, which contributes to the feeling of injustice. Another aspect of affordability discussed was frustration with the processes governed by insurance policy providers. This issue reaches beyond the scope of AT evaluation, as it constitutes a part of AT provision. Emerging evidence suggests that user involvement in AT delivery may be linked to positive outcomes, including higher level of satisfaction with the device (Brandt et al., 2020; Ranada & Lidström, 2019). As the standardized AT provision systems are being developed and refined, it is crucial to involve the end-users of ATs in this process (de Witte et al., 2018; Desmond et al., 2018).

According to Nielsen (1994), the attributes of usability include learnability, efficiency, memorability, low error rate and satisfaction. Therefore, we will discuss the criteria easy to use (rank 3) and easy to learn (rank 6) together, as they closely relate to the concept of usability and were often discussed in conjunction. Participants expressed that they would be unlikely to use ATs that are difficult in use or take a long time and effort to learn. This view is consistent with the literature, as usability is an important factor of technology acceptance frameworks such as Technology Acceptance Model (perceived ease-of-use) (Davis, 1989) and Unified Theory of Acceptance and Usage of Technology (effort expectancy) (Venkatesh et al., 2003). Despite the presence of usability in these models, it is not commonly used in AT evaluation, as only 13% evaluation tools contain constructs related to usability (Tao et al., 2020).

One of the constructs perceived as most important for long-term technology adoption is reliability or trust that the AT will consistently perform its function. Reliability has been identified as an important contributor to technology acceptance in studies with older adults (Mitzner et al., 2010) and features in a model of AT acceptability among older adults developed by McCreadie and Tinker (2005). This model, based on 67 in-depth interviews with older adults, identifies factors that contribute to acceptability of AT. Despite the evidence for the significance of reliability from end-user perspectives, this concept is not prominently featured as a criterion in AT evaluation tools (Tao et al., 2020). To better align with end-user priorities, future AT evaluation tools ought to incorporate the concept of AT reliability.

The topic of ethical considerations emerged in only one of the four meetings, but within this group was the most important criterion. Participants agreed that ethical values should underlay the process of AT development and provision. Some of the ethical concerns identified in the literature related to AT use by older adults include privacy, autonomy and obtrusiveness (Zwijsen et al., 2011). Wider adoption of ethical frameworks in all stages of technology development and adoption, such as the AT ethical adoption framework for dementia (Robillard et al., 2018), would ensure that AT provision is built on ethical values.

Recommendations for future AT evaluation tools

Based on the results of the present study and the results of a systematic review of AT evaluation tools by Tao et al. (2020), we offer recommendations for the development of an evaluation tool specific to AT use by older adults. The next step toward a tool development should be a larger-scale consultation with the user group of interest to add to the findings to the present study, which is limited in scope. Provided that the ideas expressed by research participants in this study can be generalized to the broader older adult population, user-generated criteria should be included in the resulting evaluation tool. The most important criteria that are not sufficiently represented in existing evaluation tools include AT impact on the life of the user, affordability, usability and reliability. As Czaja (2019) points out, older adults should be recognized as a distinct and valuable user group and be involved in the process of technology development and evaluation, for instance through user-centered design.

Limitations

The definition of “older adult” used in this paper (50 years old or older) was broader than the generally accepted age of 60+. As a result, a younger population could introduce a bias in criteria priority, which should be accounted for when comparing the present study with others that include older adults. We did not collect specific participant demographic information which limits our ability to generalize our findings. Due to the context in which the study was conducted, the generalization of our findings should be limited to city-dwelling older adults who are comfortable with technology such as using e-mail. Further, due to last minute changes in participant availability, the number of participants between the group meetings ranged from 3 to 9 attendees, whereas the optimal number is 5–8 (Krueger & Casey, 2009). Additionally, the research participants are AT users proficient in the use of e-mail, which could suggest that the participants in the sample are more comfortable with technology use than the general older adult population.

Conclusion

ATs have the potential to improve the quality of life, participation and health outcomes in older adults. However, existing technology evaluation tools are not centered around user priorities and experiences. To address this gap, the present study investigated what selection and use criteria are important for older adult AT users. Our findings suggest that criteria such as promotion of independence, affordability, ease of using and learning to use the AT, reliability and ethics are of the utmost importance for older adults. This study identified specific priorities of older adults, as well as their concerns about technology use. The quantitative findings are contextualized with in-depth qualitative data which will inform future AT evaluation tool development.

Author contributions

K.K., W.C.M., and J.M.R. – conception and design of the study.

K.K., M.T., O.E., J.M.R. – conducting of the nominal group meetings.

K.K., K.V., J.M.R. – data analysis and interpretation.

All authors took part in revising the work, approving the final version for publication and agree to be accountable for all aspects of the work.

Statement of ethics

This study protocol was reviewed and approved by the University of British Columbia Behavioral Ethics Board, study number H18–02825. Written informed consent to participate in the study was obtained from all participants.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are not publicly available due to their containing information that could compromise the privacy of research participants. Please contact the Corresponding Author (J.M.R.) for further information.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/10400435.2022.2113180.

Additional information

Funding

This work was supported by AGE-WELL NCE Inc [grant no. - AWCAT 2018-121] and by the Canadian Consortium on Neurodegeneration in Aging’s Ethical, Legal and Social Implications Cross-cutting Program.