Formative usability evaluation of a fixed-dose pen-injector platform device

Abstract

Background

This article for the first time presents a formative usability study of a fixed-dose pen injector platform device used for the subcutaneous delivery of biopharmaceuticals, primarily for self-administration by the patient. The study was conducted with a user population of both naïve and experienced users across a range of ages. The goals of the study were to evaluate whether users could use the devices safely and effectively relying on the instructions for use (IFU) for guidance, as well as to benchmark the device against another similar injector established in the market. Further objectives were to capture any usability issues and obtain participants’ subjective ratings on the properties and performance of both devices.

Methods

A total of 20 participants in three groups studied the IFU and performed simulated injections into an injection pad.

Results

All participants were able to use the device successfully. The device was well appreciated by all users with, maximum usability feedback scores reported by 90% or more on handling forces and device feedback, and by 85% or more on fit and grip of the device. The presence of clear audible and visible feedbacks upon successful loading of a dose and completion of injection was seen to be a significant improvement over the benchmark injector.

Conclusion

The observation that the platform device can be safely and efficiently used by all user groups provides confidence that the device and IFU in their current form will pass future summative testing in specific applications.

Keywords:

• human-factor engineering
• pen injector
• handling study
• user error
• instructions for use
• FixPen

Introduction

Pen injectors enable patients to treat themselves safely and conveniently with biopharmaceuticals through self-injection. Such devices, originally developed for use with insulin, have found widespread use across a variety of indications.^1,2 The increasing use of biological drugs that cannot be administered orally and the trend (for both cost and convenience reasons) toward patient self-administration mean that the use of such devices is expected to grow strongly in the future.² To meet the needs of different drug-dosing regimens, pen injectors with different operating principles have been developed. Variable-dose pens, where the user selects the required dose for each injection, is the largest category, with a multitude of different devices and solutions available on the market. Fixed-dose pens, where the same preset dose is delivered in every injection, is a smaller category, with only a limited number of options currently available.²

The development of safe and reliable medical devices, such as pen injectors, requires the application of knowledge of human capabilities and limitations to the design of artifacts, known as usability or human-factor engineering. The documented application of usability methods, including usability testing, throughout the design cycle is also required by regulatory authorities.^3–5 Typically, the usability-testing process during new-device development is divided into three parts. First, early formative studies are conducted during early development with the aim of providing user feedback iteratively to refine the device design and instructions for use (IFU). Late-stage formative tests are performed toward the end of development to confirm that the device is suitable for its intended use and likely to pass the usability part of design validation. Finally, summative testing, also known as usability validation, is carried out to provide objective evidence that the intended use has been met and that the device can be reliably and safely used by the intended user population.⁶

Several usability studies on pen injectors have been reported, ranging from formative work aimed at identifying and quantifying handling errors, studies focused on assessing task completion, late-stage formative studies to confirm overall safety and ease of use, to validation studies.^7–17 Although these studies provide important and valuable information on possible handling difficulties and usability issues, it is worth noting that all reported studies have dealt with variable-dose pen injectors. There is thus little information specifically related to fixed-dose pen injectors available in the literature.

This article presents a late-stage formative study of a fixed-dose pen-injector platform device. In order to provide a benchmark in terms of usability performance, a different fixed-dose pen injector available on the market was included in the study. Specifically, the study was designed and carried out with the aim of understanding whether the platform device and its IFU are suitable for users with different characteristics and whether the pen injector would be likely to pass future summative testing in specific indications.

Materials and methods

Objectives

In this late-stage formative study, the primary objective was to evaluate whether participants could use the FixPen™ fixed-dose platform prototype devices safely and effectively relying on the IFU as only support and to benchmark the device against a fixed-dose pen injector available in the market. Further objectives were to capture usability issues that may affect user performance, preference, and satisfaction and to obtain participants’ subjective ratings on properties and performance of the device.

Compliance with ethics guidelines

The study was conducted in accordance with Ypsomed AG’s standard procedures, including internal review of the study protocol and supervision of the study by medical services. The study complies with the principles laid out in the User Experience Professionals Association Code of Professional Conduct and the European Pharmaceutical Market Research Association Code of Conduct. As a simulated-use study, it did not require approval by an ethics committee.

Study devices and materials

Two devices were included in the study. The FixPen (Ypsomed AG, Burgdorf, Switzerland) is a fixed-dose disposable pen injector intended for the subcutaneous self-injection of drugs in the context of various treatments requiring relatively frequent (daily or every 2–3 days) injections of a fixed dose.¹⁸ The device contains a 3 mL cartridge and works according to the push–pull principle (pull to load a dose, push to inject). It is manually operated and designed with a gearing mechanism to optimize injection force and dose-button extension. It also provides audible feedback (“clicks” after loading a dose and completion of injection) and visible feedback (viewing window) to the user during the injection. It is presented in Figure 1A. The Forteo™ pen (Lilly France S.A.S, Fegersheim, France) injector is a device used for the delivery of teriparatide to patients suffering from osteoporosis.¹⁹ It was introduced to the market in 2008, and has similar characteristics in terms of cartridge type, injection volume, and user interface to the FixPen, with the difference that it neither has any window indicating the status of the injection nor features an audible “click” signal after completed loading of a dose. Neither of the two pens in the study requires priming before injection of the first dose. The Forteo pen is presented in Figure 1B. The Forteo pen was selected over the only other fixed-dose pen currently on the market (the Lyxumia pen, introduced in 2013) because of the Lyxumia’s shorter use history and very limited market penetration.

Figure 1 The FixPen (A) and Forteo (B) disposable pen injectors used in the study.

Notes: For the FixPen, a neutral label was used; for the Forteo, the product label was removed.

For the simulated injections, standard 31 G 6 mm injection-pen needles (Mylife ClickFine; Ypsomed AG, Burgdorf, Switzerland), water-filled 3 mL insulin-type cartridges (Sanofi, Paris, France), and injection pads (Ypsomed AG) were used. IFU were prepared in a similar format for the two devices to control for differences in learning effect. The tasks performed by the user to perform an injection are shown in Figure 2, and involved the actions of removing a protective cap, attaching a needle, loading a dose, injecting, holding for 5 seconds, removing the needle, and replacing the cap.

Figure 2 Tasks performed by the user during an injection.

Participants and groups

The FixPen has been developed as a platform device, implying that it may be used for different therapies. However, one important therapeutic area for this type of pen injector is known to be osteoporosis, which typically concerns postmenopausal women.^20,21 To reflect this situation, two user groups were defined in the study. The first user group comprised females aged ≥45 years, representing potential osteoporosis patients. As osteoporosis does not lead to any specific impairments directly relevant to using an injection pen, postmenopausal women in general (as opposed to patients suffering from osteoporosis) were selected as users. The target was to reach eight to ten participants in this group, with five as a minimum requirement. The second group consisted of male and female participants of various ages in two categories: either naïve with respect to device use or with previous experience of using pen injectors, representing potential users from other indications. In this group, the target number of participants and minimum requirement was five in each category. A minimum of five participants in each category to be evaluated is in line with current recommendations for this type of study.³

Procedure

The handling study followed standard protocols and procedures based on single-site visits. Trained staff (ie, one moderator) guided each study subject’s visit according to the procedure presented in Figure 3. Each subject first responded to a prestudy questionnaire, and then studied the IFU before performing a simulated injection into the pad situated on a desk in front of them and answering questions on the injection and the device. Having completed the three steps with the first pen, the subject then repeated them with the second pen. Subjects were randomized to test the two pens in a crossover setup to control for order, as well as learning effects. Each study subject thus performed two simulated injections in total (one with each pen). Each simulated injection was video-recorded and followed by the moderator, who was present at all times. During the injections, the moderator observed and recorded any usability issues, consisting of use errors or near misses, and intervened as soon as the study subject experienced difficulties. The set of questions at the end of each injection involved asking the subjects to rate the performance and ease of use of the study device on a 4-point Likert scale.

Figure 3 Sequence of events during the handling tests.

Abbreviations: IFU, instructions for use.

Results

Participants

The study-participant population is characterized in Table 1. A total of 20 participants were recruited and participated. The target number of participants (ten for group 1 and five each for groups 2A and 2B) was reached in all groups. Overall, 80% of the participants were female, 25% had previous experience using pen injectors, and 5% reported suffering from some kind of impairment. The mean age of the participants was 50 years, with a range of 21–74 years.

Table 1 Participant characteristics

Group	n	Sex	Age, years	Handedness	Pen experience	Self-reported impairments
1. Females >45 years of age	10	10 female	Mean 56, range 45–74	10 right-handed	All naïve	One with early arthritis
2A. Naïve users	5	3 male, 2 female	Mean 44, range 21–71	4 right-handed, 1 left-handed	All naïve	None
2B. Experienced users	5	1 male, 4 female	Mean 44, range 25–68	4 right-handed, 1 left-handed	All experienced	None

Injection success and observed user errors

The initial injection-success rate, defined as tasks 1–5 being successfully completed without any intervention by the moderator (holding after injection was not considered critical to success), was 90% for both pens across all users. All users were ultimately successful in using both devices. The initial success rates per user group are presented in Figure 4. The highest initial success rate was achieved in group 1 and the lowest in group 2B. There were no near misses. An overview of all observed use errors for all tasks is provided in Table 2. A total of 27 errors were seen for the 40 injections, corresponding to an overall error rate of 0.68 errors per injection. The needle-removal task was by far the largest source of user error, responsible for close to 60% of all observed errors. The tasks of loading the dose and holding after injection were the next major contributors, together responsible for about 30% of all errors. The two study devices exhibited comparable error rates (0.70 for the FixPen and 0.65 for the Forteo pen), and given the similarity in rates and handling sequence, the results for the two pens were pooled to look at learning effects. The data for the first and second injections in Table 2 show that except for the errors in loading the dose, there was no discernable reduction in error rate from the first to the second injection. The highest error rates were seen in group 1 and the lowest in group 2B, with experienced users in the latter group making less than half the errors of the females >45 years of age in the former group.

Figure 4 Initial injection-success rate.

Notes: Success rate defined as the fraction of injections where tasks 1–5 (cap removal, needle attachment, dose loading, needle insertion, injection) were completed by the user without any involvement by the moderator. All injections were ultimately successful. Group 1, females >45 years of age; group 2A, naïve users; group 2B, experienced users.

Table 2 Observed user errors

User task	Group 1: females >45 years of age (n=10)		Group 2A: naïve users (n=5)		Group 2B: experienced users (n=5)		Overall (n=20)
	FixPen (n=10)	Forteo (n=10)	FixPen (n=5)	Forteo (n=5)	FixPen (n=5)	Forteo (n=5)	FixPen (n=20)	Forteo (n=20)	First injection, both pens (n=20)	Second injection, both pens (n=20)
1. Identify and remove cap	0	0	0	0	0	0	0	0	0	0
2. Attach needle	0	0	0	1	0	0	0	1	1	0
3. Load dose	1	0	0	0	1	1	2	1	3	0
4. Insert needle	0	0	0	0	0	0	0	0	0	0
5. Inject dose	0	0	0	0	0	0	0	0	0	0
6. Hold	2	0	1	1	0	0	3	1	2	2
7. Remove pen	0	0	0	0	0	0	0	0	0	0
8. Remove needle	6	5	1	2	1	1	8	8	7	9
9. Attach cap	1	1	0	1	0	0	1	2	1	2
Sum over all steps	10	6	2	5	2	2	14	13	14	13
Rate (occurrence, n)	1.0	0.6	0.4	1.0	0.4	0.4	0.70	0.65	0.70	0.65

User feedback

Figure 5 presents self-reported data on convenience in using the device. Both devices were well appreciated by the subjects. The FixPen was given the highest score by ≥85% of subjects on the different convenience aspects, whereas the Forteo pen received the highest score from ≥70% or more. The FixPen received higher ratings on overall attractiveness and injection feedback (certainty that the dose had been loaded and injected), whereas the Forteo pen was given a higher rating for the grip of the dose button. The acceptability of the force to load the dose was given somewhat higher ratings for the FixPen over the Forteo Pen, whereas the acceptability of the injection force was given the highest ratings by all subjects for both pens. The perceived feedback on completion of injection was examined separately, as presented in Figure 6. For the Forteo pen, subjects reported relying mainly on the position of the dose button, whereas for the FixPen the audible click and the display window were used as additional sources of feedback from the device.

Figure 5 User-feedback scores on usability questions asked at the end of the handling test.

Notes: Users self-reported convenience on a scale of 1–4, where 4 represented the highest degree of convenience. Each bar provides a breakdown of the percentage of users assigning a certain agreement score to a question, with the absolute number of users per score given as a numeral.

Figure 6 User feedback on which pen feature indicated that the injection had been completed.

Notes: Results combined for all user groups (n=20).

Discussion

There were four user errors made during tasks 1–5 leading to an intervention by the moderator and thus to the injection being considered as initially unsuccessful. One subject experienced difficulties attaching the needle, two subjects initially forgot to load the dose, and one subject only partially loaded the dose before attempting to inject. As the moderator intervened as soon as the subjects started having difficulties, rather than waiting until he or she proved unable to continue, it is not known which user errors would have led to an injection failure in the absence of any moderator intervention. However, the fact that the moderator interventions for the errors related to needle attachment, and forgetting to load the dose consisted of merely referring the subject to the IFU, is taken as an indication that these participants eventually would have been able to complete the tasks unassisted and thus achieve a successful injection. The subject who only partially loaded the dose was directly assisted with this task by the moderator. It is believed to have been less likely that this subject would have been capable of completing the task alone.

It is worth noting that the lowest rates of initial injection success were seen with the experienced users in group 2B, whereas the injection-naïve females in group 1 had the highest success rates. The relatively low rates achieved by the experienced users in group 2B was somewhat counterintuitive, and is believed to be due to the fact that they were seen to spend less time studying the IFU before the injections and more likely to adhere to ingrained behavior compared to the other user groups.⁷

A total of 27 user errors were observed, most of which concerned needle handling (17 errors). This is in line with previous studies finding needle handling to be the main source of user error.⁷ As the needle handling and interface between the pen and the needle are standardized, such errors are by definition unrelated to the design of the individual device. Holding after injection was the second-largest source of user error (four errors). This task is known to be a major error source, and it is widely acknowledged that users often have difficulties respecting a predefined holding time.²⁰ Loading the dose was associated with three user errors: two with the FixPen and one with the Forteo pen. The FixPen errors were due to the subjects forgetting to load the dose, whereas for the Forteo pen the error was an incomplete loading of the dose. The former errors may well have been related to the IFU, whereas the latter, which was the one requiring direct assistance by the moderator, could possibly have been related to the mechanism of the pen. The Forteo pen continuously emits audible clicks during dose loading, and thus does not directly indicate a complete loading of the dose, whereas the FixPen instead clicks once the dose has been completely loaded.

With regard to user error rates, it is useful to compare the results from the present study with rates reported in the literature. The present study saw rates of 0.70 errors per injection for the first injection and 0.65 for the second injection. This can be compared with work on a variable-dose insulin-type pen, where rates of 2.39 for the first and 1.94 for second injection were found, and variable-dose pen injectors for infertility treatment and diabetes, reporting error rates of 1.75 and 1.85, respectively.^7,8,10 The handling of variable-dose pens is more complex than for the devices in the current study in that such pens typically require a priming step and selection of the dose before injection. At the simpler end of the spectrum of handling complexity, a study on an autoinjector requiring neither needle attachment nor loading of the dose reported error rates of 0.53 for the first injection and 0.14 for the second injection.²² It thus appears that the error rates seen in the current study fall neatly in between those observed for devices requiring more complex handling and those reported from devices with fewer and simpler handling steps.

The user feedback on performance was overall very positive with high ratings on all usability questions asked for both devices. The FixPen received somewhat higher ratings than the Forteo pen on general attractiveness, how well the device fit in the user’s hand, and feedback on dose loading and injection completion. The Forteo pen was rated somewhat more highly than the FixPen on grip of the dose button. These differences in rating can most likely be attributed to the differences in design between the two devices. The FixPen is smaller, and the Forteo pen has a large mushroom-shaped dose button. Furthermore, the Fix-Pen has a window indicating the progress on dose loading and injection not present in the Forteo pen (see Figure 1). It also provides audible feedback of injection completion not provided by the Forteo pen. As further indicated by the scores on the two devices reported in Figure 6, it thus appears that these distinct design features of the FixPen are well appreciated by users.

Conclusion

Both study devices were able to be successfully used by all study subjects. The observed rates of user error were similar for both devices and in line with the results reported previously for other disposable dial-and-dose systems. Most user errors concerned needle handling, with holding after injection being the second-largest source of errors. Both findings are consistent with other results reported in the literature.

The FixPen was well appreciated by all users, with maximum usability feedback scores reported by ≥90% on handling forces and device feedback, and by ≥85% on fit and grip of the device. The presence of clear audible and visible feedback on completion of injection in the FixPen was seen to be a significant improvement over the Forteo pen injector. The observation that the FixPen can be safely and efficiently used by all user groups provides confidence that the device and IFU in their current form will pass future summative testing in specific applications.

Acknowledgments

This study was conducted with funding from Ypsomed AG.

Disclosure

Both authors work for Ypsomed AG. The authors report no other conflicts of interest in this work.