Pre-ANDA strategy and Human Factors activities to de-risk pharmaceutical companies ANDA submission of drug–device combination products: case study of a formative Comparative Use Human Factors study

ABSTRACT

Background

This article presents a strategy that a Drug Delivery Device Developer (DDDD) has adopted to support Abbreviated New Drug Application (ANDA) submissions of drug–device combination products. As per the related FDA guidance, a threshold analysis should be compiled. If ‘other differences’ between the Reference Listed Drug (RLD) and the generic drug devices are identified, a Comparative Use Human Factors (CUHF) study may be requested.

Methods

The DDDD performed task analysis and physical comparison to assess the pen injector design differences. Then, a formative CUHF study with 25 participants simulating injections using both RLD and the generic pen injectors was conducted.

Results

After each participant completed four simulated injections, similar type and rates of use error between the RLD (0.70) and generic (0.68) pen injectors were observed.

Conclusion

DDDDs can support pharmaceutical companies in the ANDA submission strategy of their drug–device combination product by initiating comparative task analysis and physical comparison of the device as inputs for the threshold analysis. If ‘other differences’ are identified, a formative CUHF study can be performed. As shown in our case study, this approach can be leveraged to support the sample size calculation and non-inferiority margin determination for a CUHF study with the final combination product.

KEYWORDS:

• Comparative use human Factors
• drug delivery system
• generic
• ANDA
• combination product
• self-administration

1. Introduction

When looking to bring a drug to the US market, two common pathways to obtain Food and Drug Administration (FDA) Approval are through a New Drug Application (NDA) or an Abbreviated New Drug Application (ANDA). The NDA pathway is the pathway for novel drugs that have not been previously studied or assessed by FDA to confirm its safety and efficacy for a given indication, while an ANDA is deemed an ‘abbreviated’ NDA due to its reliance on FDA’s previous finding that a novel drug is safe and effective in obtaining its intended purpose. For an NDA to demonstrate a drug’s safety and efficacy, extensive data, which can take 10 years or longer to accumulate, are required. This may include, but is not limited to, preclinical (animal) and clinical (human) studies. In contrast, regulatory files for generic drugs reviewed through the ANDA pathway are ‘abbreviated’ because they rely on the innovator’s preclinical and clinical data. Therefore, reviews of ANDAs typically yield both a shorter development timeline and quicker regulatory review, even if it may vary depending on the complexity of the drug and the availability of information [1].

An NDA commonly introduces a New Chemical Entity (NCE) or a new indication of an existing chemical entity to the market. Pharmaceutical companies will commonly apply for patents with the U.S Patent and Trademark Office (USPTO) to secure market exclusivity. By obtaining patent protection and NCE exclusivity, pharmaceutical companies are protected from generic competition until their expiration date [2]. In the time leading up to that period, generic pharmaceutical companies can start filing for an ANDA given that they have met all the requirements set forth by FDA and the USPTO offices and agree not to commercialize prior to the exclusivity and patent protection period. Therefore, once the window of exclusivity ends, generics can enter on the market. The main purpose of this exclusivity period is to promote a balance between the new drug innovation and generic drug competition. This balance is due to the cost of developing the drug. Estimates for developing a new drug and bringing it to market range from $314 million to $2.8 billion [3]. In comparison, placing a generic drug on the market reduces costs [4,5]. Upon the expiration of the exclusivity period, it is common to see an increased number of generics enter the market [6–8].

A generic drug is defined by the FDA as a product with identical active ingredients, dosage forms, strengths, labeling, and routes of administration, as well as evidence of the drug’s bioavailability, which is a crucial factor in demonstrating bioequivalence between a generic drug and its Reference Listed Drug (RLD) [9]. Bioequivalence studies are studies where two drugs are compared to show that they have nearly equal bioavailability and pharmacokinetic and pharmacodynamic parameters.

Drugs, whether approved via the NDA or ANDA pathway are further classified by the route of administration and dosage forms. Additionally, drugs can also be combined with a device to achieve its therapeutical effect and be classified by FDA as a drug–device combination product. These products offer alternative ways to administer drugs and deliver treatment. Generic medicines include drug–device combination products and refer to pharmaceutical products that combine generic drugs with delivery devices. The FDA issued a draft guidance in 2017 intended to serve as a foundational guidance to assist potential ANDA applicants in the submission of generic drug–device combination products [10]. Besides assessing the therapeutic bioequivalence, FDA recommends evaluating whether the end-user can use the generic combination product when it is substituted for the RLD without additional intervention of a health care provider and/or without additional training prior to use. Therefore, in the early stages of development, potential applicants should carefully consider the design of the user interface of a proposed generic combination product and seek to minimize differences from the user interface for the RLD.

This paper presents a pre-ANDA strategy to de-risk potential differences in the user interface between the generic pen injector and its RLD pen injector. The expertise of the Drug Delivery Device Developer (DDDD) not only helps to identify any potential use-related risks associated with a significant difference, but also assists in designing and conducting a formative Comparative Use Human Factors (CUHF) study. Product knowledge and data gathered during the formative CUHF study can be used to help determine an appropriate sample size with statistical significance as well as help provide supporting information for the submission package and review process.

A case study is reported in this article, presenting preliminary data collection in preparation of an ANDA submission for a potential generic pen injector and its RLD pen injector (Figure 1(a)).

Figure 1. (a). Generic pen injector user interface. (b). Dose Set Knob on generic pen injector extends outward with rotation dose set knob and ‘mg’ marking inside the dose window.

2. Participants and methods

2.1. Products description

The BD Vystra™ disposable pen injector is a customizable disposable pen injector with a modular design to support a wide range of therapies that require frequent, low volume (up to 3 mL) injections or variable dosing. Therapeutic applications may include diabetes, obesity, osteoporosis, and growth hormone deficiency. This pen is intended for use by patients who may suffer from the above-mentioned conditions, as well as by their caregivers or healthcare practitioners. The user interface elements of this pen injector which include a body, a cartridge holder, a prefilled drug cartridge and a cap are detailed in Figure 1(a).

The comparator RLD, a disposable pen injector, is intended to administer Liraglutide 3.0, a glucagon-like peptide-1 receptor agonist medication for chronic weight management (Saxenda®, NovoNordisk, US FDA Approval on 23 December 2014). The intended patient population for Liraglutide 3.0 are:

• Adults with Body Mass Index (BMI) ≥ 27 kg/m² and weight-related medical problems

• Adults with BMI ≥30 kg/m²

• Children aged 12-17 years with a body weight above 132 pounds (60 kg) and obesity.

The Liraglutide 3.0 RLD pen injector is composed of a body, a cartridge holder, a prefilled drug cartridge, and a cap.

2.2. Threshold analysis – comparative analysis

Threshold comparative user interface analyses are performed based on:

1. Label comparison, which consists of side-by-side, line-by-line comparison of the full prescribing information, IFUs (Instructions For Use), and descriptions of the delivery device constituent parts

2. Comparative task analysis, where the sequence of use of RLD and generic pen injector are compared

3. Physical comparison, where audible, tactile and visual parameters of both RLD and generic pen injector will be compared [10].

The threshold analysis can have three possible outcomes:

1. No design differences

2. Minor design differences where differences in user interface do not affect an external critical design attribute

3. Other design differences where threshold analyses suggest that differences may impact an external critical design attribute that involves administration of the product [10].

If the three comparative aspects yield ‘no design difference’ the threshold analysis is sufficient for the ANDA submission. However, if ‘other design differences' are identified, additional Human Factors data may be required, such as data from a CUHF study with the final finished combination product.

The DDDD can proactively support the final drug–device combination product threshold analysis by performing two of the three threshold analyses comparisons required with the related pen injector constituent part (drug-independent): the task analysis, and the physical pen injector comparison between the generic pen injector and the RLD pen injector. This strategy is slightly different from a full threshold analysis; therefore, such comparison will be referred to as Comparative Analysis in this paper to avoid confusion. Similar to the threshold analysis, the comparative analysis outcomes can be:

1. no design difference

2. minor design difference

3. other design difference.

2.3. Formative Comparative Use Human Factors (CUHF) study

A non-interventional, randomized, single-session, simulated, comparative usability study was conducted between the generic pen injector and Liraglutide 3.0 RLD pen injector in 2022. The protocol was reviewed and approved by an Institutional Review Board (Institutional Review Board of Core Human Factors, Inc., registered with the U.S. Department of Health and Human Services).

2.3.1. Human Factors study participants

Patients, caregivers, and Health Care Professionals (HCP) were recruited, for the formative CUHF study sessions – conducted in Fort Lee, New Jersey, U.S.A.. To be eligible for inclusion, all participants were to be:

• US residents

• aged >21 years

• without evidence of acute pathology (such as fever, pain, inflammation)

• willing and able to provide valid informed consent to participate and agree on conditions defined in the informed consent form, to consent to collection of personal information including audio and video records of their participation in the study

• able to read and speak English clearly.

In addition to be eligible, patients had to:

• have a BMI of 18.5 or higher

• meet one of the three experience-related criteria:

  1. no experience with self-injection or partner and/or pets administration with a pen injector

  2. at least two years of experience with self-injection using a pen injector, other than Liraglutide 3.0 pen injector

  3. at least two years of experience with self-injection using Liraglutide 3.0 and pen injector.

Caregivers to be enrolled in the study:

• were not to be professionally trained, currently caring for someone with chronic disease or obesity

• had to meet one of the three experience-related criteria:

  1. experienced with self-injections or partner-injection with a pen injector

  2. at least two years of experience giving an injection with a pen injector, other than Liraglutide 3.0 pen injector

  3. at least two years of experience giving an injection with Liraglutide 3.0 mg RLD pen injector.

The inclusion criteria for HCP:

• were to be professionally trained and certified nurses, currently providing care for patients in-home, or in alternative care settings (i.e. nursing homes, long-term care or rehab facilities)

• had to meet one of the three experience-related criteria:

  1. with at least 2 years of experience in giving injections with pen injector, other than Liraglutide 3.0 RLD pen injector

  2. with at least 2 years of experience in giving injections with Liraglutide 3.0 RLD pen injector.

Non-Disclosure Agreement, Institutional review board approved consent form, as well as complete demographics forms were obtained from participants prior starting the session.

2.3.2. Study site

The study was conducted at one study site in the US (Fort Lee, NJ).

2.3.3. Study design

The study included participants divided into three distinct groups of intended users: patients, caregivers, and HCP.

Participants were randomized to start with either generic pen injector or Liraglutide 3.0 RLD pen injector. In addition, injection doses for both pen injectors were randomized (1.8 mg or 3.0 mg) (Supplementary File 1). All the simulated injections were performed into a foam injection pad. Both pen injectors were provided with Novo Nordisk NovoFine® Needles.

As part of evaluating drug products for safety and effectiveness, FDA will also consider whether the end-user can use the generic drug–device combination product when it is substituted for the RLD without any additional intervention by the health care provider and/or additional training prior to use.

Thus, no training was provided to participants. IFU for each pen injector was provided to all participants who were free to read it at any time during each simulated injection (Supplementary File 2). For the formative CUHF study, generic pen injector’s cartridges were filled with 3 mL of saline 0.9% NaCl (same viscosity as Liraglutide 3.0 RLD) and RLD pen injectors were filled with 3 mL of Liraglutide 3.0 mg.

This study was conducted in a simulated home environment. HCP and caregivers simulated the injections into foam pads attached to the abdomen or thigh of a manikin, seated in a chair. Patients simulated the injections into foam pads, secured to their body (abdomen or thigh) with a Velcro® strap.

After a brief introduction, participants were asked to complete the tasks appropriate to their role. The study moderator recorded the completion rates in compliance with the IFUs, noted all comments, usability performance, and number of times participants required assistance to use the pen injector appropriately.

Each participant performed two simulated injections: one at 1.8 mg and one at maximum dose (3.0 mg) with the first pen and two additional simulated injections: one at 1.8 mg and one at maximum dose (3.0 mg) with the second pen as per the full operating sequence (Supplementary File 3).

After completing four injections, root cause analysis of use errors/difficulties that occurred on critical tasks was performed by the moderator during the debriefing with the participants. Root cause analysis was performed at the session end, after the four simulated injections, to understand why the assessed task(s) was/were not performed as per the IFU. This root cause analysis was performed by a third-party moderator with a strong Human Factors expertise and trained on the use of both pens. This root cause analysis was done to:

• Ensure all foreseeable risks have been taken into account in the DDDD’s risk management file and that no unanticipated risk has been identified

• Determine any link between failure and other differences in design between both pens.

2.3.4. Study endpoints assessment

2.3.4.1. Usability: task assessment

As per FDA guidance, a critical task is a task that would or could cause serious harm to the patient or user in case of use error or failure where harm is defined to include compromised medical care. Essential tasks are the ones necessary for successful use of the product and associated with high risk but resulting in severity level below the one of critical tasks [11,12].

The usability of the pen injectors was evaluated for the following critical tasks impacted by a difference in design between the generic and the RLD pen injectors (Supplementary File 2):

• (i) ‘Rotate dose dialer to set dose to priming symbol’

• (ii) ‘Needle pointing up, press on the center of thumb button until it stops. Dose dialer will click as it counts down to zero’ (hereafter referred to as (ii) ‘Prime the pen’)

• (iii) ‘Rotate dose dialer until the proper dose aligns with the pointer in Dose window’

• (iv) ‘Insert needle into injection location and press thumb until “0” aligns with pointer’ (hereafter referred to as (v) ‘deliver the dose’).

For each simulation and each evaluated critical task, the moderator recorded successes with and without operational difficulties, close calls, use errors, assistance requested by participant and moderator intervention. Usability data is described as categorical data (N), and proportion frequencies (%). For each pen injector, generic pen injector and Liraglutide 3.0 RLD pen injector, the use error rates per participant were calculated for all four tasks assessed.

2.3.4.2. Usability analysis

Rates of use errors, success with operational difficulties, close calls, along with root cause analysis of the reason why the error occurred were considered for data analysis and data comparison.

2.3.4.3. Safety assessment

All adverse events and technical incidents were reported and qualitatively described.

2.3.4.4. Use error rates calculation

For each pen injector, all participants, for whom at least one-use error was observed during a critical task, were considered to have failed to successfully use the pen. Therefore, the number of failures over the two simulated injections was calculated with the generic and Liraglutide 3.0 RLD pen injector.

2.3.4.5. Within-subject correlation

Within subject correlation was performed, calculating the correlation between the usability results with both pen injectors for all the simulated injection of the same subjects. The within-subject correlation represents the percentage of equivalence between the generic and Liraglutide 3.0 RLD pen injectors, meaning the percentage of participants with the same usability results (either success or failure) with both pens. The within-subject correlation ($\mathit{x}$) was calculated as follows:

$\mathit{x}=\frac{\mathit{usability}\mathit{results}\mathit{with}\mathit{generic}\mathit{pen}=\mathit{usability}\mathit{results}\mathit{with}\mathit{RLD}\mathit{pen}}{\mathit{Number}\mathit{total}\mathit{of}\mathit{individus}}$

2.3.4.6. Inputs for final CUHF study done with final combined product.

Based on the formative CUHF results, the sample size and power calculation were calculated using software PASS (version 19.0.10) according to the FDA guidance (section ‘Sample size considerations’) [10].

3. Results

3.1. Comparative analysis

Similarities between the Liraglutide 3.0 RLD pen injector and generic pen injector were reported. However, some ‘other design differences’ were highlighted, due to the design differences observed in the dose set knob extension (Figure 1(b)) of the generic pen injector as compared to the Liraglutide 3.0 RLD pen injector. These differences may impact the following critical tasks: ‘(i) Set 2 units,’ ‘(ii) Prime the pen,’ ‘(iii) Set the correct dose,’ and ‘(iv) Inject the medication.’ The observed differences may result in risk of underdosing or delayed treatment. The details of the other differences, critical tasks impacted, and potential user risks are summarized in Table 1.

Table 1. Summary of ‘other differences’.

Observed difference(s)	Physical Attributes involved in the difference	Critical task impacted	Potential user risk(s)
Design mechanism is different (Generics’ dose set knob extends outward along with rotation while RLD pen injector only allows rotation).	Dose Dialer of the generic pen injector extends outward while dialing the dose: Dose dialer of the Liraglutide 3.0 RLD pen injector is fixed and does not extend.	Prime the pen	May lead to a difference in user perception for former users of RLD pen injector which could lead to a delayed treatment.
The generic pen injector has a different design mechanism where the dose set knob extends outward with rotation while the RLD pen injector only allows rotation. While turning the dose set knob, a dose may be aligned with the wrong side of the pointer, outside the dose window.	Dose marking of the generic pen injector may be misaligned while the dose set knob is extending outward during the dose dialing:	Set the correct dose	May lead to an under- or over-dosing.
For the RLD pen, the ‘mg’ mark is directly printed on the pen body, while it is printed on the dose set knob for the generic pen injector.	Dose counter marking ‘mg’ of the generic pen injector:	Set the correct dose	User may think to align the arrow with the ‘mg’ mark instead of the dose, leading to under-dosing.
Design mechanism is different because the generic’s dose set knob extends outward along with the rotation, while the RLD pen injector only allows rotation.	Distance to reach the button is higher with the generic pen injector because the Dose Dialer of the generic pen injector extends outward while dialing the dose: Dose dialer of the Liraglutide 3.0 RLD pen injector is fixed and does not extend.	Inject the medication	Possibly increased use error risk especially because of the difference in pen design for high injection dose units as the user may need to stretch thumb to reach the injection button with the generic pen injector.

The results of this comparative analysis between generic pen injector and its RLD can be provided to pharmaceutical companies and can be used as input and leveraged in pharmaceutical companies’ final threshold analysis. Thus, pharmaceutical companies will only need to perform the first requirement of the threshold analysis which consists of completing side-by-side, line-by-line labeling comparison between their generic labeling and that of the Liraglutide 3.0 RLD. The intent of this action is to ensure that pharmaceutical companies create the generic packaging and labels are as close as possible to the RLDs.

3.2. Formative Comparative Use Human Factors study (CUHF)

Because the comparative analysis highlighted ‘other design differences’ (Table 1) in the design of the user interface of the proposed generic pen as compared to the Liraglutide 3.0 RLD (dose set knob extension – Figure 1(b)), FDA may request that applicants provide additional data such as data from a CUHF study. Such CUHF study aims to assess if ‘other design difference’ in user interface may introduce a new risk that may impact the safety profile of the generic pen injector compared to the Liraglutide 3.0 RLD pen injector. A key aspect to warrant a successful study is to have a documented strategy as well as ensuring good methodological inputs (e.g. sample size and study design).

As such, DDDD can perform a formative CUHF study with the generic pen injector filled with placebo to assess the use error rate of both products on critical tasks impacted by a difference. Although not required by the FDA guidance for ANDA submission, this study has been done as per Human Factors engineering guidance including 5–8 users per user group and assessing use error rate but also root cause analysis to ensure no new risks with the use of the pen injector are identified [11,13].

3.3. Formative Comparative Use Human Factors study (CUHF)

3.3.1. Participants

The demographic characteristics of all the participants who took part in the study are summarized in Table 2.

Table 2. Demographic characteristics of the participants.

Participant Characteristic		Patients	Caregivers	HCPs	Total
Sex	Male	4 (44.4%)	4 (57.1%)	0 (0%)	8 (32%)
	Female	5 (55.6%)	3 (42.9%)	9 (100%)	17 (68%)
Dominant Hand	Right	5 (55.6%)	5 (71.4%)	6 (66.7%)	16 (64%)
	Left	3 (33.3%)	2 (28.6%)	3 (33.3%)	8 (32%)
	Ambidextrous	1 (11.1%)	0 (0%)	0 (0%)	1 (4%)
Injection Experience	Experienced with RLD pen	2 (22.2%)	0 (0%)	4 (44.4%)	6 (24%)
	Experienced with other pens	2 (22.2%)	4 (57.1%)	5 (55.6%)	11 (44%)
	Naïve	5 (55.6%)	3 (42.9%)	0 (0%)	8 (32%)
Age (years)	21–35	3 (33.3%)	2 (28.6%)	3 (33.4%)	8 (32%)
	36–50	3 (33.4%)	2 (28.6%)	2 (22.2%)	7 (28%)
	51–65	3 (33.3%)	3 (42.8%)	3 (33.3%)	9 (36%)
	66+	0 (0%)	0 (0%)	1 (11.1%)	1 (4%)

A total of 25 participants, which included naïve patients, experienced patients with Liraglutide 3.0 RLD pen injector or other pen injectors, naïve caregivers, experienced caregivers with other pen injectors and HCP (experienced with either the RLD or other pen injectors) were included in the formative study.

3.3.2. Usability: task assessment

3.3.2.1. Check the medicine flow of the pen

When using a pen injector, after the needle attachment, a priming step prior to injecting the medication is required to check the flow to ensure the needle is not blocked and ensure that the full dose will be delivered to the patient. Skipping the priming step may result in delayed treatment or underdosing [14]. For both generic and Liraglutide 3.0 RLD pen injectors, to prime the pen, users must first align the pointer to priming symbol (present on both the generic pen injector and the Liraglutide 3.0 RLD pen injector at ‘2’ units). The second step to be performed consists of pointing the pen injector with the needle up and inject the two units. The presence of a drop indicates that the pen is properly primed, while the absence of drop will require repeating the two priming steps.

3.3.2.1.1. Rotate dose dialer to set dose to the priming symbol

Across the two simulated injections performed with each pen injector, for the task ‘Rotate the dose dialer to set dose to the priming symbol’ (e.g. ‘2’ units”), 60% (N = 30) of use errors were recorded with the generic pen injector and 64% (N = 32) with the RLD pen injector. Four use errors were due to participants who forgot to set the flow check with both the Liraglutide 3.0 RLD and generic pen injectors, meaning they were aware that a pen needs to be primed but forgot to. The other use errors (N = 58) occurred when the users misunderstood they had to prime and went directly to the injection step without knowing that it was necessary to prime the pen prior the first use (Figure 2).

Figure 2. Usability assessment of generic and RLD pens for the critical tasks impacted by other design differences. Success, success with operational difficulty (SOD), close call, and use error rates are indicated for the generic and RLD pen injectors. N number of simulated injections.

An analysis per user group showed that for the generic pen injector, the rate of use errors for the caregiver group (71%, N = 10/14) was higher than for HCP (55%, N = 10/18) and patients (55%, N = 10/18) (Figure 3(a)). With the Liraglutide 3.0 RLD pen injector, the rate of use errors for the patient group (78%; N = 14/18) was higher than for caregivers (64%; N = 9/14) and HCP (50%; N = 9/18).

Figure 3. Usability assessment with generic and RLD pens per user group: (a) Task ‘Rotate dose dialer to set dose to 2’ per user group; (b) Task ‘Rotate dose dialer to set dose to 2’ per user group per user experience with pen injector (caregivers and patients); (c) Task ‘Prime the pen’ per user group and (d) Task ‘Prime the pen’ per user experience with pen injector (caregivers and patients). Success, close call, and use error rates are indicated for the generic and RLD pen injectors. N number of simulated injections.

Having prior experience with a pen injector resulted in improved performance when rotating the dose dialer to set dose to ‘2’ units” (Figure 3(b)). Including both patients and caregivers, naïve participants without experience with any pen, failed to set the dose to 2 units in 71% (N = 10/14) of the simulated injections with generic pen injector and 93% (N = 13/14) with the Liraglutide 3.0 RLD pen injector. In contrast, experienced caregivers and patients failed to set the dose to 2 units in 55% (N = 10/18) of simulated injections for both pen injectors.

3.3.2.1.2. Prime the pen

Across the two simulated injections performed with each pen injector, for the task ‘Prime the pen,’ 54% (N = 27) of use errors were recorded with the generic pen injector and 60% (N = 30) with the RLD pen injector. A total of five use errors out of the 27 use errors for the generic pen injector and six use errors out of 30 use errors for the Liraglutide 3.0 RLD pen injector occurred because the participants forgot to perform the priming of the pen (Figure 2). The other use errors occurred when the users misunderstood the sequence of the task, not knowing that the pens had to be primed prior first use and omitting doing the priming before simulating the injection.

The number of use errors were almost equivalent across the three user groups when using the generic pen injector (57%, N = 8/14 for caregivers, 55%, N = 10/18 for HCP and 50%, N = 9/18 for patients) (Figure 3(c)). With the Liraglutide 3.0 RLD pen injector, the rate of use errors for the patient group (78%; N = 14/18) was higher than for caregivers (55%; N = 10/18) and HCP (43%; N = 6/14).

Having prior experience in using a pen injector improved the success rate regarding the task of priming the pen (Figure 3(d)). Including both patients and caregivers, naïve participants without experience failed to prime the pen in 64% (N = 9/14) of the simulated injections with the generic pen injector and 86% (N = 12/14) with the Liraglutide 3.0 RLD pen injector. In contrast, experienced caregivers and patients failed to prime the pen in 45% (N = 8/18) of simulated injections for both pen injectors.

3.3.2.2. Inject the medication

To inject the dose, first users need to rotate the dose dialer to align the pointer with the requested dose and then insert needle into the skin and press down the button until ‘0’ aligns with pointer.

3.3.2.2.1. Rotate dose dialer until the proper dose aligns with the pointer in dose window

Across the two simulated injections performed with each pen injector, for the task ‘Rotate dose dialer until the proper dose,’ 2% (N = 1/50) of use errors were recorded with the generic pen injector and 6% (N = 3/49) with the RLD pen injector (Figure 2). With the generic pen injector, one user did not understand the step, while this participant initially set to correct dose (1.8 mg), this user changed to incorrect dose of 3.0 mg upon reading in the IFU that the pen can be dosed to up to 3.0 mg. With the Liraglutide 3.0 RLD pen injector the use errors were caused by participants who forgot to set the dose initially or did not notice dialer was not aligned with dose marking.

The success rates were high in all user groups (Figure 4(a)). All HCP succeeded in correctly setting the proper dose with both generic and Liraglutide 3.0 RLD pen injectors (100%; N = 18/18). Patients failed to correctly achieve the specific task in 6% of the simulated injections with the generic pen injector (N = 1/18) and the Liraglutide 3.0 RLD pen injector (N = 1/17). For the caregivers, while all of them succeeded in setting the proper dose with the generic pen injector, they encountered 14% (N = 2/14) of use errors with the Liraglutide 3.0 RLD pen injector.

Figure 4. Usability assessment with generic and RLD pens per user group: (a) Task ‘Rotate the dose dialer until the proper dose’ per user group; (b) Task ‘Rotate the dose dialer until the proper dose’ per user group per user experience with pen injector (caregivers and patients); (c) Task ‘Inject medication’ per user group and (d) Task ‘Inject medication’ per user experience with pen injector (caregivers and patients). Success, success with operational difficulty (SOD), and use error rates are indicated for the generic and RLD pen injectors. N number of simulated injections.

With both pen injectors, only non-experienced users (caregivers and patients) failed to rotate the dose dialer until the proper dose in 7% (N = 1/14) of the simulated injections with the generic pen injector and 21% (N = 3/14) with the Liraglutide 3.0 RLD pen injector (Figure 4(b)).

3.3.2.2.2. Deliver the dose

Ninety six percents of participants succeeded in injecting the medication with the generic pen injector and 98% with the Liraglutide 3.0 RLD pen injector. Two use errors (4%) were observed with the generic pen injector when one patient injected multiple times into the pad due to believing dose was not delivered due to the absence of a visual signal at the end of dose delivery (e.g. plunger rod movement with autoinjector). Since neither of the two pen injectors evaluated have a visual signal, these use errors are not attributable to a design difference between the two pens. One participant did not understand this step with the Liraglutide 3.0 RLD pen injector and injected into the pad without a needle attached to the pen (2% (N = 1/50)). Regardless of the pen injector, all use errors were observed in the patient group (Figure 4(c)). While the pattern of the use errors was equivalent for both pen injectors, the occurrence of successes with difficulties was higher for the generic pen injector (N = 2) compared to the Liraglutide 3.0 RLD pen injector (N = 0) (Figure 2). These two successes with difficulties were recorded for an HCP because pressing the generic pen injector button was more difficult and required a higher amount of force than the Liraglutide 3.0 RLD pen injector. Despite these difficulties, the HCP was able to deliver the dose.

Non-experienced patients and caregivers were more likely to fail injecting the medication than experienced users (Figure 4(d)). Without prior experience in using pen injectors, use errors when injecting the medication were observed in 14% (N = 2/14) of the simulated injections with the generic pen injector and 7% (N = 1/14) with the Liraglutide 3.0 RLD pen injector. All the caregivers and patients with prior experience with a pen injector correctly performed the task with the generic and the Liraglutide 3.0 RLD pen injectors (N = 18/18).

3.3.3. IFU impact

During the study, participants had access to the IFUs of both pen injectors and were free to read them or not. Eleven participants read the IFU for the generic pen injector and 10 participants for the Liraglutide 3.0 RLD pen injector.

For both pen injectors, the proportion of use errors decreased when participants read the IFU (Supplementary File 4). With the generic pen, 20% of use errors were observed for the participants having read the IFU and 35% for those who had not read it. When using the Liraglutide 3.0 RLD pen injector, 25% of use errors were observed by those who read the IFU against 37% for participants who did not read the IFU.

3.3.4. Use error rate calculation

The use error rate with the generic pen injector was 0.72 (N = 18/25 participants who failed at least 1 task) at first simulated injection and 0.64 (N = 16/25 who failed at least one task) at second simulated injection. Overall use error rate was 0.68 (Table 3).

Table 3. Summary of total number of errors and error rates on critical tasks for each pen injector.

	Generic pen injector	RLD pen injector
Use error rate	0.68	0.70
Difference in use error rate	0.02

For the Liraglutide 3.0 RLD pen injector, a use error rate of 0.72 (N = 18/25) at first simulation and 0.68 (N = 17/25) at second simulation were calculated, with an overall use error rate of 0.70 (Table 3).

Therefore, the calculated difference in use error rate of 0.02 can help to determine the non-inferiority margin (called ‘margin d’ in the FDA guidance). Margin d taken as reference in the FDA guidance is 10%. Considering a difference of 2%, taking 10% margin d for final CUHF study can be a safe way to get positive non-inferiority results.

Eighy percents of participants (N = 20/25) had the same usability performance on the generic and Liraglutide 3.0 RLD pen injectors (meaning they pass the 8 tasks for both and failed at least ½ tasks for both). As such, the within-subject correlation is 0.8 (Table 4).

Table 4. Within-subject correlation. The within-subject correlation was calculated based on the participants who succeeded or failed in completing all the assessed critical tasks identically with both pen injectors.

	Generic pen	Liraglutide 3.0 RLD pen	Within-subject correlation
Patient #10	Success	Failure	0.8
Patient #11	Failure	Failure
Patient #12	Success	Success
Patient #13	Failure	Failure
Patient #14	Failure	Failure
Patient #15	Success	Success
Patient #16	Failure	Failure
Patient #17	Failure	Failure
Patient #18	Failure	Failure
Caregiver #9	Failure	Failure
Caregiver #10	Success	Failure
Caregiver #11	Failure	Failure
Caregiver #12	Success	Success
Caregiver #13	Failure	Failure
Caregiver #14	Failure	Success
Caregiver #15	Failure	Failure
HCP #9	Success	Failure
HCP #10	Failure	Failure
HCP #11	Failure	Failure
HCP #12	Failure	Success
HCP #13	Failure	Failure
HCP #14	Failure	Failure
HCP #15	Success	Success
HCP #16	Failure	Failure
HCP #17	Failure	Failure

3.3.5. Inputs for final CUHF study with final combined product

Thanks to the results of this formative study, pharmaceutical companies will be able to substantiate the calculation of their sample size and non-inferiority margin determination.

As an example, for a one-sided non-inferiority test between two correlated proportions, a sample size of 104 subjects achieves approximately 80% power at a significance level of 0.05 when the non-inferiority margin is 0.10 (10%). The actual difference between the error rates is assumed to be 0.02 and agreement (proportion of participants who had same outcome from both pen injectors) is assumed to be 80%, based on formative data.

4. Discussion

Initial task analysis and physical comparison between the Liraglutide 3.0 RLD pen injector and the generic pen injector (summarized in a comparative analysis document) highlighted that both pen injectors were mostly similar apart from the dose dialing mechanism, which may impact the following critical tasks: set dose to priming symbol, prime the pen, set the proper dose, and deliver the dose. Then, to ensure that no new risks are introduced for end-users, a formative CUHF focusing on tasks that were impacted by an ‘other design difference’ between these two pen injectors was conducted.

The results of this formative CUHF study showed similar rates of use errors between the two pen injectors. The rate of use errors was 0.68 for the generic pen injector and 0.70 for the Liraglutide 3.0 RLD pen injector. Similar types of use errors were observed indicating that these use errors were likely not related to a difference between the design of the two pen injectors. The same usability performance with both Liraglutide 3.0 RLD and generic pen injectors was observed in 80% of participants.

From this study, it was possible to calculate a use error rate for each pen injector based on the participants’ usability results for the four assessed critical tasks impacted by an ‘other difference’ in design. Based off the usability results from the formative CUHF study, the calculated use error rates and within-subject correlation can be used to determine a statistically significant sample size for the final CUHF with the final combination product in addition to non-inferiority margins as required by the FDA guidance. By leveraging the data from the formative CUHF study, it is possible to de-risk the CUHF study with a final combined product, identifying potential use-related problems with the generic pen injector compared to its RLD and assessing whether the generic is a good candidate or not (Figure 5). Moreover, even if no conclusions can be drawn, an intermediate study makes it possible to highlight the potential risks that could exist during the change between RLD and generic. Indeed, based on the Human Factors methodology, a formative CUHF study allows evaluation of the interaction of a user with a user interface by observing the performance with the aim of identifying the element of the user interface involved in a use error or problem. Thus, the interest of the formative CUHF studies is to provide preliminary results which can be used to determine the final CUHF study parameters and shared with the FDA [15].

Figure 5. Pre-ANDA strategy and Human Factors activities to help de-risk pharmaceutical companies ANDA submission of drug–device combination products.

Therefore, comparative analysis allows to identify any design differences between a generic pen injector and its RLD that may introduce a use-related risk. Then, the formative CUHF study provides the preliminary data needed to build the final CUHF that might be required for the ANDA submission to provide justification on design differences. Indeed, use error rates and within-subject correlation gathered in the formative CUHF support the design the final CUHF. The strategy presented in this manuscript has been used by a pharmaceutical company, allowing it to use the data generated by the DDDD to design its final study and thus to test non-inferiority between the final drug-device generic pen injector and its RLD pen injector [16].

While design differences in dose set knob extension of the generic pen injector as compared to the Liraglutide 3.0 RLD pen injector (Figure 1(b)) were highlighted, these differences did not result in different type of use errors between the two pen injectors. Indeed, the use error profiles were similar between the two pens with a majority occurring during the priming step. Priming the pen injector after attaching a new needle is important to not only ensure that the pen injector is working correctly but also to remove potential air bubbles. Not performing or not correctly performing this task can lead to a delayed treatment or an incorrect delivered dose. However, issues with priming are known, especially in the diabetes field, where 57.4% of people using a pen injector actually primed the pen, as recommended by the drug manufacturer [14]. Similar rates of priming issues were reported in the current study, with 54% to 60% of users who did not or incorrectly prime the investigated pen injectors. Therefore, the results presented in this manuscript were aligned with prior publication reporting usability issues with insulin pen injectors.

Regarding the injection step, 96% of the users succeeded to deliver the medication with the generic pen injector and 98% with the Liraglutide 3.0 RLD pen injector. While there was no difference in terms of delivery achievement, one user reported that pressing the dose set knob with the generic pen injector was difficult and required a higher amount of force. During the study, no new use-related risk was identified due to the design difference of the dose set knob between the pen injectors (Figure 1(b)) but in real-world settings, some users may need a little time to get adapted to the use of the pen injector.

The FDA guidance states that end-users, including but not limited to lay-persons, such as patients and caregivers must use the generic product when it is substituted for the RLD without the intervention of healthcare providers. As such, we can imagine that participants of a CUHF study should be at least patients and caregivers. The guidance does not clearly say that HCP should be part of participants but HCPs through their training and professional experience are accustomed to using pen injectors, such as the RLD in this study. HCP might assume that the generic pen injector works like the RLD pen injector they have previously used, which may result in use-related risks. Thus, this population was considered in the presented formative CUHF study because it represented a considerable risk of negative transfer. As shown in this formative CUHF study, the user group did not significantly impact the usability performance and HCP did not encounter additional use errors compared to caregivers and patients. On the contrary, HCPs made slightly less errors and were therefore not disturbed by their habit with the RLD pen injector. In the final CUHF study, the participants should be chosen according to the intended user of the final combination product.

In this formative CUHF study, participants performed two simulations with each of the pen injectors, generic and RLD, in a randomized manner. The FDA guidance does not specify a number of simulated injections required. However, depending on the therapeutic application, it may be interesting to consider several simulations. Indeed, it has previously been shown that the ability to use a drug delivery device improves with injections [17]. During the second injection, users often achieve their maximum or near usability performance [18,19]. Thus, in a chronic use context in which users will be required to perform injections on a regular basis, it may be interesting to assess usability on several simulations. In addition, including more than one simulated injection may also present the opportunity to evaluate different use scenarios. In this study however, no significant learning effect was observed between the first and second simulated injection with both pen injectors. Most of the use errors occurred during priming regardless of the pen injector and simulation number 94% (N = 119/126). While there were few use errors for the tasks related to the priming and injection of medication, the number of use errors tended to decrease from the first (N = 6) to the second simulated injection (N = 1) for both pen injectors.

Preventing use errors from occurring at the first use is essential to increase drug delivery device safety and efficacy. During the formative CUHF study, even if only a minority of participants read the IFU, fewer errors were observed with them than those who did not read the IFU with both pen injectors. The IFU, when read, can therefore be an effective way to limit the number of use errors. However, the IFU alone is not sufficient to fully eliminate the use errors. In this study, the step of priming was not performed by a majority of participants regardless of the IFU, as already highlighted for other pen injectors on the market [20]. This highlights that an explanation of how to correctly use the pen injectors seems necessary to ensure a safe and effective use.

The reported formative CUHF study includes some limitations, commonly highlighted in all Human Factors studies, such as the small sample size, the simulated environment and the qualitative rather than quantitative nature of the outcomes. Nevertheless, the presented approach provides crucial information to help de-risk the final CUHF study with the final combined product by assessing for the first time the usability comparison between the intended generic pen injector and its RLD pen injector.

5. Conclusion

Overall, this case study presents the pre-ANDA strategy and activities conducted to gather data to help de-risk the final combination product’s ANDA submission. Firstly, drug delivery device developer can support pharmaceutical companies by proactively performing task analysis and physical comparison between the generic device and its RLD, to have an overview of all differences that need to be mitigated. Since other design differences might indicate the need for additional Human Factors data, conducting a formative CUHF study provides an initial assessment of the use errors of these differences and above all it provides insights to data in order to calculate the non-inferiority margin and within-subject correlation. These results ensure appropriate methodological decisions (study design and sample size) for the final CUHF study that will be conducted with the final combination product and whose results will be integrated into the ANDA submission.

Author contributions

L Brunet-Manquat, A Combedazou, A Maden, C Ramus, T Mardovina, C Frolet contributed to the conception, design and analysis of the human factors testing. B Ahuja supported the project with regulatory affairs expertise. A Combedazou wrote the paper with the support of L Brunet-Manquat and inputs from all authors. All authors reviewed the drafted and revised manuscript agreed to be accountable for all aspects of the work. C Frolet, A Maden and T Mardovina supervised the project.

Declaration of interest

All authors are employees of Becton, Dickinson and Company. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/17425247.2024.2356678

Additional information

Funding

This work was funded by Becton, Dickinson and Company.