How many testers are needed to assure the usability of medical devices?

Abstract

Before releasing a product, manufacturers have to follow a regulatory framework and meet standards, producing reliable evidence that the device presents low levels of risk in use. There is, though, a gap between the needs of the manufacturers to conduct usability testing while managing their costs, and the requirements of authorities for representative evaluation data. A key issue here is the number of users that should complete this evaluation to provide confidence in a product’s safety. This paper reviews the US FDA’s indication that a sample composed of 15 participants per major group (or a minimum of 25 users) should be enough to identify 90–97% of the usability problems and argues that a more nuanced approach to determining sample size (which would also fit well with the FDA’s own concerns) would be beneficial. The paper will show that there is no a priori cohort size that can guarantee a reliable assessment, a point stressed by the FDA in the appendices to its guidance, but that manufacturers can terminate the assessment when appropriate by using a specific approach – illustrated in this paper through a case study – called the ‘Grounded Procedure’.

Keywords::

• evaluation cohort
• five-user assumption
• medical devices
• product safety
• usability testing

Financial & competing interests disclosure

The authors acknowledge support of this work through the MATCH programme (EPSRC Grants: EP/F063822/1 EP/G012393/1), although the views expressed are entirely their own. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.

Key issues

• The assessment of usability and its place in the management of product safety is an increasingly important aspect of medical device development.

• Medical device manufacturers, especially in small companies, do not have sufficient expertise or knowledge about usability and human factors and it is hard for them suitably to address all of the steps required by relevant standards.

• Despite a rigorous framework for designing and assessing a product, an increasing number of medical devices are recalled each year because of safety concerns.

• There is substantial evidence that testing a device with mandated sample sizes could lead manufacturers to evaluate a product without having real control over the reliability of the assessment.

• A new approach to usability data management applied in this paper, called the ‘Grounded Procedure’, drives manufacturers to estimate the sample size needed to identify a given proportion of interaction problems and to inform critical product decisions.

• Using the Grounded Procedure could enable manufacturers to increase the usability and the safety of their medical devices and help practitioners to check the representativeness of the evaluation cohorts, to analyze significance of specific usability problems and to re-think the user selection criteria for validation testing.

Notes

1 In this evaluation case, the estimated p-value of the Monte Carlo re-sampling method was equal to 0.511, which was perfectly in line with the Bootstrap Discovery Behavior model. So, we have not reported the Monte Carlo method in the text.