Use of a Fractional Factorial Experiment to Assess the E-healthcare Application Design Needs of Persons With Dual Diagnosis

Abstract

Objective: The purpose of this study was to evaluate the influence of 12 e-healthcare applications’ design variables on the usability of websites for persons with a dual diagnosis of substance use disorder and severe mental illness. Methods: A 2¹²⁻⁴ fractional factorial experimental design was employed to specify the designs of 256 websites. The designs of the websites were specified to systematically vary 12 design variables, which included the number of hyperlinks, words, and content areas on a page as well as the depth of the hierarchy, that is, the number of pages one needed to navigate to find desired content. Subjects (n = 149) were adults with a dual diagnosis of substance use disorder and severe mental illness. Each participant was asked sequentially to try to find six specific pieces of information on each of eight different websites. We recorded ability and time to find each piece of information, whether a task was solved, and the time to solve each task. Analyses were completed with polychotomous logistic regression for the number of tasks solved and mixed effect regression for the mean time to solution. In both, the dependency of observations within subjects was included in the analyses. Interactions between the 12 design variables were identified with classification and regression tree analyses. Results: One of the most important variables was the depth of a website's hierarchy. Other important variables were the number of words, hyperlinks, and navigational areas per page and the use of navigational lists or navigational memory aids. There were clear differences in the usability of certain designs for these participants. Some designs were quite poor (success rate of 16%) and others quite effective (success rate of 86.5%). Conclusions: Our findings indicate that there are ways to design web-based applications that are far more effective than others for persons with a dual diagnosis and that certain variables have a far larger impact on the usability of a design than others. These are the variables that the most attention should be devoted to in creating an effective design.

Keywords:

• fractional factorial design
• e-healthcare
• e-healthcare design
• severe mental illness
• substance use disorder
• dual diagnosis

ACKNOWLEDGMENTS

This work was supported by grant D6180R from the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Rehabilitation Research and Development Service, and by the VA Pittsburgh Mental Illness Research, Education and Clinical Center (MIRECC).

The contents do not represent the views of the Department of Veterans Affairs or the United States Government.