Psychometric Properties of the Spanish Full and Short Forms of the System Usability Scale (SUS): Detecting the Effect of Negatively Worded Items

References

• AlGhannam, B. A., Albustan, S. A., Al-Hassan, A. A., & Albustan, L. A. (2018). Towards a standard arabic system usability scale: psychometric evaluation using communication disorder app. International Journal of Human–Computer Interaction, 34(9), 799–804. https://doi.org/10.1080/10447318.2017.1388099
   Web of Science ®Google Scholar
• Baharuddin, R., Singh, D., & Razali, R. (2013). Usability dimensions for mobile applications-a review. Research Journal of Applied Sciences, Engineering and Technology, 11(9), 2225–2231. https://doi.org/10.19026/rjaset.5.4776
   Google Scholar
• Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776
   Web of Science ®Google Scholar
• Blažica, B., & Lewis, J. R. (2015). A slovene translation of the system usability scale: The SUS-SI. International Journal of Human-Computer Interaction, 31(2), 112–117. https://doi.org/10.1080/10447318.2014.986634
   Web of Science ®Google Scholar
• Borkowska, A., & Jach, K. (2017). Pre-testing of Polish translation of System Usability Scale (SUS). In Z. Borzenski, L., Grzech, A., Swiatek, & J., Wilimowska (Eds.), Advances in intelligent systems and computing (Vol. 521, pp. 143–153). Springer Verlag. https://doi.org/10.1007/978-3-319-46583-8_12/TABLES/7
   Google Scholar
• Borsci, S., Federici, S., Bacci, S., Gnaldi, M., & Bartolucci, F. (2015). Assessing user satisfaction in the era of user experience: Comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience. International Journal of Human-Computer Interaction, 31(8), 484–495. https://doi.org/10.1080/10447318.2015.1064648
   Web of Science ®Google Scholar
• Borsci, S., Federici, S., & Lauriola, M. (2009). On the dimensionality of the System Usability Scale: A test of alternative measurement models. Cognitive Processing, 10(3), 193–197. https://doi.org/10.1007/s10339-009-0268-9
   PubMed Web of Science ®Google Scholar
• Borsci, S., Federici, S., Mele, M. L., & Conti, M. (2015). Short scales of satisfaction assessment: A proxy to involve disabled users in the usability testing of websites. In M. Kurosu (Ed.), 17th International Conference on Human-Computer Interaction (Vol. 9171, pp. 35–42). Springer Verlag. https://doi.org/10.1007/978-3-319-21006-3_4/TABLES/4
   Google Scholar
• Brislin, R. W. (1976). Comparative research methodology: Cross-cultural studies. International Journal of Psychology, 11(3), 215–229. https://doi.org/10.1080/00207597608247359
   Web of Science ®Google Scholar
• Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L. McClelland (Eds.), Usability Evaluation in Industry (pp. 189–194). Taylor and Francis.
   Google Scholar
• Castañeda, J. A., Muñoz-Leiva, F., & Luque, T. (2007). Web Acceptance Model (WAM): Moderating effects of user experience. Information & Management, 44(4), 384–396. https://doi.org/10.1016/j.im.2007.02.003
   Web of Science ®Google Scholar
• Dianat, I., Ghanbari, Z., & Asgharijafarabadi, M. (2014). Psychometric properties of the Persian language version of the System Usability Scale. Health Promotion Perspectives, 4(1), 82–89. https://doi.org/10.5681/hpp.2014.011
   PubMedGoogle Scholar
• DiStefano, C., & Motl, R. W. (2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling: A Multidisciplinary Journal, 13(3), 440–464. https://doi.org/10.1207/s15328007sem1303_6
   Web of Science ®Google Scholar
• Finstad, K. (2010). The usability metric for user experience. Interacting with Computers, 22(5), 323–327. https://doi.org/10.1016/j.intcom.2010.04.004
   Web of Science ®Google Scholar
• Guadagnoli, E., & Velicer, W. F. (1988). Relation of sample size to the stability of component patterns. Psychological Bulletin, 103(2), 265–275. https://doi.org/10.1037/0033-2909.103.2.265
   PubMed Web of Science ®Google Scholar
• Hilton, A., & Skrutkowski, M. (2002). Translating instruments into other languages: Development and testing processes. Cancer Nursing, 25(1), 1–7. https://doi.org/10.1097/00002820-200202000-00001
   PubMed Web of Science ®Google Scholar
• Hui, C. H., & Triandis, H. C. (1985). Measurement in cross-cultural psychology: A Review and Comparison of Strategies. Journal of Cross-Cultural Psychology, 16(2), 131–152. https://doi.org/10.1177/0022002185016002001
   Web of Science ®Google Scholar
• Hvidt, J. C. S., Christensen, L. F., Sibbersen, C., Helweg-Jørgensen, S., Hansen, J. P., & Lichtenstein, M. B. (2020). Translation and validation of the system usability scale in a Danish mental health setting using digital technologies in treatment interventions. International Journal of Human–Computer Interaction, 36(8), 709–716. https://doi.org/10.1080/10447318.2019.1680922
   Web of Science ®Google Scholar
• International Organization for Standardization (2018). ISO 9241-11:2018(en), Ergonomics of human-system interaction—part 11: Usability: Definitions and concepts. ISO. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
   Google Scholar
• Kortum, P., Acemyan, C. Z., & Oswald, F. L. (2021). Is it time to go positive? Assessing the positively worded System Usability Scale (SUS). Human Factors, 63(6), 987–998. https://doi.org/10.1177/0018720819881556
   PubMed Web of Science ®Google Scholar
• Kortum, P., & Sorber, M. (2015). Measuring the usability of mobile applications for phones and tablets. International Journal of Human-Computer Interaction, 31(8), 518–529. https://doi.org/10.1080/10447318.2015.1064658
   Web of Science ®Google Scholar
• Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57–78. https://doi.org/10.1080/10447319509526110
   Web of Science ®Google Scholar
• Lewis, J. R. (2018). The System Usability Scale: Past, present, and future. International Journal of Human–Computer Interaction, 34(7), 577–590. https://doi.org/10.1080/10447318.2018.1455307
   Web of Science ®Google Scholar
• Lewis, J. R., Brown, J., & Mayes, D. K. (2015). Psychometric evaluation of the EMO and the SUS in the context of a large-sample unmoderated usability study. International Journal of Human-Computer Interaction, 31(8), 545–553. https://doi.org/10.1080/10447318.2015.1064665
   Web of Science ®Google Scholar
• Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. In M. Kurosu (Ed.), Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics): Vol. 5619 LNCS (pp. 94–103). Springer Verlag. https://doi.org/10.1007/978-3-642-02806-9_12
   Google Scholar
• Lewis, J. R., & Sauro, J. (2017). Revisiting the factor structure of the System Usability Scale. Journal of Usability Studies, 12(4), 183–192.
   Web of Science ®Google Scholar
• Lewis, J. R., Utesch, B. S., & Maher, D. E. (2015). Measuring perceived usability: The SUS, UMUX-LITE, and AltUsability. International Journal of Human-Computer Interaction, 31(8), 496–505. https://doi.org/10.1080/10447318.2015.1064654
   Web of Science ®Google Scholar
• Liew, M. S., Zhang, J., See, J., & Ong, Y. L. (2019). Usability challenges for health and wellness mobile apps: Mixed-methods study among mhealth experts and consumers. JMIR mHealth and uHealth, 7(1), e12160. https://doi.org/10.2196/12160
   PubMed Web of Science ®Google Scholar
• Maramba, I., Chatterjee, A., & Newman, C. (2019). Methods of usability testing in the development of eHealth applications: A scoping review. International Journal of Medical Informatics, 126(November 2018), 95–104. https://doi.org/10.1016/j.ijmedinf.2019.03.018
   PubMedGoogle Scholar
• Marsh, H. W., Hau, K.-T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11(3), 320–341. https://doi.org/10.1207/s15328007sem1103_2
   Google Scholar
• Raita, E., & Oulasvirta, A. (2011). Too good to be bad: Favorable product expectations boost subjective usability ratings. Interacting with Computers, 23(4), 363–371. https://doi.org/10.1016/j.intcom.2011.04.002
   Web of Science ®Google Scholar
• Rosa, A. F., Silva, A., Rocha, N. P., Queirós, A., & Martins, A. I. (2015). European Portuguese validation of the System Usability Scale (SUS). Procedia Computer Science, 67(Dsai), 293–300. https://doi.org/10.1016/j.procs.2015.09.273
   Google Scholar
• Salazar, M. S. (2015). El dilema de combinar ítems positivos y negativos en escalas. Psicothema, 27(2), 192–199. https://doi.org/10.7334/psicothema2014.266
   PubMed Web of Science ®Google Scholar
• Sauro, J., & Lewis, J. R. (2011). When designing usability questionnaires, does it hurt to be positive? [Paper presentation]. Conference on Human Factors in Computing Systems - Proceedings, 2215–2223, Vancouver, Canada. https://doi.org/10.1145/1978942.1979266
   Google Scholar
• Schriesheim, C. A., & Hill, K. D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41(4), 1101–1114. https://doi.org/10.1177/001316448104100420
   Web of Science ®Google Scholar
• Sevilla-Gonzalez, M. D. R., Moreno Loaeza, L., Lazaro-Carrera, L. S., Bourguet Ramirez, B., Vázquez Rodríguez, A., Peralta-Pedrero, M. L., & Almeda-Valdes, P. (2020). Spanish Version of the System Usability Scale for the assessment of electronic tools: Development and VALIDATION. JMIR Human Factors, 7(4), e21161. https://doi.org/10.2196/21161
   PubMedGoogle Scholar
• Suárez-Alvarez, J., Pedrosa, I., Lozano, L. M., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018). Using reversed items in likert scales: A questionable practice. Psicothema, 30(2), 149–158. https://doi.org/10.7334/psicothema2018.33
   PubMed Web of Science ®Google Scholar
• Weston, R., & Gore, P. A. Jr. (2006). A brief guide to structural equation modeling. The Counseling Psychologist, 34(5), 719–751. https://doi.org/10.1177/0011000006286345
   Web of Science ®Google Scholar
• Wind, T. R., Rijkeboer, M., Andersson, G., & Riper, H. (2020). The COVID-19 pandemic: The ‘black swan’ for mental health care and a turning point for e-health. Internet Interventions, 20(March), 100317. https://doi.org/10.1016/j.invent.2020.100317
   PubMedGoogle Scholar
• Wolgast, M. (2014). What does the acceptance and action questionnaire (AAQ-II) really measure? Behavior Therapy, 45(6), 831–839. https://doi.org/10.1016/j.beth.2014.07.002
   PubMed Web of Science ®Google Scholar
• World Health Organization (2019). The WHO special initiative for mental health (2019–2023): Universal health coverage for mental health. In The WHO special initiative for mental health (2019–2023). WHO. https://apps.who.int/iris/bitstream/handle/10665/310981/WHO-MSD-19.1-eng.pdf?ua=1
   Google Scholar