Improving Delphi study rigour with the integration of Q-methodology

References

• Barrios, M., et al., 2021. Consensus in the Delphi method: what makes a decision change? Technological forecasting and social change, 163, 120484. doi:10.1016/j.techfore.2020.120484.
   Web of Science ®Google Scholar
• Belton, I., et al., 2019. Improving the practical application of the Delphi method in group-based judgment: a six-step prescription for a well-founded and defensible process. Technological forecasting and social change, 147, 72–82. doi:10.1016/j.techfore.2019.07.002.
   Web of Science ®Google Scholar
• Belton, I., et al., 2021. Delphi with feedback of rationales: how large can a Delphi group be such that participants are not overloaded, de-motivated, or disengaged? Technological forecasting and social change, 170, 120897. doi:10.1016/j.techfore.2021.120897.
   Web of Science ®Google Scholar
• Bloor, M., et al., 2015. Useful but no oracle: reflections on the use of a Delphi group in a multi-methods policy research study. Qualitative research, 15 (1), 57–70. doi:10.1177/1468794113504103.
   Web of Science ®Google Scholar
• Boulkedid, R., et al., 2011. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS ONE, 6 (6), e20476. doi:10.1371/journal.pone.0020476.
   PubMed Web of Science ®Google Scholar
• Brady, S.R., 2015. Utilizing and adapting the Delphi Method for use in qualitative research. International journal of qualitative methods, 14 (5), 1–6. doi:10.1177/1609406915621381.
   Web of Science ®Google Scholar
• Braswell, R.D., 2022. Comparative rotations and analyzes of Q data: a worked example. In: J.C. Rhoads, D.B. Thomas, and S.E. Ramlo, eds. Cultivating Q-methodology USA: International Society for the Scientific Study of Subjectivity, 145–181.
   Google Scholar
• Brown, S.R., 1980. Political subjectivity: applications of Q-methodology in political subjectivity. New Haven, CT: Yale University Press.
   Google Scholar
• Brown, S. [qsiteadmin], 2010a. Q-methodology Workshop Video 3 – The Q Sample [Video]. You Tube. https://www.youtube.com/watch?v=7L7q-o6Xrus.
   Google Scholar
• Brown, S. [qsiteadmin], 2010b. Q-methodology Workshop Video 5 - Q Method Software [Video]. You Tube. https://www.youtube.com/watch?v=LMETSxXV548.
   Google Scholar
• Dalkey, N., and Helmer, O., 1963. An experimental application of the Delphi Method to the use of experts. Management science, 9 (3), 458–467. doi:10.1287/mnsc.9.3.458.
   Web of Science ®Google Scholar
• Danielson, S., 2009. Q method and surveys: three ways to combine Q and R. Field methods, 21 (3), 219–237. doi:10.1177/1525822X09332082.
   Web of Science ®Google Scholar
• Danielson, S., et al., 2012. Three tools for evaluating participation: focus groups, Q method, and surveys. Environmental practice, 14 (2), 101–109. doi:10.1017/S1466046612000026.
   Google Scholar
• Danielson, S., Webler, T., and Tuler, S.P., 2009. Using Q method for the formative evaluation of public participation processes. Society & natural resources, 23 (1), 92–96. doi:10.1080/08941920802438626.
   Web of Science ®Google Scholar
• Diamond, I.R., et al., 2014. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of clinical epidemiology, 67 (4), 401–409. doi:10.1016/j.jclinepi.2013.12.002.
   PubMed Web of Science ®Google Scholar
• Durose, C., et al., 2023. Leadership and the hidden politics of co-produced research: a Q-methodology study. International journal of social research methodology, 26 (1), 1–21. doi:10.1080/13645579.2021.1960738.
   Web of Science ®Google Scholar
• Eyvindson, K., et al., 2015. Likert versus Q-approaches in survey methodologies: discrepancies in results with same respondents. Quality & Quantity, 49 (2), 509–522. doi:10.1007/s11135-014-0006-y.
   Google Scholar
• Flostrand, A., Pitt, L., and Bridson, S., 2020. The Delphi technique in forecasting– A 42-year bibliographic analysis (1975–2017). Technological forecasting and social change, 150, 119773. doi:10.1016/j.techfore.2019.119773.
   Web of Science ®Google Scholar
• Gargon, E., et al., 2019. Higher number of items associated with significantly lower response rates in COS Delphi surveys. Journal of clinical epidemiology, 108, 110–120. doi:10.1016/j.jclinepi.2018.12.010.
   PubMed Web of Science ®Google Scholar
• Geist, M.R., 2010. Using the Delphi method to engage stakeholders: a comparison of two studies. Evaluation and program planning, 33 (2), 147–154. doi:10.1016/j.evalprogplan.2009.06.006.
   PubMed Web of Science ®Google Scholar
• Gen, S. 2020. Developing the q-sample and p-set in Q-methodology: an example from policy studies. In SAGE research methods cases. doi:10.4135/9781529716092.
   Google Scholar
• Grant, S., Booth, M., and Khodyakov, D., 2018. Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies. Journal of clinical epidemiology, 99, 96–105. doi:10.1016/j.jclinepi.2018.03.007.
   PubMed Web of Science ®Google Scholar
• Gupta, U.G., and Clarke, R.E., 1996. Theory and applications of the Delphi technique: a bibliography (1975–1994). Technological forecasting and social change, 53 (2), 185–211. doi:10.1016/S0040-1625(96)00094-7.
   Web of Science ®Google Scholar
• Hall, D.A., et al., 2018. Recruiting and retaining participants in e-Delphi surveys for core outcome set development: evaluating the COMiT’ID study. Plos One, 13 (7), e0201378. doi:10.1371/journal.pone.0201378.
   PubMed Web of Science ®Google Scholar
• Hampson, D.I., Ferrini, S., and Turner, R.K., 2022. Assessing subjective preferences for river quality improvements: combining Q-methodology and choice experiment data. Journal of environmental economics and policy, 11 (1), 56–74. doi:10.1080/21606544.2021.1879682.
   Web of Science ®Google Scholar
• Hasson, F., and Keeney, S., 2011. Enhancing rigour in the Delphi technique research. Technological forecasting and social change, 78 (9), 1695–1704. doi:10.1016/j.techfore.2011.04.005.
   Web of Science ®Google Scholar
• Hasson, F., Keeney, S., and McKenna, H., 2000. Research guidelines for the Delphi survey technique. Journal of advanced nursing, 32 (4), 1008–1015. doi:10.1046/j.1365-2648.2000.t01-1-01567.x.
   PubMed Web of Science ®Google Scholar
• Ho, G.W.K., 2017. Examining perceptions and attitudes: a review of Likert-type scales versus Q-methodology. Western journal of nursing research, 39 (5), 674–689. doi:10.1177/0193945916661302.
   PubMed Web of Science ®Google Scholar
• Hsu, C.C., and Sandford, B.A., 2007a. Minimizing non-response in the Delphi process: how to respond to non-response. Practical assessment, research & evaluation, 12 (17), 1–6.
   Google Scholar
• Hsu, C.C., and Sandford, B.A., 2007b. The Delphi Technique: making sense of consensus. Practical assessment, research & evaluation, 12 (10), 1–8.
   Google Scholar
• Humphrey-Murto, S., and de Wit, M., 2019. The Delphi method—more research please. Journal of clinical epidemiology, 106, 136–139. doi:10.1016/j.jclinepi.2018.10.011.
   PubMed Web of Science ®Google Scholar
• Keeney, S., Hasson, F., and McKenna, H., 2011. The Delphi Technique in nursing and health research. Chichester: Wiley-Blackwell.
   Google Scholar
• Kezar, A., and Maxey, D., 2016. The Delphi Technique: an untapped approach of participatory research. International journal of social research methodology, 19 (2), 143–160. doi:10.1080/13645579.2014.936737.
   Web of Science ®Google Scholar
• Linstone, H.A., and Turoff, M., 1975. The Delphi method: techniques and applications. Reading, MA: Addison-Wesley Publishing Company.
   Google Scholar
• McKeown, B., and Thomas, B., 2013. Q-methodology. Thousand Oaks, CA: Sage.
   Google Scholar
• Meijering, J.V., and Tobi, H., 2016. The effect of controlled opinion feedback on Delphi features: mixed messages from a real-world Delphi experiment. Technological forecasting and social change, 103, 166–173. doi:10.1016/j.techfore.2015.11.008.
   Web of Science ®Google Scholar
• Mitchell, V.W., 1991. The Delphi Technique: an exposition and application. Technology analysis & strategic management, 3 (4), 333–358. doi:10.1080/09537329108524065.
   Google Scholar
• Niederberger, M., and Spranger, J., 2020. Delphi Technique in health sciences: a map. Frontiers in public health, 8 (457), 1–10. doi:10.3389/fpubh.2020.00457.
   PubMedGoogle Scholar
• Paré, G., et al., 2013. A systematic assessment of rigour in information systems ranking-type Delphi studies. Information & Management, 50 (5), 207–217. doi:10.1016/j.im.2013.03.003.
   Web of Science ®Google Scholar
• Ramlo, S.E., 2020. Promoting conceptual understanding of Q-methodology using insights from science education. International journal of multiple research approaches, 12 (3), 319–334. doi:10.29034/ijmra.v12n3a3.
   Google Scholar
• Ramlo, S., 2022. A science of subjectivity. In: J.C. Rhoads, D.B. Thomas, and S.E. Ramlo, eds. Cultivating Q-methodology. USA: International Society for the Scientific Study of Subjectivity, 182–216.
   Google Scholar
• Ramlo, S., 2023. Non-statistical, substantive generalization: lessons from Q methodology. International journal of research & method in education, 1–14. doi:10.1080/1743727X.2023.2173735.
   Web of Science ®Google Scholar
• Rieber, L.P., 2020. Q methodology in learning, design, and technology: an introduction. Educational technology research and development, 68 (5), 2529–2549. doi:10.1007/s11423-020-09777-2.
   Web of Science ®Google Scholar
• Rodl, J.E., Cruz, R.A., and Knollman, G.A., 2020. Applying Q-methodology to teacher evaluation research. Studies in educational evaluation, 65, 100844. doi:10.1016/j.stueduc.2020.100844.
   Web of Science ®Google Scholar
• Schmidt, R.C., 1997. Managing Delphi surveys using nonparametric statistical techniques. Decision sciences, 28 (3), 763–774. doi:10.1111/j.1540-5915.1997.tb01330.x.
   Web of Science ®Google Scholar
• Stephenson, W., 1953) The study of behavior: Q-technique and its methodology. Chicago: University of Chicago Press.
   Google Scholar
• Strasser, A., 2019. Design and evaluation of ranking-type Delphi studies using best-worst-scaling. Technology analysis & strategic management, 31 (4), 492–501. doi:10.1080/09537325.2018.1521956.
   Web of Science ®Google Scholar
• Tapio, P., et al., 2011. The unholy marriage? Integrating qualitative and quantitative information in Delphi processes. Technological forecasting and social change, 78 (9), 1616–1628. doi:10.1016/j.techfore.2011.03.016.
   Web of Science ®Google Scholar
• ten Klooster, P.M., Visser, M., and de Jong, M.D.T., 2008. Comparing two image research instruments: the Q-sort method versus the Likert attitude questionnaire. Food quality and preference, 19 (5), 511–518. doi:10.1016/j.foodqual.2008.02.007.
   Web of Science ®Google Scholar
• Toma, C., and Picioreanu, I., 2016. The Delphi Technique: methodological considerations and the need for reporting guidelines in medical journals. International journal of public health research, 4 (6), 47–59.
   Google Scholar
• von der Gracht, H.A., 2012. Consensus measurement in Delphi studies. Technological forecasting and social change, 79 (8), 1525–1536. doi:10.1016/j.techfore.2012.04.013.
   Web of Science ®Google Scholar
• Walker, A.M., and Selfe, J., 1996. The Delphi method: a useful tool for the allied health worker. British journal of therapy and rehabilitation, 3 (12), 677–681. doi:10.12968/bjtr.1996.3.12.14731.
   Google Scholar
• Watts, S., and Stenner, P., 2012. Doing Q methodological research: theory, method and interpretation. London: Sage.
   Google Scholar
• Webler, T., Danielson, S., and Tuler, S., 2009. Using Q method to reveal social perspectives in environmental research. Greenfield, MA: Social and Environmental Research Institute. https://www.researchgate.net/publication/273697977_Using_Q_Method_to_Reveal_Social_Perspectives_in_Environmental_Research.
   Google Scholar
• Whitehead, D., and Schneider, Z., 2013. Chapter 14: mixed methods research. In: Z. Schneider and D. Whitehead, eds. Nursing & midwifery research: methods and appraisal for evidence-based practice. 4th ed. Sydney: Elsevier-Mosby, 263-284.
   Google Scholar
• Williamson, K., 2002. Research methods for students, academics and professionals. 2nd ed. Wagga Wagga: Charles Sturt University.
   Google Scholar
• Wolf, A., 2022. Perspectives on Q sorting. In: J.C. Rhoads, D.B. Thomas, and S.E. Ramlo, eds. Cultivating Q-methodology. USA: International Society for the Scientific Study of Subjectivity, 217–241.
   Google Scholar