Sowing Q methodology in the rural global South: a review of challenges and good practices

Abstract

The accomplishment of Sustainable Development Goals (SDGs) is intrinsically connected to improving livelihoods in the Rural Global South (RGS). RGS livelihoods are complex, showing multiple dimensions beyond mere economic considerations. However, many related development policies (over)simplify livelihoods to income thresholds, leading to flawed interventions. Adequate strategies to address RGS livelihoods require a much deeper understanding of their various dimensions and complexities. Q methodology (Q) is a powerful participatory research technique that enables the systematic study of different viewpoints on subjective topics. Moreover, it has the potential to identify and reveal previously unheard narratives, thus allowing us to question the traditional understandings of RGS livelihoods. Yet, as a time- and assistance-intensive technique, its implementation faces methodological challenges that are currently overlooked and ought to be considered. We selected and reviewed 50 Q studies applied to different forms of RGS livelihoods. First, we discuss several on-field Q limitations associated with the physical, logistical, social, and cultural constraints. Second, we draw on good practices and strategies to cope with these limitations. Notwithstanding the limitations and strategies, we advocate building Q capacities and the gender-balanced empowerment of local researchers. This may contribute to a better understanding of the nuances and challenges of RGS livelihoods.

Keywords:

• Q methodology
• global south
• rural livelihoods
• methodological challenges

Reviewing Editor:

• Grace Spencer, Anglia Ruskin University, UK

Subjects:

• Sociology & Social Policy
• Multidisciplinary Psychology
• Development Studies
• Rural Development
• Research Methods in Development Studies

1. Introduction

The accomplishment of several Sustainable Development Goals (SDGs) is intrinsically connected to the generation and improvement of sustainable livelihoods in the Rural Global South¹ (RGS) (Tambe, 2022). As 84% of the 1.3 billion multidimensionally poor (i.e., deprived of several resources and services) live in the RGS, uplifting their means of living is cornerstone in meeting global development targets (Tambe, 2022). RGS livelihoods are varied and complex, showing multiple dimensions far beyond the typically observed economic consideration: capabilities, activities, material resources, and social assets (Nunan et al., 2023). In contrast, related development policies still resort to income thresholds [e.g., World Bank’s International Poverty Line (Lang and Lingnau, 2015; United Nations, 2021)] as the compass to evaluate and benchmark livelihood conditions (Ascher, 2021). Such definitions and metrics overshadow the context-dependent diversity and complexity of RGS livelihoods (Chambers, 2017; Nunan et al., 2023). In addition, facilitators of development programs are usually outsiders who imprint their own priorities, often related to a number of substantial biases (e.g., spatial bias, project bias, person bias, seasonal bias, etc.) (Chambers, 2017; Chambers, 1983; Datta, 2019). These limited approaches have led to a distorted comprehension of the issues that face RGS populations, and therefore to ineffective livelihoods interventions (Ascher, 2021).

Smallholder agriculture is a good example of how an inadequate understanding of RGS livelihoods’ dynamics has resulted in many failures (Fan and Rue, 2020; Waarts et al., 2021). This sector represents the most prominent livelihood in RGS economies. It is the main occupation of 70% of the RGS poor, supplies up to 80% of the food consumed in Asia and sub-Saharan Africa, and is the main activity of approximately 50% of RGS women in several countries (Poole, 2017). Investing in smallholder farming is therefore a crucial strategy in boosting RGS economies, securing and increasing incomes (SDG 1), providing decent and inclusive work (SDG 8), and supporting food security (SDG 2) (Fan et al., 2013; Giordano et al., 2019; Mellor and Malik, 2017; Poole, 2017). Unfortunately, many policies and interventions have not addressed farmers’ actual needs and expectations (Giordano et al., 2019). Contrariwise, those have resulted in failures, such as biases in agricultural mechanization and technology adoption (Devkota et al., 2020; Van Loon et al., 2020), lack of empowerment of female farmers (Akter et al., 2017; Slavchevska et al., 2019; Theis et al., 2018), and high rates of rural youth disengagement and unemployment (Gc and Hall, 2020; Hazell and Rahman, 2014).

Adequate strategies to address RGS livelihoods require a much deeper understanding of their various dimensions and multifaceted characteristics (Chambers, 2017; Lang and Lingnau, 2015; Nunan et al., 2023). Numerous research approaches and methods to unravel the complexities of RGS livelihoods have arisen in response to this need (Chambers, 2017; Nunan et al., 2023). These include quantitative approaches, quantitative and qualitative longitudinal studies, ethnographic studies, participatory rural appraisal, participatory video research, among others. Alongside these approaches and methods, Q methodology (henceforth referred to as Q) has emerged as a powerful participatory research technique that enables the study of human subjectivity. It allows researchers to shift from single (and perhaps oversimplified) definitions around a particular topic or phenomenon (e.g., RGS livelihoods) to the systematic analysis of diverse perspectives about it (Previte et al., 2007). Simultaneously, Q embraces this diversity while maintaining a reductionist approach. This results in consistently clustered viewpoints that represent the spectrum of individual perceptions. Moreover, by systematically encompassing grassroots voices throughout its four stages (Figure 1), Q helps identify and reveal previously unheard narratives, hence potentially allowing us to question traditional and/or dominant understandings of RGS livelihoods. Owing to these reasons, we argue that Q holds a strong potential to study the complex nature of and support interventions on RGS livelihoods. More background information on Q can be found in Appendix A of the Data availability statement.

Figure 1. Stages and steps of Q, adapted from (Zabala et al., 2018).

The potential of Q has been systematically assessed in review articles focused on psychology and behavioral studies (Dziopa and Ahern, 2011), conservation research (Zabala et al., 2018), healthcare research (Churruca et al., 2021), and even on its methodological choices across a wide range of disciplines (Dieteren et al., 2023). Despite this potential and its effective implementation across disciplines (Watts & Stenner, 2012; Zabala and Pascual, 2016), Q is seldom engaged with a focus on RGS livelihoods. Even in such cases, the on-field methodological choices and points of attention are seemingly underreported. Hardly any study has critically elaborated on the methodological implications of Q in these contexts, let alone in those related to any form of (smallholder) agriculture. Considering the challenges of conducting fieldwork in the RGS (Breman, 1985; Casale et al., 2013; Chacko, 2004; Potnis and Gala, 2020; Strijker et al., 2020), and as part of a larger Q-led doctoral project conducted at Delft University of Technology (Intriago et al., 2018), in this article we aim to analyze and discuss: (1) methodological challenges of implementing Q to study RGS livelihoods; and (2) the best (reported) practices to cope with these challenges, with emphasis on the stages that imply on-field methodological choices (i.e., research design, data collection, and interpretation). Through this study, we expect to make two key contributions: first, to expand the understanding about the methodological implications of Q in RGS settings; and second, to help researchers make informed methodological choices when engaging Q to study RGS livelihoods.

2. Methodology

We employed a semi-systematic approach in this review. As (Snyder, 2019) argues, this is an appropriate strategy to review mixed qualitative/quantitative information and identify knowledge gaps in the literature. Our approach enabled us to synthesize state-of-the-art knowledge on the application of Q in RGS settings, its intrinsic methodological issues, and best practices.

2.1. Sources of information

We chose database search as the preferred technique to search for references. Because this review focuses on the application of Q across different fields instead of discipline-specific studies, we opted for two multidisciplinary scientific databases, namely Scopus and Web of Science. Complementarily, we triangulated these databases using Google Scholar to prevent location bias. In addition to the database search, we also used snowball sampling (through bibliographic references and hyperlinks) to identify additional documents that did not appear in the iterative searches.

2.2. Search criteria

To search for the literature in the respective scientific databases, we used the terms “Q methodology” and “Q-methodology”, in combination with any/some of the following terms: “rural”, “farm”, “farmer”, “farming”, “smallholder”, “agriculture”, “irrigation”, “water”, “forest”, “forestry”. We acknowledge the possible biases in the review as a consequence of screening literature using terms exclusively in the English language. We believe that our results provide sufficient details and discussion to accept this language-based restriction in our review.

We searched for references between April and August 2020, and within the publishing period of 2010–2020. Through iterative searches, it became apparent that prior to that period, very few studies fit within the scope of this review.

2.3. Selection criteria

Within the scope of the present study, we employed the following inclusion criteria to determine the relevance of selected documents:

1. Application of Q as (one of the) main research technique(s);

2. Addressing topics around RGS livelihoods, with particular emphasis on any form of (smallholder) agriculture;

3. Direct involvement of RGS dwellers during the methodological cycle of Q, with a specific emphasis on smallholder farmers; and,

4. Given the incipient and unfamiliar use of Q in RGS settings, a peer-reviewed scientific article, published in a SCImago-indexed journal, with emphasis on Q1/Q2 impact factor quartiles.

Notwithstanding the above inclusion criteria, the final selection of studies was made based on our judgement. In our discussion below, we left aside five studies that, although fulfilled the set of criteria, showed a lack of (Q) methodological clarity (Dingkuhn et al., 2020; Leong and Lejano, 2016; Nijnik et al., 2017), or considered RGS livelihoods from the perspective of non-rural actors (i.e., extension officers) (Bond, 2016; Easdale et al., 2020).

2.4. Analytical methods and abstracted data

We analyzed the selected documents through a content analysis. Using this technique, we abstracted two types of information: descriptive information and the effects and findings of each study (Snyder, 2019). The former comprised general characteristics of studies, that is, subject of study, category of Q study, (non) open access, and geographical foci of both study areas and researchers’ affiliations. This information contributed to revealing possible underlying Q research gaps between Global South and Global North. The latter consisted of Q methodological choices and their consequent findings, in accordance with the four methodological stages of Q pointed out above (Figure 1), with special emphasis on fieldwork, that is, research design and data collection.

3. Main findings

We selected in total 50 studies based on the above selection criteria. Table 1 summarizes the data extracted from these studies. The complete dataset with qualitative and quantitative information obtained during the semi-systematic review process can be found in Appendix B of the Data availability statement.

Table 1. Data extracted from the 50 selected studies.

Nr. (Ref.)	Location	Aim	Concourse		Q-set size	P-set		Sorting grid		Administration	Sorting time	Analysis	Factors			Salient reported difficulties
			Sources	RT		ST	Size	Criteria	Scale				Results	Presentation	Labelling
1 (Alexander et al., 2018)	Laos: Savannakhet and Champasak provinces	To identify drivers and constraints affecting smallholder farmers’ decision-making when considering introduced technologies	PD: NE-UN/SD: GL-UN	EJ	16	SS	T: 35 F: 16 M: 19 RD: 100%	Not important to Very important	−3 to +3 (7p)	T: Face-to-face L: N/A A: By-step guidance; clarification of statements	∼1 h	S: PQMethod E: PCA R: Varimax	NF: 2 VE: 40%	Z-score table	Long societal scenarios	Semi-literacy of respondents
2 (Anderson and Jacobson, 2018)	Ecuador: Tabuga village (Manabí province)	To identify and categorize barriers to teaching environmental education	SD: SL-GL	EJ	25	PS, CS	T: 25 F: 7 M: 18 RD: N/A	Very unimportant barriers to Very important barriers	−4 to +4 (9p)	T: Face-to-face L: Schools A: Pre-sort instructions	∼35 m	S: PQMethod E: N/A R: N/A	NF: 3 VE: 44%	Factor table	Explanatory labels	N/A
3 (Astari and Lovett, 2019)	Indonesia: South Sumatra, Riau and West Kalimantan provinces	To analyze discourses around the Indonesian Sustainable Palm Oil policy	PD: EX-PR/SD: SL-GL	DD, MM, SW	54	N/A	T: 27 F: N/A M: N/A RD: 0%	Most disagree to Most agree	−5 to +5 (11p)	T: Face-to-face L: N/A A: Clarification of statements	N/A	S: PQMethod E: PCA R: Varimax	NF: 5 VE: 48%	Factor table	Long societal scenarios	N/A
4 (Barbosa et al., 2020)	Brazil: Lagoa Grande village (Mato Grosso do Sul state)	To identify rural women’s viewpoint about the elements that would facilitate them to eventually take over the family farm	SD: SL-GL	CD	30	CS, SS	T: 28 F: 28 M: 0 RD: 39%	Strong disagreement to Strong agreement	−4 to +4 (9p)	T: Face-to-face, individually L: N/A A: N/A	N/A	S: PQMethod E: PCA R: Varimax	NF: 5 VE: 64%	Factor table	Societal scenarios	Time restrictions did not allow to conduct post-sorting interviews
5 (Brannstrom, 2011)	Brazil: Far west of Bahía state	To analyze discourses regarding environmental governance held amongst key actors in a region of expanding high-input, high-output agriculture	PD: EX-PR	CT	26	PS, SS	T: 21 F: 4 M: 17 RD: 24%	Most disagree with my views to Most agree with my views	−4 to +4 (9p)	T: N/A L: N/A A: N/A	∼25 m	S: PQMethod E: N/A R: Varimax	NF: 4 VE: 67%	Z-score table	Societal scenarios	Anguishing of respondents after some difficult choices in sorting
6 (Bumbudsanpharoke et al., 2009)	Thailand: Ping river basin (Chiang Mai province)	To explore perspectives of environmental best practices in Thai agriculture	PD: EX-OR/SD: UN	MM	36	PS	T: 72 F: N/A M: N/A RD: 100%	Disagree to Agree (not clear)	−4 to +4 (9p)	T: Face-to-face L: N/A A: By-step guidance	N/A	S: PQMethod E: PCA R: Varimax	NF: 4 VE: N/A	Factor table	Behavioral adjectives	N/A
7 (Cammelli et al., 2019)	Brazil: Paragominas municipality	To explore smallholders’ perceptions of fire use, fire control, and firefighting in a post-frontier region of the eastern Amazon, and how these relate to fire risk perceptions and governance preferences	PD: NE-PR/SD: UN	MM	17	SS	T: 51 F: N/A M: N/A RD: 100%	Strongly disagrees to Strongly agrees	−2 to +2 (5p)	T: Face-to-face, individually L: Farmers’ houses A: By-step guidance; reading of statements	∼1/2 h	S: N/A E: PCA R: Varimax	NF: 4 VE: 55%	Modified Z-score table	Long societal scenarios	Low literacy levels of respondents
8 (Carmenta et al., 2017)	Indonesia: Riau province	To illustrate the diverse stakeholders’ perceptions of peat fires and fire management interventions across sectors and scales of governance	PD: EX-PR/SD: SL-GL	IR	70	PS	T: 219 F: N/A M: N/A RD: 80%	Least important to Most important/Least effective to Most effective	−3 to +3 (8p)	T: N/A L: N/A A: On-demand reading of statements	N/A	S: qmethod for R E: PCA R: Varimax	NF: 9 (4/5) VE: 47%/39%	Factor table and Z-score diagram	Long societal scenarios	N/A
9 (Forouzani et al., 2013)	Iran: Marvdasht county	To study stakeholders’ perceptions toward agricultural water poverty	PD: EX-PR/SD: UN	MM	54	PS	T: 75 F: N/A M: N/A RD: 67%	Strongly disagree to Strongly agree	−2 to +2 (5p)	T: Face-to-face, individually L: N/A A: N/A	N/A	S: PQMethod E: CFA R: Varimax	NF: 7 (4 – farmers/3 – specialists) VE: 42%/45%	Z-score table	Long behavioral adjectives	N/A
10 (Frate and Brannstrom, 2015)	Brazil: Andalucía village (Nioaque municipality, Mato Grosso do Sul state)	To determine empirically social perspectives regarding ethanol-agrarian reform conflicts, focusing specifically on food security and safety	PD: EX-PR	EJ	26	PS	T: 26 F: N/A M: N/A RD: N/A	Most agree with my views to Most disagree with my views	+4 to −4 (9p)	T: Face-to-face L: N/A A: By-step guidance	∼40 m (sorting); ∼55 m (interview)	S: PQMethod E: N/A R: N/A	NF: 4 VE: 63%	Combined Z-score and factor table	Long societal scenarios	N/A
11 (Giannichi et al., 2018)	Brazil: Mato Grosso state	To identify viewpoints of sellers and buyers of an emerging forest certificate trading scheme	PD: EX-UN/SD: SL-GL	CD	39	PS, SS	T: 59 F: N/A M: N/A RD: 100%	Strongly disagree to Strongly agree	−4 to +4 (9p)	T: Face-to-face L: N/A A: Pre-sort instructions	∼1 h	S: qmethod for R E: PCA R: Varimax	NF: 6 (3/3) VE: 44%/46%	Factor table	Behavioral adjectives	Remote and nonaffiliated farmers were likely undersampled
12 (Hamadou et al., 2016)	Niger: N’dounga, Namaro and Bitinkodji communes	To examine the viewpoints of stakeholders in the management and conservation of farm animal biodiversity in Niger	PD: EX-NR/SD: SL-GL	N/A	48	PS	T: 20 F: N/A M: N/A RD: 35%	I totally disagree to I fully agree	−3 to +3 (7p)	T: Face-to-face L: N/A A: Pre-sort instructions	N/A	S: qmethod for R E: PCA R: Varimax	NF: 3 VE: 61%	Factor table	Behavioral adjectives	Low educational level of several respondents
13 (Hilhorst et al., 2012)	Nepal: Beldangi I refugee camp (Damak municipality)	To explore the mutual imaging of aid workers and refugees in humanitarian crisis contexts	PD: EX-PR/SD: UN	N/A	47	SRS	T: 28 F: N/A M: N/A RD: N/A	Strongly disagree to Strongly agree	N/A (11p)	T: Face-to-face L: N/A A: N/A	N/A	S: PQMethod E: N/A R: N/A	NF: 4 VE: N/A	N/A	Generic labels	N/A
14 (Hu et al., 2018)	China: Rizhao, Tai’an, Xuzhou, Lu’an, Xinyang, Zhumadian, Fuyang, Shangqiu and Luoyang cities	To evaluate the attitudes of Chinese farmers in the period of agricultural supply-side structural reform	N/A	N/A	33	N/A	T: 61 F: 27 M: 35 RD: 100%	Strong disagreement to Strong agreement	−4 to +4 (9p)	T: Face-to-face L: N/A A: N/A	30 m–3 h	S: PQMethod E: PCA R: Varimax	NF: 3 VE: 44%	Combined Z-score and factor table	Behavioral adjectives	N/A
15 (Huaranca et al., 2019)	Argentina: Salta province	To identify and describe social perspectives on deforestation, land use change and economic development	SD: SL-GL	CT	68	PS	T: 60 F: N/A M: N/A RD: 17%	Less agreement to More agreement	−5 to +5 (11p)	T: Face-to-face/Online (POETQ) L: N/A A: N/A/Software-led	N/A	S: PQMethod E: PCA R: Varimax	NF: 4 VE: 54%	Z-score table	Societal scenarios	N/A
16 (Hugé et al., 2016)	Malaysia: Mangrove Forest Reserve, Perak state	To identify perspectives on the management of Matang Mangrove Forest Reserve	PD: EX-NR/SD: SL-GL	CT	48	SS	T: 29 F: N/A M: N/A RD: 31%	Least agreement to Strongest agreement	−3 to +3 (7p)	T: Face-to-face L: N/A A: N/A	N/A	S: PQMethod E: CFA R: Varimax	NF: 3 VE: 41%	Factor arrays	Long societal scenarios	Participants had difficulties at times while sorting Q was difficult or impossible to administer to some rural dwellers, due to social, educational and cultural barriers Some respondents tended to please researchers rather than answering authentically
17 (Jaung et al., 2016)	Indonesia: West Lombok district	To analyze diverse perspectives on the certification of a payment for watershed services scheme	PD: EX-NR/SD: SL	N/A	48	PS, SS	T: 19 F: N/A M: N/A RD: 21%	Most disagreed to Most agreed	−6 to +6 (13p)	T: Face-to-face L: N/A A: By-step guidance	N/A	S: PQMethod E: PCA R: Varimax	NF: 3 VE: 51%	Factor table	Long societal scenarios	N/A
18 (Jiren et al., 2020)	Ethiopia: Southwestern Ethopia (Jimma zone)	To examine alternative approaches to food security and biodiversity conservation	PD: EX-NR/SD: SL-GL	CT	32	PS	T: 50 F: N/A M: N/A RD: 2%	Most important to Least important	+4 to −4 (vertical) (9p)	T: Face-to-face, individually L: N/A A: Pre-sort instructions	N/A	S: qmethod for R E: PCA R: Varimax	NF: 4 VE: 48%	Z-score table	Societal scenarios	N/A
19 (Kopytko and Pruneddu, 2018)	Ukraine: Kyiv, Kherson, Ternopil, Lviv and Chernihiv municipalities	To reveal barriers to crop rotation	PD: EX-PR	N/A	27	PS, SS	T: 10 F: N/A M: N/A RD: 20%	Strongly agree to Strongly disagree	+3 to −3 (7p)	T: Online (Q sortware) L: N/A A: Software-led	N/A	S: qmethod for R 32.2 E: N/A R: Varimax	NF: 3 VE: 53%	Factor table and Z-score diagram	Societal scenarios	Respondents were difficult to reach
20 (Lairez et al., 2020)	Laos: Leng, Lé, Xay, Nadou villages (Kham district, Xieng Khouang province)	To understand farmers’ perceptions of soil fertility	PD: NE-UN	N/A	47	N/A	T: 19 F: N/A M: N/A RD: 100%	Disagree to Agree	−5 to +5 (11p)	T: Face-to-face L: Respondents’ houses A: By-step guidance; reading of statements; on-demand clarification	N/A	S: PQMethod E: CFA R: Varimax	NF: 3 VE: N/A	Factor table	Mixed (scenarios and adjectives)	N/A
21 (Lansing, 2013)	Costa Rica: Talamanca region	To examine perspectives about baseline constructions of land-use change, and the potential role of offsets in an indigenous community	PD: EX-PR/SD: SL-GL	CT	30	PS	T: 15 F: 4 M: 11 RD: 67%	Most disagree to Most agree	−4 to +4 (9p)	T: Face-to-face L: N/A A: N/A	N/A	S: PQMethod E: N/A R: Varimax	NF: 3 VE: 46%	Factor table	Behavioral adjectives	N/A
22 (Leite et al., 2019)	Brazil: Santa Catarina region	To analyze the perception of farmers on the landscape and its dynamics	SD: SL-GL	N/A	16	SS	T: 90 F: 45 M: 45 RD: 100%	Higher agreement to Less agreement	+3 to −3 (7p)	T: Face-to-face L: N/A A: N/A	N/A	S: PQMethod E: PCA R: Varimax	NF: 3 VE: 48%	Factor table	Societal scenarios	N/A
23 (Mayett-Moreno et al., 2017)	Mexico: La Preciosita Sangre de Cristo village (Puebla state)	To understand young farmers’ views of a community-based ecotourism project	PD: EX-OR	CA	36	PS	T: 16 F: 5 M: 11 RD: 100%	I don’t identify with to I identify with	−5 to +5 (11p)	T: Face-to-face, collectively L: N/A A: N/A	N/A	S: PQMethod E: PCA R: Varimax	NF: 3 VE: N/A	Z-score table, factor arrays and factor table	Societal scenarios	N/A
24 (Moros et al., 2020)	Colombia: Bogotá, Cali, Buenaventura, Medellín municipalities	To explore the discourses that underpin Payment for Ecosystem Services debates and practice in Colombia	SD: SL-GL	CD, CT, EJ	36	PS, SS	T: 41 F: N/A M: N/A RD: N/A	Strongly disagree to Strongly agree	−4 to +4 (9p)	T: Face-to-face L: N/A A: Pre-sort instructions; by-step guidance; on-demand clarification	∼1 h	S: qmethod for R E: PCA R: Varimax	NF: 3 VE: 59%	Combined Z-score and factor table	Societal scenarios	N/A
25 (Nguyen et al., 2018)	Vietnam: Phuoc Duc and Tam Lanh communes (Quang Nam province)	To examine community perceptions concerning the impact of gold mining	SD: UN	N/A	25	SS	T: 137 F: 49 M: 88 RD: 88%	Least important to Most important	−4 to +4 (9p)	T: Face-to-face, individually L: N/A A: Pre-sort instructions	N/A	S: N/A E: PCA R: Varimax	NF: 8 (5/3) VE: 51.23%/43%	Factor table	Generic labels	N/A
26 (Nhem and Lee, 2020)	Cambodia: Oddar Meanchey province	To assess the perceptions of local stakeholders towards the quality of governance of the first REDD + pilot project in Cambodia	SD: SL-GL	CD	40	PS	T: 52 F: N/A M: N/A RD: 100%	Most strongly disagree to Most strongly agree	−4 to +4 (9p)	T: Face-to-face, individually L: Respondents’ houses A: Pre-sort instructions; by-step guidance	N/A	S: PQMethod E: PCA R: Varimax	NF: 4 VE: 35%	Z-score table	Societal scenarios	N/A
27 (Nhem and Lee, 2019)	Cambodia: Oddar Meanchey province	To investigate the views of local experts on the sustainability of community-based forestry management	SD: SL-GL	EJ	43	SS	T: 52 F: 8 M: 44 RD: 100%	I Most Disagree to I Most Agree	−5 to +5 (11p)	T: Face-to-face, individually L: N/A T: By-step guidance	∼50 m	S: PQMethod E: PCA R: Varimax	NF: 4 VE: 37%	Modified Z-score table	Societal scenarios	N/A
28 (Nordhagen et al., 2017)	Papua New Guinea: Eastern Highlands and Morobe provinces	To examine the motivations for growing diverse crops amongst semi-subsistence rural farmers in Papua New Guinea	PD: NE-PR	CD	31	RS, SS	T: 92 F: 50 M: 42 RD: 100%	Don’t agree at all to Agree very much	−3 to +3 (7p)	T: Face-to-face L: N/A A: By-step guidance; reading of statements	N/A	S: PQMethod E: PCA R: Varimax	NF: 5 VE: N/A	Factor table	Behavioral adjectives	Low literacy level of respondents
29 (Pereira et al., 2016)	Brazil: Mato Grosso do Sul state	To assess the diversity of values and goals amongst Brazilian commercial-scale progressive beef farmers	SD: UN	N/A	49	PS, SRS	T: 26 F: 0 M: 26 RD: 100%	Strong disagreement to Strong agreement	−4 to +4 (9p)	T: Face-to-face L: N/A A: N/A	∼1.5 h	S: PQMethod E: PCA R: Varimax	NF: 4 VE: 62%	Factor table	Behavioral adjectives	Respondents were all male; they were the main decision makers, although some women were the legal owners of the farms
30 (Pirard et al., 2016)	Indonesia: East Kalimantan province	To study the general perceptions of rural populations toward industrial plantations	PD: NE-PR	EJ	30	SRS	T: 31 F: 12 M: 19 RD: 100%	Most disagree with to Most agree with	−3 to +3 (8p)	T: Face-to-face, three-person rounds L: N/A A: Pre-sort instructions; on-demand clarification	20 m–1 h	S: PQMethod E: PCA R: Varimax	NF: 3 VE: N/A	Factor arrays and factor table	Mixed (scenarios and adjectives)	Respondents had different levels of fluency in Indonesian
31 (Rijneveld and Marhaento, 2020)	Indonesia: Getas village (Blora district), Pitu village (Ngawi district)	To explore stakeholder perceptions of the effects of agroforestry and mono-cropping systems on water use	SD: SL-GL	N/A	17	N/A	T: 33 F: N/A M: N/A RD: N/A	Fully agree to Fully disagree	+2 to −2 (5p)	T: Face-to-face; individual and collective L: N/A A: N/A	N/A	S: qmethod for R E: N/A R: N/A	NF: 3 VE: N/A	Z-score diagram	Generic labels	N/A
32 (Rodriguez-Piñeros et al., 2012)	Mexico: La Preciosita Sangre de Cristo village (Puebla state)	To identify perspectives on sustainable management of a community-owned forest reserve	PD: EX-OR	CT	36	N/A	T: 20 F: 9 M: 1 RD: 100%	Unlike me to Like me	−4 to +4 (9p)	T: N/A L: N/A A: N/A	N/A	S: PQMethod E: PCA R: Varimax	NF: 3 VE: 65%	Z-score and factor tables	Societal scenarios	N/A
33 (Rodríguez-Piñeros et al., 2018)	Colombia: Cimitarra, Magangué, Guapi, Pasto and Tuluá municipalities	To understand stakeholders’ values about Colombian forest legislation and its implications for legal timber trade	PD: EX-UN	EJ, SW	34	PS	T: 27 F: 5 M: 22 RD: 19%	Less affect me to Most affect me	−4 to +4 (9p)	T: Face-to-face, individually L: N/A A: N/A	N/A	S: PQMethod E: PCA R: Varimax	NF: 4 VE: 52%	Z-score table	Long societal scenarios	N/A
34 (Rodríguez-Piñeros and Mayett-Moreno, 2015)	Mexico: La Preciosita Sangre de Cristo village (Puebla state)	To examine perceptions of small forest owners to build infrastructure in their forest as part of their community-based ecotourism project, and, to explore their underlying reasons to retain the forest for ecotourism	PD: EX-OR/SD: SL-GL	CA	32	PS	T: 14 F: 7 M: 7 RD: 100%	Unlike me to Like me	−4 to +4 (9p)	T: Face-to-face L: Respondents’ houses A: N/A	N/A	S: PQMethod E: PCA R: Varimax	NF: 3 VE: 56%	Z-score table	Long societal scenarios	N/A
35 (Rust, 2017)	Namibia	To investigate whether a diverse range of stakeholders could agree on how to mitigate conflict between carnivores and livestock farmers in Namibia	PD: EX-PR/SD: GL	CD	36	PS, SS	T: 35 F: N/A M: N/A RD: 40%	Strongly disagree to Strongly agree	−3 to +3 (7p)	T: Online (SurveyMonkey) L: N/A A: Software-led	N/A	S: PQMethod E: PCA R: Varimax	NF: 2 (2 in each of three rounds) VE: 51%/45%/57%	Z-score table	Generic labels	N/A
36 (Schneider et al., 2015)	Brazil: São Felix do Xingu (Pará state)	To study small-scale farmers’ needs to end deforestation	PD: NE-UN	N/A	29	SS	T: 14 F: 0 M: 14 RD: 100%	Strongly disapproves to Strongly approves	−2 to +2 (5p)	T: Face-to-face L: Farmers’ villages A: N/A	∼30 m	S: N/A E: PCA R: Varimax	NF: 3 VE: N/A	Z-score table	Behavioral adjectives	Due to remoteness researchers had to rely on an organization to reach participants Communications difficulties hampered higher participation rates Some women farmer refused to participate due to shyness Some participants found difficult to make sorting choices
37 (Schuman et al., 2018)	South Africa: Jouberton and Ikageng communities, Ventersdorp municipality	To study the influence of religious beliefs on the understanding and experience of climate change adaptation	PD: EX-OR	N/A	40	PS, SS	T: 25 F: N/A M: N/A RD: N/A	Strongly agree to Strongly disagree	+3 to −3 (7p)	T: Face-to-face L: N/A A: Pre-sort instructions	N/A	S: PQMethod E: PCA R: N/A	NF: 5 VE: 58%	Factor table	Mixed (scenarios and adjectives)	N/A
38 (Stoudmann et al., 2017)	Madagascar: Alaotra-Mangoro and Analanjirofo regions	To gain a better understanding of farmers’ definitions and experiences of change	PD: EX-OR	N/A	52	SS	T: 30 F: N/A M: N/A RD: 100%	Most disagree to Most agree	1 to 11 (11p)	T: Face-to-face, individually L: N/A A: N/A	N/A	S: PQMethod E: N/A R: Varimax	NF: 5 VE: 38%	Combined Z-score and factor table	Societal scenarios	Possible dissimilarities in terms translated between languages and local dialects
39 (Sumberg et al., 2017)	Ghana: Tepa (Ashanti region), Savelugu (Northern region)	To understand young people’s attitudes towards farming; and, to understand what should be done about rural young people and farming	PD: NE-UN/SD: SL-GL	N/A	32	SS	T: 38 F: 20 M: 18 RD: 100%	Most disagree to Most agree	−3 to +3 (7p)	T: Face-to-face L: N/A A: Pre-sort instructions; by-step guidance; on-demand clarification	N/A	S: PQMethod E: PCA R: Varimax	NF: 8 (4/4) VE: N/A	Factor table	Long societal scenarios	N/A
40 (Taheri et al., 2020)	Iran: Khuzestan province	To understand how farmers perceive the impact of dust phenomenon on agricultural production activities	PD: EX-PR/SD: GL	CT, EJ	48	PS, SS	T: 60 F: 0 M: 60 RD: 100%	Strongly disagree to Strongly agree	−2 to +2 (5p)	T: Face-to-face L: N/A A: N/A	N/A	S: PQMethod E: CFA R: Varimax	NF: 3 VE: 54%	N/A	Behavioral adjectives	N/A
41 (Truong et al., 2017)	Vietnam: Long An and Tay Ninh provinces	To explore farmers’ perceptions of foot-and-mouth disease vaccination	PD: EX-OR	N/A	46	STS	T: 46 F: 13 M: 29 RD: 100%	Strongly in disagreement to Strongly in agreement	−3 to +3 (7p)	T: Face-to-face, individually L: Usually farmers’ houses A: Pre-sort instructions	>1 h	S: qmethod for R E: PCA R: Varimax	NF: 3 VE: 57%	Factor arrays and Z-score table	Societal scenarios	Low degree of farmer participation Some participants did not follow instructions Too short post-sorting answers due to tiredness after long sorting times Possible response bias due to pre-sorting discussions Some participants complained about unclear or complicated statements
42 (Truong et al., 2019)	Vietnam: Dong Thap province (Mekong delta)	To identify poultry farmers’ drivers to use high amounts of antimicrobials	PD: EX-PR	CT	46	RS, SS	T: 54 F: 6 M: 48 RD: 52%	Extremely disagree to Extremely agree	−3 to +3 (7p)	T: Face-to-face L: Usually farmers’ houses A: Pre-sort instructions	∼1 h	S: R E: PCA R: N/A	NF: 7 (4/3) VE: 55%/50%	Factor arrays and factor table	Societal scenarios	Sampled target population changed due to unpredictable field constraints Some participants misunderstood instructions or were not willing to complete the sorting Husbands were usually the respondents, although wives were the ones conducting the farm work
43 (Tuokuu et al., 2019)	Ghana: Tarkwa-Nsuaem municipality	To understands areas of agreements and disagreements amongst critical stakeholders as a way to improve environmental policy development and implementation	PD: EX-PR/SD: SL-GL	CD	31	PS	T: 15 F: N/A M: N/A RD: N/A	Least effective to Most effective	−3 to +3 (7p)	T: Face-to-face L: N/A A: By-step guidance	N/A	S: R E: N/A R: Varimax	NF: 3 VE: 46%	Factor table and factor arrays	Societal scenarios	N/A
44 (Vargas et al., 2019)	Colombia: Luruaco and Santa Catalina municipalities	To explore the understanding of the conservation-development relationship by a rural community in Colombia	PD: EX-UN	CT	40	RS, PS	T: 38 F: 20 M: 16 RD: 100%	Disagree most to Agree most	−4 to +4 (9p)	T: Face-to-face, collectively L: N/A A: N/A	N/A	S: PQMethod E: PCA R: Varimax	NF: 4 VE: 51%	Crib sheets	Societal scenarios	N/A
45 (Vela-Almeida et al., 2018)	Ecuador: Quito municipality; Tundayme and Los Encuentros parishes (Zamora Chinchipe province); San Juan Bosco municipality (Morona Santiago province)	To assess subjective perceptions of key stakeholders regarding mining projects	SD: UN	N/A	30	SS	T: 47 F: N/A M: N/A RD: N/A	Agree the most to Disagree the most	N/A	T: N/A L: N/A A: Pre-sort instructions	N/A	S: PQMethod E: CFA R: Varimax/Judgmental	NF: 4 VE: 62%	Factor table	Societal scenarios	N/A
46 (Weldegiorgis and Ali, 2016)	Rwanda: Murambi and Masoro communities (Rulindo district)	To quantitatively analyze qualitative perspectives regarding impacts of mining-led development at a district level in Rwanda	PD: EX-PR/SD: GL	CT	46	PS, SS	T: 49 F: 15 M: 34 RD: 71%	Fully Disagree to Fully Agree	−6 to +6 (13p)	T: Face-to-face L: Communities, offices A: Pre-sort instructions	N/A	S: PQMethod E: PCA R: Varimax	NF: 3 VE: 41%	Factor arrays and factor table	Long societal scenarios	Some respondents sorted out of the distribution grid; at times they were unable to prioritize statements Some respondents criticized the forced distribution
47 (Wijaya and Offermans, 2019)	Indonesia: North Luwu, Bondowoso and Pelalawan districts	To identify sustainability perspectives in agriculture	PD: EX-PR/SD: SL-GL	CT, CD	37	PS	T: 64 F: 13 M: 51 RD: 52%	Strongly disagree to Strongly agree	−4 to +4 (9p)	T: Face-to-face L: N/A A: Pre-sort instructions; by-step guidance; reading of statements; on-demand clarification	N/A	S: PQMethod E: PCA R: Varimax	NF: 2 VE: 33%	Modified Z-score table	Behavioral adjectives	Gender diversity was hard to accomplish in the field due to cultural constraints
48 (Yeboah et al., 2017)	Ghana: Tepa (Ashanti region), Savelugu (Northern region)	To explore perspectives of job desirability	PD: NE-UN/SD: SL-GL	N/A	59	SS	T: 72 F: 29 M: 43 RD: 100%	Most disagree to Most Agree	−4 to +4 (9p)	T: Face-to-face L: N/A A: Pre-sort instructions; by-step guidance; on-demand clarification	65 m (students); 50 m (parents)	S: PQMethod E: PCA R: Varimax	NF: 19 (8/11) VE: N/A	Factor summary table	Long societal scenarios	The initial aim was to interview students and their parents; because some parents could not or were unwilling to participate, the authors resorted to parents of non-participant students
49 (Zabala et al., 2017)	Mexico: Chiapas state	To identify smallholders’ views with regard to silvopastoral practices in a community in the forest frontier in Chiapas	PD: EX-PR/SD: UN	CT	26	PS	T: 32 F: 0 M: 32 RD: 100%	Most disagree to Most agree	−3 to +3 (7p)	T: Face-to-face L: N/A A: N/A	N/A	S: qmethod for R E: PCA R: Varimax	NF: 3 VE: N/A	Z-score diagram	Behavioral adjectives	Lack of female respondents because male head of household is the decision maker
50 (Zobeidi et al., 2016)	Iran: Hamidiyeh district	To investigate and understand farmers’ perceptions regarding climate change	PD: EX-PR/SD: SL-GL	MM	42	PS, SS	T: 46 F: N/A M: N/A RD: 100%	Most disagree to Most agree	−2 to +2 (5p)	T: Face-to-face, individually L: Extension offices, homes, farms A: By-step guidance	∼80 m	S: PQMethod E: CFA R: Varimax	NF: 3 VE: 56%	Factor table	Mixed (scenarios and adjectives)	N/A

List of acronyms in the table:

In Concourse – Sources column: primary data (PD); not-exclusive for the study (NE); exclusive for the study (EX); unspecified sources (UN); partially rural dwellers sources (PR); only rural dweller sources (OR); no rural dweller sources (NR); secondary data (SD); grey literature (GL); scientific literature (SL).

In Concourse – Reduction Technique (RT) column: expert judgement (EJ); division in discourses (DD); matrix method (MM); software (SW); categorization (CT); iterative refinement (IR); combination and deletion of similar statements (CD); content analysis (CA).

In P-set – Sampling Technique (ST) column: snowball sampling (SS); purposive sampling (PS); convenience sampling (CS); random sampling (RS); stratified random sampling (SRS); structured sampling (STS).

In P-set – Size column: total (T); female (F); male (M); percentage of rural dwellers (RD).

In Sorting grid – Scale column: sorting points (p).

In Administration column: administration technique (T), location (L), assistance (A).

In Analysis column: software (S); factor extraction method (E); factor rotation method (R); Principal Component Analysis (PCA); Centroid Factor Analysis (CFA).

In Factors – Results column: number of retained factors (NF); total variance explained (VE).

3.1. Characteristics of studies

The selected studies belonged mainly to the subjects² of environmental studies (n = 15), conservation (n = 7), forest and forestry (n = 5), agriculture (n = 4), and international development (n = 4) (Figure 2a). Whereas rural studies have traditionally focused on these subjects (Strijker et al., 2020; Wang and Liu, 2014), these leave aside other relevant yet still neglected (Q) research themes in the RGS, including subjects such as rural health, women empowerment, food safety, environmental justice, responsible mechanization, and education. Furthermore, we categorized the 50 selected studies according to the Q themes proposed in (Zabala et al., 2018). Most of these studies are within the category of management alternatives (n = 36), with two other categories worth mentioning being conflict resolutions (n = 4) and policy appraisals (n = 10) (Figure 2b).

Figure 2. Characteristics of the selected studies. (a) Number of studies across disciplines of agriculture (AG), international development (ID), environmental studies (ES), conservation (CO), forests and forestry (FF), veterinary sciences (VS), and others (OT). (b) Number of studies per category of Q study [as defined by (Zabala et al., 2018)] as conflict resolution (CR), management alternatives (MA) and policy appraisal (PA). (c) Number of studies published as (non)open-access documents. (d) Number of studies per geographical region, across Australasia (AA), East Asia (EA), Eastern Europe (EE), Middle East (ME), Central America (CA), South America (SA), South Asia (SAs), Southeast Asia (SEA) and sub-Saharan Africa (SSA); solid dark gray, solid light gray and diagonal-line patterns on each bar represent the proportions of low-, lower-middle- and upper-middle-income countries, respectively.

Only 16% (n = 8) of the selected papers were published as open-access (Carmenta et al., 2017; Giannichi et al., 2018; Mayett-Moreno et al., 2017; Rust, 2017; Schuman et al., 2018; Sumberg et al., 2017; Truong et al., 2017; Truong et al., 2019) (Figure 2a). Given the financial, legal, and technical restrictions faced by low- and middle-income countries, open access to scientific knowledge and data is crucial in the development of their research (Arunachalam, 2017; Chan et al., 2005; Serwadda et al., 2018; Zachariah et al., 2014). It seems paradoxical that, to a large extent, the selected studies, which can directly benefit (Q) researchers in Global South countries, are not (easily) accessible to these scholars.

3.2. Geographical foci

Despite their strong focus on RGS populations, only two publications (Schuman et al., 2018; Truong et al., 2019) were authored by researchers exclusively affiliated with institutions in their respective target countries. As illustrated on the world map in Figure 3, most studies were conducted by (main) authors exclusively (n = 25) or partially affiliated (n = 6) with organizations located in countries of the Global North. The selected studies showed a strong emphasis on Southeast Asia (n = 15), South America (n = 14), and sub-Saharan Africa (n = 9) (Figures 2d and 3). As represented in Figure 2d, only 10% of the studies (n = 5) (Hamadou et al., 2016; Hilhorst et al., 2012; Jiren et al., 2020; Stoudmann et al., 2017; Weldegiorgis and Ali, 2016) aimed specifically at low-income countries, which bear the weakest economic category, where the livelihoods of RGS dwellers face more profound subsistence challenges. Moreover, none of the studies that focused on low-income countries were (exclusively) carried out by researchers and institutions within their national boundaries, nor from any other Global South country (Figure 3). This might reflect the access and equality issues that researchers from these geographical areas have to confront³. This clear decoupling between the places where the study has been envisaged and carried out, and where the data have been collected, could pose even further constraints for the (still limited) research capacities in the Global South. According to (ESSENCE on Health Research, 2014), research capacity building should be a long-term, explicit process that must go beyond the temporal scope of a single project or grant, whereas (Shucksmith and Brown, 2016) advocates the international co-production of knowledge between a number of (non)academic actors whose outputs must be more accessible and understandable for wider audiences. In addition, this detachment, which leads to sporadic, spatially biased contacts, could play against robust relationships and trust between researchers and communities, which are key requirements in rural studies (Chambers, 2017; Chambers, 1983).

Figure 3. Geographic location of main authors’ affiliations and studies per the theme of Q study [as defined by (Zabala et al., 2018)].

3.3. Research design

3.3.1. Concourse development

A minority (n = 15) of the selected studies relied purely on primary data for the development of their concourse, either exclusively for their respective studies (Brannstrom, 2011; Frate and Brannstrom, 2015; Kopytko and Pruneddu, 2018; Mayett-Moreno et al., 2017; Rodriguez-Piñeros et al., 2012; Rodríguez-Piñeros et al., 2018; Schuman et al., 2018; Stoudmann et al., 2017; Truong et al., 2017; Truong et al., 2019; Vargas et al., 2019) or as part of a larger umbrella project (Lairez et al., 2020; Nordhagen et al., 2017; Pirard et al., 2016; Schneider et al., 2015). Moreover, only five of these studies relied solely on RGS dwellers (Mayett-Moreno et al., 2017; Rodriguez-Piñeros et al., 2012; Schuman et al., 2018; Stoudmann et al., 2017; Truong et al., 2017). Approximately half of the studies (n = 23) employed a mixed primary/secondary data approach (Alexander et al., 2018; Astari and Lovett, 2019; Bumbudsanpharoke et al., 2009; Cammelli et al., 2019; Carmenta et al., 2017; Forouzani et al., 2013; Giannichi et al., 2018; Hamadou et al., 2016; Hilhorst et al., 2012; Hugé et al., 2016; Jaung et al., 2016; Jiren et al., 2020; Lansing, 2013; Rodríguez-Piñeros and Mayett-Moreno, 2015; Rust, 2017; Sumberg et al., 2017; Taheri et al., 2020; Tuokuu et al., 2019; Weldegiorgis and Ali, 2016; Wijaya and Offermans, 2019; Yeboah et al., 2017; Zabala et al., 2017; Zobeidi et al., 2016), whereas 11 studies used only secondary data (Anderson and Jacobson, 2018; Barbosa et al., 2020; Huaranca et al., 2019; Leite et al., 2019; Moros et al., 2020; Nguyen et al., 2018; Nhem and Lee, 2020; Nhem and Lee, 2019; Pereira et al., 2016; Rijneveld and Marhaento, 2020; Vela-Almeida et al., 2018) (Figure 4a). The concourses varied in size from as small as 42 (Brannstrom, 2011) to as large as 419 statements (Bumbudsanpharoke et al., 2009) (Figure 4b).

Figure 4. Characteristics of the concourse. (a) Number of studies per source for concourse construction, based on primary data (PD), secondary data (SD) and mixed sources (PD/SD). (b) Size of the constructed concourse in a number of statements across studies. (c) Number of studies per concourse reduction technique, comprising software (SW), matrix method (MM), iterative refinement (IR), expert judgement (EJ), division in discourses (DD), categorization (CT), combination and deletion of similar statements (CD), content analysis (CA). (d) Concourse reduction ratio across studies, expressed as the decreasing percentage between the concourse and the Q-set.

The development of the concourse requires time and rigor to ensure that the eventual Q-set represents an acceptable range of voices involved in the topic under study (Simons, 2013; Watts & Stenner, 2012). Although the concourse can be built purely from secondary data (Donner, 2001), it makes sense to incorporate primary data to guarantee proper representation of the range of discourses (Simons, 2013). When addressing understudied topics, geographic areas, and/or human groups, primary data collection for concourse development from RGS dwellers might become the only (or at least main) option. Seven studies (Nordhagen et al., 2017; Rodriguez-Piñeros et al., 2012; Rodríguez-Piñeros et al., 2018; Schuman et al., 2018; Stoudmann et al., 2017; Truong et al., 2017; Truong et al., 2019) are remarkable examples of such cases, especially because of their exhaustive primary data sources. In certain cases, however, RGS dwellers may be located in too remote—or ultimately almost unreachable—areas, or their political-cultural values or legal status could hide potential participants (e.g., lower-caste individuals, refugees and displaced groups, women of particular societies, individuals involved in illegal activities). Moreover, purely primary data collection for the concourse is not always applicable nor is perhaps the best approach when (financial) resources are a main limiting factor (Barbosa et al., 2020; Schneider et al., 2015) or when it is difficult to (re)visit participants (Giannichi et al., 2018; Kopytko and Pruneddu, 2018; Schneider et al., 2015; Truong et al., 2019; Yeboah et al., 2017).

Considering these possible limitations, three strategies for concourse development should be considered. First, (partially) resort to reliable secondary data, mainly if produced around the same study area or population. Second, reuse primary data from previous fieldwork activities, especially when they were part of a larger research program, as applied by (Alexander et al., 2018; Cammelli et al., 2019; Schneider et al., 2015). Third, as reported in (Astari and Lovett, 2019; Hamadou et al., 2016; Jiren et al., 2020; Kopytko and Pruneddu, 2018; Pirard et al., 2016; Rust, 2017; Taheri et al., 2020; Truong et al., 2019; Wijaya and Offermans, 2019), to build the concourse based on proxies’ discourses (i.e., experts, advisors, scholars, etc.), although researchers must be aware of its potential compromise in the accuracy and representativeness of the viewpoints (Cobb, 2018).

3.3.2. Concourse reduction

There is no specific recipe or fixed methodology on how to reduce the collected concourse to statements, let alone the number of statements required by the study. An appropriate approach is to consider the coverage and balance of the statements in such a way that they become as equally representative and balanced as possible across the different discourses (Watts & Stenner, 2012; Zabala et al., 2018). The reduction process should not eliminate any relevant statement of certain discourse(s), given that it will provoke further biases in the later sorts. Here, it may be good to remind ourselves that the Q set aims to create possible combinations between statements as expressions of diverse perspectives; as such, individual statements should represent sufficient diversity themselves but would not need to cover every possible perspective as such.

Most of the selected studies (Brannstrom, 2011; Huaranca et al., 2019; Hugé et al., 2016; Jiren et al., 2020; Lansing, 2013; Rodriguez-Piñeros et al., 2012; Truong et al., 2019; Vargas et al., 2019; Weldegiorgis and Ali, 2016; Zabala et al., 2017) relied on a reductionist technique of categorization, that is, classification into different categories within the found discourses, to filter statements out of the general concourse. Other studies (Barbosa et al., 2020; Giannichi et al., 2018; Nhem and Lee, 2019; Nordhagen et al., 2017; Rust, 2017; Tuokuu et al., 2019) applied a basic method of combining similar statements and deleting duplicates, redundant, and/or unclear ones. Other less frequent methods for selection of statements were purely expert judgement (Alexander et al., 2018; Anderson and Jacobson, 2018; Frate and Brannstrom, 2015; Nhem and Lee, 2019; Pirard et al., 2016; Rodríguez-Piñeros et al., 2018), matrix method (Astari and Lovett, 2019; Bumbudsanpharoke et al., 2009; Cammelli et al., 2019; Forouzani et al., 2013; Zobeidi et al., 2016), content analysis (Mayett-Moreno et al., 2017; Rodríguez-Piñeros and Mayett-Moreno, 2015), division of statements according to found discourses (Astari and Lovett, 2019), and funnel-like iterative refinement (Carmenta et al., 2017). Other authors (Moros et al., 2020; Taheri et al., 2020; Wijaya and Offermans, 2019) have employed combinations of these techniques. Moreover, (Astari and Lovett, 2019) and (Rodríguez-Piñeros et al., 2018) were the only studies that used specific qualitative data analysis software (Nvivo 11 and ATLAS.ti 7.5.9, respectively) to make a systematic selection of statements (Figure 4c).

Q studies dealing with conflict resolution may produce an unbalanced representation of discourses, typically in favor of the most powerful voices, while reducing the concourse. This could be more exacerbated when involving less-empowered RGS individuals compared to other stronger actors (Vela-Almeida et al., 2018; Weldegiorgis and Ali, 2016). Here, the matrix method becomes interesting, as it aims to capture several dimensions of both discourses and categories of statements, thereby ensuring representativeness across viewpoints. Three studies (Astari and Lovett, 2019; Forouzani et al., 2013; Zobeidi et al., 2016) enriched this technique using political discourse theory, as explained by (Dryzek and Berejikian, 1993).

There is no ideal concourse reduction percentage; it largely depends on the concourse type, number of sources, and amount of information extracted into the initial statements. As such, this percentage has been found to be not uniform across the selected studies. Of the 23 studies that provided sufficient information to calculate this reduction, two (Anderson and Jacobson, 2018; Brannstrom, 2011), 13 (Forouzani et al., 2013; Giannichi et al., 2018; Hugé et al., 2016; Lansing, 2013; Nhem and Lee, 2020; Nhem and Lee, 2019; Pereira et al., 2016; Rust, 2017; Taheri et al., 2020; Tuokuu et al., 2019; Vargas et al., 2019; Zabala et al., 2017; Zobeidi et al., 2016), six (Barbosa et al., 2020; Frate and Brannstrom, 2015; Moros et al., 2020; Rodriguez-Piñeros et al., 2012; Rodríguez-Piñeros et al., 2018; Wijaya and Offermans, 2019), and two (Bumbudsanpharoke et al., 2009; Leite et al., 2019) reported reductions of <50%, 50%–75%, 75%–90% and even up to >90%, respectively (Figure 4d).

3.3.3. Q-set (size)

The size of the Q-set across studies ranged from 16 to 70, although most were around 30-50 (Figure 5a). The decision on the Q-set size should not be underestimated, nor should it be considered as a mere output of the concourse reduction process. Some authors have reported ideal sizes as high as 40–80, ≥40, ≥60 and 60–90 (Watts & Stenner, 2012). Large Q-sets enlarge the (already time-consuming) sorting process, thereby possibly discouraging respondents and eventually increasing the dropout rate (Previte et al., 2007; Simons, 2013; Stone et al., 2017). In light of these two antagonistic positions and considering RGS-related constraints for Q (e.g., illiteracy, improper site conditions, exposure to elements), researchers may be inclined to keep a highly reduced number of statements (Alexander et al., 2018; Cammelli et al., 2019; Nordhagen et al., 2017; Sumberg et al., 2017), without compromising the representativeness of the discourses.

Figure 5. Characteristics of research design. (a) Q-set size in a number of statements across studies. (b) Number of studies per P-set sampling techniques, including convenience sampling (CS), purposive sampling (PS), random sampling (RS), stratified random sampling (SRS), snowball sampling (SS) and structured sampling (STS). (c) P-set size in number of participants across studies. (d) Q-set/P-set ratio across studies.

3.3.4. Q-set (presentation of statements)

The vast majority of the selected studies (n = 47) presented statements solely in written form. Exceptions to this are (Carmenta et al., 2017) which also included images (though not specified) next to written statements; (Barbosa et al., 2020) which would suggest the use of illustrations along with the wording; and (Alexander et al., 2018) which was the only one conducted with a photo-based Q-set supported by proxy statements. The latter was intentionally chosen, along with just 16 statements, to reduce the complexity of engaging semi-literate Laotian farmers. Researchers may encounter other potential limitations besides illiteracy. For instance, participants with visual conditions (e.g., visual impairment and color blindness) would require visual items to be carefully implemented. Some authors have employed high-contrast designs and even statements written in Braille (Huang and Yu, 2013; Salaj and Kiš-Glavaš, 2017), whereas others advocate for non-conventional audiovisual-based Q-sets (Nazariadli et al., 2019). It is noteworthy that the latter are usually attached to digital tools and software such as VQMethod (Nazariadli, 2020), whose availability and/or applicability could be compromised in RGS contexts.

Most of the selected studies (n = 36) presented their statements written in a Latin script language (Afaan Oromo, English, French, Malagasy, Malay, Indonesian, Kinyarwanda, Portuguese, Spanish, Tok Pisin, Tswana, and Afrikaans). From these, (Astari and Lovett, 2019) and (Hugé et al., 2016) worked with a combination of Indonesian/English and Malay/English, respectively, whereas (Stoudmann et al., 2017) presented a unique successive translation of French, Malagasy, Sihanaka dialect, and Betsimisaraka dialect. In contrast, 14 studies (Alexander et al., 2018; Bumbudsanpharoke et al., 2009; Forouzani et al., 2013; Hilhorst et al., 2012; Hu et al., 2018; Kopytko and Pruneddu, 2018; Lairez et al., 2020; Nguyen et al., 2018; Nhem and Lee, 2020; Nhem and Lee, 2019; Taheri et al., 2020; Truong et al., 2017; Truong et al., 2019; Zobeidi et al., 2016) were conducted in non-Latin script languages (Khmer, Lao, Mandarin, Nepali, Persian, Thai, and Ukrainian). Although the latter does not seem to pose any inconvenience for the administration of hand-written Q-sets, it certainly might bear further limitations for researchers willing to rely on digital/electronic platforms and tools (Nazariadli, 2020; University of Birmingham, 2010; SurveyMonkey, 2020; Pruneddu, 2011). For example, current popular software has limited use (or none at all) of certain non-Latin script languages, which tend to belong to Global South cultures. For some Asian languages, complex and rare characters are not even defined for digital systems (Lee, 2019). Trivial operations, such as operating files of written statements across several platforms and throughout different software products (word processing, spreadsheets, design, CAD, etc.), might create spontaneous modifications in non-Latin characters, thus possibly rendering statements in rather meaningless wording. This digital constraint might further limit the applicability of the aforementioned inclusive audiovisual tools.

The sole use of a national/official language and/or lingua franca, even among native speakers, does not entail immediate accuracy and/or bias reduction. (Pirard et al., 2016) highlighted that language could be an issue across several ethnic groups in the study area, and, although relying on a lingua franca (Bahasa Indonesia, in this case) as a solution, respondents still presented different levels of fluency. Other authors (Schuman et al., 2018; Stoudmann et al., 2017), who had to deal with successive translations throughout a series of languages and dialects, resorted to the committee approach [as defined in (Buil et al., 2012)], in which discussions between researchers and translators aimed to use the most suitable terms for each statement, thereby reducing the likelihood of misinterpretation. Although (Barbosa et al., 2020) and (Zabala et al., 2017) conducted studies in Brazil and Mexico with purely native Portuguese and Spanish speaker research teams, respectively, special care was given to adapting the statements to local terms through extensive iterative piloting with on-site experts and community members.

3.3.5. P-set (sampling techniques)

The P-sets were mainly sampled through purposive sampling (n = 15) (Bumbudsanpharoke et al., 2009; Carmenta et al., 2017; Forouzani et al., 2013; Frate and Brannstrom, 2015; Hamadou et al., 2016; Huaranca et al., 2019; Jiren et al., 2020; Lansing, 2013; Mayett-Moreno et al., 2017; Nhem and Lee, 2020; Rodríguez-Piñeros et al., 2018; Rodríguez-Piñeros and Mayett-Moreno, 2015; Tuokuu et al., 2019; Wijaya and Offermans, 2019; Zabala et al., 2017), snowball sampling (n = 11) (Alexander et al., 2018; Cammelli et al., 2019; Hugé et al., 2016; Leite et al., 2019; Nguyen et al., 2018; Nhem and Lee, 2019; Schneider et al., 2015; Stoudmann et al., 2017; Sumberg et al., 2017; Vela-Almeida et al., 2018; Yeboah et al., 2017), and a dual-method approach, which is usually a combination of the first two (n = 10) (Brannstrom, 2011; Giannichi et al., 2018; Jaung et al., 2016; Kopytko and Pruneddu, 2018; Moros et al., 2020; Rust, 2017; Schuman et al., 2018; Taheri et al., 2020; Weldegiorgis and Ali, 2016; Zobeidi et al., 2016) or in combination with convenience sampling (Anderson and Jacobson, 2018; Barbosa et al., 2020), random sampling (Nordhagen et al., 2017; Truong et al., 2019; Vargas et al., 2019) and stratified random sampling (Pereira et al., 2016). (Hilhorst et al., 2012; Pirard et al., 2016) and (Truong et al., 2017) are the only ones that rely solely on an exclusive approach of stratified random sampling and structured sampling, respectively (Figure 5b).

Both purposive and snowball sampling have become practical methods to recruit potential Q respondents. The selected studies applied these techniques relying on contacts of governmental representatives (Alexander et al., 2018; Forouzani et al., 2013; Nguyen et al., 2018), (local) organizations (Anderson and Jacobson, 2018; Cammelli et al., 2019; Giannichi et al., 2018; Jiren et al., 2020; Leite et al., 2019; Nhem and Lee, 2019; Schneider et al., 2015), local experts (Jaung et al., 2016), local community leaders (Alexander et al., 2018; Cammelli et al., 2019; Forouzani et al., 2013; Hugé et al., 2016; Nguyen et al., 2018; Nhem and Lee, 2020; Nordhagen et al., 2017; Stoudmann et al., 2017), and recruited respondents themselves (Barbosa et al., 2020; Hugé et al., 2016; Schuman et al., 2018; Vela-Almeida et al., 2018). Their main shortcoming is that researchers may end up with undesirably homogeneous P-sets (Truong et al., 2017; Watts & Stenner, 2012; Watts and Stenner, 2005) associated with the prevalence of existing networks (Cohen and Arieli, 2011; Sadler et al., 2010). This homogeneity can ultimately leave hard-to-reach RGS respondents aside (Woodley and Lockard, 2016), possibly biasing the analyzed viewpoints. For instance, although (Pereira et al., 2016; Truong et al., 2019) aimed at a gender-balanced P-set, their snowball sampling resulted only in male respondents due to a lack of engagement with/of women. (Schneider et al., 2015) acknowledged potential biases in the respondents because of their closeness to a local farmers’ aid organization. (Stoudmann et al., 2017) reported that snowballing through village heads was a matter of cultural etiquette, which could lead to other types of unforeseen cultural interactions. (Truong et al., 2017) remarked that sampling through key local informants resulted in a limited representation of certain perspectives, thereby hampering their interpretation. Variations in snowball sampling could be suitable for reducing these biases; for instance, turning initial key informants from selectors to legitimators of the spread voice (Sadler et al., 2010), or increasing the trust of the desired networks by emphasizing the integrity, transparency, and sensitivity of (local) researchers (Cohen and Arieli, 2011).

3.3.6. P-set (size)

Q does not rely on large P-sets but on their diversity of viewpoints (Simons, 2013; Stenner et al., 2017; Watts & Stenner, 2012). Hence, there is not an ideal minimum number of participants. According to (Watts & Stenner, 2012), some authors advocate for ranges of 40–60 participants; others favor Q-set/P-set ratios higher than 1, with the number of respondents being lower than the number of statements. P-set sizes across the selected studies ranged from 10 to 219, although the majority were concentrated around 30–50 (Figure 5c). The Q-set/P-set ratios varied from 0.18 to 2.70, with most of them being around 1.0–1.5 (Figure 5d). From the selected studies, only two of them antagonistically elaborated on it: (Jaung et al., 2016) appealed to the <1 ratio as an indicator of ideal P-set size, whereas (Wijaya and Offermans, 2019) pointed out that a ratio larger than <1 would have increased the likelihood of finding a correlation between loaded respondents.

3.3.7. P-set (gender)

Gender representativeness, particularly concerning women’s participation, did not prove to be an active P-set criterion across the selected studies (Figure 6a). The aggregated female/male ratios⁴ of the selected studies (Figure 6b) revealed that female participants were typically about half of their male counterparts. Honorable exceptions are (Barbosa et al., 2020), whose focus was exclusively on a female phenomenon, and (Leite et al., 2019; Mayett-Moreno et al., 2017; Rodríguez-Piñeros and Mayett-Moreno, 2015), which considered a strong gender dimension in conducting their studies and interpreting viewpoints. In addition, only two studies (Stoudmann et al., 2017; Vargas et al., 2019) were explicit about gender balance, whereas (Pirard et al., 2016; Sumberg et al., 2017; Tuokuu et al., 2019; Weldegiorgis and Ali, 2016; Wijaya and Offermans, 2019; Yeboah et al., 2017) aimed towards proper gender diversity and women representation.

Figure 6. P-set size and genders. (a) Number and gender of participants per each of the 50 studies. Numbers on the X-axis correspond to the references. Solid dark gray, solid light gray and diagonal-line patterns on each bar represent the proportions of female participants, male participants, and gender-unspecified participants, respectively. Dashed line, dotted-dashed line and dotted line represent the average of female, male and total participants across studies, respectively. (b) Female/Male ratio across the selected studies. Barbosa et al. (2020) considered female participants only, thus is not represented here.

Four studies (Pereira et al., 2016; Schneider et al., 2015; Taheri et al., 2020; Zabala et al., 2017) relied exclusively on male participants (Figure 6a). These numbers should not be taken exclusively as unawareness from the researchers but also as a result of potential political and cultural ideas. For instance, (Pereira et al., 2016; Truong et al., 2019) pointed out that although some women had stronger (legal) attachment to their farms, they gave up responding in favor of their husbands. Schneider et al. (2015) and Wijaya and Offermans (2019) indicated that women were too shy to talk or faced cultural constraints, ultimately declining their participation. Contrarily, Vargas et al. (2019) highlighted the higher number of female participants, though not offering any plausible explanation, whereas (Nordhagen et al., 2017) argued that men usually being absent from the village/farm resulted in slightly skewed female participation. Taking into account the particular challenges RGS women must face in accessing resources (Poole, 2017; Giordano et al., 2019), gender imbalance can cause further biases and/or incompleteness of the topic that researchers expect to understand. Therefore, it is key for Q researchers in RGS settings to adopt cross-cutting, gender-sensitive approaches in their studies, primarily when dealing with male-dominated societies.

3.3.8. Sorting grid

There are no rules to ascertain the sorting grid in which the Q-set must be sorted. Typical shapes include quasi-normal (pyramid) and inverted quasi-normal (inverted pyramid) forced-sorting grids. In this regard, 17 and 20 selected studies provided the former and the latter, respectively (Figure 7). In contrast, Jiren et al. (2020) used a unique, double-pyramid or diamond shape. This matrix, unlike typical grids, bears a principle of inverted axes. The ranking is performed across a vertical scale, whereas the rows, distributed symmetrically, hold for statements with the same value. A non-forced grid was used in Rijneveld and Marhaento (2020) (not depicted), although the authors did not explain the reason for its use (nor its subsequent analytical process). Lansing (2013) piloted a non-forced distribution that was discarded in favor of a forced grid; the authors argued that the forced approach led participants to reflect more while sorting. In contrast, Hugé et al. (2016) allowed its respondents to deviate from the forced distribution as a way to cope with decision issues while sorting.

Figure 7. Sorting grids of the selected studies. ¹(Anderson and Jacobson, 2018; Brannstrom, 2011; Forouzani et al., 2013; Huaranca et al., 2019; Hugé et al., 2016) did not report the orientation of their grids. They were assumed as inverted distributions. ²The shaded area with thicker border represents the second grid used in (Carmenta et al., 2017). ³(Hilhorst et al., 2012) did not use a quantitative scale; instead, its authors reported graphical hints (happy/sad faces). ⁴Reported grids of (Rodríguez-Piñeros and Mayett-Moreno, 2015; Schneider et al., 2015) did not match with their respective number of statements. ⁵(Sumberg et al., 2017; Yeboah et al., 2017) resorted to two grids; only information of one of them was provided.

The shape of the sorting grid does not influence the reliability of the method. The forced distribution should be considered as a mere device to encourage respondents to perform a systematic analysis of each item (McKeown and Thomas, 1988; Watts & Stenner, 2012). However, unless properly designed and explained, the (inverted) pyramidal shape, with the strongest load of statements in the central column, might transmit to the participant the impression of importance, in which the apex of the pyramid should match the most critical statement(s). From this perspective, the diamond grid used by (Jiren et al., 2020) would offer a more natural, easy-to-read, top-to-bottom hierarchy, which can be further underpinned by providing graphical hints or ideograms (e.g., sad/happy faces) depicting the degrees of agreement along with the ranking scale (Cammelli et al., 2019; Hilhorst et al., 2012; Schneider et al., 2015).

The sorting grids in Q are structured through two ordinal scales: qualitative and quantitative. The former typically comprises a wording-based scale to measure the level of agreement. The latter, matching with the qualitative one, generally makes use of odd symmetric scales [(Carmenta et al., 2017; Pirard et al., 2016) become rare even-scale exceptions] with negative and positive sides and several sorting points, whose center corresponds to the neutral position, also referred to as ‘distensive zero’ (Watts & Stenner, 2012). Most of the selected studies (n = 39) employed qualitative scales, with (a variation in) the typical disagree/agree scale. Others resorted to (variations of) importance (Alexander et al., 2018; Anderson and Jacobson, 2018; Carmenta et al., 2017; Jiren et al., 2020; Nguyen et al., 2018), effectiveness (Carmenta et al., 2017; Tuokuu et al., 2019), (dis)approval (Schneider et al., 2015), affection (Rodríguez-Piñeros et al., 2018), and self-identification (Mayett-Moreno et al., 2017; Rodriguez-Piñeros et al., 2012; Rodríguez-Piñeros and Mayett-Moreno, 2015). The latter group employed highly personal approaches (I don’t identify with/I identify with, Unlike me/Like me), even though they studied perceptions of external phenomena (i.e., sustainable management of a community-owned forest reserve and related tourism infrastructure) rather than deeply intrinsic subject-wise affairs. In these cases, sorting impersonal statements like ‘Agriculture is not profitable’ (Mayett-Moreno et al., 2017), ‘The reserve should have more wild animals’ (Rodriguez-Piñeros et al., 2012) or ‘Ecotourism is a way to preserve the forest’ (Rodríguez-Piñeros and Mayett-Moreno, 2015), could become sources of confusions. Researchers should pay close attention to the possible mismatches between the wording of statements and the grid’s qualitative scales to prevent respondents from being biased by a false sense of doubt or neutrality.

The quantitative scales of the selected studies ranged from five to 13 points, although most of them were concentrated on seven (n = 13) and nine (n = 19). Although the number of sorting points enlarges/shortens the continuum through which respondents make ranking decisions on a given Q-set (Watts & Stenner, 2012), little is mentioned about their impact on the difficulty level of the sorting process. It is logical to think that the more sorting points are offered, the more time the respondent will take to position every single statement, and consequently, the more burdensome the process could become. In turn, this can negatively impact the in-sorting motivation, possibly decreasing the number of well-thought responses, as well as participation and completion rates.

Most of the selected studies (n = 41) employed a negative-to-positive order of the quantitative scales. Exceptions are studies with absolute (Stoudmann et al., 2017) and positive-to-negative (Frate and Brannstrom, 2015; Kopytko and Pruneddu, 2018; Leite et al., 2019; Rijneveld and Marhaento, 2020; Schuman et al., 2018) [and its vertical variation (Jiren et al., 2020)] scales. Absolute scales are used to prevent discomfort in the participants due to seemingly forced positive/negative choices while sorting; for example, participants do not necessarily have to feel disagreement, but a lower level of agreement in a negatively ranked statement (Watts & Stenner, 2012). Positive-to-negative scales could entail confusion in participants from sociolinguistic contexts with right-to-left reading languages (e.g., Persian, Arabic, Hebrew, Urdu, etc.), where the direction of the scale can enter into conflict with the respondents’ approach to reading and thus understanding (Bergen and Chan Lau, 2012).

The range of these scales, in combination with the different Q-set sizes, resulted in a wide diversity of sorting grids of both size and shape. These can be categorized according to the number of statements and kurtosis (Table 2). Most of the selected studies used mesokurtic sorting grids (n = 26), consistent with the traditional shapes depicted in introductory studies to Q (Watts & Stenner, 2012); 10% (n = 5) and 16% (n = 8) employed less common platykurtic (flat) and leptokurtic (steep) shapes, respectively (Figure 7).

Table 2. Characteristics of sorting grids with regard to their size and kurtosis.

Q-set	Platykurtic (flat)	Mesokurtic	Leptokurtic (steep)
10–30	(Kopytko and Pruneddu, 2018)	(Alexander et al., 2018; Anderson and Jacobson, 2018; Barbosa et al., 2020; Brannstrom, 2011; Cammelli et al., 2019; Lansing, 2013; Nguyen et al., 2018; Pirard et al., 2016; Zabala et al., 2017)	(Schneider et al., 2015)
31–50	(Moros et al., 2020; Weldegiorgis and Ali, 2016)	(Bumbudsanpharoke et al., 2009; Giannichi et al., 2018; Hilhorst et al., 2012; Jaung et al., 2016; Jiren et al., 2020; Lairez et al., 2020; Mayett-Moreno et al., 2017; Nhem and Lee, 2020; Nhem and Lee, 2019; Nordhagen et al., 2017; Rodríguez-Piñeros et al., 2018; Rodríguez-Piñeros and Mayett-Moreno, 2015; Sumberg et al., 2017; Tuokuu et al., 2019)^b	(Hugé et al., 2016; Taheri et al., 2020; Truong et al., 2017; Truong et al., 2019; Zobeidi et al., 2016)
51–70	(Astari and Lovett, 2019; Carmenta et al., 2017)^a^,^b	(Huaranca et al., 2019; Stoudmann et al., 2017; Yeboah et al., 2017)^b	(Carmenta et al., 2017; Forouzani et al., 2013)^a^,^b

^a Carmenta et al. (2017) was the only study that offered two sorting grids from two different kurtosis categories.

^b Carmenta et al. (2017), Sumberg et al. (2017), and Yeboah et al. (2017) provided two Q-sets to the participants; the table registers the sum of their statements.

According to (Watts & Stenner, 2012), targeting the correct size and kurtosis of the sorting grid is key to making participants feel comfortable during the sorting process. Two complementary factors that should lead the choices are the complexity or specialized nature of the topic and the related level of knowledge of participants. Steeper grids allow for larger neutrality and less decision making. By contrast, flatter ones are suitable for participants and/or topics that require more fine-grained decisions. Most of the selected studies seemingly made arbitrary choices of sorting grids; scarcely, three offered justifications for their grid choices. Astari and Lovett (2019) implemented platykurtic grids owing to the knowledgeability of the respondents [consistent with (Watts & Stenner, 2012)]. (Carmenta et al., 2017) preferred a platykurtic shape (one of the two grids) to enable subtle discrimination throughout many agreed statements. Nordhagen et al. (2017) opted for a mesokurtic grid to diminish low-literacy cognitive barriers by allowing for more neutral positions. In addition, although Hamadou et al. (2016) did not depict the sorting grid, its authors argued its simplicity was chosen because of the low educational level of the respondents.

3.4. Data collection

3.4.1. Location and materials

Q is typically a space-demanding technique that requires controlled environments and large flat workspaces. The use of appropriate, robust, and resistant materials can cause a substantial difference during their administration (Donner, 2001; Watts & Stenner, 2012). Only 20% of the studies (n = 10) reported their respective locations where sorting occurred. Most of these (Cammelli et al., 2019; Lairez et al., 2020; Nhem and Lee, 2020; Rodríguez-Piñeros and Mayett-Moreno, 2015; Truong et al., 2017; Truong et al., 2019; Zobeidi et al., 2016) mentioned respondents’ houses or farms, whereas others pointed out generically each village or community (Schneider et al., 2015; Weldegiorgis and Ali, 2016), offices of stakeholders (Weldegiorgis and Ali, 2016; Zobeidi et al., 2016), and schools (Anderson and Jacobson, 2018). In RGS contexts, particularly in remote and scattered areas where it is not feasible to gather participants at specific locations, ideal site conditions cannot be easily met and controlled. If the sorting location is the main workplace of the dweller, exposure to the elements (i.e., sun, wind, rain, and moisture) will certainly imply further constraints for researchers (Cheema et al., 2018). Lack of proper furniture (e.g., large tables and chairs for participants) is another point of concern that must not be overlooked, as it can hamper the engagement of participants. Probably due to the unavailability of these facilities, some of the selected studies were sorted directly on the floor (Cammelli et al., 2019; Lairez et al., 2020).

Although 30 of the selected studies indicated certain use of materials, most of these referred only to generic instruments such as ‘cards’ and ‘boards’. Others provided further specifications, such as paper (Huaranca et al., 2019; Nhem and Lee, 2020; Nhem and Lee, 2019; Rodríguez-Piñeros and Mayett-Moreno, 2015; Sumberg et al., 2017; Yeboah et al., 2017), (thin) paper/cardboard (Barbosa et al., 2020; Cammelli et al., 2019; Lairez et al., 2020; Schneider et al., 2015), laminated cards and board (Jiren et al., 2020), magnetic cards and board (Schuman et al., 2018), and a combination of paper, pencil, and eraser (without cards and sorting board) (Weldegiorgis and Ali, 2016). Three studies (Huaranca et al., 2019; Kopytko and Pruneddu, 2018; Rust, 2017) did not (partially) use any of these kinds of materials because of the use of online platforms. Materials such as mere paper and/or cardboard can result in damaged instruments if sorting is conducted outdoors during drizzling periods, and too lightweight materials could be compromised in the case of winds, becoming an additional burden to the respondent. The selection of adequate materials can help researchers cope with these unforeseen conditions; thus, these logistical issues should not be underestimated. Jiren et al. (2020) and Schuman et al., (2018) are good examples of the proper management of materials that facilitate interaction between researchers and respondents. The former used laminated cards, thereby becoming waterproof and highly durable throughout the field journeys. Furthermore, it implemented a system of hook and loop fasteners, hence being windproof and rendering it prone to be used vertically (coping with lack of flat horizontal space). Finally, the board was designed in a foldable layout, thus becoming more portable, in a (seemingly) waterproof material. The latter provided similar benefits through magnetic materials, although these could be more costly and scarce in certain (rural) settings.

3.4.2. Administration technique

Most of the selected studies were conducted face to face. Of these, 15 were done individually with each respondent (Anderson and Jacobson, 2018; Barbosa et al., 2020; Cammelli et al., 2019; Forouzani et al., 2013; Hu et al., 2018; Jiren et al., 2020; Nguyen et al., 2018; Nhem and Lee, 2020; Nhem and Lee, 2019; Rijneveld and Marhaento, 2020; Rodríguez-Piñeros et al., 2018; Stoudmann et al., 2017; Truong et al., 2017; Zabala et al., 2017; Zobeidi et al., 2016), thereby being more time-consuming for the research teams. For this reason, studies (Nguyen et al., 2018; Nordhagen et al., 2017), and (Carmenta et al., 2017), with 92, 137, and 219 effective respondents, respectively, became impressive cases of collected sorts for this type of study. In contrast, studies (Mayett-Moreno et al., 2017; Pirard et al., 2016), and (Vargas et al., 2019) were collectively conducted during a 16-person community meeting, in rounds of three people simultaneously, and in a 39-person deliberative workshop, respectively. Considering that (Vargas et al., 2019) poses an exceptional setup for RGS contexts, it would have been interesting to understand how it was executed; unfortunately, the authors did not provide any details on the process or locations. In addition, it is worth recalling whether Q is administered individually or collectively can influence the results (Buil et al., 2012). Since it is intended to capture personal viewpoints, undesired group opinions—especially when involving dominant individuals and/or in collectivistic cultures—could steer some respondents’ own perspectives (Stone et al., 2017).

Online-administered Q is an acceptable alternative to its face-to-face version (Watts & Stenner, 2012). Some authors (Ormerod, 2017; Davis and Michelle, 2011; Westwood and Griffiths, 2010; Hermans et al., 2012; Raadgever et al., 2008) have successfully conducted online-administered Q sorts, although this is still a rare choice nowadays. From the selected studies, only three (Huaranca et al., 2019; Kopytko and Pruneddu, 2018; Rust, 2017) were (partially) conducted by means of online tools, namely Partnership Online Evaluation Tool with Q methodology – POETQ (University of Birmingham, 2010; Jeffares and Dickinson, 2016), Qsortware (Pruneddu, 2011) and SurveyMonkey (SurveyMonkey, 2020); none of them took place in a low-income country. Online administration at times might be the only feasible technique, for instance, when addressing an international community (Bordt, 2018)⁵ or in view of exceptional yet plausible limited-access scenarios, such as the COVID-19 global pandemic crisis (Omary et al., 2020). In RGS settings, the online administration of Q is certainly restricted by much more than merely the researcher’s willingness to use it. RGS dwellers worldwide face a serious lack of access to the Internet (Villapol et al., 2018), deeply limited access to equipment and electricity (Armey and Hosman, 2016), and (technological) illiteracy (Jere et al., 2013).

On one hand, the (digital) gap between researchers and RGS populations demands building and/or reinforcing local (Q) research capacities. On the other hand, circumstances like those of the ongoing pandemic may pose a sudden and unforeseen turn towards remote research (Omary et al., 2020) that renders that gap more acute and critical than ever. The way forward during the latter should not be limited to relying on local networks (e.g., NGOs, cooperatives, village development centers, extension officers) as a way to bridge the gap. The crisis must foster the development of innovative, open-source tools to make Q more accessible and with fewer shortcomings, especially under the light of an increasing access to and use of mobile phones in the Global South (Loo and Ngan, 2012).

3.4.3. Assistance and facilitation

Q is an assistance-intensive technique; therefore, for RGS dwellers who might bear further cultural- and literacy-related constraints (Roser and Ortiz-Ospina, 2018), appropriate facilitation is crucial. Most of the selected studies relied on pre-sort instructions (i.e., explanation of the purpose and whole process) (Anderson and Jacobson, 2018; Giannichi et al., 2018; Hamadou et al., 2016; Jiren et al., 2020; Moros et al., 2020; Nguyen et al., 2018; Nhem and Lee, 2020; Pirard et al., 2016; Schuman et al., 2018; Sumberg et al., 2017; Truong et al., 2017; Truong et al., 2019; Vela-Almeida et al., 2018; Weldegiorgis and Ali, 2016; Wijaya and Offermans, 2019; Yeboah et al., 2017), normally accompanied by step-by-step oral guidance (Alexander et al., 2018; Bumbudsanpharoke et al., 2009; Cammelli et al., 2019; Frate and Brannstrom, 2015; Jaung et al., 2016; Lairez et al., 2020; Moros et al., 2020; Nhem and Lee, 2020; Nhem and Lee, 2019; Nordhagen et al., 2017; Sumberg et al., 2017; Tuokuu et al., 2019; Wijaya and Offermans, 2019; Yeboah et al., 2017; Zobeidi et al., 2016). Other complementary, more time-consuming activities were reading of (almost) every statement by the research team (Cammelli et al., 2019; Carmenta et al., 2017; Lairez et al., 2020; Nordhagen et al., 2017; Wijaya and Offermans, 2019), especially because of low levels of literacy and on-demand iterative clarification of statements (Alexander et al., 2018; Astari and Lovett, 2019; Lairez et al., 2020; Moros et al., 2020; Pirard et al., 2016; Sumberg et al., 2017; Wijaya and Offermans, 2019; Yeboah et al., 2017). In-depth explanations and interactions may smoothen the sorting process and reduce the risk of participants misunderstanding instructions and misinterpreting statements; however, this may also increase sorting times and interviewer bias, which can seriously affect the respondents’ engagement and validity of the findings. Moreover, the status of the researchers (i.e., origin, gender, age, etc.) may provoke unexpected behavior from participants; in these cases, proper selection, training, and supervision of (local) assistants is highly advisable (Buil et al., 2012).

When researchers are not (native) speakers of the P-set language(s), as it occurred with 20% (n = 10) of the selected studies (Anderson and Jacobson, 2018; Brannstrom, 2011; Bumbudsanpharoke et al., 2009; Hilhorst et al., 2012; Hugé et al., 2016; Kopytko and Pruneddu, 2018; Lansing, 2013; Nguyen et al., 2018; Nordhagen et al., 2017; Rodriguez-Piñeros et al., 2012)⁶, they will likely rely on translators and interpreters. In this case, particular emphasis should be placed on biases beyond the mere accuracy of the terms in statements. Interpreters and assistants must first thoroughly understand the dynamics of the methodology and the topic under investigation, so they can provide a more accurate explanation to participants (Cheema et al., 2018). Similarly, they must be aware of not influencing the respondents’ sorts with their own opinions while facilitating. This potential limitation again links to discussing the need to build (Q) research capacities in the local contexts of the Global South. By intensively involving local scholars, universities, and institutes, these studies could be conducted by relying on native speakers and will also empower those who can better understand the demands of their local realities.

Online-administered studies (Huaranca et al., 2019; Kopytko and Pruneddu, 2018; Rust, 2017) do not allow—nor should require—face-to-face facilitation. Their respective platforms give the participant the chance to read written instructions as many times as needed to understand the required dynamics. Three main downsides are that they require participants to have access to the required equipment, demand a certain degree of (ICT) literacy, and entirely rely on each respondent’s interpretation of the provided statements.

Sorting a set of statements holistically through a (relatively) large grid can be a daunting and cumbersome process, especially if respondents are not vastly knowledgeable on the topic under study. The so-called three-pile technique is a popular way among researchers to cope with this burden (Watts & Stenner, 2012). It consists of a primary rough sorting in which the participant distributes all statements based on three criteria: agree, neutral, and disagree. This preliminary rough sort is thereafter refined by positioning the statements to the sorting grid. From the selected studies, 48% (n = 24) (Alexander et al., 2018; Astari and Lovett, 2019; Barbosa et al., 2020; Cammelli et al., 2019; Forouzani et al., 2013; Hamadou et al., 2016; Jaung et al., 2016; Jiren et al., 2020; Kopytko and Pruneddu, 2018; Lairez et al., 2020; Moros et al., 2020; Nhem and Lee, 2020; Nhem and Lee, 2019; Nordhagen et al., 2017; Rodríguez-Piñeros et al., 2018; Stoudmann et al., 2017; Sumberg et al., 2017; Truong et al., 2017; Truong et al., 2019; Tuokuu et al., 2019; Wijaya and Offermans, 2019; Yeboah et al., 2017; Zabala et al., 2017; Zobeidi et al., 2016) resorted to this technique. Jaung et al. (2016) implemented an interesting two-step modification, where participants sequentially sorted into three and nine sub-piles (three per each first pile), thus enabling a smoother transition to the final grid distribution.

3.4.4. Sorting times

Required sorting times across the selected studies were reported to be as low as 25 min (Brannstrom, 2011), and as high as 1.5 h (Frate and Brannstrom, 2015; Pereira et al., 2016) and (up to) 3.0 h (Hu et al., 2018) (Figure 8a). Considering that these times are highly interrelated with the Q-set size, we can define a sorting time ratio expressed in seconds (s) per statement (st). These ratios varied from approximately 58 s st⁻¹ (45) to 225 s s^t-1 (Alexander et al., 2018), although most ratios were concentrated around 100 s st⁻¹ (Figure 8b). A third variable that influences the time required for sorting, which is usually overlooked in Q studies, is the number of sorting points throughout the grid. Larger Q-sets, distributed over a wider range of sorting choices, naturally take respondents longer times than otherwise. Accounting for this third variable, we define another ratio as the required time in seconds (s) per statement (st) per sorting point (sp). Most of the selected studies were within ratios of <10 s st⁻¹ sp⁻¹ (Anderson and Jacobson, 2018; Brannstrom, 2011; Nhem and Lee, 2019) and 10–20 s st⁻¹ sp⁻¹. Giannichi et al., (2018), Moros et al., (2020), Pereira et al., (2016), Pirard et al., (2016), Schneider et al., (2015), Truong et al., (2017), Truong et al., (2019), Yeboah et al., (2017) (Figure 8c); others had higher ratios of >20 s st⁻¹ sp⁻¹ (Cammelli et al., 2019; Frate and Brannstrom, 2015; Hu et al., 2018; Zobeidi et al., 2016), and even an exceptionally high ratio of 32 s st⁻¹ sp⁻¹ (Alexander et al., 2018).

Figure 8. Sorting times across the selected studies. (a) Absolute sorting time in minutes. (b) Sorting time ratio, expressed in seconds per statement. (c) Sorting time ratio, expressed in seconds per statement per sorting points.

Only Truong et al., (2017) elaborated on the consequences of (too) long sorting times hampering the Q process. Regarding the high ratios of (Alexander et al., 2018; Cammelli et al., 2019), they found their origins in the reported illiteracy conditions of their respective respondents. Other unexplained yet salient time-related facts from certain studies are worth remarking. Although Alexander et al., (2018) presented the smallest Q-set, presented in the form of pictures instead of written statements, it counterintuitively resulted in the highest sorting time ratios. Its images could facilitate the sorting flow yet could also turn into subjective instruments that perhaps demanded more extended interpretation and discussion times. In contrast, although Nhem and Lee, (2019) had a large Q-set that had to be additionally sorted vertically (concerning the strength of feeling of each statement within a given sorting point), it turned into barely 6 s st⁻¹ sp⁻¹, the lowest reported ratio.

Except for Yeboah et al., (2017), none of the studies with the largest Q-sets (>50 statements) (Astari and Lovett, 2019; Carmenta et al., 2017; Forouzani et al., 2013; Huaranca et al., 2019; Stoudmann et al., 2017) indicated sorting times. From these, two particularly interesting cases to analyze would have been (Huaranca et al., 2019), which presented 68 lengthy written statements, and (Carmenta et al., 2017) which asked each respondent to sort two 30- and 40-statement Q-sets in a single sitting.

3.4.5. Complementary information

To provide Q studies with an accurate and holistic interpretation of viewpoints, authors normally collect qualitatively rich complementary information (e.g., sociodemographic data, reasoning on sorting, etc.) (Watts & Stenner, 2012; Zabala et al., 2018). The most commonly used technique is the post-sorting interview about the placement of the (most extreme) elements and related topics (Alexander et al., 2018; Anderson and Jacobson, 2018; Barbosa et al., 2020; Brannstrom, 2011; Bumbudsanpharoke et al., 2009; Cammelli et al., 2019; Carmenta et al., 2017; Forouzani et al., 2013; Frate and Brannstrom, 2015; Giannichi et al., 2018; Hamadou et al., 2016; Hilhorst et al., 2012; Huaranca et al., 2019; Hugé et al., 2016; Jaung et al., 2016; Jiren et al., 2020; Kopytko and Pruneddu, 2018; Lairez et al., 2020; Leite et al., 2019; Mayett-Moreno et al., 2017; Moros et al., 2020; Nguyen et al., 2018; Nhem and Lee, 2020; Nhem and Lee, 2019; Nordhagen et al., 2017; Pereira et al., 2016; Pirard et al., 2016; Rijneveld and Marhaento, 2020; Rodriguez-Piñeros et al., 2012; Rodríguez-Piñeros et al., 2018; Rodríguez-Piñeros and Mayett-Moreno, 2015; Rust, 2017; Schneider et al., 2015; Stoudmann et al., 2017; Sumberg et al., 2017; Taheri et al., 2020; Truong et al., 2017; Truong et al., 2019; Tuokuu et al., 2019; Vargas et al., 2019; Vela-Almeida et al., 2018; Weldegiorgis and Ali, 2016; Wijaya and Offermans, 2019; Yeboah et al., 2017), as well as its collective variant in the form of focus group discussions (Weldegiorgis and Ali, 2016; Wijaya and Offermans, 2019). Other less common techniques include in-sorting interviews (about clarifying and sorting statements) (Astari and Lovett, 2019; Hu et al., 2018; Nguyen et al., 2018), pre-sorting interviews (Giannichi et al., 2018; Schneider et al., 2015), sociodemographic surveys (Sumberg et al., 2017; Zobeidi et al., 2016), and secondary information from prior interviews (Schuman et al., 2018). It is worth noting that according to (Truong et al., 2017), (too) long sorting times led participants to provide poor-quality complementary information during exit interviews.

3.4.6. Data recording

Q studies require adequate data recording of both the sort itself and any other information that contributes to the interpretation (Watts & Stenner, 2012). Less than half of the selected studies (n = 19) provided relevant information. Most of them used any form of (audio) recording for interviews (Alexander et al., 2018; Anderson and Jacobson, 2018; Cammelli et al., 2019; Giannichi et al., 2018; Hu et al., 2018; Jiren et al., 2020; Pereira et al., 2016; Schuman et al., 2018; Truong et al., 2019; Vela-Almeida et al., 2018), answer sheets for recording the sorts (Bumbudsanpharoke et al., 2009; Nhem and Lee, 2020; Nhem and Lee, 2019; Weldegiorgis and Ali, 2016; Zobeidi et al., 2016) and written notes (Cammelli et al., 2019; Jiren et al., 2020; Sumberg et al., 2017; Yeboah et al., 2017). Less reported techniques include photos of sorts (Alexander et al., 2018), structured questionnaires (Nhem and Lee, 2020), and even a unique approach of collecting notes written by the participants themselves (Rodríguez-Piñeros et al., 2018). Regarding web-based Q studies (Huaranca et al., 2019; Kopytko and Pruneddu, 2018; Rust, 2017), their respective platforms offered their own data-recording methods. Moreover, these studies did not limit the application of other online methods such as email-based follow-up interviews (Kopytko and Pruneddu, 2018).

Means of recording could be more restricted, in both quality and quantity, in RGS settings. Although none of the studies pointed out any related limitations, it makes sense to resort to methods that fulfil certain context-friendly properties: portable and lightweight, particularly for journeys between remote areas with low accessibility; elements-resistant, so rain, dust, heat, and humidity do not compromise recorded data; off-the-grid operation, either through long-life batteries for electronic equipment and/or by using non-electronic media. Moreover, a good strategy for reducing the risk of on-field data loss is to rely on several complementary and redundant recording methods.

3.5. Analysis and results

As this Q research stage typically does not imply on-field methodological choices, the respective findings can be found in Appendix C of the Data availability statement.

3.6. Interpretation

Given the subjectivity that interpreting viewpoints entails, and the particularities of each of the selected studies, the contents of the interpretations themselves were not considered within the scope of this review. Nevertheless, some commonalities can be identified regarding the labelling and framing of the interpreted factors. Although labelling is not a mandatory step in Q, it is certainly a common practice among Q methodologists. These labels are intended to deliver, in a nutshell, what characterizes each viewpoint and makes it unique compared to one another (Donner, 2001; Simons, 2013; Watts & Stenner, 2012). Because these labels depend mostly on the creativity of the researchers, there are virtually endless options to define them; however, some approaches are recognizable. Some labels assign behavioral characteristics to respondents, whereas others focus on defining a given situation or even providing a short explanation of certain positions.

Most of the selected studies relied on labels for societal scenarios, either in their compact (n = 17) (Barbosa et al., 2020; Brannstrom, 2011; Huaranca et al., 2019; Jiren et al., 2020; Kopytko and Pruneddu, 2018; Leite et al., 2019; Mayett-Moreno et al., 2017; Moros et al., 2020; Nhem and Lee, 2020; Nhem and Lee, 2019; Rodriguez-Piñeros et al., 2012; Stoudmann et al., 2017; Truong et al., 2017; Truong et al., 2019; Tuokuu et al., 2019; Vargas et al., 2019; Vela-Almeida et al., 2018) or longer forms (n = 12) (Alexander et al., 2018; Astari and Lovett, 2019; Cammelli et al., 2019; Carmenta et al., 2017; Frate and Brannstrom, 2015; Hugé et al., 2016; Jaung et al., 2016; Rodríguez-Piñeros et al., 2018; Rodríguez-Piñeros and Mayett-Moreno, 2015; Sumberg et al., 2017; Weldegiorgis and Ali, 2016; Yeboah et al., 2017), behavioral adjectives (n = 11) (Bumbudsanpharoke et al., 2009; Giannichi et al., 2018; Hamadou et al., 2016; Hu et al., 2018; Lansing, 2013; Nordhagen et al., 2017; Pereira et al., 2016; Schneider et al., 2015; Taheri et al., 2020; Wijaya and Offermans, 2019; Zabala et al., 2017), or their combinations (n = 4) (Lairez et al., 2020; Pirard et al., 2016; Schuman et al., 2018; Zobeidi et al., 2016). Few authors resorted to longer, descriptive versions of behavioral adjectives (n = 1) (Forouzani et al., 2013) and explanatory labels (n = 1) (Anderson and Jacobson, 2018). Other studies (n = 4) (Hilhorst et al., 2012; Nguyen et al., 2018; Rijneveld and Marhaento, 2020; Rust, 2017) reported the use of generic nameless labels, distinguished by the use of numbers or letters.

Interpreted factors should ideally be validated through ulterior interaction with respondents. By iteratively providing participants with draft interpretations, they can offer further feedback that contributes to refining the narratives (Robbins and Krueger, 2000; Robbins, 2005). This appears to have been amply overlooked (or underreported) in Q studies. Of the selected studies, only four (Brannstrom, 2011; Kopytko and Pruneddu, 2018; Lansing, 2013; Schuman et al., 2018) mentioned that they had resorted to this technique. Regarding RGS settings, where even one-time (sorting) contact with respondents could already be limited, validation seems to become a less likely choice. Under such circumstances, an alternative could be to validate the narratives with at least the highest loaded respondents for each factor (Kopytko and Pruneddu, 2018; Lansing, 2013).

3.7. Challenges and the way forward

By reviewing and analyzing the 50 selected studies, we have seen the potential of Q in unraveling diverse narratives on different forms of RGS livelihoods. We have seen its application across topics related to decision-making in smallholder farming systems, conflicts between environmental governance and RGS livelihoods, conflicts of mining projects and RGS populations, environmental management and conservation, conditions of refugees in humanitarian crisis contexts, among others. At the same time, we have evidenced that deploying Q in RGS settings is a planning-, time-, and facilitation-intensive process (Previte et al., 2007; Simons, 2013; Stone et al., 2017; Ho, 2017). The first two stages of Q, namely research design and data collection, are the ones requiring interactions between researcher and the (RGS) participant; hence, the ones that concentrate most of the identified methodological challenges (Table 3). As such, during its implementation in the RGS—particularly in low-income settings—along with its (non)human-dependent constraints, it will almost certainly result in limitations and improvisations. Paradoxically, most of the Q scientific literature keeps looping on the portion that has already been exhaustively reported: analysis and interpretation (Dziopa and Ahern, 2011; Zabala and Pascual, 2016; Watts and Stenner, 2007). From the selected studies, only four (Hugé et al., 2016; Schneider et al., 2015; Truong et al., 2017; Weldegiorgis and Ali, 2016) critically elaborated on on-field methodological issues. An unawareness of these challenges could undermine the successful implementation of Q with RGS dwellers.

Table 3. Summary of discussed challenges due to the implementation of Q in RGS settings and respective good practices.

Stage	Aspect	Challenge	Good practices
Q stage 1: Research design	Concourse development and reduction	^aInaccessibility to RGS dwellers as primary data sources	Resort to reliable secondary data
			Reuse of primary data from larger research program
			Rely on proxies’ discourses
		^aUnbalanced representation of discourses, in detriment of less-empowered individuals	Apply matrix reduction method
	Q-set	Too large Q-sets might discourage participation	Keep small number of statements without compromising representativeness
		^aIlliteracy of participants	Use other (visual) techniques (e.g. pictures/illustrations)
		Incompatibility of non-Latin script languages of Global South cultures with certain electronic platforms	Use of hand-written material, which can be ultimately digitalized
			Development and definition of non-Latin script languages
		Multiple languages or dialects involved	Apply a reliable translation method (Buil et al., 2012)
			Iterative piloting of statements in the local context
	P-set	^aToo homogeneous P-set due to biased snowball/purposive sampling	Use variations of snowball sampling (Cohen and Arieli, 2011; Sadler et al., 2010)
		Underrepresentation of female respondents	Adoption of cross-cutting gender-sensitive approaches in the research process
	Sorting grid	Confusion due to mismatches between qualitative scale and wording of statements	Make sure both elements hold to one another consistently
		Burden caused by (too) many sorting points in quantitative scale	Limit the number of sorting points depending on the topic under study and characteristics of respondents
		Mismatch between direction of quantitative scale and the sociolinguistic context of participants	Define a qualitative scale sensitive to the linguistic context
		Inappropriate shape of sorting grids	Define the shape of sorting grids based on nature of topic and level of knowledge of participants
Q stage 2: Data collection	Location and materials	^aInappropriate locations and unavailability of adequate furniture	Use of right materials for boards (sorting grid) and cards (statements)
	Administration technique	Data collection is too time consuming for researchers	Administer sorts simultaneously to small groups of (Lang and Lingnau, 2015; United Nations, 2021) respondents
		Undesired interactions between respondents in collective sorts	Consider small groups of (Lang and Lingnau, 2015; United Nations, 2021) respondents, preventing their interaction
		^a(Digital) gap between researchers and studied populations	Long-term Q capacity building in the Global South
		^aLimited use of online-administered Q	Development of open-source, mobile-friendly Q platforms
	Assistance and facilitation	Biases in responses due to prolonged assistance	Provide concise pre-sort instructions and clear (short) statements
			Limit assistance to the sorting mechanism rather than interpretation of statements
			Use the three-pile technique (or its nine-pile variant) as auxiliary sorting method
		Biases due to status of researchers	Training and empowerment of local researchers and/or assistants
		Biases due to translation and interpretation	Proper training of interpreters about both the method and the topic
			Long-term Q capacity building in the Global South
	Sorting times	Too long sorting times affecting response rates and validity	Control number and type of statements, as well as the number of sorting points
	Complementary information	Poor quality of collected information	Use structured instruments (surveys, forms) to reduce total interaction times and burden of respondents
	Data recording	Possible loss of data	Use of context-sensitive recording means
			Rely on mutually complementary and redundant recording methods
Q stage 4: Interpretation	Validation	^aImpossibility of validation due to remoteness	(Remote) validation with at least the highest loaded respondent per factor
Post-Q: Knowledge production	Access to knowledge	Q studies with limited access to Global South researchers	Open-access publishing
		^aSocio-geographical decoupling of researchers and studied populations	Long-term Q capacity building in the Global South

^aChallenges that may be more profoundly present in RGS settings.

Most challenges across the selected studies were related to the difficulty in reaching (female) respondents, thereby possibly underrepresenting viewpoints (Anderson and Jacobson, 2018; Brannstrom, 2011; Giannichi et al., 2018; Kopytko and Pruneddu, 2018; Pereira et al., 2016; Rust, 2017; Schneider et al., 2015; Truong et al., 2019; Wijaya and Offermans, 2019; Yeboah et al., 2017; Zabala et al., 2017). This was not exclusively limited to physical/geographical unreachability, but also to social and cultural barriers that excluded at times female and other less socially connected participants. In addition, some research teams faced particular constraints due to their dependency on local third parties (e.g., NGOs, farmer associations), and thus lack of on-field autonomy, to reach the desired P-set (Schneider et al., 2015).

Other authors (Alexander et al., 2018; Cammelli et al., 2019; Hamadou et al., 2016; Hugé et al., 2016; Nordhagen et al., 2017) reported illiteracy, semi-literacy, and low education as limiting factors in conducting sorting sessions more successfully. Such limitations likely lead to (too) long sorting interactions, which in turn could lead to a number of challenges. These include post-sorting time restrictions for researchers (Barbosa et al., 2020), thereby compromising the quality of collected complementary data (Truong et al., 2017); response biases due to short, not-well-thought sorts (Jaung et al., 2016; Truong et al., 2017); decrease in the level of engagement of respondents (Brannstrom, 2011; Schneider et al., 2015; Truong et al., 2017; Truong et al., 2019); and even ultimately drop-out problems (Cammelli et al., 2019; Vargas et al., 2019; Zabala et al., 2017). These potential limitations become much more salient when focusing on sub-Saharan Africa and South Asia, the regions with the highest illiteracy rates among adults worldwide (Szmigiera, 2015; Roser and Ortiz-Ospina, 2018).

Another identified issue was the lack of methodological clarity of the administered Q, which evolved towards inaccurate or invalid responses. For instance, Truong et al. (2017) and Truong et al., (2019) pointed out that some participants could not follow sorting instructions and at times found statements too complicated or contradictory, whereas Hugé et al., (2016) and Weldegiorgis and Ali, (2016) reported that some respondents who were uncomfortable with the forced distribution tended to sort out of the grid. Perhaps these difficulties become more understandable if Q is compared with other more familiar, more economical, and easier-to-administer attitudinal measuring instruments, such as the Likert scale (Ho, 2017; ten Klooster et al., 2008). Linguistic problems, such as different degrees of fluency in both researchers and participants (Pirard et al., 2016), as well as mismatches and misunderstandings in provided terms and wordings (Stoudmann et al., 2017), might aggravate this methodological obscurity.

Based on the selected documents, we also identified and discussed several good practices that could help in coping with the issues mentioned above (Table 3). Researchers can immediately adopt and implement these practices. For example, the design of an appropriate sorting grid is a costless and quicker process with substantial positive impacts. Other measures, however, demand longer participation and commitment of many more actors (e.g., Q capacity building in the Global South and development of more compatible Q electronic platforms). Moreover, the implementation of identified good practices must sometimes undergo trade-off decisions; for instance, complex translations and piloting of statements are not ideal when time and financial restrictions condition the study.

Beyond the discussed challenges, it is worth noting that (Nordhagen et al., 2017; Schneider et al., 2015) argued that their participants found Q an original and engaging technique. This is consistent with (Stone et al., 2017; ten Klooster et al., 2008), whose (non RGS) P-set enjoyed sorting, and even deemed Q ‘a welcome change to the usual research practices’. Other selected studies (Barbosa et al., 2020; Cammelli et al., 2019; Hilhorst et al., 2012; Truong et al., 2017) framed it to their respondents, perhaps intentionally, as a game rather than a survey method. Perhaps these perceptions and strategies are yet to be exploited to reduce the burden on participants.

Finally, although no single study reported any ethical conflicts of Q with cultural values, it also appears as an overlooked topic among researchers. Only (Kopytko and Pruneddu, 2018; Leite et al., 2019; Stoudmann et al., 2017; Truong et al., 2017; Truong et al., 2019) scantily touched upon the clearance and compliance with ethical standards. Nonetheless, this might represent just the tip of a much more complex (cross-)cultural iceberg [for example, the multi-cultural mining conflicts in the Ecuadorian Amazon reported in (Vela-Almeida et al., 2018)]. This could be the result of a (still) too Eurocentric, culture-insensitive way of conducting Q research (Stone et al., 2017). For example, it should call our attention when (Laney and Turner, 2015) points out that they gave up on using Q in northeast Madagascar after some villagers perceived it as a form of sorcery. Perhaps more subtle forms of cultural conflict occur in the RGS, and the research community is simply not aware of it (or does not document it). Another instance is the rising and mismanagement of RGS dwellers’ (monetary) expectations, especially after exposing them to recurrent and sustained interventions by (non)academic organizations (Cheema et al., 2018). Unfortunately, the data gathered here has not allowed us to elaborate much more in-depth on these topics, yet certainly is a way worth exploring.

4. Concluding remarks

Q can be considered as a flexible, innovative, and powerful technique for assessing differences in viewpoints across groups. Through the analysis of the selected studies, we have observed its strong potential to better understand the dynamics of the RGS livelihoods beyond oversimplified and stereotypical narratives (i.e., mere economic considerations). Hence, it can become a valuable tool to support context-sensitive and sustainable development interventions. At the same time, conducting Q studies in RGS settings may pose particular onsite methodological challenges and limitations. These, unless properly addressed in the planning and execution, may hamper Q’s effectiveness in revealing discourses on RGS livelihoods that are faithful to respondents’ perceptions and opinions. Such inaccurate and distorted discourses may eventually lead to flawed decisions and actions. As a response, in this review we have highlighted good Q methodological practices whereby researchers could cope with those challenges and limitations, thereby ensuring a better comprehension of the discourses emerging from the studied phenomenon (e.g., RGS livelihoods). We encourage Q researchers, particularly those engaging with RGS studies, to implement the strategies presented here.

Notwithstanding limitations and good practices, we advocate the construction of robust Q capacities and the gender-balanced empowerment of local researchers, along with the indispensable provision/production of open access and inclusive scientific knowledge, data, and tools. These efforts may contribute to closing geographical, social, and cultural gaps, such as the ones we have analyzed throughout the present work.

Authors contributions

Table

Author	Statement
Juan Carlo Intriago Zambrano	Conception and design of the work; acquisition, analysis, and interpretation of data Drafting of the work Final approval of the version to be published Agreement to be accountable for all aspects of the work
Jan Carel Diehl	Interpretation of data Critical review for important intellectual content Final approval of the version to be published Agreement to be accountable for all aspects of the work
Maurits W. Ertsen	Interpretation of data Critical review for important intellectual content Final approval of the version to be published Agreement to be accountable for all aspects of the work

Acknowledgements

We express our gratitude to Zabala et al., (2018), whose work inspired the framing of this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The dataset related to this study can be found at https://doi.org/10.34894/K252KB, an open-source online data repository hosted at DataverseNL (Intriago Zambrano, 2020).

Additional information

Funding

This work was supported by the TU Delft | Global Initiative, a program of the Delft University of Technology to boost Science and Technology for Global Development.

Notes on contributors

Juan Carlo Intriago Zambrano

Juan Carlo Intriago Zambrano is pursuing a PhD degree at the Water Resources department at Delft University of Technology, the Netherlands. He holds an MSc degree in Urban Environmental Management from Wageningen University, and a BArch degree from the University San Gregorio de Portoviejo. His scholar interests span across (agricultural) technological innovations in the Global South, as well as trans-disciplinary, multi-cultural studies with a gender approach. He is fond of bottom-up, participatory research methods like Q methodology.

Jan Carel Diehl

Jan Carel Diehl is professor in Design for Inclusive Sustainable Healthcare at the Faculty of Industrial Design Engineering of Delft University of Technology and Medical Delta Professor at Erasmus Medical Center in Rotterdam. He received his MSc degree and PhD in Design for Sustainability from Delft University of Technology. He is steering board member of the Delft Global Initiative and part of the LDE Center for Frugal Innovation in Africa. His general aim is to develop design tools and methods to create inclusive and sustainable product (service) systems solutions.

Maurits W. Ertsen

Maurits W. Ertsen is Associate Professor in the Water Resources department at Delft University of Technology, the Netherlands. He holds a PhD degree from DUT and an MSc degree from Wageningen University. He studies how longer-term water practices emerge from short-term actions of human and non-human agents in current, historical, and archaeological periods in places ranging from Peru to the Near East, in close cooperation with universities, NGO’s, and the private sector.

Notes

1 In this document, the Global South comprises low- and middle-income countries, as classified by the United Nations (142).

2 According to the classifications of journals of Ulrichsweb™ Global Serials Directory (http://ulrichsweb.serialssolutions.com/); in cases of journals bearing more than one discipline, the more representative was assigned to the respective document.

3 We do realize that our own research endeavors can be labelled in similar terms. In itself, we would argue that involvement of researchers from the GN in itself is not necessarily to be avoided – but we do argue that the balance of research power between GN and GS is in need of correction, including the labelling of GN and GS itself.

4 Barbosa et al. (2020) was not accounted due to its exclusivity of female participants.

5 The author gave up the option of Q due to lack of feasible web-based alternatives.

6 Assumed after the authors’ countries of affiliations and language employed during the studies.