Ten statements related to qualitative research in IS: true or false?

When Professor Bansal, the EIC of JITCAR, asked me to write a guest editorial for JITCAR, I was delighted. However, as my submission deadline approached, I wondered what my editorial would be about. After some thought, I decided to write this short piece to highlight 10 frequently held beliefs about qualitative research in our information systems (IS) community, some of which are not necessarily true. As an author, reviewer, and reader of JITCAR, you may find it interesting to see where you stand with respect to the statements listed below. I realize that many of you will find this list to be elementary. Nevertheless, I suggest you try to assess the following statements, by marking them as True or False, before consulting the discussion that follows.

Table

Ten Statements Related to Qualitative Research in Information Systems

Qualitative studies in IS are exploratory in nature. Qualitative studies in IS involve analysis that is inductive in nature. Qualitative research in IS can be best characterized as being interpretivist. Qualitative case-study-based manuscripts in IS need multiple (at least 4) units for cases to be considered rigorous. Qualitative studies in IS need to use formal coding procedures to be considered of high quality. The use of computer-based data management and coding is considered superior practice by qualitative scholars in IS. Interviews must be recorded and transcribed to be considered valid data in IS. For a study to be credible in IS, it is important to have between 50 and 100 interviews. Furthermore, more the no. of interviews, better the manuscript is likely to be received in the review process. Eisenhardt (1989), Yin (1994), Strauss and Corbin (1998), and Klein and Myers (1999) are mandatory references for any qualitative research study in IS. Theoretical contributions in the form of a set of propositions are the desired output for qualitative manuscripts in IS.

Let us look at the first statement. Sure, a lot of qualitative research explores some phenomena, and hence tends to be labeled as exploratory, but qualitative studies can have other objectives as well. For example, according to Yin (1994, p. 1), case studies, a popular form of qualitative research, can be “exploratory,” “explanatory,” or “descriptive.”

So, let us not assume that cases are necessarily exploratory, or should be characterized as such. In fact, Sarker et al. (2018b) discourage using the “exploratory” label for qualitative work whenever possible, since it indicates, rightly or wrongly, a sense of tentativeness and lack of clarity about the objectives, nature of results, and the way it is conducted; indeed, Yin (1993, p. 5) believes that exploratory case studies tend to follow “intuitive paths” and are “perceived by others as sloppy.”

The second statement is somewhat related to the first, and it refers to the nature of reasoning underlying the analysis of data. The common approaches mentioned in the literature are induction, deduction, and abduction (e.g., Van de Ven, 2007). Often times, it is assumed that qualitative research adopts an inductive approach – that is, takes some specific instances of data (e.g., cases) related to a focal phenomenon and seeks to make general statements about the phenomenon grounded in the data (e.g., Eisenhardt, 1989; Walsham, 1995). However, qualitative research can also be used to test the theory (Lee, 1989) using the logic of modus tollens (Lee & Hubona, 2009). The idea is elaborated upon in Lee and Sarker (2021):

In general, the major premise, minor premise, and conclusion of modus tollens are, respectively, ‘if p is true, then q is true,’ ‘q is not true,’ and ‘therefore, p is not true.’ In theory testing, ‘p’ stands for the general statements making up a theory, and ‘q’ stands for the statements describing an observable consequence of the theory applied in a particular setting. (One example of the statements making up ‘q’ are the statements describing a prediction that follows from the theory being tested.) Data may play the role of indicating whether or not q is true. If the data indicate that ‘q is not true,’ then it follows that ‘therefore, p is not true’ or, in other words, the theory is rejected.

Abduction, according to Van de Ven (2007), is “a creative form of reasoning” (p. 140) that involves a mental leap triggered by data; it “usually begins with a surprising observation or experience … Abduction is an inferential procedure in which we create a conjecture that, if correct, would make the surprising anomaly part of our normal understanding of the world” (p. 101). It is worth noting that Van de Ven believes that “researchers and practitioners create or discover theories through a process of abduction – not by induction or deduction” (p. 105). As we can see, it would be inaccurate to assume that qualitative research is necessarily inductive in nature.

With respect to the third statement, qualitative research in IS has been conducted based on various perspectives, including “positivist” (Dubé & Paré, 2003; Eisenhardt, 1989), “interpretive” (Klein & Myers, 1999; Walsham, 1995), and “critical realist” (e.g., D. Wynn & Williams, 2012),¹ and there are excellent examples related to each of the perspectives in the literature. In other words, qualitative research does not have to be “interpretive.”

The fourth statement probably has origins in a very well-known article by Eisenhardt (1989), where she states “while there is no ideal number of cases, a number between 4 and 10 cases usually works well. With fewer than 4 cases, it is often difficult to generate theory with much complexity, and its empirical grounding is likely to be unconvincing, unless the case has several mini-cases within it … ” (p. 545).

Yet, to achieve the depth that is expected of case research, many scholars prefer to see a small number of cases in a study. Having too many cases in a study makes it seem like a survey study that lacks rigor. Sarker et al. (2013), in their review of case study articles published between 2001 and 2012 in MIS Quarterly, Information Systems Research, Journal of the AIS, and Journal of MIS (often considered to be the four leading mainstream journals in IS), found that “about 52% used one [1] case, and about 22% used 2 or 3 cases”; they further add “A single case is absolutely acceptable, if done well. Using a large number of cases in a study, in itself, does not imply that the study is of high quality” (p. x).

The fifth statement is related to the often-made assumption that coding is essential to qualitative research. While coding is recommended within certain qualitative research traditions such as Grounded Theory Methodology (GTM), and can be seen by some reviewers and editors as indicative of rigor, the following quote from Walsham (2006, p. 325), a leading exponent of interpretive research, is worth considering in this regard:

In terms of learning from the data itself, grounded theory offers one way of doing this, although the ‘coding’ is a subjective process to some extent, because the researcher chooses the concepts to focus on. I tend to use a looser approach myself, where I write impressions during the research, after each interview, for example. I generate more organized sets of themes and issues after a group of interviews or a major field visit. I then try to think about what I have learnt so far from my field data. If this sounds a rather subjective and relatively unplanned process, well it is. I believe that the researcher’s best tool for analysis is his or her own mind, supplemented by the minds of others when work and ideas are exposed to them.

Indeed, in the review of the articles published in the four leading journals referred to above, Sarker et al. (2013) note that only 43% of the qualitative studies reported that coding procedures were used. So, it would be incorrect to assume that coding is essential to qualitative research. Of course, for some genres of qualitative research (say, GTM or positivist case studies), it may be more desirable than in others.

The sixth statement is about the use of computer-based data management and coding. While historically, many qualitative researchers have chosen not to use computer-based systems such as NVivo, we believe that such systems can help in managing data in a systematic manner, in creating transparency, and in assisting in the research process. Such software can also enable mixed methods studies by allowing for the use of statistical analysis and even sentiment analysis in conjunction with the qualitative analysis. However, the use of tools is not a requirement for qualitative research.

The seventh statement related to recording interviews and transcribing them. Given the technologies available today, I would recommend that in general it is a good idea that all researchers consider doing this. In the past, sometimes authors partially transcribed the interviews given the prohibitive costs of transcription. It is also true that certain subjects will refuse to be interviewed and many others will avoid speaking about the “real” issues on record. Based on the review of the IS literature (referred to earlier), Sarker et al. (2013, p. x) report that “about 64% [of the studies] mentioned that their interviews were at least partially recorded and transcribed. Interestingly, about 7% justified not recording the interviews.”

The eighth statement has to do with the number of interviews. Interviews can be seen and conducted from various perspectives, ranging from a structured process of elicitation of expertise or specific facts to even unstructured conversations. For example, Sarker et al. (2018a, p. 758) draw upon past work and note that “Interviews can be of a rational type, with the assumption that a skilled interviewer can find the objective truth, or interviews can be of the creative type, where the goal is for the researcher to go beyond the interviewee’s ‘rational façade’ and understand the interviewee’s feelings and thoughts (Fontana & Frey, 2000, p. 663).” With respect to the number of interviews, according to the review of the IS literature by Sarker et al. (2013, p. viii), “Interviews were found to be the most common technique of qualitative data collection. The average number of interviews [in a study] conducted was 40, with a maximum number of 175 and a minimum number of 6.” My personal opinion is that the quality of interviews and the selection of a range of appropriate stakeholders tend to be more valued more than the sheer number of interviews by perceptive reviewers. The number of interviews needed may also be related to the nature of the study.

The ninth statement has to do with the references that are considered to be credible and appropriate in IS. There is no doubt that all of the authors mentioned in the list have been important contributors to the methodological thinking among those conducting qualitative research in IS. In fact, Sarker et al. (2018b, see p. 912) find “Yin, Walsham, Eisenhardt, Klein and Myers, Miles and Huberman, and Strauss and Corbin” as the most popular authors/references in the literature between 2001 and 2017 in four leading mainstream IS journals (MIS Quarterly, Information Systems Research, Journal of the AIS, and Journal of MIS) overall. However, they note that different references are more applicable to different types or genres of qualitative research (Sarker et al., 2018b, p. 913). For example, Walsham (1995) and Klein and Myers (1999) are appropriate references for interpretive case studies but not for positivist case studies. In summary, references used should depend on the nature of the qualitative study, and none of the references in the statement should be considered mandatory for (all) qualitative research.

Finally, the tenth statement is related to how contributions are to be presented. Sarker et al. (2013, p. ix) note, “..we found that authors used several strategies to present their findings”: in “table format (30%),” “in the form of a model (36%),” whether process or variance, “as propositions or abstractions (9%),” “as a map (5%),” and “as text (21%).” In other words, there are many effective ways of representing the contributions in qualitative research manuscripts, and propositions (abstractions) are just one. Thus, authors should not feel obliged to develop propositions in every qualitative study, and reviewers should not expect to see propositions in every manuscript. However, I note that authors of different types of qualitative research may find some representation approaches more effective than others.

In summary, I hope that the “ten statements” and the surrounding discussion provide some clarity about qualitative research and how this form of research is sometimes misunderstood by authors as well as reviewers. This is despite the fact that qualitative research has had a long history in IS and related disciplines such as HCI (e.g., E. Wynn & Hult, 2019). As a former EIC of JITCAR and an ardent supporter of the qualitative inquiry in our discipline, I wish the entire qualitative IS research community the best in publishing high-quality work that investigates the many interesting and transformative IS-related phenomena all around us.

Acknowledgments

I thank Dr. Xiao Xiao (Copenhagen Business School) and Dr. Tanya Beaulieu (University of Maine) for their collaboration that played a role in the formulation of these statements. Some of the content of this editorial draws on the editorial we wrote for MIS Quarterly (Sarker et al., 2013). The reader is referred to the editorial for detailed recommendations about crafting articles based on qualitative research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Suprateek Sarker

Suprateek Sarker (“Supra”) is Rolls-Royce Commonwealth Commerce Professor at the McIntire School of Commerce, University of Virginia. Professor Sarker currently serves on the University-level Provost’s Promotion & Tenure Committee. His work has been funded by the National Science Foundation, and he has published in many leading journals and has given talks at many universities across the globe. He has served (or serves) as Senior Editor of ISR, MISQ, Decision Sciences, and Information & Organization, and as an Editorial Board member or Associate Editor for JMIS, IT & People, and IEEE Transactions on Engineering Management. He has also been the Editor-in-Chief of JITCAR and JAIS. He is a Fellow and currently the President-Elect of the Association for Information Systems (AIS), and he has been awarded honorary doctorates from two reputed European universities.

Notes

¹ I use the terms positivist, interpretivist, and critical realist in the way they are used in mainstream IS literature in the context of qualitative research.