Design-based research: What it is and why it matters to studying online learning

Figures & data

Table 1. Comparing DBR with qualitative and experimental traditions.

Tradition	Defining characteristics	Epistemological stance	End points
Qualitative participant- observer research	Researchers engage with participants to gain greater insight into the meaning associated with activities or behaviors. (Platt, 1983)	Interpretivist. Reality is observable but needs to be interpreted in light of theory and the observer’s subjectivity. Issues of positionality are key for establishing validity and reliability.	After-the-fact knowledge: descriptive and explanatory (Bakker, 2018), such as thick descriptions of groups, people, social systems or processes.
Experimental Research	The gold standard for drawing causal inference and, therefore, universal prediction. A hypothesis is formulated and tested. Conditions do not change during the experiment. (Bakker, 2018)	Positivist. Reality is observable and theory is built through isolating and testing particular variables, leading to causal claims. Strong claims may only be made about replicable phenomena.	After-the-fact knowledge: explanatory and predictive (Bakker, 2018), such as Null hypothesis disconfirmation and comparison among intervention types or particular variables.
DBR	Suitable for ill-structured problems (Lewis et al., 2020), combines qualitative traditions with quantitative (Collins et al., 2004). Conjectures are not fixed for approval/rejection, but updated on each iteration to match existing data at the moment.	Pragmatist. Designs can embody questions that challenge the status quo (Zimmerman, 2007). Knowledge is contextually grounded but general enough to be transferred to similar situations (Bakker, 2018).	Usable knowledge: design knowledge including domain theories, patterns or principles. Transformation: changes in human and organizational capacity toward design goals (Kali, 2016)

Figure 1. A process model for DBR.

Table A1. Methods that share characteristics with DBR.

Paradigm	Defining characteristics	Epistemological stance	End points
Design-Based Research (DBR)	Research is the focus and design is its means (Fallman, 2007). Flexible, iterative, interventionist. Conjectures are constantly updated. Conditions might need to be changed during the course of the experiment.	Neither theory nor interventions alone are sufficient to generate knowledge (Easterday et al., 2014), especially when problems are ill-structured (Rittel & Webber, 1973). Adaptation to local situations is necessary. Theory can be actionable and, although bound to a domain, transferable to similar environments.	Theories and their Means: "Useful products and effective theory for solving individual and collective problems of Education." (Easterday et al., p. 319) Design knowledge: Principles, patterns, criteria, guidelines, advice, etc. as well as transformation of designers’ own knowledge (Kali, 2016).
Research-Based Design (RBD)	An exploratory process that has design as the focus and research as its means (Fallman, 2007). Not oriented by a research question but the pursuit of a solution.	Dialogue and participation of users may lead to designs that are more attuned to their needs and practices.	A novel design, not necessarily concerned with theory building.
Formative Evaluation (FE)	Assessment instruments that allow designers to identify gaps or areas of uncertainty. Organized as a continuous process of design, evaluation, and revision (Edelson, 2002).	The evaluative process is conducted while materials are being developed. Knowledge produced in one cycle will be utilized to inform the approach of subsequent iterations (Egan et al., 1989).	What works best—FEs show whether a design feature works better than conventional approaches (Egan et al., 1989). Alone, FEs, are not enough to base scientific claims.
Develop-mental Research (DR)	DR understands the development of learning environments (design) and the development of learning theories (research) as two highly intertwined processes (Bakker, 2018).	Research scope is focused on "how some ideal could be achieved; for example, “how can a problem-solving attitude be promoted?" (Bakker, 2018, p. 31).	Domain– or subject-specific theories and materials (e.g., learning module).
Participatory Action Research (PAR)	Orientated toward action generating critical consciousness among participants. PAR is "part pedagogy, part organizing, and part research" (Anderson, 2017, p. 440).	Participants are insider “agents” rather than objects of study. Research focus and questions are formulated with participants (Anderson, 2017). Praxis: research and action, as well as theories and their applications, are intertwined. Knowledge is not just an explanatory tool but a means of social transformation.	Creation, circulation and enactment of socially-relevant knowledge. Community capacity-building. (Anderson, 2017)

Table A2. Other design-oriented approaches to research methods.

Paradigm	Defining characteristics	Epistemological stance	End points
Design-Based Implementation Research (DBIR)	Focused on persistent problems of practice negotiated between practitioners and researchers, and concerned with systematic implementation. Iterative co-design is key and policy issues are often considered (Penuel et al., 2011).	Neither theory nor interventions alone are sufficient to generate knowledge (Easterday, Lewis, and Gerber, 2014), especially when problems are ill-structured. Theory can be actionable and, although bound to a domain, transferable to similar environments.	Large-scale design principles and policy guidelines.
Formative Interventions (FI) and Change Laboratories (CL)	Focused on transforming and/or creating new instruments, models or processes for key activities in organizations. A variation of FI, CL is an interventionist research toolkit aimed at generating actionable knowledge in activity systems facing major transformations. (Sannino & Engeström, 2017).	Agency of learners toward hypotheses are a source of information (Engeström, 2011). Knowledge is actionable and future-oriented. Double stimulation: Research stimuli are based upon (1) problematic situations that trigger participants’ agency through (2) symbolic artifacts. (Engeström et al., 2014) Abstract to concrete: participants identify theoretical principles that are later developed into concrete manifestations.	Possibility knowledge: future-oriented perspectives, and designs of new activities or instruments in social organizations. (Sannino & Engeström, 2017). (For CL) A model of the contradictions at play and a map of the organization’s zone of proximal development, showing how to enact change.
Improvement Science (IS)	Problem-specific, participatory and cyclical, IS integrates interventions with local knowledge-building at educational sites. Researchers and practitioners cooperate in communities (Bryk et al., 2015).	Knowledge exists not only in the intervention but also in the system and people. Research involves a combination of domain-specific (e.g., Mathematics), organizational (e.g. a district) and systemic forms of knowledge (Lewis, 2015). Variation in implementation is a source of information.	Organizational knowledge building, with ownership by participants (Lewis, 2015) Customization and fit between designed intervention and educational site.
Social Design Experimentation	Social design experimentation links ecological psychology research, DBR, and transformation of institutions toward social justice with an explicit focus on marginalized communities (Gutiérrez & Jurow, 2016)	Participants and researchers alike take a critical stance, examining historicity and creating syncretic conversation between marginalized communities and stakeholders in marginalizing institutions; co-construction of texts or interpretations is key.	Social transformation by increasing agency of participants, partly through insights gained in a DBR process.

Platt, J. (1983). The development of the “participant observation” method in sociology: Origin myth and history. Journal of the History of the Behavioral Sciences, 19(4), 379–393. https://doi.org/10.1002/1520-6696(198310)19:4<379::AID-JHBS2300190407>3.0.CO;2-5

 Web of Science ®Google Scholar

Bakker, A. (2018). Design research in education: A practical guide for early career researchers. Taylor & Francis Group.

 Google Scholar

Lewis, D. R., Carlson, S., Riesbeck, C., Lu, K., Gerber, E., & Easterday, M. (2020). The logic of effective iteration in design-based research. In M. Gresalfi and I. S. Horn (Eds.), The interdisciplinarity of the learning sciences, 14th International Conference of the Learning Sciences (ICLS) 2020) (Vol. 2, pp. 1149–1156). International Society of the Learning Sciences.

 Google Scholar

Collins, A., Bielaczyc, K., & Joseph, D. (2004). Design experiments: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 15–42. https://doi.org/10.1207/s15327809jls1301_2

 Google Scholar

Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research in HCI. In Conference on Human Factors in Computing Systems – Proceedings, 493–502. https://doi.org/10.1145/1240624.1240704

 Google Scholar

Kali, Y. (2016). Transformative learning in design research: The story behind the scenes [Keynote Presentation] International Conference of the Learning Sciences, 2016. https://doi.org/10.13140/RG.2.2.21293.33760

 Google Scholar

Fallman, D. (2007). Why research-oriented design isn’t design-oriented research: On the tensions between design and research in an implicit design discipline. Knowledge, Technology & Policy, 20(3), 193–200. https://doi.org/10.1007/s12130-007-9022-8

 Google Scholar

Easterday, M. W., Lewis, D. R., & Gerber, E. M. (2014). Design-based research process: Problems, phases, and applications. In J. L. Polman, E. A. Kyza, D. K. O’Neill, I. Tabak, W. R. Penuel, A. S. Jurow, K. O’Connor, T. Lee, and L. D’Amico (Eds.). Learning and becoming in practice: The International Conference of the Learning Sciences 2014 (Volume 1., pp. 317–324). International Society of the Learning Sciences.

 Google Scholar

Rittel, H. W., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. https://doi.org/10.1007/BF01405730

 Web of Science ®Google Scholar

Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105–121. https://doi.org/10.1207/S15327809JLS1101_4

 Web of Science ®Google Scholar

Egan, D. E., Remde, J. R., Gomez, L. M., Landauer, T. K., Eberhardt, J., & Lochbaum, C. C. (1989). Formative design evaluation of superbook. ACM Transactions on Information Systems, 7(1), 30–57. https://doi.org/10.1145/64789.64790

 Web of Science ®Google Scholar

Anderson, G. (2017). Participatory action research (PAR) as democratic disruption: New public management and educational research in schools and universities. International Journal of Qualitative Studies in Education, 30(5), 432–449. https://doi.org/10.1080/09518398.2017.1303211

 Web of Science ®Google Scholar

Penuel, W. R., Fishman, B. J., Haugan Cheng, B., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331–337. https://doi.org/10.3102/0013189X11421826

 Web of Science ®Google Scholar

Sannino, A., & Engeström, Y. (2017). Co-generation of societally impactful knowledge in Change Laboratories. Management Learning, 48(1), 80–96. https://doi.org/10.1177/1350507616671285

 Web of Science ®Google Scholar

Engeström, Y. (2011). From design experiments to formative interventions. Theory & Psychology, 21(5), 598–628. https://doi.org/10.1177/0959354311419252

 Web of Science ®Google Scholar

Engeström, Y., Sannino, A., & Virkkunen, J. (2014). On the methodological demands of formative interventions. Mind, Culture, and Activity, 21(2), 118–128. https://doi.org/10.1080/10749039.2014.891868

 Web of Science ®Google Scholar

Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America’s schools can get better at getting better. Harvard Education Press.

 Google Scholar

Lewis, C. (2015). What is improvement science? Do we need it in education? Educational Researcher, 44(1), 54–61. https://doi.org/10.3102/0013189X15570388

 Web of Science ®Google Scholar

Gutiérrez, K. D., & Jurow, A. S. (2016). Social design experiments: Toward equity by design. Journal of the Learning Sciences, 25(4), 565–598. https://doi.org/10.1080/10508406.2016.1204548

 Web of Science ®Google Scholar