How research-based learning affects students’ self-rated research competences: evidence from a longitudinal study across disciplines

ABSTRACT

The ongoing implementation of research-based learning (RBL) within the higher education core curriculum leads to a growing number of students who conduct their own research projects during their course of study. Although previous studies have indicated that research-based learning has positive outcomes for students, their causal link remains unclear given a focus on specific disciplines, and the lack of pre-post designs and control groups. To fill this gap, the aim of our study is to examine how research-based learning, specifically the conduct of the research process by students on their own, affects their self-rated research competences with a longitudinal research design that was conducted in a wide range of disciplines. We used panel data from Germany (N = 520) and applied fixed-effects panel regression models to analyze students within student variance to the treatment effect of research-based learning. Results suggest two main findings. First, engagement in research-based learning significantly increases students’ self-rated research competences relating to reviews of the state of research, methodology, reflection on research findings, communication, and content knowledge. Second, the effectiveness of research-based learning differed among students enrolled in bachelor's and master's degree programs.

KEYWORDS:

• Research-based learning
• research competences
• longitudinal data
• comparison of teaching methods
• program effectiveness
• fixed effects

Introduction

Research-related teaching is a central pillar in academic education that can occur in different ways. One of the most sophisticated ways is research-based learning (RBL), in which students actively participate in a research process. The aims of RBL are not only to promote students’ research competences but also to augment their general professional qualifications by teaching them key competences like communication, presentation, and problem-solving skills (Wessels et al. 2021; Thiem, Preetz, and Haberstroh 2020b; Linn et al. 2015; Taraban and Logue 2012; Stanford et al. 2017). Accordingly, RBL is considered a ‘panacea for addressing a range of demands within higher education’(Wessels et al. 2021, 2595). In many degree programs, learning through the conduct of research is described as an ‘undergraduate research experience’ (URE), which is, by definition, limited to undergraduate students. We use the term ‘research-based learning’ (Healey and Jenkins 2009) to describe an instructional format for Bachelor and Master students who conduct a research project (full research cycle) (Mieg and Deicke 2022; Pedaste 2022; Wessels et al. 2021).¹

Healey and Jenkins (2009) systematized various possible combinations of linking research and teaching along two axes. One axis of their model showed research content versus research processes, and the second axis showed to what extent students are treated as either participants or as an audience. The following linkages were identified according to the two dimensions of the model (Healey and Jenkins 2009):

• Research-led: students are the audience, and the focus is on research content;

• Research-tutored: students are participants, and the focus is on research content;

• Research-oriented: students are the audience, and the focus is on research processes and problems; and

• Research-based: students are participants, focusing on research processes and problems.

Despite its popularity in (under-)graduate research, this model neither specifies the exact nature of students’ involvement in research nor does it present a clear definition of RBL (Wessels et al. 2021). Based on Healey and Jenkins (2009), Rueß, Gess, and Deicke (2016) provided a more pronounced classification concerning students’ involvement in research-related teaching as they differentiated the students’ activity level (receptive learning, applied learning, learning by research) and the thematic focus (research results, research methods, research process) (see Figure 1). The model was tested in a curriculum analysis across various disciplines in which modules, characterized by research-related teaching, were identified and categorized according to these two dimensions (Rueß, Gess, and Deicke 2016) (see also Wessels et al. 2021). This classification allows for more conceptual clarity and offers a theoretically derived and empirically tested re-definition of RBL. In this model, RBL is located in the cell ‘research process/learning by research’ (see Figure 1). It thus activates students to undertake their own research process. By pursuing their own research questions and inquiries, students become active producers of knowledge rather than simply being consumers. A central element of RBL entails producing insights that are new not only to the student but also to the community as a whole. Herein lies a fundamental difference between this learning method and other activating learning strategies, such as inquiry-based learning (Lopatto 2009).

Figure 1. Classification matrix of research-based learning and other research-oriented teachings.

RBL has been implemented within many degree programs across various disciplines because students’ independent research projects are ‘high impact’ educational practices that significantly benefit them (Kilgo, Ezell Sheets, and Pascarella 2015). However, little is known about the actual effects of this approach (Linn et al. 2015). Given the extensive requirements of resources (e.g. time and support provided by lecturers) for implementing RBL, the question of its effectiveness is pertinent. A key question is whether the benefits associated with RBL are a causal effect of this teaching method. This study addresses this question by applying a longitudinal, quasi-experimental research design in a multitude of disciplines. Specifically, we investigated whether RBL, in which students conduct a whole research process, affects the development of their self-rated research competences.

What is currently known about the effects of research-based learning?

Several benefits are associated with RBL, most of which can be categorized as research competences. Thus, students report a better understanding of the research process as a whole, enhanced skills in interpreting results, a higher tolerance for obstacles, and increased proficiency in analyzing data (Lloyd, Shanks, and Lopatto 2019), as well as greater confidence in their research skills (Russell, Hancock, and McCullough 2007; Seymour et al. 2004). In addition, students demonstrate enhanced abilities to assess their own research (Taraban and Logue 2012; Kardash 2000), research knowledge (Wessels et al. 2021; Seymour et al. 2004), research interest (Gess, Rueß, and Deicke 2014), and a deeper understanding of their research findings and of conducting research (Bauer and Bennett 2003). Other positive developments include growing interest in the subject, which can foster a scientific career or interest in RBL in general (Ward, David Clarke, and Horton 2014; Seymour et al. 2004). Furthermore, studies have shown that key competences, such as communication, problem-solving, and teamwork skills, are promoted through RBL (Ward, David Clarke, and Horton 2014; Taraban and Logue 2012; Seymour et al. 2004; Bauer and Bennett 2003; Kardash 2000).

However, little is known about the actual causal effects of RBL on acquiring research competences because the design of most of the studies does not allow for statements on causality, which require experimental or at least quasi-experimental designs with pre-post measurements and a control group. Existing studies are either based on qualitative interviews (Seymour et al. 2004), retrospective alumni assessments of undergraduate research experience (Bauer and Bennett 2003), or self-reports (Lloyd, Shanks, and Lopatto 2019). Wessels et al. (2021) assessed research knowledge, but their study is limited to the social sciences. Most quantitative studies lack a pre-post design, control group (Wessels et al. 2021), or both (Taraban and Logue 2012). The few existing studies with a quasi-experimental design are of limited scope because they are restricted to specific disciplines (Lloyd, Shanks, and Lopatto 2019; Ward, David Clarke, and Horton 2014). Moreover, some studies (e.g. Kardash 2000) may also suffer from (self-)selection effects, as they focus on courses or programs for which students participate voluntarily or even have to apply.

The current study

The aim of our study is to investigate the effects of research-based learning on several self-rated research competences. By developing their knowledge through RBL and transferring it to their individual knowledge structure, students acquire skills that extend beyond mere knowledge acquisition (Lopatto 2009). In light of these considerations, we expect that students who have completed an entire research project will rate their research competences higher than those who have not completed an entire research process.

Our study contributes to the literature in three ways: (1) the use of longitudinal panel data and methods, (2) a setting across disciplines, i.e. in a multitude of disciplines, and (3) investigating differences between undergraduate and graduate students.

First, we used longitudinal panel data covering the duration of the student's life cycles to analyze their self-related research competences before and after they experienced RBL. We also considered the research competences of students in a control group who did not experience RBL. Because previous studies are often based on qualitative or retrospective interviews or lack a pre-post design, a control group, or both, the causal effects of RBL remain unclear. The setting of our study, a German university, offers all of the necessary conditions for investigating the causal effects of RBL. Approximately 16,000 students at this university are enrolled in a wide range of disciplines, from the humanities and cultural studies to economics, law, social sciences, mathematics, computer science, the natural sciences, and medicine. Educational formats grounded in RBL are a key curricular element in the university's degree programs and not just within its undergraduate programs. At the University, RBL is the conduct of research projects by students within particular courses or modules (i.e. independently from their bachelor's or master's theses). Thus, all processual steps in a research project should be included, from defining a research question, gathering and analyzing data or material, to evaluating and presenting the project's findings. This conceptualization of RBL is in line with that of Rueß, Gess, and Deicke (2016) (see also Wessels et al. 2021; Pedaste 2022).

At a general level, this definition adheres to the pragmatic theory of science, according to which the unity of science lies in the unity of academic research practice or in the unity of the criteria of scientific rationality (Butts 1991; Akademie der Wissenschaften zu Berlin 1991), which is expressed in the (theoretically and methodologically based) research process. Thus, the common aim is to generate new knowledge based on empirical data. This approach allows for the domain-specific implementation of RBL according to the scientific traditions of the particular domains.

Second, most previous studies have focused on students within specific disciplines, but our study adopted a cross-disciplinary approach. We operationalized research competences as self-rated competences with reference to the cross-disciplinary RMRC-K model formulated by Thiel and Böttcher (2014) and the R-Comp instrument derived from this model (Böttcher and Thiel 2018; 2016). Both the model and the instrument are based on a pragmatic theory of science (Butts 1991; Dewey 1938), and the operationalization of research competences occurs according to the research process.

A cross-disciplinary model of research competences shares certain features of modern research, namely the independence of different subject cultures. Such research entails a methodically reflective examination of a research topic, consideration of existing theories and findings on the topic, and documentation of the research process and results that are comprehensible to the relevant research community (Thiel and Böttcher 2014).

From this definition, four basic dimensions of the research process can be identified and operationalized as concrete research competences (Böttcher and Thiel 2018; Thiel and Böttcher 2014):

• - skills in reviewing the state of research (R),

• - methodological competences (M),

• - reflection skills (R), and

• - communication skills (C)

In addition to these four dimensions of research competence, the model also incorporates the dimension of content knowledge (K). This dimension has been included, given the importance of comprehensive knowledge of the central theoretical constructs, paradigms, methods, and standards of the respective subject for the acquisition of research competence (Thiel and Böttcher 2014, 120; Böttcher and Thiel 2018).

The concept of competence in the RMRC-K model is based on a functional and pragmatic view of competences (Klieme and Hartig 2008) and an understanding of cognitive dispositions (Mayer 2003; Simonton 2003). Accordingly, students’ research competences have the following characteristics: they can be learned at university; they are helpful in the research process; and they are cognitive performance dispositions that functionally relate to situations and requirements, for example, during research-oriented teaching or while working on the final thesis; and they are in the domain of science/research. Motivational or volitional aspects are explicitly excluded in this concept (Böttcher and Thiel 2018, 2016).

The R-comp instrument relies on self-reports of research competences, thus allowing to apply of this instrument for students of all disciplines. Self-assessments may be biased depending on personality (John and Robins 1994) or social desirability (Ziegler and Buehner 2009). Students also have to assess their competences in comparison to immediate peers (Gess, Geiger, and Ziegler 2019). Therefore, self-reports depend, among other things, on the abilities of the other students in the course, and biases may arise due to the composition of the course (e.g. big-fish-little-pond effect; Seaton, Marsh, and Craven 2010). In addition, inexperienced individuals tend to overestimate their competences, which can paradoxically lead to a decrease in the level of competences in the course of an intervention (Kruger and Dunning 1999) (for this discussion, see also (Gess, Geiger, and Ziegler 2019)). However, due to the lack of a general objective research competence test that can be used in all academic disciplines in the same way, we decided to use self-rated competences.

Third, our study extends the existing literature on research-based learning by considering differences between undergraduate and graduate (master's level) students. Previous studies have mainly focused on undergraduate research. Therefore, little is known about differences in the effectiveness of RBL between students enrolled in bachelor's and master's degree programs. The effectiveness of RBL could be related to different levels of expertise or competence (Böttcher and Thiel 2018; Simonton 2003; Mayer 2003). At the beginning of their degree programs, students are regarded as novices who may benefit most from RBL when they learn and experience the elements of a research process for the first time. As their degree program continues, the effect of RBL on the students’ research competences may decrease because their current level of experience does not result in further advancement of their research competences. For example, their communication or presentation skills may not show further improvement after they have made a specific number of presentations.

Data and methods

Participants

We used longitudinal data from the project ‘Research-Based Learning in Focus Plus’. The aim of this project was to evaluate the ongoing implementation of RBL in the degree programs offered by the university across all disciplines. Starting in April 2018, we administered an online, questionnaire-based survey within a cohort of students enrolled in bachelor's and master's degree programs to elicit their experiences with RBL. Over two years, the students were invited to answer follow-up questionnaires after every semester, resulting in five survey waves. Online surveys with questions about the student's participation in RBL, entailing reflection on the previous semester, were conducted at the beginning of each new semester. For example, students were asked if they had engaged in RBL during the previous winter semester at the beginning of the summer semester. We used this strategy to capture all kinds of RBL, especially those that continued up to the end of the semester, such as writing or presenting final papers.

A total of 749 students participated in at least two waves and provided longitudinal information. To uncover the effects of RBL on students’ self-rated research competences, we applied a quasi-experimental research design focusing on individuals’ within-variation by following up with students before and after they had experienced RBL. The basic idea behind this approach was to analyze how the occurrence of a treatment (research-based learning) resulted in increasing or decreasing outcomes (self-rated research competences). We analyzed data from students who had not experienced RBL during the first panel wave to obtain a baseline measurement before RBL occurred. We followed up with these students over time. Due to this restriction, our final sample comprised 520 students with 1,405 observation time points. The students came from a wide range of disciplines such as social and natural sciences, the humanities and cultural studies, economics, engineering science, mathematics, computer science, sport science, and medicine. A total of 53 different study programs are covered in the sample. Most of the students (73.8%) were at the beginning of their studies in the first wave. Female students made up 73.4% of the sample, and nearly two-thirds (62.5%) of all students were enrolled in a bachelor's degree program. Half of all students (N = 212; BA students = 107; MA students = 105) experienced RBL in subsequent waves.

Data analysis

We use fixed-effects (FE) panel regression models to investigate the causal effects of research-based learning on students’ self-rated research competences (Allison 2009). Standard regression models result in biased estimates of causal effects when there are unobserved confounders, whereas fixed-effects panel regression provides unbiased estimates when modeling causality because of the use of persons’ within-person variation (Brüderl and Ludwig 2014). FE models do not infer causal effects from group comparisons of different persons but from comparing within-person changes induced by a treatment event (Brüderl and Ludwig 2014; Allison 2009). The main advantage of this method is that its focus on persons’ within-variation enables individuals’ unobserved heterogeneity to be controlled for, thus preventing confounding effects that could result from persons’ between-variation. In more detail, instead of conducting a between-variation comparison of students who did or did not experience RBL, we identified the effect of this learning method on students’ self-rated research competences, according to differences between individuals’ competences observed before and after they underwent RBL. Stable characteristics or heterogeneity between individuals caused, for example, by their sex or course characteristics, no longer influenced the effect of RBL on self-rated research competences. Self-selection and between-variance did not bias results. Thus, the transformation of a standard between-regression model into an FE panel regression model that reveals differences over time illustrates the advantage of using only within-variation: (1) ${\gamma }_{it}=\text{β}{x}_{it}+\gamma {\mathrm{Z}}_i+{\alpha }_i+{ϵ}_{it}$(1) (2) $\mathrm{\Delta }{\gamma }_i=\text{β}\mathrm{\Delta }{x}_i+\mathrm{\Delta }{ϵ}_i$(2) ${\gamma }_{it}$ is the dependent variable (self-rated research competences) that changes for different individuals ($i$) and time points $\left(t\right)$, and $x_{it}$is a vector of an independent variable or treatment (research-based learning) that varies with individuals and over time. Conversely, ${\mathrm{Z}}_i$ is a vector of time-constant variables that do not vary over time, and $\text{β}$ and $\gamma$ are vectors of coefficients. The two error terms are ${\alpha }_i$ and ${ϵ}_{it}$, with ${\alpha }_i$ representing the unit-specific effects of all unobserved time-constant variables and ${ϵ}_{it}$ representing random variation for individuals and different time points that is statistically independent from other variables (Feldhaus and Kreutz 2021; Brüderl and Ludwig 2014). In the less restrictive FE model, any correlations between ${\alpha }_i$ and ${\mathrm{Z}}_i$ and $x_{it}$ are allowed and have the effect of controlling for time-invariant observed and unobserved variables. Given that this study focused on changes in the dependent measure ${\gamma }_{it}$ (self-rated research competences) caused by changes in the time-variant independent measure $x_{it}$ (research-based learning), equation (1(1) ${\gamma }_{it}=\text{β}{x}_{it}+\gamma {\mathrm{Z}}_i+{\alpha }_i+{ϵ}_{it}$(1) ) can be rewritten as equation (2(2) $\mathrm{\Delta }{\gamma }_i=\text{β}\mathrm{\Delta }{x}_i+\mathrm{\Delta }{ϵ}_i$(2) ). By using the differences scores Δ, all time-constant information and errors were netted out of the equation, and only the within-variation information was considered. The FE approach offers unbiased estimates of a time-varying predictor or treatment variable for a specific outcome that is not confounded by interindividual variation (Allison 2009).

It is essential to control for time as an independent variable in the FE models to consider possible time trends within the control group of those who do not experience a treatment (Brüderl and Ludwig 2014). In this study, it seems plausible that students’ self-rated research competences increased not only when they underwent RBL. Competences like content knowledge or communication skills may have also increased over the course of their studies in light of more general experiences acquired in several different classes. Without controlling for time trends, the treatment effect could be overestimated. To uncover the causal effects of RBL, we considered these general time trends by including time dummies for every panel wave and considered the control group of non-treated students that did not experience RBL. The time trend contributed to the FE estimation when it was differenced out from the within comparison in the treatment group.

We estimated several FE models, with each self-rated research competence considered as a dependent variable and RBL and time as independent and control variables. While it is neither possible nor necessary to estimate time-constant effects in FE models, we used interactions between experiencing RBL and the student's enrollment in bachelor's or master's degree programs to investigate differences for both stages in the student's life cycles. Within panel data, observations from the same individual over several time points were not independent of each other. We used panel-robust standard errors in all FE models to correct for heteroscedasticity and arbitrary clustering of time series due to serial correlation (Brüderl and Ludwig 2014; Stock and Watson 2008). Furthermore, we used a specific extension of the widely used Jarque-Bera test to test the assumption of normality. While this test detects departures from normality in the form of skewness and excess kurtosis for standard regression models, (Alejo et al. 2015) extended the test for linear panel data. Normality of the residuals of each presented FE model was tested with the null hypothesis that our sample data was not significantly different from a normal population. For all models, test results accept the null hypothesis and normality. The lowest significance value was p = 0.137 for skills in reflecting on research findings.

Dependent measures

Our dependent variables were self-ratings of five different research competences based on the RMRC-K model and the R-Comp questionnaire developed by Böttcher and Thiel (2018). This model and associated questionnaire were aimed at capturing several key research competences that included all aspects of an entire research process experienced during RBL. All items were phrased in a way that encompassed a multitude of disciplines, entailing interdisciplinary conceptualization and enabling the analysis of a sample of students across a wide spectrum of academic disciplines. Overall, students assessed their research competences via self-ratings for 32 grouped items. An example of an item under skills in reviewing the state of research was ‘I am able to systematically review the state of research regarding a specific topic.’ An example of an item under methodological skills was ‘I can apply different research methods appropriate to my research question.’ An example of an item under skills in reflecting on research findings was ‘I am able to adequately interpret my own research findings by relating them to key theories in the subject area.’ An example of an item under communication skills was ‘I am able to prepare research findings for a presentation at a research colloquium.’ Lastly, an example of an item under content knowledge was ‘I have a good overview of the main (current) research findings in my discipline.’ The response scale for every item ranged from 1 = strongly disagree to 5 = strongly agree. Table S1 in the Supplementary lists all of the items with their means. All research competences showed high reliability levels, with Cronbach's alpha ranging from .84 to .91 (Table 1).

Table 1. Description of research competences.

Research competencies	Mean	Std	Cronbachs Alpha
State of research	2.97	.87	0.86
Methodology	2.89	.70	0.85
Reflection	3.21	.83	0.91
Communication	2.85	.89	0.84
Content knowledge	2.58	.80	0.91

Data: Waves 1-5; N = 520 students with 1405 observations

Independent measures

To analyze RBL as a treatment, we asked students whether they had performed and completed an entire research process during the previous semester, which entailed developing a research question, applying research methods and analysis, and presenting results. The time trend of students’ research competences independent of conducting RBL was measured by including time dummies for every panel wave.

Results

We began with a number of FE panel regression models, shown in Table 2. Separate models were developed for each research competence, indicating how RBL (i.e. completing an entire research process) resulted in changes in self-rated research competences. The results shown in models 1–5 indicate that the experience of engaging in RBL led to an increase in several self-rated research competences. Students’ skills in reviewing the state of research, methodological skills, skills in reflecting on research findings, communication skills, and content knowledge increased significantly after they had experienced an entire research process. For example, students’ self-rated skills in reviewing the state of research increased significantly by a factor of .179 when they engaged in RBL. The significant positive time effects for all research competences illustrate the importance of considering time trends in longitudinal models. The results indicated an overall increase in students’ self-rated research competences over the course of their degree programs. Apart from this general time trend, the significant treatment effects are indicative of a further increase in self-rated research competences as a result of experiencing RBL.

Table 2. Fixed-Effects Panel Regression for students’ research competences.

	(1)	(2)	(3)	(4)	(5)
	State of research	Methodology	Reflection	Communication	Content knowledge
Treatment
Entire research process	0.179***	0.144***	0.158***	0.137**	0.189***
	(0.0762; 0.282)	(0.0558; 0.232)	(0.0508; 0.265)	(0.0243; 0.249)	(0.0990; 0.280)
Time
Wave 2 vs. Wave 1	0.100***	0.0198	0.0787**	0.102***	0.118***
	(0.0317; 0.169)	(−0.0362; 0.0758)	(0.0121; 0.145)	(0.0322; 0.172)	(0.0571; 0.178)
Wave 3 vs. Wave 1	0.221***	0.108***	0.209***	0.205***	0.249***
	(0.124; 0.318)	(0.0318; 0.185)	(0.102; 0.317)	(0.106; 0.305)	(0.164; 0.333)
Wave 4 vs. Wave 1	0.391***	0.186***	0.245***	0.259***	0.272***
	(0.249; 0.532)	(0.0476; 0.324)	(0.0793; 0.410)	(0.0953; 0.423)	(0.147; 0.398)
Wave 5 vs. Wave 1	0.352***	0.182**	0.228**	0.350**	0.427***
	(0.153; 0.552)	(0.0100; 0.354)	(0.0196; 0.436)	(0.0725; 0.627)	(0.246; 0.608)
Constant	2.841***	2.829***	3.107***	2.734***	2.437***
	(2.804; 2.877)	(2.798; 2.860)	(3.067; 3.146)	(2.695; 2.774)	(2.403; 2.471)
Rwithin-Variance	0.104	0.056	0.069	0.062	0.126
NObservations	1,405	1,405	1,405	1,405	1,405
NStudents	520	520	520	520	520

Robust 95% CI in parentheses; *** p < 0.01, ** p < 0.05, * p < 0.1; Data: Waves 1-5; own calculation

In the next step, we analyzed whether the positive effects of RBL on students’ self-rated research competences varied according to whether students were enrolled in bachelor's or master's degree programs. Students enrolled in university undergraduate programs may experience RBL for the first time within small projects, whereas master's students may experience more complex research processes. This difference in research processes could lead to different outcomes or pathways of self-rated research competences that change over time (Böttcher and Thiel 2018). Table S2 in the Supplementary shows the results of independent FE models for each research competence with interactions between the treatment, completing an entire research process, and if the student is in a bachelor's or master's degree program.

In the interaction model, the treatment's main effect shows how bachelor students’ self-rated research competences changed after completing an entire research process. The interaction effect revealed the difference between the RBL outcomes of undergraduate and graduate students. Figure 2 graphically illustrates the results of the interaction models, differentiating the effects of RBL on these two categories of students. Both groups of students developed enhanced skills in reviewing the state of research and methodological skills as a result of undergoing RBL, with significant improvements evident after they had completed the research process. Undergraduate students’ self-rated skills relating to reflection on research findings did not significantly increase after undergoing RBL. The communication skills and content knowledge of students in master's degree programs showed no significant changes after completing the research process.

Figure 2. Fixed-Effects Panel Regression for interaction between students’ reseach competences and B.A./M.A.

Discussion

The rise of RBL within higher education has prompted a growing number of studies that have analyzed the outcomes and effectiveness of RBL for students (Wessels et al. 2021; Lloyd, Shanks, and Lopatto 2019; Ward, David Clarke, and Horton 2014; Taraban and Logue 2012; Russell, Hancock, and McCullough 2007; Thiem, Preetz, and Haberstroh 2020a).

The aim of our longitudinal study was to investigate the impact of RBL, and specifically the effects of conducting an entire research process, on various self-rated research skills. Our cross-disciplinary approach, encompassing the students’ life cycles, yielded general insights on the effects of RBL that were not limited to a specific discipline or specific types of students (undergraduate vs. graduate). Moreover, it allowed a comparison between these two types of students. Using a longitudinal data set from a German university, we explored the progress of students’ self-rated research competences throughout their student life cycle, i.e. from the first semester until well into their studies or even until graduation. The availability and use of panel data allowed us to model the effect of RBL by applying fixed-effects panel regression models in a more causally focused manner than in previous studies. FE models focus on individuals’ within-variation and allow to control for unobserved heterogeneity caused by between variance. Therefore, the results are not biased because of self-selection or between variance.

Our results revealed two main findings. First, due to the use of a quasi-experimental research design, the results show that students’ research experience is effective, and previous findings are not only the result of self-selection effects (Linn et al. 2015). We found a statistically significant increase in the self-rated research competences of students who experienced RBL. Even after we controlled for general time trends over the course of the study program by including a control group of students who did not participate in RBL, the effects on students’ self-rated cognitive competences remained significantly positive. The consideration of a control group allowed us to differentiate between general increases in students’ self-rated competences over their course of studies and the specific benefits of experiencing RBL. Our results proved general rises in self-rated research competences for all students but further increases in competence levels for those who participated in RBL. Completing an entire research process increased self-rated skills in reviewing the state of research, methodology, reflection on research findings, communication, and content knowledge. Taken together, the implementation of research-based learning pays off.

Second, our results revealed differences in the effectiveness of RBL among students enrolled in bachelor's and master's degree programs. Completing an entire research process led to enhanced skills in reviewing the state of research and methodology among both groups of students. However, the communication skills and content knowledge of Master students were not significantly improved through their engagement in RBL. Students at the master's level should have a higher level of expertise than undergraduate students, given their prior experiences in RBL (Böttcher and Thiel 2018; Simonton 2003; Mayer 2003). Consequently, they may reach a peak in specific research competences that do not increase further with frequent repetitions. Students should have practiced developing communication skills, notably writing and presenting research findings acquired in previous courses, or at least in developing a bachelor's thesis. Additionally, RBL within master's degree programs may be more focused and based on previous content knowledge, exploring an innovative research question, and developing a research design that includes the use of advanced methods rather than simply gaining familiarity with and performing every step of a research process at a more basic level, like undergraduate students (for examples see, e.g. Ruf, Ahrenholtz, and Matthé 2019; Kaufmann 2019). This difference may explain why skills in content knowledge did not increase among master's level students and could also account for the lack of improvement in undergraduate students’ skills in reflecting on research findings. The focus for these students may have been more on learning how to conduct a research process and try it out rather than on reflecting upon it.

Practical and pedagogical implications

The implementation of RBL requires not only extensive resources (e.g. time-consuming supervision of students) it also goes along with a student-centered approach of learning. Since in RBL, students are responsible for their learning process, the role of the lecturer is to provide support as a learning facilitator or coach rather than an instructor who teaches knowledge. However, also in RBL, lecturer-centered units are not uncommon. But they aim to convey methodological and content knowledge to facilitate the student research process (Reichow 2021). From a pedagogical perspective, this requires lecturers who are able to adapt to these different forms of teaching. At many universities, seminars on RBL in higher education didactics have been implemented to prepare lecturers for RBL. In these seminars, central aspects of designing and planning teaching for RBL are addressed, such as the degree of independent work by students, the role of teachers, group work, and the development of research questions or suitable examination formats. Furthermore, the participants can jointly identify the difficulties and challenges of RBL and develop concepts for them.

Limitations and future research

Despite the advantages of our study's cross-disciplinary, quasi-experimental design, it also had some limitations. First, even though it covers 53 study programs, it presented evidence from only one university in Germany. Whereas on the one hand, the choice of one location removed institutional variability, on the other hand, the extent to which the results could be generalized to other institutions as well remains to be demonstrated (Stanford et al. 2017). However, the cross-disciplinary RMRC-K model and the operationalization of RBL as the completion of an entire research process allows for unmodified extension to other institutions.

Second, an approach that covers a wide range of disciplines entails one limitation: it relies on students’ self-reports of their research competences rather than on a test of their competences. It cannot be ruled out that the students’ self-reports of research competences when answering the R-comp questionnaire capture their expectations of research competences rather than their actual research competences. In this case, the self-reports may rather display research-related self-efficacy expectations. The concept of research-related self-efficacy is ‘understood to describe the individuals’ beliefs in their ability to perform certain tasks such as conducting sound empirical research and disseminating the research findings’ (Lambie et al. 2014, 142). Whereas self-reported competences indicate the current state of capabilities, self-efficacy expectations refer to future behavior. Moreover, while both constructs seem closely related, empirical findings on this relation are still lacking. Thus, the question of how research competence tests (that still are to be developed for a cross-disciplinary context), self-reported research competences, and self-efficacy expectations are related has not yet been answered. So far, research has been inconclusive: Whereas several studies find only a weak or no correlation at all between indirect measures of individuals’ cognitive competences and more objective test scores (Mabe and West 1982; Schladitz, Ophoff, and Wirtz 2018), other studies have demonstrated that self-assessments of competences provide valid measurements of students’ cognitive abilities (Caspersen, Smeby, and Aamodt 2017; Douglass, Thomson, and Zhao 2012).

A third limitation is that, like the samples used in previous studies, ours did not allow observations of multiple research processes per student. Future studies could examine the cumulative effects of repeatedly performed research processes or the effects of carrying out single elements of the research process on research competences (Reichow 2021). However, the results suggest that even master's students with prior research experience benefit from conducting a complete research process. In conclusion, our results, which are based on a longitudinal research design, contribute to advancing understanding of the effectiveness of RBL for strengthening students’ research competences.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Raw data were generated at the University of Oldenburg, Germany. Derived data supporting the findings of this study are available from the author Janina Thiem on request.

Additional information

Funding

The research presented here was carried out with financial support from the German Federal Ministry of Education and Research (BMBF) (grant number 01PL16056).

Notes

1 When referring to the evidence presented in studies on URE, we confirmed that this approach accords with our definition of RBL (for a similar approach, see also (Wessels et al. 2021)).