Is research-based learning effective? Evidence from a pre–post analysis in the social sciences

ABSTRACT

Research-based learning (RBL) is regarded as a panacea when it comes to effective instructional formats in higher education settings. It is said to improve a wide set of research-related skills and is a recommended learning experience for students. However, whether RBL in the social sciences is indeed as effective as has been postulated for other disciplines has not yet been systematically examined. We thus administered a pre–post-test study to N = 952 students enrolled in 70 RBL courses at 10 German universities and examined potential changes in cognitive and affective-motivational research dispositions. Latent change score modelling indicated that students increased their cognitive research dispositions, whereas most affective-motivational research dispositions decreased. The instructors’ interest in the students’ work served as a significant predictor of changes in research interest and joy. Practical implications for designing RBL environments can be inferred from the results.

KEYWORDS:

• Research-based learning
• research knowledge
• research literacy
• affective-motivational research dispositions
• social sciences didactics

Introduction

Teaching and research can be linked through a variety of well-defined instructional formats. One of these is research-based learning (RBL), in which students conduct their own research with the help of a supervisor. RBL is currently seen as a panacea for addressing a range of demands within higher education, e.g. a lack of meaningful learning experiences and the need for stimulating instructional formats. Accordingly, several authors and institutions claim that RBL should be incorporated into the curriculum of many if not every academic study programme (e.g. Healey and Jenkins 2009). Indeed, a growing number of programmes have attempted to implement RBL in a range of disciplines and forms, e.g. the REU programme by the US National Science Foundation. The main goal of these endeavours is to provide students with an opportunity to experience participation in research. In science, technology, engineering, and mathematics (STEM) disciplines, there is evidence that RBL does indeed live up to its promises and constitutes an effective learning experience (Linn et al. 2015). However, outside the STEM disciplines, it is still unclear which research dispositions RBL fosters. Thus, this study aims to examine whether RBL’s effectiveness regarding the acquisition of various cognitive and affective-motivational research dispositions can be generalised to the social sciences.

Theoretical background

Positioning research-based learning in relation to other forms of research-related teaching

Teaching and research can be linked in different ways. In a popular model, Healey and Jenkins (2009) distinguish among different instructional formats for engaging students in research along two axes. The first axis describes whether the research results or the research process is emphasised. The other axis describes whether students take on an active role as participants or a passive role as audience. These two axes can be combined into four different formats: research-tutored, research-led, research-oriented and research-based learning. In RBL, teaching focuses on the research process, and students actively conduct research and inquiry. However, this description fails to describe the exact nature of students’ involvement in research. Huber (2014) further defines RBL as an instructional format in which students work through the entire research process in a self-regulated manner, guided by their own research questions. The instructor takes on a facilitating role. This theoretically derived definition was replicated in an empirical classification of research-related formats (Rueß, Gess, and Deicke 2016) and serves as the underlying definition of RBL in the current study.¹

The effectiveness of research-based learning

Conducting one’s own research project involves various cognitive, behavioural, and affective experiences (Lopatto, 2009, 29), which in turn lead to a wide range of benefits associated with RBL.

RBL is associated with long-term societal benefits because it can foster scientific careers: Students participating in RBL reported a greater interest in pursuing postgraduate education or PhDs (Lopatto 2007; Russell, Hancock, and McCullough 2007) and were more likely to be engaged in scientific careers six years after graduation (Hernandez et al. 2018).

In addition, RBL fosters research skills that are also necessary for occupations outside academia (British Academy 2012). RBL is said to facilitate the development of a ‘researcher’s mindset’ – the ability to objectively examine data or a situation and finding enjoyment in solving problems (Wood 2003). A researcher’s mindset can be effective in a wide range of professional activities. For example, in the field of psychotherapy, therapists could draw upon their research knowledge to consult evidence on new therapeutic approaches (Levant et al. 2006). Hence, the acquisition of research-related knowledge and skills is a prerequisite for successfully engaging in both scientific and non-scientific careers – making it an appropriate focus for our article.

Successfully engaging in a task requires both cognitive dispositions, such as knowledge, and affective-motivational dispositions to put this knowledge into practice (Blömeke, Gustafsson, and Shavelson 2015). Disposition serves as an umbrella term to denote a range of latent, personal resources (e.g. attitudes, traits and abilities) that determine how an individual will normally act in a certain situation (Schmidt-Atzert and Amelang 2012, 63). Accordingly, competent performance in the research domain requires various cognitive (e.g. knowledge) and affective-motivational (e.g. interest) research dispositions. Whether RBL is effective at facilitating the development of different cognitive and affective-motivational research dispositions has been the focus of previous studies. The existing evidence will be introduced in the following sections.

Cognitive gains

Most empirical studies on the effectiveness of RBL focus on cognitive research dispositions. However, the majority of these studies assessed STEM students (e.g. Linn et al. 2015; Seymour et al. 2004), with only a few studies investigating the effect of RBL in the social sciences. In a study from the field of social work, students gained domain-general research knowledge (Whipple, Hughes, and Bowden 2015). Taraban and Logue (2012) found evidence for a range of cognitive benefits of psychology students’ participation in research, such as improved research methods skills. Participation in RBL can also lead to increased understanding of the scientific process as a whole (Lloyd, Shanks, and Lopatto 2019).

Other researchers have examined specific skills pertaining to individual research steps, e.g. the ability to use statistics software (Whipple, Hughes, and Bowden 2015) and communicating and presenting one’s research (Stanford et al. 2017). RBL also seems to facilitate more general cognitive dispositions like critical thinking (Hunter, Laursen, and Seymour 2007; Kilgo, Ezell Sheets, and Pascarella 2015) and the ability to work independently (Stanford et al. 2017).

Thus, while RBL in the social sciences seems to be effective at facilitating a range of different cognitive dispositions, these results can only serve as preliminary evidence. A problem concerning the interpretability of these and other studies in the field lies in their methodological designs: Most existing studies focus on subjective ex-post assessments and self-evaluated skill gains (e.g. Stanford et al. 2017). However, self-assessments are often distorted by personality (John and Robins 1994) or skill levels themselves (unskilled students overestimate their abilities, see Kruger and Dunning 1999). Large-scale investigations using objective measures provide more substantial conclusions, but have so far only been completed for STEM students (e.g. Russell, Hancock, and McCullough 2007). Linn et al. (2015) note that the underlying problem is a lack of valid measures to objectively investigate the effectiveness of RBL. To address this problem, the Social-scientific Research Competency Test, an objective measure of cognitive research dispositions in the social sciences, was developed by Gess and colleagues (Gess, Geiger, and Ziegler 2018; Gess, Wessels, and Blömeke 2017). The instrument is based on a coherent model of different areas of research knowledge necessary to conduct critical steps in the research process (see Appendix 1, online supplemental data). In validation studies, the instrument has been shown to be suitable for evaluating social-scientific research education and could serve as an objective measure of the cognitive benefits of RBL.

Affective-motivational gains

Higher education research is increasingly acknowledging the importance of affective-motivational aspects for learning (e.g. Postareff and Lindblom-Ylänne 2011). Reflecting this general trend, affective-motivational gains have also drawn increased attention in research on RBL.

Evidence on RBL’s potential to alter affective-motivational research dispositions often stems from studies with multidisciplinary samples. Demonstrated benefits include higher research self-efficacy (Deicke, Gess, and Rueß 2014; Whipple, Hughes, and Bowden 2015), increased intellectual curiosity (Bauer and Bennett 2003) and a higher tolerance for obstacles in the research process (Lloyd, Shanks, and Lopatto 2019). Furthermore, a study with STEM students demonstrated a greater desire to learn and an increased disposition towards working with ambiguity (Ward, Bennett, and Bauer 2003).

The few existing studies all examine individual affective-motivational research dispositions, often in an exploratory manner. However, conducting research is an especially demanding task that requires students to handle uncertainties and manifold frustrations (John and Creighton 2011). Thus, it can be assumed that successfully conducting research requires a range of different affective-motivational dispositions to cope with the challenges of the research process. A coherent, empirically grounded model of the affective-motivational research dispositions necessary for student research in the social sciences has been recently developed (Wessels et al. 2018). It encompasses dispositions that are necessary to begin and to sustain the research process: for example, research interest is needed to initiate a research process, while sustaining it requires frustration tolerance to cope with inevitable setbacks. It is unclear whether RBL is effective in developing these research dispositions.

Overall, studies on the nature and effectiveness of RBL in the social sciences are generally scarce and often based on weak methodological designs – in contrast to studies from other disciplines. However, one cannot assume that the evidence gained in studies with STEM students easily translates to the social sciences. First, research seems more important to university programmes in the natural sciences than in the social sciences (cf. Taraban and Logue 2012). Second, most research experiences within STEM disciplines occur in structured lab environments that might have a different pedagogical culture (Rand 2016). Third, if discipline-specific outcome variables are to be investigated, a study needs to be conducted in that specific discipline.

Another open question pertains to the processes by which RBL in the social sciences affects changes in different research dispositions. In studies with STEM students, the main predictors of learning gains are the duration and intensity of the research experience: longer-lasting and more intense research experiences lead to stronger increases in skill levels (Bauer and Bennett 2003). Another study found that students with higher levels of autonomy in the research process, e.g. the autonomy to make their own methodological decisions, showed stronger learning gains (Gilmore et al. 2015). However, which characteristics of RBL courses in the social sciences affect changes in different research dispositions has not been studied yet.

Research questions and hypotheses

The objective of this paper is to analyse the effectiveness of RBL courses in the social sciences. Two main research questions guided our work: (1) Does research-based learning have a positive effect on cognitive and affective-motivational research dispositions? (2) How do different course characteristics relate to changes in these research dispositions?

Pertaining to the first research question, the following hypotheses were tested:

Hypothesis 1a: As previous studies have found associations between student research experiences and self-evaluated knowledge gains (Taraban and Logue 2012), we predict that students will have significantly higher post-test scores than pre-test scores for research knowledge (knowledge of methods, knowledge of methodologies and research process knowledge).

Hypothesis 1b: As previous studies have found associations between student research experiences and a higher tolerance for obstacles in the research process (Lloyd, Shanks, and Lopatto 2019) as well as an increased ability to work with ambiguity (Ward, Bennett, and Bauer 2003), we predict that students will have significantly higher post-test scores than pre-test scores for affective-motivational research dispositions.

Pertaining to the second research question, the following hypotheses were tested:

Hypothesis 2a: Since studies in STEM disciplines have demonstrated that longer and more intense research experiences (Bauer and Bennett 2003) have a positive influence on the effect of participation in RBL, we predict that the intensity of the research experience, i.e. the number of research steps performed, will influence changes in research knowledge.

Hypothesis 2b: Since studies in STEM disciplines have demonstrated that higher levels of autonomy in the research process (Gilmore et al. 2015) positively impact the effect of participation in RBL, we predict that students’ autonomy, i.e. ability to freely choose a research question and a research method, will positively affect changes in affective-motivational research dispositions.

Hypothesis 2c: We predict that different motivating factors, e.g. students’ self-efficacy, the perception that they are doing ‘real research’, perceived instructor interest in the students’ work, and the perceived usefulness of RBL for their later career will positively affect changes in affective-motivational dispositions.

Methods

To answer our research questions, paper-based measurements were conducted at the beginning and the end of RBL courses offered in different social scientific disciplines at 10 different universities.

Procedure

As the objective was to study comparable RBL courses in the social sciences, only the curricula of study programmes employing empirical social science research methods were considered. These included sociology, political science, psychology, and education science (see also Gess, Wessels, and Blömeke 2017).

Suitable RBL courses were identified via their course descriptions. Only courses that allowed students to experience a full research cycle in a self-regulated manner were considered, in line with our definition of RBL. The instructors of 146 courses were contacted via email and asked to participate in the study; 65 agreed to participate, 50 did not wish to participate, mostly due to time constraints in the course, and the remaining 31 instructors did not respond. Pre-tests were scheduled for the first two weeks of the course, and post-tests for the last two weeks of the course.

Altogether, pre- and post-measurements were conducted in N = 70 RBL courses at 10 universities across Germany. All universities included were state-funded public universities with 10,000-50,000 students offering degrees in a wide range of disciplines.

The testing itself was conducted during class time by one of the authors of this article, who explained the procedure and general purpose of the study. The questionnaires were administered in the form of printed booklets. A personal 6-digit code based on non-sensitive information, e.g. birthday month, was used to match pre- and post-test questionnaires while granting anonymity. Filling in the questionnaire took approximately 25 min. The post-test followed the same procedure. Additionally, a brief instructor survey on characteristics of the course instruction was administered.

Sample

The sample encompassed N = 952 students (74.1% female, 23.5% male), of which 881 participated in the first measurement and 539 participated in the second measurement. Higher participation rates at the first measurement point were due to higher course attendance at the beginning of the semester.

The mean age of the participating students was M = 24.38 years (SD = 4.79). 61.6% of the students were enrolled in a bachelor’s programme, while 29.5% were enrolled in a master’s programme. Fifty students were enrolled in other study programmes, such as the traditional German university diploma, and were treated as either bachelor’s or master’s students depending on their study progress. Bachelor’s students were near the end of their second year of study on average; the mean number of semesters completed was M = 3.33 (SD = 1.67). Master’s students were at the beginning of their second year of master’s studies on average, with M = 2.57 (SD = 1.63) semesters of the degree completed on average.

The students were enrolled in different fields of study, namely educational science (31.4% of the students), psychology (22.4%), sociology (10.3%), communication science (8.6%), and political science (5.5%) The remaining students were studying other, more specific social scientific subjects (i.e. media studies).

The students were enrolled in one of 70 RBL courses. Participation was often a mandatory part of the students’ study programmes: 41.8% of the students were required to enrol in this specific course; an additional 35.7% could have chosen a different RBL course, while only 17.6% could have chosen a course not involving the instructional format of RBL. The average number of participants per course was M = 13.54 (SD = 12.62). The majority of students were enrolled in one-semester courses (77.7%); 22.3% of the students were enrolled in two-semester courses. The courses were led by 65 different instructors or co-teaching teams. Fifty two of these instructors participated in the instructors’ survey at the end of the course.

Measures

Research knowledge

A 9-item short version of the social-scientific research competence measure by Gess, Wessels, and Blömeke (2017) was used to assess research knowledge in the social sciences. This test assesses knowledge of research methods, knowledge of methodologies and research process knowledge with items referring to both quantitative and qualitative research. The test uses short vignettes coupled with multiple choice questions on different research problems (see sample item in Appendix 1, online supplemental data). The instrument has gone through several validation studies and is suitable for the evaluation of research courses in the social sciences in both bachelor’s and master’s degree programmes (Gess, Geiger, and Ziegler 2018; Gess, Wessels, and Blömeke 2017). Since the full 27-item measure takes 35 min to complete and in-class time was sparse, a 9-item short version reflecting the full breadth of the original test in terms of content areas was developed based on the discrimination parameters, item difficulty, reliability and correlation with the long version. The correlation of the person scores for the short version and the person scores for the long version is r = 0.86, which indicates that the two versions measure a similar construct. However, it must be noted that the short version of the test has not undergone the same validation procedure as the long version. The students’ answers were coded as either correct (1) or incorrect (0), such that the final data consisted of 9 dichotomous items. The reliability was acceptable, with weighted omega h = 0.69 (see Table 1).

Table 1. Sample items, means, standard deviation and weighted omega of the variables at T1 and T2.

Disposition with sample item		Number of items	Mean (SD)		Weighted Omega h
1.	Research knowledge – T1 Sample item see Appendix 1, online supplemental data	9	0.46	(0.16)	.69
2.	Research knowledge – T2	9	0.50	(0.16)	.66
3.	Value-related interest in research – T1 ‘Compared to other topics, I assign a high value to research.’	5	3.96	(0.46)	.80
4.	Value-related interest in research – T2	5	3.86	(0.50)	.80
5.	Joy in working with scientific literature – T1 ‘I enjoy reading the scientific literature on a topic.’	4	3.20	(0.80)	.77
6.	Joy in working with scientific literature – T2	4	3.03	(0.85)	.82
7.	Joy in working with empirical data – T1 ‘I enjoy analyzing data.’	4	3.44	(0.47)	.68
8.	Joy in working with empirical data – T2	4	3.45	(0.57)	.74
9.	Uncertainty tolerance – T1 ‘I find it disturbing that before I start my research project, I don't know whether everything will work out as I imagine it will.’	4	2.71	(0.73)	.73
10.	Uncertainty tolerance – T2	4	2.83	(0.78)	.75
11.	Frustration tolerance – T1 ‘If my data analysis turns out to be incorrect and I have to start all over again, I would probably despair.’	4	2.57	(0.54)	.71
12.	Frustration tolerance – T2	4	2.55	(0.56)	.76

Affective-motivational research dispositions

The model of affective-motivational research dispositions (Wessels et al. 2018) encompasses nine necessary dispositions for pursuing research in the social sciences, of which four were selected to be investigated in the present study. (1) Value-related interest in research subsumes beliefs about the usefulness of research. (2) Finding joy in conducting research denotes the joy experienced with respect to different research activities. (3) Research-related uncertainty tolerance is the disposition to handle uncertainties in the research process. (4) Research-related frustration tolerance is the disposition to endure setbacks in the research process.

Self-assessment scales (sample items and basic descriptive data can be found in Table 1) were developed in a multistep process following deductive and inductive test construction procedures (Burisch 1984). First, at least 20 items per disposition were constructed according to fixed theory-driven construction principles (Wilson 2005). The items were selected and refined based on a pilot study with N = 250 students from the social sciences. The final instruments encompass 4 or 5 items per disposition and exhibit acceptable or good reliabilities (weighted omega h = 0.68–0.82). The response format for all affective-motivational measures was a five-point Likert scale ranging from 1 (completely disagree) to 5 (completely agree).

Instructor and course characteristics

Student survey

During the pre- and post-test, students were asked for additional information. At pre-test, this included their self-assessed research self-efficacy (6 items on a 5-point scale, e.g. ‘I am sure I can find suitable assessment tools for a quantitative study, even if the main variable is difficult to operationalize’). At post-test, students were asked about the research steps (e.g. searching for relevant literature) they had completed so far, their perception of the instructor’s interest in their research project, their perception of whether they were doing ‘real’ research, and the perceived usefulness of the course for their later career (all measured with one item each on a 5-point scale).

Instructor survey

The post-test was also used to gather information about the course’s instructional concept from the instructor’s perspective. A 5-minute questionnaire distributed to the instructors asked about students’ autonomy in choosing their own research question and method (two items on a five-point scale).

Statistical analysis

In a first step, students’ pre- and post-test data were matched via their personal six-digit code. We used SPSS 23 to conduct data checks and descriptive analyses of the manifest variables. To investigate changes in the different variables over time, we employed latent change score modelling (McArdle 2009; LCM) and multiple regressions. LCM and all necessary preceding analyses were performed with Mplus version 8 (Muthén and Muthén 2017). The following three steps were performed:

Dimensionality tests

To confirm the assumed factor structures and allow for a meaningful interpretation of the data, we conducted confirmatory factor analyses on all variables (see Appendix 2, online supplemental data). For almost all variables, the unidimensional model exhibited better model fit. The only exception was the variable ‘Finding joy in conducting research’, which exhibited inadequate model fits in both the unidimensional and the three-dimensional solution. Hence, subsequent analyses were conducted with two separate factors for this construct to ensure a meaningful interpretation of the data. The first factor describes ‘joy in working with scientific literature’, while the second describes ‘joy in working with empirical data’.

Measurement invariance tests

A prerequisite for latent change score modelling is strong factorial invariance (McArdle 2009). Only if strong factorial invariance is given can all factor loadings and intercepts be fixed to the same values for all measurement points. Following Meredith and Horn (2001), the CFI values of increasingly constrained models were compared (see Appendix 3, online supplemental data). For all variables, either strong factorial invariance or partial measurement invariance was established, meaning that the subsequent analyses can be meaningfully interpreted.

Latent change score modelling

We then employed LCMs to examine changes in our variables over time. In LCM, change is modelled with latent difference variables that express the change across two or more measurement points (see Figure 1). This approach enables us to observe interindividual differences in intraindividual change free from measurement error (McArdle 2009).

Figure 1. Illustrative latent change model for two measurement points and three items.

LCM analyses were performed in two steps: In the first step, we specified univariate LCMs (with two measurement points, T1 and T2) for each variable. The latent change variable indicates intraindividual changes from T1 to T2. Therefore, this variable was interpreted to test hypotheses H1a and H1b (effectiveness). The variance of the latent change variable indicates interindividual differences, i.e. whether students’ research dispositions develop in different ways. When significant interindividual differences were found, in a second step, the latent change variable was regressed on six different course characteristics. The regression coefficients were then interpreted to test hypotheses H2a, H2b and H2c (impact of course characteristics).

To account for the nested structure of the data (N = 952 students nested in 70 courses), we used the course as a cluster variable with the Mplus command TYPE = COMPLEX. Additionally, auto-correlated errors were included to account for method variance resulting from the use of the same items over the two measurement points. Missing data were handled using full-information maximum likelihood estimation (FIML). The criteria suggested by Hu and Bentler (1999) were used as a reference point for determining good model fit: models with a CFI > 0.95 and a RMSEA < 0.06 were considered to have adequate fit.

Results

Univariate latent change score models: changes in individual cognitive and affective-motivational research dispositions over time (hypotheses H1a and H1b)

Research knowledge

The LCM for research knowledge exhibited good model fit (see Table 2). The mean of the change variable was small but significant (ΔM = 0.04, p < .01), indicating a significant change from T1 to T2. This means that after taking the RBL course, students were able to correctly answer 0.45 questions more on average (out of nine questions) than at T1. Thus, the data supported hypothesis H1a. The variance of the change variable was very small and not significant (σ² = 0.001, p = .8), indicating that there were no interindividual differences.

Table 2. Model fits of all univariate latent change score models.

Model	Χ^2(df)		p	RMSEA	CFI
1. Research knowledge	199.22	(141)	.001	0.02	0.93
2. Value-related interest in research	132.61	(37)	.001	0.05	0.95
3. Joy in working with scientific literature	62.18	(21)	.001	0.05	0.98
4. Joy in working with empirical data	91.69	(20)	.001	0.06	0.92
5. Uncertainty tolerance	39.29	(21)	.01	0.03	0.98
6. Frustration tolerance	38.37	(20)	.01	0.03	0.98

The univariate LCMs for all affective-motivational dispositions had very good model fits (see Table 2). The dispositions differed in their development from T1 to T2:

Value-related interest in research

The results revealed a significant decrease from T1 to T2 (ΔM = −0.14, p < .01). The significant variance of the change variable (σ² = 0.33, p < .01) indicates the presence of interindividual differences in changes in interest.

Joy with respect to research activities

As described above, this variable consisted of two distinct factors whose development was examined individually. The results suggest a significant decrease in ‘joy in working with scientific literature’ from T1 to T2 (ΔM = −0.17, p < .01). The significant variance of the change variable (σ² = 0.36, p < .01) indicates that there were differences in students’ trajectories. No significant change was observed for the second factor, ‘joy in working with empirical data’ (ΔM = −0.05, p = .25). The significant variance indicates the presence of interindividual differences in students’ trajectories (σ² = 0.15, p < .01).

Uncertainty tolerance

The results suggest a significant increase from T1 to T2 (ΔM = 0.12, p < .01). The significant variance (σ² = 0.38, p < .01) indicates that there were substantial interindividual differences in students’ trajectories.

Frustration tolerance

The results show that frustration tolerance did not change significantly from T1 to T2 (ΔM = 0.03, p = .24). The significant variance was indicative of interindividual differences (σ² = 0.12, p < .01).

Therefore, the data supports hypothesis H1b only with respect to uncertainty tolerance. For value-related interest in research and joy in working with scientific literature, significant decreases were found.

Influence of other variables on changes in different research dispositions over time (hypotheses H2a, H2b and H2c)

Next, predictors of the change variable were analysed for the research dispositions for which the univariate LCMs showed evidence of interindividual differences. This was the case for value-related interest for research, joy in working with scientific literature and uncertainty tolerance.

Value-related interest for research

The multiple regression revealed two significant and positive predictors of the latent change in value-related interest in research: the perceived usefulness of the course for one’s later career and the instructor’s perceived interest in the students’ work. The overall variance explained by this regression model was 10% (see Table 3).

Table 3. Multiple regression analysis for affective-motivational research dispositions.

	Change of value-related interest in research		Change of joy in working with scientific literature		Change of uncertainty tolerance
Predictor variables (and time point of measurement)	B (SE)	β	B (SE)	β	B (SE)	β
Research self-efficacy (T1)	−0.01 (0.01)	−0.08	−0.01 (0.00)	−0.01	−0.03 (0.01) **	−0.22 **
Number of research steps performed (T2)	−0.02 (0.03)	−0.04	<−0.01 (0.03)	<−0.01	0.02 (0.03)	0.04
Usefulness of the course for a later profession (T2)	0.10 (0.03) **	0.24 **	0.10 (0.04) **	0.22 **	−0.01 (0.04)	−0.03
Student autonomy (T2 – lecturer survey)	0.02 (0.02)	0.04	0.01 (0.04)	0.03	−0.04 (0.03)	−0.09
Lecturers interest in students’ work (T2)	0.09 (0.04) *	0.16 *	<−0.01 (0.05)	−0.01	0.03 (0.04)	0.05
Perception of conducting ‘real’ research (T2)	−0.04 (0.04)	−0.10	0.00 (0.04)	0.01	−0.01 (0.03)	−0.02
AIC	31,375		28,945		29,746
R^2 (SE)	0.10 (0.04)		0.05 (0.03)		0.06 (0.04)

Note: B = unstandardised coefficients, SE = standard error; β = standardised coefficients.

*p < .05.

**p < .01.

Joy in working with scientific literature

The perceived usefulness of the course served as a significant predictor of the latent change in joy from T1 to T2. Students who perceived the course as useful for their later career experienced greater increases in joy in working with scientific literature. The full regression model explained 5% of the variance in the change in joy (see Table 3).

Uncertainty tolerance

Uncertainty tolerance was significantly predicted by research self-efficacy at T1. Self-efficacy served as a negative predictor: the higher a student’s self-efficacy, the more uncertainty tolerance decreased or the less it increased. The overall variance explained by this regression model was 6% (see Table 3).

These findings are in line with hypothesis H2c, which examined the influence of additional motivating factors. Hypotheses H2a and H2b were not supported.

Discussion and implications

Our study examined the effectiveness of RBL in the social sciences. By applying pre–post measurements in 70 courses, we examined changes in different cognitive and affective-motivational research dispositions through participation in RBL. Research knowledge increased significantly, but no interindividual differences were observed that could be further investigated. Research-related uncertainty tolerance increased, whereas research interest and joy in working with scientific literature decreased over the course of RBL participation. Subsequent regression analyses showed that the change in uncertainty tolerance was significantly predicted by research self-efficacy. The changes in interest and joy were predicted by the perceived usefulness of the course for one’s later career, while the change in interest was also predicted by the instructor’s perceived interest in the students’ work.

Contrary to our expectations, the number of research steps performed and the autonomy students were given during the RBL experience did not have an effect on changes to any of the affective-motivational research dispositions.

Research knowledge

Overall, research knowledge increased significantly over the course of RBL participation (see hypothesis 1a). Previous studies with students from individual social scientific disciplines have reported comparable results (e.g. Taraban and Logue 2012). We were able to confirm these findings using an objective test instrument assessing three sub-areas of research knowledge: knowledge of methods, knowledge of methodologies and research process knowledge in the social sciences.

However, the students in our sample did not exhibit substantial interindividual differences in their improvement and no further analyses could be conducted to explain differences in the observed change with reference to other variables. This lack of interindividual differences might have been due to similar answering patterns on the knowledge items. We used a 9-item short version of a longer test, which might not have been sufficient to identify substantial differences between students. In future projects, we would recommend using the 27-item test form or another objective measurement that yields more variance in students’ answers.

Affective-motivational research dispositions

A significant change from the first to the second measurement point was found for three out of the four affective-motivational research dispositions examined.

In line with our expectations (see hypothesis 1b), uncertainty tolerance increased over the course of RBL participation. This change in uncertainty tolerance was significantly predicted by research self-efficacy (see hypothesis 2c). However, self-efficacy served as a negative predictor: the higher a student’s self-efficacy, the smaller the positive change in uncertainty tolerance. Students with low levels of research self-efficacy might exhibit stronger increases in uncertainty tolerance because these students have less research experience and thus benefit more strongly from participation in RBL. A high level of uncertainty tolerance is important for coping with the unpredictable nature of the research process. Some claim that uncertainty tolerance is vital not only for conducting research but also for facing an increasingly complex world in general (Brew 2010). In this sense, uncertainty tolerance not only assists students in pursuing scientific careers but also prepares students for other professions. How students’ uncertainty tolerance can be changed is currently a subject of debate in several fields. In the health sciences, it has been suggested that medical students’ uncertainty tolerance can be enhanced by monitoring and controlling emotional processes related to uncertainty (Iannello et al. 2017). Translating this recommendation to research in the social sciences, we suggest integrating guided reflections on experienced emotions related to uncertainty in the research process. One way of doing so would be to use reflective learning diaries (Nevalainen, Mantyranta, and Pitkala 2010). However, we did not test for reflective processes related to uncertainty in our sample. We can only assume that some instructors reflected on and discussed research-related uncertainties. Further research investigating the influence of guided reflection processes on the development of uncertainty tolerance in RBL courses would be necessary to come to a more comprehensive conclusion.

Interest and joy in research exhibited high mean values during both the pre- and post-test, indicating that the participating students are generally very fond of research and related activities. However, unlike uncertainty tolerance, interest and joy decreased over the course of RBL participation (see hypothesis 1b). There are several possible explanations for this. Perhaps students gain a more realistic idea of what research is during the course. At the beginning of their studies, students’ conceptions of research might be influenced by the predominant view of research in their society: in Germany, the public perceives research as interesting and trustworthy (Wissenschaft im Dialog 2018). Thus, realising how small the explanatory power of a single research project is might be frustrating or disillusioning. Gaining a more realistic understanding of the nature and practice of research might lead to decreased interest or joy in research, while simultaneously serves as an indication of what others have termed ‘becoming a scientist’ (Hunter, Laursen, and Seymour 2007).

The regression analyses showed that certain course variables served as significant predictors (see hypothesis 2c): changes in students’ interest in research were significantly predicted by the instructor’s perceived interest in the students’ research and the perceived usefulness of the course for their later career (both rated by the students). Perceiving that the instructor is interested in their work might be motivating for students and increase their own interest in research. As a practical implication, this does not mean that instructors should pretend to be interested in students’ work. It could suffice for instructors to choose topics for RBL courses that are of genuine interest to them – for example, their own research topics. Bringing one’s own research topics into the classroom, thereby combining one’s teaching and research, has often been recommended as a useful practice for instructors (Vicens and Bourne 2009). One of the main arguments for this is that it saves valuable time for instructors involved in both teaching and research. Our results additionally suggest that combining teaching and research comes with benefits for students, who feel more motivated by their instructors’ interest in the topic.

Changes in joy were significantly predicted by the perceived usefulness of the course for students’ later careers: those students who perceived the course as useful for their future career gained more joy in research. For students who do not aspire to academic careers, it might be beneficial to emphasise or enhance the course’s usefulness for outside academia, e.g. by choosing research topics that are of interest in non-academic careers or applying service learning (Potter, Caffrey, and Plante 2003). In this way, more students might perceive conducting their own research projects as useful for careers outside academia and therefore find greater joy in doing research.

Contrary to our expectations, the number of research steps performed and the autonomy students were given during the RBL experience did not have an effect on changes to any of the affective-motivational research dispositions (see hypothesis 2a and 2b). This indicates that even working on pre-defined research problems or completing only a limited amount of research steps has a positive effect on students.

Overall, the regression models used to predict changes in different affective-motivational variables accounted for 5% (joy in working with scientific literature), 6% (uncertainty tolerance) and 10% (research interest) of the latent change variable’s variance. While these effect sizes can be classified as small (Cohen 1988), it is important to put these values into perspective: given that answering the questionnaires on the predictor variables took students only 1–2 min, the cost-value ratio of these regression analyses can be considered very positive. From a more fundamental perspective, it must be noted that affective-motivational dispositions are complex, multidimensional phenomena that are influenced by a range of external variables, such as current mood or personal life events. The variables examined in this study (e.g. student autonomy, instructors’ interest) are not sufficient to accurately predict changes in different affective-motivational research dispositions over an entire course. However, they did partially serve as significant predictors and thus provide practical new ideas for designing RBL courses.

Limitations and implications for future research

A problem with our and other studies in the field is the lack of a control group (cf. Lopatto 2004). Without an adequate comparison group, it remains unclear whether the research experience itself is effective or whether it is the type of student who participates in RBL courses (Linn et al. 2015). Some authors claim that students who seek out RBL courses have higher academic abilities and are more motivated than other students in the first place (Carter et al. 2016). In our sample, participation in the RBL course was often a mandatory part of the students’ study programme; thus, a strong self-selection bias in our sample can be ruled out. Nevertheless, a meaningful, matched control group is still necessary to draw final conclusions on the effectiveness of RBL, e.g. by examining study programmes with a waiting list for RBL courses.

Another limitation concerns the testing time point. Since the post-measurement was conducted in the classroom towards the end of the course, our results do not reflect the effect of writing final papers or presenting research results. However, giving a public presentation on one’s research has been described as particularly motivating by students (Cuthbert, Arunachalam, and Licina 2012) and thus might influence the learning outcomes associated with RBL. Future research should incorporate the effects of final assignments by using later or follow-up measurements.

Our study’s quantitative set-up meant that the students’ personal perspectives on their research projects, individual reactions to challenges in the research process and additional thoughts on their instructors’ behaviour could not be addressed. A future project could further explore and validate the preliminary findings of this study and the resulting implications by incorporating students’ perspectives via in-depth interviews.

The aim of this study was to examine the effectiveness of RBL courses in the social sciences for enhancing cognitive and affective-motivational research dispositions. Based on the results, we can conclude that RBL is an effective instructional format for enhancing research knowledge and research-related uncertainty tolerance. RBL courses proved especially effective when students thought the RBL experience was useful for their later career.

The question of whether RBL is an effective instructional format has so far been dominated by studies from the field of STEM, while evidence from the social sciences remains scarce. Our study sought to provide a systematic account of the effectiveness of RBL among students from different social scientific disciplines for enhancing discipline-specific measures using a pre–post design. While the chosen procedure was suitable for extending existing evidence in the field, a range of open questions remain that should be addressed in further research endeavours.

Acknowledgements

We would like to thank Christoph Geiger, Frederic Lenz and Luise Behm for their valuable help in conducting this study. We acknowledge support by the Open Access Publication Fund of Humboldt-Universität zu Berlin.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was supported by the German Federal Ministry of Education and Research [grant number 01PB14004B].

Notes

1 What we call RBL has different names elsewhere, e.g. ‘undergraduate research experiences’ (URE), ‘summer undergraduate research experiences’ (SURE) or ‘course-based undergraduate research experiences’ (CURE). Most of these terms describe the context (‘during the summer’) or the type of students (‘undergraduates’) rather than the instructional set-up per se. We chose the term ‘RBL’ to denote a specific instructional approach independent of the exact duration or the participating students. We do, however, use evidence from studies examining ‘CURE’ or ‘URE’. We carefully checked that the students’ research experiences aligned with our notion of RBL.