Exploring the potential of using undergraduates’ knowledge, skills and experience in research methods as a proxy for capturing learning gain

Abstract

Learning gain is a politicised concept within contemporary HE, and as such has been aligned with agendas of teaching excellence and learning outcomes but the extent to which it captures actual learning has yet to be clarified. Here, we report the outcomes of a learning gain study which examines how students’ knowledge, skills and experiences as researchers develops throughout their studies. We examine data from a self-reporting survey administered across a university and college-based HE providers during students’ second year of undergraduate study. The data highlight disciplinary differences in student engagement with research methods and the significance of perceived relevance of research methods to students’ learning. These findings do have a bearing on the development of measures of learning gain as they are demonstrating the complexity of capturing student learning across disciplines. Our findings can be employed to develop a method of capturing learning gain that can be integrated into undergraduates’ research methods education.

Keywords:

• Self-efficacy
• research preparedness
• research training
• student learning

Introduction

Learning gain is an emerging concept in higher education (HE) discussed in terms of the distance travelled by a student. To some (e.g. Coates, 2009; Schleicher, 2016) connections are made between the idea of learning gain and value added, but unlike value added which is centred on comparing ‘performance predicted at the outset of studies and actual performance achieved’ (McGrath, Guerin, Harte, Frearson, & Manville, 2015, p. xi), learning gain represents ‘the difference between the skills, competencies, content knowledge and personal development demonstrated by students at two points in time’. (McGrath et al., 2015, p. xi). Internationally, there are three drivers underpinning current rhetoric around learning gain – accountability, teaching enhancement and student learning (Varsavsky, Matthews, & Hodgson, 2014). Existing measures of learning gain such as the Collegiate Learning Assessment and the Assessment of Higher Education Learning Outcomes have attempted to address the first two of these drivers (Coates, 2009; Klein, Benjamin, Shavelson, & Bolus, 2007). The extent to which these serve to promote student learning has been questioned (e.g. Douglass, Thomson, & Zhoo, 2012). Nevertheless, through careful framing and explicit integration of a discourse of learning gain into undergraduate study, there is considerable potential for measures of learning gain to stimulate student learning (Varsavsky et al., 2014).

This paper reports interim findings of a Higher Education Funding Council for England’s (HEFCE) pilot project funded to evaluate different approaches to measure a student’s learning gain. There are many challenges to measuring learning gain, not the least of which are a lack of uniformity in entry-level qualifications into university, non-comparable entry and exit tests at university (McGrath et al., 2015; Varsavsky et al., 2014). This project focused on examining how students develop knowledge, skills and expertise of research methods, as they progress through their degrees, exploring the potential for students’ research experiences to serve as a meaningful way to measure learning gain. This paper will explore the rationale for using research methods, introduces the methods used to capture students’ development in this area and report on the outcomes generated from a bespoke self-reporting survey. We conclude by examining the potential of using research methods as a proxy for learning gain and discuss development of an innovative research methods toolkit to stimulate dialogue around learning gain to direct future learning and capture student performance.

Research methods and their role in undergraduate education

Developing the research capacity of undergraduates, provides them with the abilities to generate, apply and adapt new knowledge; these are skills that are integral not only to their success as undergraduates but also to the knowledge economy (Davis, Evans, & Hickey, 2006). The knowledge economy is reliant on graduates having a flexible knowledge base, underpinned by their ability to seek out and identify relevant knowledge, reconfigure and evaluate this knowledge to respond to the demands or challenges they are facing (Green, Hammer, & Star, 2009). As Jenkins, Healey, and Zetter (2008, p. 3) commented: ‘teaching students to be enquiring or research-based in their approach is not just a throwback to quaint notions of enlightenment or liberal education but central to the hard-nosed skills required of the future graduate workforce’. These themes resonate with the wider discourse around learning gain. In developing an agenda to measure and capture learning gain, the UK government want to provide employers with clearer information about the knowledge, skills and experience graduates possess (Schleicher, 2016).

Rhetoric relating to research methods education has foregrounded policy and practice in HE internationally, leading to policy recommendations triggering investment in infrastructure and development activities in UK universities (Healey & Jenkins, 2009; Levy & Petrulis, 2012). This has led to the emergence of a body of work relating to the pedagogy of research methods education. This highlights the gains to students’ research methods education of early exposure to research activities (Levy & Petrulis, 2012), and the role of active pedagogies that involve students practising and rehearsing essential research skills (Benson & Blackman, 2003; Earley, 2014; Wagner, Garner, & Kawulich, 2011; Winn, 1995). But the focus on research-based curricula does not represent a recent innovation; equipping students with the skills and abilities to think as researchers underpins many undergraduate programmes (Earley, 2014). Indeed, this is evidenced through the extensive use of the final year dissertation (Ashwin, Abbas, & McLean, 2017; Todd, Bannister, & Clegg, 2004). More widely, students’ development as researchers is framed by credit level descriptors used by UK HE providers to ‘define the level of complexity, relative demand and autonomy expected of a learner’ (SEEC, 2016). Credit level descriptors, such as those of the Southern England Education Consortium for Credit Accumulation and Transfer (SEEC) which are applied in UK HE contexts, and map to the European Qualifications Framework (European Commission, n.d.), provide a generic overview of the knowledge and understanding students should develop as they progress through their studies, as well as indicating the cognitive, intellectual, practical and transferable skills they should gain (SEEC, 2016). These descriptors increase in complexity over the duration of an undergraduate’s study, as shown in Table 1, and can be used by HE providers to inform curriculum and assessment design, as well as communicating expectations of learners (SEEC, 2016).

Table 1. Undergraduates developing knowledge, skills and experience in research as categorised through SEEC level descriptors.

Level of undergraduate study	Expected level of complexity, relative demand and autonomy as a research
Level 4	Identifies a well-defined focus for enquiry, plans and undertakes investigative strategies using a limited and defined range of methods, collects data from a variety of sources, and communicates results effectively in an appropriate format
Level 5	Undertakes research to provide new information and/or explores new or existing data to identify patterns and relationships. Uses appropriate theoretical models to judge the significance of the data collected recognising the limitations of the enquiry
Level 6	Demonstrates confidence and flexibility in identifying and defining complex problems. Identifies, selects and uses investigative strategies and techniques to undertake a critical analysis, evaluating the outcomes

Notes: This is the UK-specific credit level descriptors framework which aligns to the European Qualifications framework that providers generic descriptors that undergraduate provision, see levels 4–6. However, these are not as detailed as the SEEC level descriptors, hence the focus on this framework.

The progression in the knowledge, skills and experiences HE providers can expect of undergraduates as they progress through their studies is clearly reflected in the UK level descriptors and the broader European Qualifications Framework (Table 1). However, developing students’ skills and abilities as researchers is challenging (Deem & Lucas, 2006; Shaw, Holbrook, & Bourke, 2013). Many students, particularly early on in their studies, have a weak understanding of the epistemological foundations of research, which often manifests as a fixed conception of knowledge and the acceptance of facts as truth. This may often be confounded by limited experience or knowledge of critical reflection and evaluation practices (Murtonen, 2005; Schommer, 1990). This fixed conception of knowledge needs to be transformed through their studies in order for them to appreciate how knowledge evolves and changes, and also to equip them with the transferable skills of critical thinking, analysis and evaluation (Murtonen, 2015; Schommer, 1990). This transformation can be achieved via pedagogies such as enquiry or problem-based learning, and early exposure to research (Brew, 2013). More widely, students need to gain a comprehensive understanding of research methodologies in order to be able to rigorously and effectively function as researchers, reflect on their strengths and articulate these to future employers (Davis et al., 2006). The role of research methods within SEEC level descriptors is implicit, experienced through the increasing complexity of the knowledge and tasks students engage with through their studies (SEEC, 2016; Table 1).

Often the term ‘research methods’ is taken to narrowly represent methods of data collection or analysis (Murtonen, 2015). However, research methods represent a complex domain of knowledge encompassing the general principles of science, research paradigms, research approaches and methods, as well as addressing issues relating to the theoretical framing and philosophical underpinnings of knowledge (Murtonen, 2015). It means that research methods represent a combination of knowledge domains and practices. Students need to develop an understanding of these, as well as gain proficiency in specific practical research skills, in order to effectively conduct research (Earley, 2014). The challenges students face when engaging with this subject are often reported in terms of issues such as statistical anxiety (Chamberlain, Hillier, & Signoretta, 2015; Sizemore & Lewandowski, 2009), negative dispositions towards research (Murtonen, 2015), or students failing to recognise the relevance of research methods courses (Deem & Lucas, 2006; Papanastasiou & Zembylas, 2008). These issues can be exacerbated by curriculum marginalisation of research methods training which can potentially undermine students’ engagement with, and development as, researchers (MacInnes, 2012; Williams et al., 2016). These challenges have resulted in considerable attention being paid to this area, with researchers attempting to identify solutions in order to promote or better understand students’ learning about research methods (e.g. Earley, 2014; Howard & Brady, 2015; Kilburn, Nind, & Wiles, 2014; Nind, Kilburn, & Wiles, 2015).

One solution has been to consider the role of self-efficacy in research methods education (Forester, Kahn, & Hesson-McInnis, 2004; Seymour, Wiese, Hunter, & Daffinrud, 2000; Shaw et al., 2013). The concept of self-efficacy is widely applied in work focused on vocational theory (e.g. social cognitive theory) and career development studies, as it has been identified as shaping individuals’ career choices and persistence in a chosen profession (van Dinther, Dochy, & Seges, 2011; Zimmerman, Bandura, & Martinez-Pons, 1992). It has been suggested that researcher self-efficacy relates to students developing the confidence in performing research tasks, which in the long term promotes their learning and engagement with research methods (Forester et al., 2004; Shaw et al., 2013). Self-efficacy also determines students’ motivation and attainment in HE, and these in turn influence students’ self-regulation and the strategies they employ to achieve their own learning goals (van Dinther et al., 2011; Zimmerman et al., 1992). Self-efficacy therefore is highly relevant to research methods education, as students who exhibit high levels of self-efficacy are more likely to achieve the skill and knowledge goals they set for themselves (van Dinther et al., 2011; Shaw et al., 2013). Indeed Forester et al. (2004) identified self-efficacy in research tasks as effective in predicting students’ interest in conducting research, and that confidence measures provided a reliable measure of researcher self-efficacy. Similar conclusions were reached in related work by Shaw et al. (2013). As self-efficacy shapes an individual’s attitudes, beliefs, thoughts and feelings it does have direct relevance to a measure of a student’s learning; a student with a strong sense of self-efficacy can demonstrate higher levels of effort, persistence and resilience (van Dinther et al., 2011). Progression in a student’s sense of self-efficacy is achieved through mastery of specific skills or knowledge as this provides evidence of success. Students then draw on this sense of success to develop their capacity to perform tasks in the future, with more challenging tasks fostering a stronger sense of self-efficacy as they signal the benefit of maintained effort and persistence (Bandura, 1997; Palmer, 2006).

Self-efficacy has also been applied in a number of learning gain studies (e.g. Arico, 2016; Cox & Lemon, 2016; Lim, Hosack, & Vogt, 2012) which have used students’ self-assessment of their learning, knowledge and/or skills and demonstrated that students’ self-assessment of their learning can provide a robust measure of learning. Ongoing application of these measures in classroom settings (as explored, for example by Arico, 2016) have demonstrated learning gain over time. These studies are founded on the principles of self-efficacy, and given the connections that have been demonstrated by Forester et al. (2004)and Shaw et al. (2013) between self-efficacy and researcher development, this study centred on piloting a measure of learning gain based on researcher self-efficacy to explore new methods of capturing learning gain in HE.

Developing connections between learning gain and research methods

Researchers have indicated that learning gain is a complex and context-dependant construct, influenced by a range of factors such as student attitudes and entry profiles (Kandiko Howson, 2016). Attempts to capture learning gain have been shaped by a variety of definitions and methodological approaches (e.g. Corlu & Aydin, 2016; Cox & Lemon, 2016; Lim et al., 2012; Varsavsky et al., 2014), which make comparative work in this area challenging to undertake (McGrath et al., 2015).

The idea of learning gain is being widely critiqued by academics and students (Gourlay & Stevenson, 2017; TSEP, 2016). Capturing learning gain through metrics such as employability, student satisfaction or student performance data, as is the focal point of some contemporary work, threatens to simplify HE. It overlooks its transformative power and the need for higher level study to stimulate risk taking, to challenge students and in some cases to allow students to learn from failure (Gourlay & Stevenson, 2017). There is a concern that focusing on learning gain, as part of the wider discourse of teaching excellence, could lead to a measure of mediocrity (Gourlay & Stevenson, 2017). Wood and Su (2017) state that any outcome measure should be based on an ethical and relational concept that promotes growth and development, as advocated by Nixon (2007). This would allow for institutional context to be accommodated as well as student need, and this would rely on the pedagogic relationship underpinning learning to be considered. These writers have shaped our work, and as we demonstrate, capturing learning gain through the development of pedagogic relationships is integral to this study.

Based on the literature, we have identified the following as ‘success criteria’ which need to be addressed in order for a measure of learning gain to be useful. Any measure should:

•	evidence its validity as a measure of student learning and recognition of other student characteristics and environmental factors which might influence its outcomes;
•	generate student data which is representative and comparable at different stages of HE;
•	allow for meaningful comparison across institutions, while also recognising the diversity in the goals of HE institutions.

Drawing on the outcomes of the extensive body of work centred on student learning through engagement with research and research-related activities, research methods, and examining how a student’s knowledge in this subject develops and changes over their studies. It provides a useful starting point to develop a valid and useful measure of learning gain, which we present here drawing on data resulting from this first phase of this study.

As demonstrated by the SEEC level descriptors, all undergraduate programmes include an element of research training, and whilst the specific focus of this training may vary according to disciplines, it will cover key elements of the definition of research methods presented above from Murtonen (2015). This broad application of research methods, and academic progression (again, as framed by the SEEC level descriptors), creates the potential for developing a measure of learning gain that can be used across a number of academic levels, therefore capturing the temporal element of the McGrath et al. (2015) definition of learning gain, but also creates the potential for this measure to be applied across a range of disciplines. It is on this basis the following research questions were developed:

•	How do undergraduates’ attitudes towards, and confidence in, research methods change through their undergraduate study?
•	What pedagogic approaches are used to develop undergraduates’ knowledge, skills and expertise in research methods?
•	Can research methods be used as an effective measure of learning gain and what measures can be used to assess this?

Research design

Overall, the study employs a longitudinal research design to capture students’ knowledge, skills and experience of research methods as they progress through their undergraduate studies. This would potentially account for phenomena such as ‘sophomore slump’ – a term used to describe the underperformance of second-year undergraduates (Lemons & Richmond, 1987). Our work is ongoing, and here we present the self-reported survey data from the first phases of fieldwork. Due to sample size and coverage, cross-sectional analyses are presented. This study gained ethical approval from the University of Plymouth, Faculty of Business Research Ethics Committee.

A self-reporting survey represented the primary mechanism for data collection. These are widely used in research methods studies (Forester et al., 2004; Sizemore & Lewandowski, 2009; Williams, Payne, Hodgkinson, & Poade, 2008; Williams et al., 2016) and also by those examining learning gain (e.g. McGrath et al., 2015; Seymour et al., 2000; Varsavsky et al., 2014; Vogt, Atwong, & Fuller, 2005).

The research team have previously examined the role of research methods teaching in college-based HE. Here, we adapted a survey developed by Williams et al. (2008, 2016) to explore research methods teaching in Sociology programmes. These two surveys provided the basis of a data collection instrument that included a range of confidence measures and attitudinal scales that had been extensively tested in college and university-based HE that could be incorporated into the current study. Confidence measures and attitudinal scales have both been applied in studies examining research methods education (e.g. Williams et al., 2016) and self-efficacy (Shaw et al., 2013) as both have a significant impact on students’ development. Drawing on work by Jenkins et al. (2008) and Levy and Petrulis (2012), we then integrated further measures to capture students’ experiences and approaches to learning research methods, paying particular attention to the pedagogic practices (i.e. whether they were learning about research methods through structured or lecturer-led activities, practiced in a simulated task/research activity instigated by a lecturer, or through independent, student-led research. These descriptors were derived from discussions within the literature relating to effective pedagogic practice around research methods education (e.g. Levy & Petrulis, 2012; Williams et al., 2016). Confidence scales were used to capture students’ self-efficacy across a range of research approaches, practice and skills. We also captured data on students’ definitions and perceptions of research methods to generate evidence of their epistemological and methodological development as researchers, and pays heed to the work of Murtonen (2015) who observed that studies into research methods education need to look beyond skill development. Essential demographic information was captured and individual identifiers allocated to track individual students. Given that prior educational experiences have been identified as influential (Shaw et al., 2013), data on students’ previous academic qualifications were also collected. Consequently, the survey primarily captured quantitative data, with students responding to a series of statements or questions presented as Likert scales.

Despite widespread use of self-reporting surveys, some commentators have noted limitations in their application (e.g. Porter, 2013). Concerns have been raised the extent to which they capture actual learning and the ease at which students are able to access the experiences they need to draw upon to accurately rate their learning. To mediate any potential effects of this, the survey was prefaced with a definition of research, research methods and other key terminology. This decision was based on the recommendations of Porter (2013) and Tourangeau, Rips, and Rasinski (2000) regarding the use of self-reporting surveys to capture learning gain. We also did not want to cause undue anxiety amongst respondents around a lack of knowledge on aspects of research methods they had yet to study. Therefore, it was explained to participants that some of the areas of research methods may only relate to specific disciplines or stages of study.

The first iteration of the learning gain self-reporting survey underwent extensive development through consultation with programme teams to ensure that the survey was accessible and applicable to a range of disciplines and then piloted with a group of level five Business & Social Science students. The students’ feedback on the survey design, question style and ease of completion informed final revisions before the survey was administered.

Participant recruitment

A novel aspect to this project is that it spans a range of UK HE providers including a university and further education colleges delivering higher education courses. Established network of further education (FE) colleges that deliver HE (in the form of foundation and full honours degrees) validated by the University. College-based HE echoes the community college model found in the United States (Dougherty, 2009; Gray, 2016). FE colleges have long been involved in the delivery of HE. However, provision expanded considerably as part of moves to widen participation and promote engagement for underrepresented groups in the later 1990s (Gray, 2016). Students studying on a college-based HE course can either graduate following completion of a foundation degree, which represents the first two years of a full degree, or progress on the honours level either within the college, if provision is available locally, or to the validating partner (Gray, 2016). Foundation degrees are required to conform to the Foundation Degree Characteristics statement (QAA, 2015a) in providing vocationally focused HE at levels four and five. Predominantly (but not exclusively) delivered by FE colleges, these short cycle degree programmes provide a work-focused emphasis whilst still meeting the other requirements of any undergraduate subject at these levels (such as research methods training). Therefore, programmes based at the University and across its FE college network were selected to participate within the project in order to provide a measure of learning gain that has been tested across different HE contexts.

As research methods feature in all undergraduate programmes, it created the potential for the resulting measure of learning gain to be applied across disciplines. Therefore, a range of programmes were purposefully selected to be involved in this study. Sixteen programmes across one university and six FE colleges were chosen to represent the following disciplinary areas; Arts & Humanities (n = 6; university = 2 and college-based HE = 4), Business & Social Sciences (n = 8; university = 2 and college-based HE = 6) and Science (n = 2; university = 1 and college-based HE = 1). More college-based programmes were selected to compensate for smaller class sizes in colleges. The subject groupings of the Higher Education Academy were used to identify and align the disciplinary programmes. Programmes were selected for which there was mirrored provision at the University and the FE college network. This was considered essential to facilitate comparisons. Each programme within our sample draws on the SEEC level descriptors to shape the curriculum, representing an integral aspect of the quality assurance protocols of the validating university, hence their application in the framing of the research methods provision and the survey. To facilitate analysis, data are reported on a disciplinary basis only; future analysis will centre on disciplinary groupings and educational contexts.

Survey administration

There is debate on the most effective way to administer surveys, with observable variations in the profile of survey respondents according to whether surveys are administered online or paper-based (Nulty, 2008). Another challenging issue that persists, regardless of the method of administration, is response rates (Nulty, 2008). As the data resulting from this study would inform a learning gain toolkit, the research team were keen to maximise response rate, and therefore made the decision to administer paper-based surveys in class time. Three administration points were identified, ‘survey one’ in autumn 2015 where level five students would be prompted to reflect on their knowledge, experience and confidence in research methods following level four, ‘survey two’ at the end of level five, and ‘survey three’ at the end of level six. Most research methods teaching is concentrated in level five to support students planning their level six dissertation, therefore administering the surveys at these time frame would potentially capture resulting gains in learning. Here, we focus on data from survey one and two.

Data analysis

Following each period of data collection, data were input into SPSS by two members of the research team. Following the recommendations independent data checking was undertaken with 10% of the sample (Bryman, 2008). Descriptive analyses were undertaken of item statements at the level of study and by discipline group with a focus on baseline measures and mean differences.

Reliability analysis

The survey was subjected to reliability checks using level 4 data from the first sampling point. Item pools with illustrative example statements, alongside Cronbach’s α values are shown in Table 2. Since each item pool represented different dimensions, separate reliability analysis was undertaken on five of the identified pools. The item pool on engagement activities collected categorical multiple responses and was consequently omitted from reliability analysis. Self-reported confidence and self-efficacy: research skills evidenced higher levels of internal consistency (α > .90). Research orientation: emotion/feelings (a = .664) and learning motivations: frequency of studying (a = .674) showed levels of internal consistency below the accepted .7 cut-off for point. Since we are reporting research data mid-way through the study we will review when the data from all three data collection points is available. Research orientations: perceptions of research methods had lower levels of internal consistency (α = .445) with Corrected Item-Total Correlations of less than .3 suggesting that this subscale has less internal consistency (Field, 2013, p. 709). Preliminary Principal Component Analysis, PCA, (rotated varimax) was undertaken on five of the item pools, with factors retained where eigenvalue were >1. The Kaiser–Meyer–Olkin measure for sampling adequacy was applied and found to be mediocre for three item pools, and meritorious for two item pool. Investigation of the individual items clustering on the factors suggested some identifiable subscales, however given the size of the sample, and that the measurement tool was designed to reflect the learning gain in research methods across levels 4, 5 and 6, the research team took the conservative decision to undertake the remaining data collection for levels 5 and 6 prior to any future latent variable analysis.

Table 2. Item pools, reliability analysis and preliminary principal component analysis.

Item pool	No. items	Example item (scale)	α	PCA (No. of factors) (% of variance) (KMO)
1. Research orientation: emotion/feelings	4	Easy -> challenging (9-point scale)	.664	1
				50.75%
				.67
2. Research orientation: general perceptions	5	Research always begins with a question (4-point scale. 1 = strongly agree –> 4 strongly disagree)	.445	2
				52.03%
				.62
3. Self-efficacy in research skills	44	Manage and store all data/information collected in line with ethical guidelines and professional codes of conduct (10-point scale. 1 = Not heard of -> 10 Expert)	.965	7
				75.46%
				.87
4. Self-report confidence	14	Apply facts, theories or methods to practical problems or new situations (5-point scale. 1 = No confidence -> 5 = Extremely confident)	.911	3
				65.67%
				.90
5. Learning motivations: frequency of studying	12	Combined ideas from different modules when engaged in research-related tasks (4-point scale. 1 = Very Often -> 4 = Never)	.674	5
				65.74%
				.61
6. Learning motivations: engagement activities	14	Reviewing various information sources to gain an understanding of existing research and evidence around a particular topic	–	–
		Multiple response categories: (not studied, studied as a structured learning activity, Studied as independent reading, Practised in a simulated task, Applied in own research)

Findings

Here, we present an overview of the sample profile, and report data relating to respondents at survey points one and two in one University and six college-based HE providers. Cross-sectional results from the first two surveys are presented. As we report ongoing work we are yet to undertake longitudinal analyses involving matched case analysis including academic performance data, though this is a long-term ambition as these data becomes available to the research team. We also intend to control for institutional context, however, where relevant disciplinary trends are considered. These future analyses will facilitate a more detailed examination of the complexity of measuring learning gain through measures of pedagogic engagement, self-efficacy and performance outcomes at the level of the individual, and provide clear insights into the long-term viability of using research methods as a proxy for learning gain. The analysis presented here allows us to begin to assess these factors at a cohort/disciplinary level and provide a valuable basis on which to shape subsequent data collection.

Sample profile

To date 312 surveys have been collected from 219 students (74 M, 131 F, 14 prefer not to say/unspecified) (see Table 3 for a breakdown of respondents across survey points and disciplinary areas). At level 4 this represents a response rate of 48.0%, and level 5, 33.9%. Participant engagement with the study has been variable across each disciplinary area (Table 3), despite the best efforts of the research team. Attrition in learning gain work is an ongoing challenge (Kandiko Howson, 2016; Porter, 2013), and has resulted in many studies adopting cross-sectional design or undertaking cross-sectional analyses (McGrath et al., 2015). Indeed accessing Arts and Education participants for the second administration point was initially unsuccessful. As data collection are ongoing, we are monitoring respondents’ engagement and exploring strategies to maintain and promote participant engagement (e.g. extending data collection beyond the current academic year and recruiting additional cohorts in disciplines where attrition is concerning). Once this work has been completed a profile analysis of non-participation can be undertaken to provide insights into non-administrative barriers to engagement, and potential impacts this has on representativeness considered (De Vaus, 2014). Furthermore, we plan to consider response profiles alongside summative academic performance. The research team also intend to undertake an in-depth analysis of all data once all surveys have been administered which will allow the analysis of matched cases to be considered.

Table 3. Number of survey responses across each data collection point by disciplinary area.

Survey response profile	L4 survey (n)	L5 survey (n)
Business and social sciences	70	58
Sciences	34	32
Art and education	79	39
Total	183	129

The participants’ ages ranged from 18 to 50 years with a mean of 23.5 (SD = 7.14). The majority of respondents (69.8%) identified A-levels as their most recent qualification, 9.3% had entered based on their professional/vocational qualifications, 6.0% had completed Access to HE courses and 5.5% studied GCSE’s. The varied entry profile represents the breadth of educational contexts included in this study as college-based HE is recognised as attracting vocationally orientated students or those entering with non-traditional qualifications (Lea & Simmons, 2012). However, the prominence of A-level qualifications indicates that some respondents from this group are likely to have completed these before commencing HE study, even though they are based within an FE college where these qualifications are not generally perceived to be the norm (Lea & Simmons, 2012).

Research orientations: student attitudes and perceptions of research methods

Based on their level 4 experiences 92.9% of respondents stated they thought studying research methods and developing research skills was an essential part of their course, dropping slightly to 88.4% at Level 5. This is a notable finding; a major issue for research methods educators is students failing to perceive the relevance of research methods provision, which impacts on their motivation to learn the subject and their level of interest (Earley, 2014). This may reflect changing attitudes to research methods more widely – a consequence of the focus placed upon this area by policy-makers and universities (Kilburn et al., 2014; Nind et al., 2015). This trend mirrored across discipline groups (Table 4). The level 4 to level 5 reduction was not unanticipated; students enter HE with preconceived ideas of research (Murtonen, 2005). Those with prior family experience of HE have been observed to possess more realistic understandings of research (Bangera & Brownell, 2014). However, even these individuals experience a change as they begin to engage with research and research-related activities. First years often perceive activities such as literature searching as research (Sizemore & Lewandowski, 2009). Engaging with research methods courses fosters realistic understandings of research. Earley (2014) suggests that students adjust their expectations according to the aspects of research methods they engage with, although this can depend on the pedagogic approaches used. Integral to this is the extent to which students formulate an awareness of the wider applicability of research methods, for example, whether they are taught primarily research skills or the wider applicability to their discipline (Murtonen, 2015). The impact of this is evident as we examine learning motivations drawing on respondents’ pedagogic experiences and practising of research methods (Table 5).

Table 4. Studying and developing research skills is essential by level and discipline group.

	Business & social science (%)	Science (%)	Arts & education (%)
Level 4	89.9	100.0	92.4
Level 5	81.0	96.9	92.3

Table 5. Pedagogical learning experiences at Level 4 and Level 5 (%).

	Level 4			Level 5
	S	P	A	S	P	A
Reviewing various information sources to gain an understanding of existing research and evidence	66.3	42.5	32.6	45.7	55.8	54.3
Constructing a research question	39.7	33.5	24.6	52.7	37.2	38.0
Considering research design and methodology/critical approaches and critical theories	61.9	37.0	19.3	58.1	44.2	45.7
Reviewing measurement tools used for data collection through consideration of reliability and validity issues/	43.6	24.6	15.1	50.4	27.9	34.1
Identifying and avoiding bias in research	58.9	31.7	25.0	51.2	29.5	41.1
Research ethics and professional codes of conduct	48.6	28.2	19.9	47.3	34.1	36.4
Considering methods for data collection/ identification of source materials	53.9	37.1	21.9	55.8	38.8	46.5
Identifying conclusions from research data	55.0	40.6	28.9	60.5	35.7	46.5
Reporting research	37.4	30.7	21.2	49.6	37.2	44.2
Qualitative data analysis of text, images and videos	58.7	29.6	20.7	51.2	29.5	38.0
Quantitative data analysis focusing on descriptive analysis	44.1	30.5	16.9	49.6	30.2	36.4

Notes: Multiple response questions (Level 4 valid n = 177–181 & Level 5 valid n = 129).

S = structured learning activity; P = practised in a research activity/project set by teaching staff; A = applied as independent research.

Pedagogic engagement with research methods education

Students were asked to identify how they had studied and practiced essential aspects of the research process (Table 5). Consistent with the move from level 4 and level 5, and as students accrue and recognise their gain in knowledge and experience of different learning activities in research methods training, the survey data demonstrate a move towards more practice and applied learning activities. For example, at level 4 ‘considering research design and methodology/critical approaches and critical theories’ were more likely to be studied through structured learning activities (61.9%) than a practice-based activity (37.0%) or applied to independent research (19.3%) (Table 5). By level 5, there is a shift towards higher proportions of practice (44.2%) and applied (45.7%) activities. Active engagement with research methods is essential to fostering positive attitudes to research methods. Deem and Lucas (2006) identified a relationship between how students are taught about and engage with research methods and perceived relevance. In their early encounters with research skills, especially when encountered through didactic approaches, students can perceive research methods as abstract; their appreciation only matures through active engagement with research (Murtonen, 2015). Active engagement with research methods can also uncover the limits to their knowledge, and this is where attention should be focused to promote future learning and avoid limitations in a student’s confidence in an aspect of research methods undermining their future engagement – an ongoing challenge for the quantitative research methods educators in particular (Chamberlain et al., 2015).

Some research areas are reported as having less coverage in the curriculum (e.g. ‘constructing a research question’, and ‘reporting research’; Table 5). It is likely that as students progress through their studies (and equally this project), and they complete their dissertation, that their familiarity with these topics may increase. At levels 4 and 5 students’ ‘reporting’ of research is driven by assessment guidelines that focus their attention towards specific activities (e.g. report writing) and provided prescriptive templates that seem removed from how they may commonly encounter primary research in journals. Interestingly though, growing in prominence, undergraduate research conferences represent a recent innovation (Walkington, Hill, & Kneale, 2017) which may explain why this is an underdeveloped aspect of research methods education.

Disciplinary differences were evident in the pedagogical approaches used, which to some extent may be expected, as they reflect the research traditions of disciplines. There is a clear rationale for teaching research methods within the context of a substantive discipline to promote relevance and engagement amongst students (Bridges, Gilmore, Pershing, & Bates, 1998). For example, level 5 respondents in Science reported practising and engaging in activities that involved them ‘considering research design and methodology/critical approaches and critical theories’ practising and applying, which were higher than those in Business & Social Science and Arts & Education (Table 6). Likewise engagement with methods of qualitative data analysis of text, images and videos was greatest in Arts & Education, with application of these skills reported (Table 6). Business & Social Science students tend to encounter their research methods through structured activities, with limited opportunities to practice or apply research skills reported (Table 6). This could potentially indicate a connection between the pedagogic framing of research methods within the discipline and student engagement.

Table 6. Pedagogical learning experiences at Level 4 and Level 5 (%) around specific aspect of research methods/discipline.

	Business & Social Science						Science						Arts & Education
	Level 4 (%)			Level 5 (%)			Level 4 (%)			Level 5 (%)			Level 4 (%)			Level 5 (%)
	S	P	A	S	P	A	S	P	A	S	P	A	S	P	A		S	P	A
Considering research design and methodology/critical approaches and critical theories	52.9	20.6	4.4	57.9	31.6	33.3	67.6	67.6	32.4	59.4	81.3	75.0	67.1	38.0	26.6		59.0	33.3	41.0
Reviewing measurement tools used for data collection through consideration of reliability and validity issues /	39.7	16.2	7.4	53.4	13.8	24.1	58.8	47.1	32.4	65.6	56.3	56.3	40.3	22.1	14.3		33.3	25.6	30.8
Considering methods for data collection/ identification of source materials	53.0	30.3	12.1	53.4	25.9	34.5	58.8	67.6	35.3	68.8	65.6	65.6	53.9	29.5	24.4		48.7	35.9	48.7
Identifying conclusions from research data	50.0	27.9	20.6	62.5	25.0	32.1	61.8	70.6	32.4	78.1	71.9	62.5	56.4	38.5	34.6		46.2	23.1	56.4
Qualitative data analysis of text, images and videos	50.7	22.4	11.9	52.6	22.8	22.8	57.6	42.2	24.2	59.4	50.0	50.0	65.8	30.4	26.6		43.6	23.1	51.3
Quantitative data analysis focusing on descriptive analysis	41.2	22.1	8.8	51.7	22.4	25.9	57.6	54.5	33.3	71.9	62.5	53.1	40.8	27.6	17.1		28.2	15.4	38.5

Notes: [Level 4 valid n = 177–181; Level 5 valid n = 127–129).

S = structured learning activity; P = practised in a research activity/project set by teaching staff; A = applied as independent research.

These trends reflect the positioning of research methods in QAA Subject Benchmark Statements. Research methods are foregrounded in the Benchmark Statements for Arts (QAA, 2015c), Education (QAA, 2014a) and Science (QAA, 2014b). Research methods feature in the description of the discipline reflecting characteristics of the discipline; e.g. Science students are expected to engage with ‘quantitative and qualitative approaches to acquiring and interpreting data’ (QAA, 2014b, p. 7). These are reinforced by the intellectual and practical skills graduates in these disciplines are expected to demonstrate. Interestingly, the role of research and research methods in the Business Subject Benchmark is less explicit. Here, the emphasis is on students being able to ‘analyse and evaluate business data’ and ‘evidence informed decision making’ (QAA, 2015b, p. 8). These skills naturally lead towards more passive engagement with research methods as students are less likely to be involved in research design and reviewing methods of data collection, and instead working with existing data-sets. This is evidenced through the extent to which many key aspects of research methods are engaged with through structured, lecturer-led activities and the limited application of research methods (Table 7). This contrasts the student experience in Arts & Education and Science where there is an overall move towards the application of research methods. This may explain why the Business students report a reduction in their perception of ‘studying and developing research as essential’ (Table 4), as the absence of active engagement with research methods means it remains an abstract skill. The absence of a pedagogical practice around research methods teaching in Business & Social Science disciplines may relate to a wider issue recognised by Wagner et al. (2011) regarding the absence of a pedagogical culture surrounding research methods teaching in social sciences. Collectively, these factors are likely to impact significantly on students’ motivation and willingness to engage with research methods.

Table 7. Studying or practising research methods by discipline (very often/often%) at Level 4 and 5.

	Business & Social Science		Science		Arts & Education
	Level 4%	Level 5%	Level 4%	Level 5%	Level 4%	Level 5%
Asked another student to help you understand course material	53.0	50.9	41.2	62.5	50.6	53.9
Asked a staff member to help you understand course material	53.0	46.6	41.1	37.5	50.6	66.7
During research training, connected ideas to your prior knowledge and experiences	48.3	50.0	77.4	65.6	57.9	48.8
Combined ideas from different modules when engaged in research-related tasks	62.5	60.3	75.8	78.2	57.9	48.7
Examined the strengths and weaknesses of your own views on a topic or issue	44.6	50.9	46.9	50.0	59.8	46.1
Reached conclusions based on your own analysis of numerical information (numbers, graphs, statistics, etc.)	37.8	56.2	66.7	78.1	26.9	46.1
Evaluated what others have concluded in their research (either through critically reasoned arguments or numerical analysis)	32.2	35.7	39.4	53.2	50.6	23.1
Discussed research or your research assignments with peers outside the classroom	34.5	54.4	70.6	71.9	56.5	64.1

Notes: [Level 4: Valid n = 161–180, Level 5: Valid n = 127–129].

Students were asked to report how often they engaged with a range of learning activities (e.g. pre-course reading, accessing support from peers/lecturers and collaborative working; Table 7). Again there are disciplinary trends. For example, Science students increasingly report reaching conclusions based on their own analysis of numerical information, as do Business & Social Science students (Table 7). The extent to which there is a limited change across some of the categories may indicate the increasing sophistication of the research skills they are learning and the extent to which they are mastering these skills (e.g. limited progress is made across each disciplinary area with respect to the survey items ‘during research training, connected ideas to your prior knowledge and experiences’ and ‘combine ideas from different modules when engaged in research-related tasks’). However, this may also be influenced by the foundation on which they are building. In some areas, the Business & Social Science students appear to begin from a lower base of engagement compared to their peers in other disciplines (Table 7).

Student confidence as researchers: self-efficacy

Studies that have examined how students’ sense of self-efficacy in research methods changes identify that as students become more competent as researchers they grow in confidence (Forester et al., 2004; Shaw et al., 2013). Consequently, students were asked to rate their confidence on a five-point scale across a range of research-related activities. Mean confidence across the statements varied between 2.44–3.09 (level 4) and 2.73–3.42 (level 5). Positive mean differences were found across all areas, and the greatest confidence gains are in activities such as applying research techniques to new areas of study (+.39), organising, planning and managing a research process (+.37), correctly citing existing published research (+.36), reading and understanding research articles (+.29), developing research proposals (+.27), presenting research (+.27) and developing research proposals (+.27).

Gains in areas such as developing research proposals may reflect recent engagement with a module to prepare them for their level 6 dissertation. However, growing confidence in planning research and applying research techniques to new areas of study are notable, and resonate with the findings of Forester et al. (2004) to compare and contrast measures of undergraduate research self-efficacy. This potentially indicates the emergence of more sophisticated understandings of knowledge and research, potentially also, a sense of themselves as researchers – a move towards being ‘research prepared’. Making this transition is challenging for many undergraduates, and where this does not occur, it can lead to the persistence of negative attitudes towards research methods (Murtonen, 2015; Shaw et al., 2013). It is often dependant on how research methods are framed within a degree programme i.e. are they taught within the confines of a standalone module and not integrated or embedded across the whole curriculum (Williams et al., 2016; Table 8).

Table 8. Confidence in research methods (Level 4 to Level 5).

	Level 4 Mean Median Std. Dev. (n = 169–178)	Level 5 Mean Median Std. Dev. (n = 115–125)	Level 4 to Level 5 Mean difference
Evaluate a point of view, decision or information source	2.99	3.12	.13
	3.00	3.00
	.85	.86
Form a new idea or understanding from various pieces of information	2.94	3.11	.17
	3.00	3.00
	.93	.86
Analyse an idea, experience or line of reasoning in depth by examining its parts	2.85	2.99	.14
	3.00	3.00
	.97	.88
Read and understand research findings in academic journals	2.90	3.19	.29
	3.00	3.00
	.91	.97
Critically evaluate research articles for sound methodology, data analysis and interpretation	2.55	2.73	.18
	2.00	3.00
	.97	.94
Critically evaluate research evidence encountered in the media	2.66	2.91	.25
	3.00	3.00
	.99	.96
Apply research techniques studied in new areas of your discipline; incorporate ideas from related disciplines in your research	2.44	2.83	.39
	2.00	3.00
	.91	.94
Apply facts, theories or methods to practical problems or new situations	2.82	3.07	.25
	3.00	3.00
	.95	.99
Explain and defend your research through a presentation to a small audience	2.68	2.95	.27
	2.00	3.00
	1.09	1.12
Appropriately cite and reference other research studies in written reports and presentations	3.06	3.42	.36
	3.00	3.00
	1.09	1.03
Undertake collaborative research and work cooperatively as part of a team	3.09	3.19	.10
	3.00	3.00
	1.03	.99
Develop a research proposal for your own or your team’s research	2.87	3.14	.27
	3.00	3.00
	1.12	1.09
Organise, plan and manage a research project – time, resources, etc	2.87	3.24	.37
	3.00	3.00
	1.12	1.00
Debate whether a proposed research study is ethical or unethical	2.83	3.05	.22
	3.00	3.00
	.97	.99

Once again, there were disciplinary trends (Table 9) evident through changes in mean confidence from level 4 to 5. Skills students reported studying or practising ‘very often/often’ at level 4 (Table 7) show lower level of change in mean confidence from level 4 to 5. This is interesting; it may be anticipated that as students become familiar with research activities they experience increased confidence. However, this could also relate to the academic complexity of these skills, in that they have yet to gain a sense of ‘mastery’. Business & Social Science students report considerable gains in confidence across numerous items (e.g. +.64 ‘apply research techniques studied in new areas of your discipline’, and + .45 ‘read and understand research findings in academic journals’). As noted above, research methods appear to be less explicit in the Business curriculum, and this impacts on student engagement and confidence in these areas. The Science students are actively exposed to research methods from level 4 therefore in some areas only small gains in confidence are reported (e.g. no change in confidence ‘form a new idea or understanding from various pieces of information’) whereas again engaging activities related to developing a dissertation proposal are positively impacting their confidence (e.g.+.38 ‘develop a research proposal for your own or your team’s research’ and + .16 ‘apply facts, theories or methods to practical problems or new situations’). Science students reported negative confidence gain in ‘undertaking collaborative research and working cooperatively as a team’ (−.21) and may reflect wider students concerns about group work and summative assessment (Allan, 2017). Arts & Education show reductions in confidence in some areas (Table 9) despite reporting the application and practising of many aspects of research methods (Table 7). This perhaps indicates students previously possessing incorrect or limited understanding in items such as ‘debate whether a proposed research study is ethical’ (−.10), ‘apply facts, theories or methods to practical problems or new situations’ (−.06), ‘critically evaluate research articles for sound methodology, data analysis and interpretation’ (−.06). (Table 9). This could have previously overestimated their confidence, as once they practised and realised what these skills actually entailed they reassessed their level of confidence. This is not unexpected; partly due to the issues of ‘sophomore slump’ but also as this is a finding reported in related work on research methods education (e.g. Chamberlain et al., 2015; Sizemore & Lewandowski, 2009).

Table 9. Mean change in confidence in research methods by discipline group: Level 4 to Level 5.

	Business and Social Science			Science			Art and Education
	Level 4 Mean Median Std. Dev. (n = 65–68)	Level 5 Mean Median Std. Dev. (n = 54–57)	Mean change	Level 4 Mean Median Std. Dev. (n = 30–32)	Level 5 Mean Median Std. Dev. (n = 27–31)	Mean change	Level 4 Mean Median Std. Dev. (n = 72–78)	Level 5 Mean Median Std. Dev. (n = 33–37)	Mean change
Evaluate a point of view, decision or information source	3.09	3.16	.07	3.06	3.13	.07	2.88	3.05	.17
	3.00	3.00		3.00	3.00		3.00	3.00
							.81	.88
	.89	.84		.88	.90
Form a new idea or understanding from various pieces of information	3.07	3.25	.18	3.03	3.03	.00	2.79	2.97	.18
	3.00	3.00		3.00	3.00		3.00	3.00
	.97	.85		.86	.80		.92	.91
Analyse an idea, experience, or line of reasoning in depth by examining its parts	2.93	3.07	.14	2.94	3.10	.16	2.74	2.78	.04
	3.00	3.00		3.00	3.00		3.00	3.00
	.99	.91		.88	.79		.97	.89
Read and understand research findings in academic journals	2.62	3.07	.45	3.22	3.55	.33	3.03	3.06	.03
	3.00	3.00		3.00	4.00		3.00	3.00
	.88	.92		.94	1.03		.86	.96
Critically evaluate research articles for sound methodology, data analysis and interpretation	2.35	2.77	.42	3.00	2.94	−.06	2.53	2.47	−.06
	2.00	3.00		3.00	3.00		2.00	2.50
	.89	.91		.95	1.00		.99	.90
Critically evaluate research evidence encountered in the media	2.62	3.07	.45	2.90	2.90	.00	2.60	2.67	.07
	3.00	3.00		3.00	3.00		2.00	3.00
	.98	.88		.96	1.03		1.02	.99
Apply research techniques studied in new areas of your discipline; incorporate ideas from related disciplines in your research	2.31	2.95	.64	2.84	2.81	−.03	2.38	2.66	.28
							2.00	3.00
	2.00	3.00		3.00	2.00		.89	.91
	.87	1.03		.93	.98
Apply facts, theories or methods to practical problems or new situations	2.82	3.28	.46	2.97	3.13	.16	2.75	2.69	−.06
	3.00	3.00		3.00	3.00		3.00	3.00
	1.08	.94		.97	1.06		.82	.92
Explain and defend your research through a presentation to a small audience	2.76	3.27	.51	2.68	2.80	.12	2.60	2.58	−.02
	3.00	3.00		3.00	3.00		2.00	2.50
	1.14	1.10		1.11	1.10		1.06	1.05
Appropriately cite and reference other research studies in written reports and presentations	2.88	3.25	.37	3.63	3.74	.11	2.97	3.43	.46
	3.00	3.00		4.00	4.00		3.00	3.00
	1.23	.93		.79	1.00		1.00	1.14
Undertake collaborative research and work cooperatively as part of a team	3.07	3.21	.14	3.50	3.29	−.21	2.93	3.06	.13
	3.00	3.00		3.50	4.00		3.00	3.00
	1.05	.91		.98	1.01		.99	1.13
Develop a research proposal for your own or your team’s research	2.97	3.16	.19	2.97	3.35	.38	2.40	2.92	.52
	3.00	3.00		3.00	3.00		2.00	3.00
	1.17	1.10		1.17	1.05		1.00	1.11
Organise, plan and manage a research project – time, resources, etc.	3.17	3.27	.10	3.17	3.35	.18	2.77	3.11	.34
	3.00	3.00		3.00	3.00		2.00	3.00
	1.15	.92		1.15	1.05		1.11	1.09
Debate whether a proposed research study is ethical or unethical	2.91	3.24	.33	2.91	3.07	.16	2.83	2.73	−.10
	3.00	3.00		3.00	3.00		3.00	3.00
							1.08	1.01
	1.28	.88		1.28	1.14

Measures of self-efficacy are valuable in directing future learning and encouraging goal setting, hence why many studies centred on motivations and attitudes to research methods examine it (e.g. Shaw et al., 2013). Students that exhibit high levels of self-efficacy are usually self-regulated learners who are good at identifying future learning needs (Zimmerman et al., 1992). However, not all students, particularly those with diverse entry profiles, have the expertise or capacity to identify their learning goals (Hsieh, Sullivan, & Guerra, 2007). Encouraging students to assess their confidence or expertise may highlight the focus of future learning, and within a supportive or directive framework, may motivate students to achieve (Hsieh et al., 2007).

Conclusions

In this paper, we report findings of a HEFCE Learning Gain pilot to capture data on undergraduates’ research methods education to develop a measure of learning gain. Given the challenges students face with research methods, it may not appear an obvious area on which to develop a learning gain measure. Indeed, whilst we do identify positive learning gains, these need to be considered in relation to disciplinary framing and pedagogic practice underpinning research methods education. Evidence from our data demonstrates that there are clear trends towards disciplinary differences in the extent to which students are making judgements on the validity of existing knowledge, engaging in the generation of new knowledge through the application of research methods. However, as is increasingly been reported, learning gain is a complex and context-dependant construct (Kandiko Howson, 2016). Therefore, it is likely that any resulting measure will need to pay heed to student profile, disciplinary practice and institutional focus.

Our learning gain study is founded on the concept of self-efficacy, which is increasingly been applied in learning gain studies. Lim et al. (2012) identified value in students’ self-assessing through a measure of learning gain, as it can encourage students to connect with the material being ‘measured’. This can have two benefits, as well as overcoming some of the limitations associated with other learning gain measures (e.g. lack of student engagement with standardised tests). Firstly, if the measure connects with a student’s future learning (i.e. they gain feedback on current performance and receive guidance on how to direct their future learning to maximise performance through, for example, targeted resources, recommendations on learning strategies or readings), they should be able to set future learning goals (Hsieh et al., 2007). Such an approach resonates with work from Arico (2016), who explores how in-class self-assessment and reflection activities can, over the course of a module, promote and capture learning gain. It would also provide a measure of learning gain that goes beyond institutional accountability and teaching enhancement, to consider the process of student learning itself (Varsavsky et al., 2014). For this to be successful, it requires explicit integration of a learning gain measure into the curriculum and for students to engage in a dialogue around their growing knowledge, skills and experience in the area the measure is centred, which in this case would be research methods. The extent to which this self-assessment, reflection and dialogue needs scaffolding requires consideration, particularly for those in environments where research may not be an explicit feature of institutional practice (e.g. in college-based HE). However, this does create opportunities for relationships to be fostered to promote learning. Recent work (e.g. HEA, 2011; Hagenauer & Volet, 2014) identified the integral role of pedagogic relationships, (between and across students and educators) as creating safe spaces that encourage risk taking and experimentation, and in the long-term student engagement and learning. Good pedagogical relationships are identified by Wood and Su (2017) as a sign of ‘excellence’. Here we propose, therefore, that self-assessment, reflection and dialogue around research methods education could foster relationships that scaffold and capture learning.

Additionally, encouraging students to self-assess, reflect and discuss their current/future learning of research methods would highlight the role of research for disciplinary and academic development. This may overcome the basic challenges faced by many research methods educators of students struggling to perceive the relevance of the subject, or failing to recognise the progress they are making (e.g. Murtonen, 2015; Shaw et al., 2013). Also, participating in a dialogue around their engagement with, and application of, research methods within their studies may better prepare students for applying such expertise in their professional lives. This could also work towards addressing the concerns of research methods educators, employers and policy-makers regarding undergraduate preparation for working within the knowledge economy (Nind et al., 2015). Indeed, though our findings demonstrate that disciplinary groups may differentiate the focus of the research methods content and learning, there is a commonality of purpose in underlying skills across disciplines. A more explicit examination of the applicability of skills in employment/vocational contexts would be a welcome addition to the curriculum.

Our approach would allow programme teams and institutions to build up longitudinal data relating to the learning gain students make in research methods. Indeed, these are the premises on which the Toolkit are based, and form the intended outcome of this project. In developing a measure of learning gain centred on research methods, we intend to create a ‘learning toolkit’ which allows students, with the support of research methods educators, to self-assess, reflect on and direct their future learning. Though the focus here is upon research methods, given that many HE providers are integrating research-based pedagogies across the curriculum (Earley, 2014; Healey, Jenkins, & Lea, 2014), which may sit alongside or be integrated into research methods provision, it may prompt wider reflections on learning and development. Self-assessment of current levels of knowledge, skills and experiences in research methods, reflection on the outcomes with a tutor, signposting to relevant resources and consideration of methods of learning/pedagogic engagement, to stimulate future learning gain is the basis of this resource and will be the focus of the ongoing work.

The need for institutions to evidence students’ skills acquisition for employers is a significant factor underpinning the development of measures of learning gain (McGrath et al., 2015). However, in developing such measures, institutions and programme teams should be cautious. It is unlikely that there will be a single solution, and institutions will need to adapt and contextualise any learning gain measure they employ. This is evident in the presentation of data obtained in our project. The differential framing of research methods in disciplines shapes students’ pedagogic engagement with research methods, which equally impacts on their research orientations and students’ sense of self-efficacy. This is before factors such as entry profile are controlled for. Overlooking the significance of context, and also student need, could undermine the potential of a learning gain measure to promote actual student learning. To some, student learning has been overtaken by a discourse of excellence and accountability. Maintaining the central role of student learning in studies of learning gain is essential and will support to development and application of measures that have credibility across the sector.

Funding

This work was supported by the Higher Education Funding Council for England to pilot and evaluate measures of learning gain [grant number G0177/IPLY02/LG1516].

Disclosure statement

No potential conflict of interest was reported by the authors.