The influence of curriculum on master’s students’ perceived abilities in four domains of graduateness

Abstract

Potentially all university graduates, regardless of the discipline they have studied, are expected to have obtained generic learning outcomes, which we refer to as ‘graduateness’. This study investigates the extent to which learning programmes’ emphasis on graduateness affects students’ perceived abilities in the domains of graduateness. Four domains of graduateness are considered: reflective thinking, scholarship, moral citizenship and lifelong learning. Based on curriculum maps, master’s programmes were clustered according to the emphasis placed on each domain. Unexpectedly, there appeared to be no difference in students’ perceived competence in the four domains of graduateness between master’s programmes that placed little to no emphasis on reflective thinking, moral citizenship or lifelong learning and master’s programmes that placed more emphasis on these domains. Only in the scholarship domain was a difference found in students’ perceived competence; surprisingly, it was in the opposite direction. In conclusion, we can say that the relation between emphasis on the domains of graduateness and students’ perceived abilities in these domains were not found across a large sample of study programmes.

Keywords:

• Graduateness
• curriculum
• reflective thinking
• scholarship
• moral citizenship
• lifelong learning

Introduction

The importance of explicating graduateness has been well established in recent years. Whereas graduateness used to be considered an attribute that students acquired while they studied their disciplines, there is now increased awareness of the university’s role in stimulating students’ development of generic graduate attributes (Barrie 2005; Biggs 1999; Booth, McLean, and Walker 2009; Hughes and Barrie 2010; Vaatstra and de Vries 2007). In this article, we use the terms ‘graduateness’ and ‘generic graduate attributes’ to refer to students’ academic intellectual development. Specifically, graduateness refers to the stage in students’ development at which they have acquired a certain set of generic graduate attributes. It is beyond the scope of the present article to discuss the graduateness stage in detail (see Steur, Jansen, and Hofman [2012] for an elaboration of our theoretical model of graduateness). Our model distinguishes four domains of graduateness: reflective thinking, scholarship, moral citizenship and lifelong learning. In this article, we investigate whether students in master’s programmes that place little to no emphasis on a specific domain of graduateness report lower abilities in this domain than students in master’s programmes that place more emphasis on that particular domain.

All university students are expected to have attained graduateness by the time they graduate, regardless of the discipline they have studied (Bridges 1993; Glover, Law, and Youngman 2002; Yorke and Harvey 2005). However, Booth, McLean and Walker (2009) previously found that an emphasis on employability leads to lower reported academic intellectual development among students, indicating that graduation does not per se convey graduateness. In addition, in Steur, Jansen and Hofman (2016), we found that although most students have attained graduateness by the time they graduate, a substantial group (approximately 30% of the population) does not. This raises the question whether differences in study programmes explain differences between students’ graduateness. We argue that the attention paid to domains of graduateness influences students’ reported abilities in those domains. To investigate this hypothesis, we conducted our study across different disciplines within a research-intensive university. The broad range of disciplines considered in this study – business and economics, spatial sciences and behavioural and social sciences – allowed us to draw conclusions that apply to a variety of disciplines. This university has committed itself to the Bologna process but has no explicit policy concerning graduateness as we understand the term.

In this study, we use a general model of graduateness that we previously developed (Steur, Jansen, and Hofman 2012) to provide a theoretical framework for graduateness that is embedded in existing research traditions. We use this model in the current study to describe both curriculum characteristics and students’ learning outcomes in terms of graduateness domains. This model can be applied to different disciplines. The following section describes the model and the research questions. The third section explains the research method and instruments, and the results are presented in section 4. In the final section, conclusions are drawn and the results are discussed.

Theory

Academic intellectual development: graduateness

There is a widely shared notion that university education provides more to students than discipline-specific knowledge and skills (Barrie 2005; Bridges 1993; Glover, Law, and Youngman 2002; Yorke and Harvey 2005). This is sometimes referred to as the formative function of university education (UNESCO 1998) and is in addition to the other functions of universities; namely, research and the professional preparation of students. In The Netherlands, the formative function is addressed by the Dutch Higher Education and Research Act (2010). Nevertheless, there are almost as many interpretations of graduateness as there are lecturers (Barrie 2004; Holmes 2013). Moreover, existing studies apparently fail to establish the level at which graduates should function to claim graduateness (Davies and Hogarth 2010). Therefore, we must elaborate our understanding of graduateness. We take the formative function of university education as a starting point.

The formative function of university education refers to an intellectual transformation in students (e.g., Jansen 2009; Perry 1970; Stevenson 2003; Van Rossum and Hamer 2010). This is the essence of our model of graduateness (Steur, Jansen, and Hofman 2012). This transformation implies that freshmen and graduates are at different stages of intellectual development (Perry 1970; Van Rossum and Hamer 2010). According to this view, graduateness is the specific stage at which students have acquired an exclusive set of complex generic graduate attributes that are consistent with what Barrie (2004) identifies as enabling. From the enabling perspective, the acquired knowledge and skills enable students to develop new knowledge and to shape their own personal development. Graduates must function at this level to become experienced professionals (Jansen 2009; Schön 1983) or, in the words of Schön, to become reflective practitioners. The development of professional skills is insufficient to acquire graduateness. Moreover, there are indications that universities’ inclusion of professional preparation in their curricula has occurred at the expense of intellectual development (Booth, McLean, and Walker 2009; Glover, Law, and Youngman 2002). Therefore, we explicitly exclude professional skills from our model.

Four domains of graduateness are defined in our model: reflective thinking, scholarship, moral citizenship and lifelong learning. Reflective thinking refers to higher-order thinking that enables one to reflect on complex and/or unfamiliar situations as well as on one’s own judgments of these situations. At this stage, students are aware of the relativity of knowledge (Perry 1970) and form judgments by translating and integrating information from different angles (King and Kitchener 2004). The second domain, scholarship, refers not only to the mastery of research skills but also to a scholarly attitude. Whereas the concrete research skills that are required for various disciplines are likely to differ (for example, the skills required for research in the natural sciences differ from the skills required for research in the social sciences), scholarship refers to the development of a scholarly stance towards the world. The domain of moral citizenship combines an awareness of social responsibility (Nussbaum 1997) with moral development (Kohlberg 1973). Finally, the domain of lifelong learning comprises an attitude and a set of learning skills that allow a student to learn effectively during the university years and beyond. Lifelong learning enables graduates to respond adequately to future situations that require knowledge and skills that are yet to be acquired by the graduate.

In conclusion, we consider graduateness to be a specific stage in a student’s intellectual development that is most likely achieved by the time the student graduates. However, this timing is not ironclad; a portion of students might not achieve graduateness by graduation (Van Rossum and Hamer 2010). We are interested in whether differences in reported abilities in the graduateness domains can be explained by differences in the emphasis on graduateness in study programmes. This will provide guidance for curriculum reforms related to graduateness.

Graduateness and the curriculum

There has been some previous discussion of the influence of curriculum on student learning outcomes (e.g., Biggs 1999; Lizzio, Wilson, and Simons 2002). Certain students learn what they are supposed to learn regardless of the learning environment, whereas other students need effective teaching to reach the same level of understanding (Biggs 1999). According to Biggs (1999), the lecturer can influence the manner in which students process subject matter. Thus, the approach to learning influences the student’s level of understanding (Biggs 1999; Marton and Saljo 2005). In this paper, we focus on the attention paid to graduateness in the curriculum because for most students, it is difficult to acquire knowledge, skills and an appropriate attitude if the curriculum does not support this process (Biggs 1999).

There are different means by which a curriculum can be supportive. For example, Lizzio, Wilson and Simons (2002) showed that good teaching and independence positively influence both academic achievements and aspects of graduateness. Vaatstra and de Vries (2007) showed that an active learning environment has a positive effect not only on graduates’ generic competences, such as planning and working in teams, but also on more complex competences, such as problem solving and reflective thinking. Both of these studies focus on how a subject is taught as opposed to what is taught. With respect to curriculum content, the findings of Visser-Wijnveen (2009) showed that learning contexts with a strong emphasis on scholarship stimulate students to develop an academic attitude. In addition, Robley, Whittle and Murdoch-Eaton (2005) showed that students reported learning in the domains of graduateness that were included in a curriculum. We propose that to develop a better understanding of why, upon graduation, some students are further along than others in terms of academic intellectual development, it is necessary to take a closer look at the content of study programmes, particularly with respect to the emphasis on graduateness domains.

The emphasis on graduateness domains in the curriculum is important because students are more likely to achieve learning objectives that are deemed important by lecturers (e.g., Biggs 1999). Given the diversity of interpretations of graduateness, it is likely that study programmes also differ in terms of the emphasis placed on graduateness. Both Barrie (2005) and Van Rossum and Hamer (2010) stress that lecturers’ interpretations of graduateness influence student learning outcomes with regard to the domains of graduateness. For example, lecturers who believe that generic graduate attributes should be developed in a separate curriculum – the compulsory perspective of Barrie’s study (2004) – are less likely to contribute to students’ academic intellectual development towards graduateness (the enabling perspective). We assume that lecturers who expressly identify a specific graduateness domain as an important element of their respective courses will incorporate this domain into their lectures.

Research questions

In this study, we consider both the emphasis in university curricula on all four graduateness domains and students’ perceived abilities in each of these domains to establish whether there is a relationship between the two. Thus, the following research questions are addressed:

(1)	Do study programmes from different disciplines address the domains of graduateness to the same extent? Given the absence of a university policy on graduateness that could enhance lecturers’ perspectives on graduateness, we expect that the curricula of study programmes differ in terms of the emphasis placed on domains of graduateness.
(2)	Do students of different faculties differ from each other with regard to reported abilities in domains of graduateness? We expect no particular differences between the reported abilities of students across different faculties. Emphasis on specific domains of graduateness is not restricted to certain faculties; as stated above, there is no particular university policy on graduateness.
(3)	Do students who study in programmes that place a greater emphasis on a particular domain of graduateness report higher abilities in that domain compared with students in programmes that place less emphasis on that domain? We expect that a greater emphasis on a particular domain of graduateness results in students reporting higher abilities in that domain.

Methods

Data for this study were collected from master’s programmes within three faculties of a Dutch research-intensive university. These faculties offer both bachelor’s and master’s programmes in the fields of behavioural and social sciences, business and economics, and spatial sciences. There is no explicit university policy regarding graduateness, which means that all departments are free to emphasise the learning outcomes that they deem important. The lack of a university-wide policy on graduateness also means that there is no common frame of reference. The studied faculties differ in terms of size and history. One faculty went through a reorganisation in which two faculties were combined into one. Another faculty is increasing in size from small to medium. All faculties have a relatively large proportion of students who received their bachelor’s degrees from institutions that provide higher professional education, as opposed to university bachelor education.

We chose to study master’s programmes for two reasons. First, because our interpretation of graduateness refers to a transformation in students, we wanted to optimise the likelihood that a substantial group of students have experienced this transformation, recognising that not all students will have completed this transformation. The second reason is of a more practical nature; that is, the duration of master’s programmes in The Netherlands allowed us to study programmes of minimum length. The longer the study programme, the more opportunities students have to engage in extra-curricular activities that could contribute to their academic intellectual development. In one-year master’s programmes, these opportunities are limited, making it more likely that the findings can be attributed to the curricula.

The study comprises two parts. First, to establish the emphasis placed by master’s programmes on domains of graduateness, curriculum maps were drawn based on lecturers’ course descriptions. These short descriptions are prepared by lecturers to apprise students of the learning objectives of the course units. Although lecturers are not explicitly requested to address learning objectives related to graduateness, we were confident that lecturers address these objectives in their course descriptions when the objectives are deemed to be important aspects of the course units. Second, students’ perceived abilities in graduateness domains were established by an online questionnaire; this questionnaire was administered prior to the study for quality assurance purposes. This questionnaire contained items related to all four domains of graduateness. By using these data, we were able to include a large number of disciplines in our study. In contrast, most other studies on graduateness were conducted either within a single discipline or across a limited number of disciplines (e.g., Van Rossum and Hamer 2010). Further details regarding the research method are provided in the following sections.

Analyses

The data in this study were analysed using text analysis (curriculum maps) to determine which aspects of graduateness domains are emphasised. This analysis yielded a set of variables at the study programme level, and these data were combined with the student-level data. We used ANOVAs (Analysis of Variance) to determine differences between groups, both with respect to faculties and to clusters of study programmes. This technique was also used to check for differences across faculties related to each domain, both at the curriculum level and at the student level.

Curriculum mapping

Curriculum maps were drawn for 12 departments representing 12 different disciplines across the three faculties. Certain departments offered multiple master’s programmes for various specialisations within a particular discipline. In this study, each of these specialisations is considered a separate study programme, resulting in 28 separate study programmes. The study programmes comprise compulsory and elective courses. We considered only the compulsory course units because these units reflect the extent to which the academic development of students is embedded in the curriculum. Including the electives in this study would have misrepresented the course programmes because of the effect that a single elective chosen by only one student would have had on the curriculum map of the entire study programme. On average, 50 European Credit Transfer System (ECTS) credits per 60-ECTS-credit master’s programme were considered in the curriculum maps. To facilitate comparisons between master’s programmes, the scores were corrected to account for the number of ECTS credits that were considered.

To map the curriculums, course descriptions were coded for the four domains of graduateness. The coding procedure was performed using ATLAS.ti 5. First, three master’s programmes were coded to develop a list by which all master’s programmes could be coded. Certain codes in the final list (Table 1) were similar to the generic skills items in the questionnaire. Second, a random selection of five course units was coded by multiple raters to prevent single-rater biases in the coding. The interrater reliability for these courses was .58, which can be considered a moderate reliability (Landis and Koch 1977).

Table 1. Code list used in curriculum mapping.

Category	Codes in mapping
Reflective thinking	Ability to integrate
	Abstract/analytical thinking
	Academic thinking
	Academic attitude
	Connect subject matters
	Critical thinking
	Critical attitude
	Develop new ideas
	Identify problems
	Reflection skills
Scholarship	Apply knowledge, methods and concepts
	Contribute to the development of the discipline
	Design a research proposal
	Design a (theoretical) model
	Design skills
	Design theories
	Formulate clear hypotheses or research questions
	Logically connect different theories and/or research fields with each other
	Methodology skills
	Model development
	Perform research
	Research skills
	Understand new scientific developments and technologies
	Understand research in its appropriate context
	Understand theories
	Write a scientific paper
Moral citizenship	Anticipate changes
	Connect subject matter with current affairs
	Debate skills
	Explain discipline to laymen
	Formulate a clear opinion on important issues within discipline
Lifelong learning	Address incomplete information
	Information processing skills
	Gaining insight
	Gaining knowledge
	Information and communications technology skills
	Initiative and entrepreneurship
	Plan one’s own activities
	Select scientific literature and evaluate it critically

The final coding list was used to code all 28 master’s programmes. The list appeared to be appropriate for all course units, although it was difficult to assign a degree of importance to text fragments; therefore, we decided to count whether a code appeared in the course description. This approach means that the number of codes per domain represents the maximum score per course unit in that domain. For example, there are 10 codes in the reflective thinking domain, which means that this domain has a maximum score of 10 per course unit, which is obtained when all codes are present in a course description. The scores are corrected for the relative weight of the course unit, which is based on the total number of ECTS credits that are included in that programme. For example, if a five-ECTS-credit course unit includes ‘understand theories’, ‘perform research’ and ‘design theories’ in its curriculum map, and this master’s programme comprises 50 ECTS credits of compulsory course units, the score for this course unit in the scholarship domain is 3*(5/50) = 0.30. The domain score for a study programme is calculated by adding the scores for all course units in that domain (see Table 2 for an overview of the programmes’ scores). These scores represent the relative importance of the domain in the compulsory section of the master’s programmes. To provide a better understanding of the coding process, some typical text fragments are presented here (the P-numbers serve as identification numbers for the study programmes):

This knowledge forms the building blocks for developing a critical stance towards the relevant literature. (P14: coded as select relevant literature and review it critically)

In the literature and the lectures, a number of modern integrative perspectives and themes are considered. (P20: coded as understand new scientific developments and technologies)

… developing a critical attitude towards theoretical and clinical usability of [cognitive models of psychopathology]. (P21: coded as critical attitude)

Table 2. Curriculum mapping scores for graduateness domains.

Programme^*	Reflective thinking		Scholarship		Moral citizenship		Lifelong learning		N^**
	Score	Group	Score	Group	Score	Group	Score	Group
F1D1	2,03	+	1,06	-	0,12	0	0,26	-	14
F1D2	0	-	1,22	-	0	-	0,2	-	25
F1D3	0,5	0	1,3	0	0	-	1	+	13
F1D4	0,5	0	0,88	-	0,08	0	0,42	-	17
F1D5a	1	0	1,8	0	0,4	+	1,2	+	37
F1D5b	0,71	0	0,71	-	0,14	0	0,43	-	18
F1D5c	0,33	-	2,33	+	0	-	1,17	+	7
F1D5d	0	-	1,67	0	0	-	1,17	+	9
Mean F1 (SD)	0,63 (.655)		1,37 (.533)		0,09 (.137)		0,73 (.442)
F2D1a	0,42	0	1,5	0	0	-	0.67	0	1
F2D1b	0,33	-	1,33	0	0,08	0	0,42	-	6
F2D2a	1	0	2,67	+	0,17	0	1	+	4
F2D2b	0,25	-	1,5	0	0	-	0,75	0	3
F2D2c	0,6	0	1,8	0	0,2	0	0,7	0	4
F2D2d	0,55	0	1,91	0	0,36	+	0,73	0	5
F2D2e	0,2	-	1,4	0	0,2	0	0,6	0	5
F2D2f	0,2	-	1,8	0	0,4	+	0,8	0	1
F2D2 g	0,2	-	2	0	0,2	0	0,8	0	1
F2D2 h	0,11	-	1,67	0	0,11	0	0,44	-	2
F2D2i	0,5	0	1,67	0	0,17	0	0,67	0	4
F2D2j	0	-	1,6	0	0,2	0	0,8	0	3
F2D2 k	0	-	1,67	0	0	-	0,44	-	0
F2D3	0,83	0	1,67	0	0,17	0	1	+	13
F2D4	0,57	0	2,71	+	0,71	+	1	+	6
Mean F2 (SD)	0,44 (.282)		1,80 (.420)		0,21 (.183)		0,74 (.183)
F3D1	1,4	+	3,3	+	0,3	+	0,7	0	10
F3D2	1,35	+	2,47	+	0,32	+	0,6	0	6
F3D3	1,5	+	2,56	+	0,06	0	1,32	+	2
F3D4	1,48	+	2,49	+	0,16	0	0,67	0	7
F3D5	0,78	0	3,78	+	0,22	0	0,89	0	9
Mean F3 (SD)	1,30 (.298)		2,92 (.591)		0,21 (.105)		0,84 (.290)

^* F = faculty and D = department within the faculty; additional letters (a–k) refer to specialisations within departments.

^** N refers to the number of respondents refers to the questionnaire data; + = highest scoring group; 0 = mid-scoring group; - = lowest scoring group.

Questionnaire

Perceived abilities in the domains of graduateness were also considered in this study. Vaatstra and de Vries (2007) showed that perceived ability could be used as a proxy for competence in domains of graduateness. The questionnaire items covered reflective thinking, scholarship, moral citizenship and lifelong learning. Using questionnaire data allowed us to collect data from a broad selection of students, which was necessary to compare a relatively large number of programmes. Furthermore, these data allowed us to include a large number of disciplines, whereas only four study programmes in two disciplines were included in our original study. This difference between the studies required some adjustments to operationalisation, because none of the instruments that we used in the earlier study to operationalise the theoretical model were present in this dataset. Nevertheless, we argue that all of the concepts could be operationalised in the current dataset; although it is not optimal, this operationalisation is sufficient. When we tested the model on the data used in the current study, we found an adequate fit of the model to the data (Standardised RMR = .094; RMSEA = .065; GFI = .87), and the coefficients were similar to the original model (Steur, Jansen, and Hofman 2012) for which the instruments were selected based on the theoretical model of graduateness.

To assess reflective thinking, students’ perceived abilities to manage knowledge and information were measured using items from the ASSIST (Approaches and Study Skills Inventory for Students) questionnaire (Entwistle, McCune, and Tait 1997). A typical item from this questionnaire is: ‘Ideas in course books or articles often set me off on long chains of thought of my own.’ For scholarship, students’ perceived ability to, for example, ‘contribute to the development of the discipline’ are considered. Four items were used to measure moral citizenship, one of which asked students to rate their perceived ability to ‘formulate a clear opinion on important issues within the discipline’. Finally, lifelong learning, which relates to students’ abilities to use the appropriate set of skills to facilitate learning, is measured in terms of the students’ learning goal orientation (Button, Mathieu, and Zajac 1996). A typical item for this domain is: ‘When I have difficulty solving a problem, I enjoy trying different approaches to see which one will work.’ Table 3 presents the descriptive statistics for these instruments. Based on the recommendation of Nunnally and Bernstein (1994) that instruments used in basic research should have reliability coefficients of approximately .70 or better, it appears that all instruments used in this research have an appropriate internal consistency (using Cronbach’s alpha).

Table 3. Descriptive statistics for scales per domain of graduateness.

Scale	N	M	SD	Cronbach’s alpha	Number of items
Reflective thinking	253	3.12^1	.421	.749	7
Scholarship	246	6.95^2	.974	.786	5
Moral citizenship	244	6.90^2	.962	.761	4
Lifelong learning	253	3.23^1	.430	.820	8

¹ Scale ranges = 1–4.

² Scale ranges = 1–10.

Results

Graduateness: emphasis in the curriculum

Table 2 provides an overview of how the programmes scored in each domain of graduateness. The number of respondents across study programmes made it necessary to cluster study the programmes (only two study programmes had more than 25 students complete the questionnaire). In each domain, the programmes were grouped into three categories (low, middle and high) based on their relative scores in the curriculum maps (given the number of questionnaire respondents, three was the maximum number of groups). The groups were determined based on changes in slope when the programmes were ranked from highest to lowest. The cut-off criteria are presented in Table 4, together with the other descriptive statistics for the curriculum maps. (With this categorisation, the group sizes for each domain range from 54 to 97 respondents.)

Table 4. Descriptive statistics study programmes per domains of graduateness.

	Descriptives			Cut-offs groups
	MIN	MAX	M	SD	Low	High
Reflective	.00	2.03	.62	.534	<.334	>1.344
Scholarship	.71	3.78	1.87	.701	<1.224	>2.334
Moral citizenship	.00	.71	.17	.163	<0.1	>.294
Lifelong learning	.20	1.32	.74	.290	<.444	>.994

The faculties appear to differ with respect to the emphasis placed on all four domains of graduateness. An ANOVA analysis of the data presented in Table 2 reveals that Faculty 3 differs from the other two faculties in the domains of reflective thinking (F = 7.944; p < .002) and scholarship (F = 15.949; p < .000). Both domains are emphasised more by Faculty 3 than by the other two faculties; the other two faculties do not differ from each other. The data also show that scholarship is the domain most prominently represented in the master’s programmes, whereas moral citizenship receives the least emphasis; in 25% of the programmes, there is no reference to moral citizenship. In addition, several master’s programmes make no mention of reflective thinking.

Graduateness: students’ reported abilities

First, differences between students from each of the three faculties are investigated for each domain by means of ANOVA analysis. The only difference found between the students of the three faculties is in the scholarship domain (F = 19,199; p < .000), where students in Faculty 3 report lower competences than students in the other faculties (Table 5). The means in all domains are located at the positive end of the scale, indicating that, on average, students report that they have acquired these generic graduate attributes.

Table 5. Means (SD) of students’ perceived competence in domains of graduateness.

Domain	Faculty 1	Faculty 2	Faculty 3	Total
Reflective thinking	3.14 (.445)	3.08 (.415)	3.13 (.345)	3.13 (.423)
Scholarship	7.07 (.856)	7.11 (.717)	6.04^* (1.310)	6.92 (.978)
Moral citizenship	6.81 (.981)	7.19 (.733)	6.67 (1.270)	6.89 (.986)
Lifelong learning	3.24 (.405)	3.26 (.461)	3.14 (.474)	3.23 (.430)

^* Significant at p < .000.

Finally, ANOVAs were performed to establish whether students in departments with curricula that place more emphasis on domains of graduateness report higher acquired abilities in these domains. Each domain is analysed separately. The results are presented in Table 6. A significant difference between the group with the lowest emphasis and the group with the greatest emphasis is found only for the scholarship domain. In this domain, students in departments whose curricula place the least emphasis on domains of graduateness report greater abilities (see Table 7).

Table 6. Results ANOVA for students’ perceived ability and programme clusters for domains of graduateness.

Domain		Sum of Squares	df	Mean Square	F	Sig.
Reflective thinking	Between groups	.705	2	.353	1.974	.141
	Within groups	39.665	222	.179
	Total	40.371	224
Scholarship	Between groups	7.013	2	3.506	3.664	.027^*
	Within groups	206.709	216	.957
	Total	213.722	218
Moral citizenship	Between groups	1.629	2	.815	.827	.439
	Within groups	210.855	214	.985
	Total	212.484	216
Lifelong learning	Between groups	.817	2	.408	2.216	.111
	Within groups	40.902	222	.184
	Total	41.719	224

^* Significant at p < .05.

Table 7. Means for students’ perceived ability in domains of graduateness for each cluster of programmes.

Domain	Means			N
	Low	Middle	High
Reflective thinking^1	3.13	3.05	3.17	225
Scholarship^2*	7.10	7.06	6.72	219
Moral citizenship^2	6.80	7.03	6.88	217
Lifelong learning^1	3.19	3.18	3.31	225

^* Difference is significant.

¹ Scale = 1–4.

² Scale = 1–10.

Conclusion and discussion

The purpose of this study is to contribute to a better understanding of factors that influence the development of graduateness in university education by investigating the relationship between the emphasis on graduateness and students’ perceived competences in four domains of graduateness. We aspire to contribute to the body of knowledge on graduateness by investigating this relationship across different disciplines. Most existing studies draw conclusions based on a limited number of course units or study programmes (e.g., Van Rossum and Hamer 2010), which limits the possibility of drawing more general conclusions because opportunities to compare different course units or programmes are limited or even absent. By using questionnaire data as a proxy for levels of graduateness in students, we were able to investigate 28 master’s programmes across three different faculties within a research-intensive university. The breadth of our study allowed us to obtain a more robust sample of study programmes to investigate whether the relationship between specific course characteristics and students’ perceived abilities in graduateness holds across a broad range of disciplines.

Some authors dispute the notion of generic graduateness; they argue that the study of graduateness should be discipline specific because the aspects of graduateness are interpreted differently within each discipline (Coetzee 2014; Jones 2013). Nevertheless, the theoretical model we use for graduateness appears to fit the various disciplines included in our study. There were no indications in either the curriculum maps or the questionnaire data that our model articulates an interpretation of graduateness that is limited to a specific discipline, and there was no study programme for which none of the domains could be measured to some extent.

Our first research question concerns whether study programmes within different faculties differ from each other with respect to the emphasis placed on each domain of graduateness. Our study indicates that study programmes differ with respect to the attention paid to the four domains of graduateness; we even found differences between study programmes within the same department. Some departments had specialisations both in the group that places a low level of attention on a particular domain of graduateness and in the group that places a high level of attention on the same domain. The specialisations of Department 5 in Faculty 1 demonstrate this phenomenon; it also occurs in other disciplines. The study programmes within Faculty 3 are more similar, inasmuch as they place greater emphasis on the domains of reflective thinking and scholarship compared with the other faculties. Apparently, the importance of these domains is well established among the lecturers on this faculty, which has attracted an increasing number of students, including a larger proportion of students with bachelor’s degrees from institutions that place less emphasis on scholarship. The increase in number of students from higher professional education institutions enrolling in this faculty could have initiated a re-evaluation of the importance of scholarship in the master’s curricula in this faculty.

Regarding the second research question, our study indicates that students report similar perceived abilities across faculties. The only exception occurs in the domain of scholarship, where students from Faculty 3 report significantly lower abilities than students in the other faculties do. This is remarkable because the course descriptions of Faculty 3 place the greatest emphasis on scholarship. It could be that because of the emphasis placed on scholarship by this faculty, the standards for scholarship are set higher for these students, and the students reported their ability based on these higher standards.

Finally, for our third research question, we combined the data obtained from curriculum maps and students’ self-reports. We expected to find that students in study programmes that place greater emphasis on domains of graduateness report higher levels of ability in these domains. However, according to our findings, the effects of the learning environment on students’ perceived abilities in the domains of graduateness are limited. The only significant effect we found was in the domain of scholarship, and it was in an unexpected direction: students in programmes that pay less attention to scholarship reported slightly higher competences in scholarship compared with students in programmes that pay more explicit attention to this domain. This finding contradicts the findings of Visser-Wijnveen (2009) concerning the relationship between attention to scholarship and students’ scholarly attitudes; however, her study was performed in an arts faculty, which is a type of faculty not included in our study. Our results also contradict the findings of McNeil et al. (2012), who found that a curriculum developed around the domains of graduateness contributed to students’ development in these domains. We therefore encourage further investigation into the potential for stimulating students’ academic intellectual development through curricula. One important issue that must be addressed is the means by which students’ abilities can be judged by a universal standard. One proposed method to investigate this development is the use of hierarchical scaling, for example, the Mokken or Rasch model.

Obviously, this study, and therefore the results of this study, have certain limitations. First, we used student self-reports to establish learning achievements in the domains of graduateness, which has been done in many previous studies of graduateness and related notions, such as generic graduate attributes and generic skills (McNeil et al. 2012). There are known limitations to this type of data. For example, Kruger and Dunning (1999) showed that low-performing students overestimate their performance. This penchant for overestimation by low-performing students could also explain the results of this study in the scholarship domain. However, we checked the data and found no indication of structural biases in students’ self-reports across the four domains. Moreover, other studies have shown that students are capable of adequately reporting their abilities (e.g., Vaatstra and de Vries 2007), especially in domains that lecturers identify as essential (Benton, Duchon, and Pallett 2013).

Second, course descriptions were used to establish the emphasis placed by course units on domains of graduateness. These course descriptions were not written specifically for this study; consequently, the descriptions differed in terms of length and level of detail. Nonetheless, there are no indications that differences in the lengths of descriptions influenced the results. First, we scored each aspect dichotomously, so any repetition within a description did not influence the score. Second, just as there are no study programmes that received the highest scores in all domains, there are also no study programmes that received the lowest scores in all domains. The differences in detail are regarded as an expression of where the emphasis is placed in a particular course unit. We chose to use course descriptions to reveal the emphasis placed on the domains of graduateness and thereby to cluster the master’s programmes rather than a survey of lecturers. One advantage of using course descriptions is that one can assume that items included in course descriptions are valued by the lecturer. In addition, lecturers’ course descriptions were not restricted by a given list of skills or attributes or by the researchers’ interpretation of graduateness. However, the data did not allow us to assess the extent to which course descriptions accurately represent lecturers’ actual classroom practices. De la Harpe and David (2012) found that the topics and issues that lecturers claim to find important are not necessarily reflected in their lectures. This anomaly should be addressed in subsequent research.

Finally, the results are limited to one university and to a limited number of faculties within that university. Our findings might be typical for a research-intensive university. Although we found some variation across study programmes, the extent to which our samples adequately reflect the possible range of variation in each domain remains unclear. Data from different institutions with different orientations will help to put the differences found in this study into perspective. We thus encourage research lines that invest in the development of adequate proxies for graduateness based on our theoretical model. Not only will this investment enhance departments’ abilities to effectively develop curricula that stimulate students to achieve graduateness, it will also help to generate better quantitative data for quality assurance purposes. These data will in turn provide departments with guidance regarding study programme improvements.

Notes on contributors

Jessica Steur, MSc, worked as a PhD candidate at the University Centre for Teaching and Learning. She currently works as a quality assurance policy worker in the faculty of Behavioural and Social Sciences within the University of Groningen.

Ellen Jansen holds the position of associate professor in teacher education at the University of Groningen. Her expertise relates to the fields of teaching and learning, curriculum development, factors related to excellence and study success, social (policy) research and quality assurance in higher and secondary education.

Adriaan Hofman is appointed as professor of education, especially higher education, at Groningen University in the Netherlands. He specialises in school and teacher effectiveness, higher education, education in developing countries, research methods, urban education and learning cities.

Disclosure statement

No potential conflict of interest was reported by the authors.