Towards graduateness: exploring academic intellectual development in university master’s students

ABSTRACT

Our research aims to contribute to the body of knowledge on graduateness by proposing a model that explicates the expected level performance of graduates. In this study, the model is elaborated for 3 graduateness domains: reflective thinking, scholarship, and moral citizenship. We used data on students’ perceived abilities in these domains that were collected at both the beginning and end of 1-year master’s programmes in 3 faculties at a research-intensive university. The model appears to be suitable for investigating students’ academic intellectual development. Not all students appeared to achieve the expected level of graduateness by the end of the master’s programme. However, the results revealed an increase in the proportion of students meeting the thresholds for graduateness. The students’ reports reveal growth in reflective thinking and scholarship during the master’s programme.

KEYWORDS:

• Academic intellectual development
• graduateness
• reflective thinking
• scholarship
• moral citizenship

Introduction

University graduates of all disciplines are expected to have something in common as a result of their time at university (Barrie, 2005; Perry, 1970; Van Rossum & Hamer, 2010; Wheelahan, 2003). However, there is no consensus on what is expected of graduates (Barrie, 2006), at what level it is expected (Davies & Hogarth, 2002), and when it should be achieved by students (Van Rossum & Hamer, 2010). This lack of clear expectations makes it challenging to investigate whether study programmes succeed in establishing this “something” in graduates, particularly because interpretations of this “something” range from providing students with key skills (Gonzalez & Wagenaar, 2003) to supporting students’ personal (intellectual) development (Fuller, 1999; Glover, Law, & Youngman, 2002; Perry, 1970; Van Rossum & Hamer, 2010). To ensure readability in this paper, we adopt the term “graduateness” to refer to what graduates are expected to have in common. This term represents a specific stage in students’ academic intellectual development (Steur, Jansen, & Hofman, 2012). By introducing developmental theories to determine the level at which graduateness occurs and by applying this model to the development of master’s students towards graduateness, we aim to contribute to the international debate on what universities offer students to support their intellectual development.

Do all students achieve graduateness upon completion of their master’s programme? Do they show progress in their academic intellectual development during their master’s year? Alternatively, is graduateness achieved after students obtain their bachelor’s degree? Answers to these questions provide further insight into the dynamics of students’ academic intellectual development and reveal the extent to which universities succeed in addressing their formative function in a world with increasing emphasis on employability (e.g., Moreau & Leathwood, 2006). However, the lack of a shared theoretical framework and, consequently, the lack of shared terminology make it difficult for researchers and scholars to investigate graduateness (Barrie, 2006; Bennett, Dunne, & Carré, 1999; Green, Hammer, & Star, 2009). Additionally, these factors make it difficult for curriculum designers to design a curriculum that optimally supports students in their academic intellectual development towards graduateness (Hughes & Barrie, 2010). With this study, we aspire to investigate students’ growth in graduateness and its domains by applying a model of graduateness (Steur et al., 2012) that is enriched with insights from developmental theories in three graduateness domains.

Theoretical framework

Graduateness

A variety of terms is used to refer to outcomes of university education, such as key skills, transferable skills, graduateness, generic skills, (generic) graduate attributes, and (academic) intellectual development. The terminology is not solely a matter of speech; these terminologies also accentuate different functions of university education. For example, many terms emphasise the employability and professional skill development of graduates (e.g., Barrie, 2005; Bennett et al., 1999; Gonzalez & Wagenaar, 2003; Green et al., 2009; Schön, 1983; Yorke & Harvey, 2005). Certain researchers place more emphasis on domain-related elements, particularly those concerning research skills and attitudes (e.g., Van der Rijst, 2009; Visser-Wijnveen, 2009), whereas others emphasise students’ intellectual or personal development when referring to the generic learning outcomes of university education (e.g., Booth, McLean, & Walker, 2009; Glover et al., 2002; Perry, 1970; Van Rossum & Hamer, 2010). We do not claim that this is an exhaustive enumeration; rather, it illustrates the diversity in the field.

Similar terminology is sometimes used for different aspects of generic learning outcomes of university education. For example, “graduateness” (Wheelahan, 2003), our preferred term, is used to refer to both professional skill development (employability) and students’ personal intellectual development. Despite the ambiguity associated with this term, we adopted this term because we find it to be the most appropriate term to express that university education also has a formative function (e.g., UNESCO, 1998). Graduateness has the connotation of representing something generic that is developed through the university experience and is expected to be achieved by graduates. Although we acknowledge employability to be an important aspect of university education, we do not include it in our interpretation of graduateness because there are some indications that focusing on employability in university education can occur at the expense of students’ intellectual development (Booth et al., 2009; Glover et al., 2002). Hence, they are indeed two separate concepts in higher education, and they should be treated as such when investigating the generic learning outcomes of university education.

The trouble with most interpretations of graduateness is that they generally lack sufficient theoretical foundation (Holmes, 2013). The graduateness model that is proposed here attempts to overcome this flaw by basing the graduateness model on developmental theories. In our interpretation, graduateness is the product of students’ intellectual development. With other authors (e.g., Barrie, 2004; Biggs, 1999; Booth et al., 2009; Glover et al., 2002), we acknowledge that university education can provide students with experiences that enhance their higher order cognitive abilities. This developmental perspective implies a transformation in students as opposed to acquiring skills that broadens the number of tools at hand but does not necessary change the way students interpret the world (Stevenson, 2003). Moreover, it explicitly posits the value of knowledge as an inseparable element of graduateness, which provides depth for skills that otherwise would have remained superficial (Barnett, 2009; Peters, 1975; Stevenson, 2003). Approaching graduateness from this perspective distances it from unfounded lists of generic skills and attributes, for it opens up the opportunity to apply developmental theories to interpret graduateness. This is a major advantage over other interpretations of graduateness, like, for example, Barrie’s work in this field (e.g., Barrie, 2004, 2005, 2006). At the top level (enabling) of Barrie’s (e.g., 2004, 2005, 2006) hierarchy, students have transformed such that they are able to determine their positions in current (public) debates, to create new knowledge, to shape their intellectual development, and to engage in different types of reflection. However, despite providing a hierarchical model (Barrie, 2004), the developmental aspects remain underexposed in Barrie’s work (2004, 2005, 2006). For example, the manner in which students evolve from the translating level to the enabling level remains unclear. We argue that additional perspectives are needed to illuminate the transformative aspects of academic intellectual development towards graduateness.

Graduateness domains

Before we delineate the developmental aspects, we first introduce the domains of graduateness that are considered in this study. As in Peters (1975) and Barrie (2004), we consider graduateness as a multidimensional construct that encompasses interwoven domains and enables different emphases on different aspects in different disciplines. At the heart of intellectual development lies reflective thinking (Baehr, 2013; King & Kitchener, 2004; Pascarella & Terenzini, 2005; Perry, 1970; Schön, 1983; Van Rossum & Hamer, 2010). We follow this idea by situating reflective thinking at the core of graduateness, as an underlying concept to other graduateness domains. For the other domains, we want to propose domains that stay as close as possible to the current ideas in higher education. This does not mean that any domain that is covered by university education is included in the proposed model for graduateness. We already mentioned, for example, that employability, albeit an increasingly important aspect of university education, is not included in the model because findings imply that time spent on employability goes actually at the expense of intellectual development (Booth et al., 2009; Glover et al., 2002). Domains that are known for counter-interactions with intellectual development cannot be part of graduateness.

There are various domains associated with university education. Pascarella and Terenzini (2005) presented a detailed overview of potential generic yields of university education, including subject-matter competence, intellectual development, critical thinking, creativity, attitudes, and economic benefits. The extensiveness of this list illustrates that choices need to be made on domains that are covered by graduateness to prevent that it simply comprises everything a student has learned in university education. Other authors have proposed a limited number of domains that they consider to cover graduateness or intellectual development. Barrie’s (2004) model, for example, includes three domains: scholarship, lifelong learning, and global citizenship. Peters (1975) also mentioned scholarship and morality as domains where intellectual development takes place. It is probably no coincidence that these domains also refer to different perspectives on university education as it is represented by, respectively, Bildung and Liberal Education. By including reflective thinking, scholarship, and moral citizenship in a model for graduateness, a substantial range of the intellectual development of university students is covered. This perspective results in the following interpretations of the three domains of graduateness:

• Reflective thinking – refers to higher order thinking abilities, including meta-cognitive thinking.

• Scholarship – refers to a set of knowledge, skills, and attitudes that is associated with research in practice.

• Moral citizenship – refers to the idea of students as global citizens, including and highlighting students’ moral development.

Graduateness and development

Using a developmental approach, we aspire to acknowledge some of the associated complexity that cannot be addressed by listing different skills and attitudes under the flag of graduateness. Such lists lack theoretical foundation because they are often simply based on previous lists (Holmes, 2013). As a consequence, any set of skills and attributes is as good as any other. Moreover, such lists fail to address the level at which students are expected to perform (Davies & Hogarth, 2002). They often provide a general impression of the expected overall level required in specific (degree) programmes, but they offer only indirect guidelines regarding the expected level for each generic skill, attribute, or competence. Furthermore, development is not merely covered by performance at a higher level, that is, within an increasingly complex context; rather, (academic) intellectual development implies a transformation in students (Stevenson, 2003). This transformation is marked by a different way of thinking, a different way of approaching the world, and/or a different way of positioning oneself (Perry, 1970). For example, at some point in their educational careers, students come to recognise the uncertainty of knowledge, realising that what is taught by their teachers may not be the one and only truth but that other teachers, researchers, or scholars may hold different positions on the same subject matter. When students acknowledge this fact, they will find it difficult to return to a previous stage; their way of thinking has changed. By contrast, skill development enables students to address increasingly complex situations, but this does not require a change in thinking.

Although development is underexplored in the field of graduateness and generic learning outcomes, in epistemological traditions, development is theorised and investigated in different graduateness domains (e.g., reflective thinking, moral and ethical behaviour). These models are concerned with the nature of knowledge and individuals’ experiences in constructing knowledge in different domains (Boyles, 2006; King & Kitchener, 2004). Several researchers have found evidence for developmental stages to occur in these domains (King & Kitchener, 2004; Kohlberg, 1973; Perry, 1970), through which individuals develop at their own pace and according to their own capabilities. This indicates that not everyone achieves a specific stage at the same age and that the higher stages may be achieved only by a few people (King & Kitchener, 2004; Kohlberg, 1973; Perry, 1970). However, because university students, especially master’s students, are a specific group within the human population, high expectations appear to be justified. Therefore, it seems reasonable to regard the graduateness level as referring to advanced stages in the developmental models.

Using a developmental approach allows us to identify where the stage of graduateness occurs in students’ academic intellectual development. We achieve this goal by determining the graduateness level in developmental models in the fields in our domains of graduateness. One should, however, be aware that this reflects the minimum level of development that needs to be achieved to speak of graduateness. Perry (1970) was one of the first to design a developmental model of students’ intellectual and ethical development, but we found models developed for specific aspects of intellectual development to be more suitable for our purpose. Certain aspects of graduateness, including scholarship, are not addressed in Perry’s model. Moreover, for reflective thinking, a more specific model is available (King & Kitchener, 2004). Below, we briefly describe the models that we used.

Reflective thinking

Inspired by the cognitive-developmental tradition (e.g., Perry, 1970) and constructive-developmental perspectives (e.g., Fischer, 1980), King and Kitchener (2004) developed a developmental model for reflective thinking, the reflective judgement model. Central in their model is how knowledge is constructed and the role of authority in this process. This model distinguishes seven stages of reflective thinking divided into three clusters: pre-reflective judgement, quasi-reflective judgement, and reflective judgement. In this latter cluster, we place the graduateness level of reflective thinking at Stage 6 of the reflective judgement model. This stage is characterised by the ability of students to recognise that knowledge is uncertain and that they must translate and integrate information from different angles to reach a conclusion. Furthermore, concerning knowledge, there is no such thing as “truth” being something that is known to an authority (in case of education, the teacher). A student functioning at the graduateness level in the reflective thinking domain is aware that knowledge is constructed by collecting information and opinions from different sources and views.

Scholarship

To the best of our knowledge, there are few models concerning development in scholarship devoted to the stages of becoming experienced researchers. Of course, a scholarly attitude includes some aspects of reflective thinking (Schön, 1983; Van der Rijst, 2009). However, we believe that it is valuable to use a separate developmental model for scholarship to acknowledge its domain-specific aspects, such as research skills and an inquiring attitude. Willison and O’Regan (2007) developed the Research Skill Development Framework that distinguishes seven levels based on the extent of students’ autonomy according to six facets of research (e.g., Find & Generate, Evaluate & Reflect, and Analyse & Synthesise). Levels 6 and 7 are dedicated to research contributing to the disciplinary body of knowledge and are thus unlikely to apply to master’s students. For a master’s education, we expect students to perform at least at Level 5, which is characterised by “students research within self-determined guidelines that are in accord with discipline or context” (Willison & O’Regan, 2008). This expectation entails that students are capable of self-directing their research in all facets, ranging from the ability to translate experience and expertise into researchable research questions to independently designing an appropriate research design and engaging in insightful reflection on one’s own research and that of others.

Moral citizenship

Where Barrie (2004) used the term global citizenship, we prefer the term moral citizenship to emphasise that this domain includes not only students’ stance on the world (Nussbaum, 1997) but also their moral development (Kohlberg, 1973; Perry, 1970), which is considered an important element of students’ development in university education (Gottlieb, Handelsman, & Knapp, 2008). Moral citizenship is concerned with both students’ ethical and moral development and their development towards global citizenship. In a previous work, Kohlberg and Hersh (1977) noted that ethical and moral development is closely related to the concept of “citizenship”. Nussbaum (1997) described global citizenship as having knowledge about other cultures and being able to place oneself in the position of other people who live different lives. An explicit purpose of liberal education, which promotes global citizenship, is to continuously reconsider one’s opinions and beliefs (Nussbaum, 1997). In the same tradition as Perry (1970) and King and Kitchener (2004), Kohlberg (1973) developed a six-stage model for moral development. This model is criticised primarily because of certain strong claims that Kohlberg made in the model, such as the universality of the model and the lack of consistent empirical evidence for the sixth stage. However, other ideas behind the model still prove to be valuable, such as the developmental aspect of the model. Rest and colleagues developed this model into a neo-Kohlbergian approach to moral development (Rest, Narvaez, Thoma, & Bebau, 2000; Thoma, 2014) with three so-called schemas: Personal interest, Maintaining norms, and Postconventional schema. These schemas represent how new information is processed (Rest et al., 2000). Graduateness is situated in the postconventional schema. Moral citizenship is a broad concept covering different aspects ranging from ethical development to opinion forming and social awareness. In this study, we emphasise a specific aspect of moral citizenship, namely, acknowledging societal relevance. In terms of the postconventional schema, this means that students are capable of engaging with controversial topics, contributing to a scientific and social debate, and acknowledging that all opinions are open to scrutiny.

Model of graduateness

Despite the claim of genericity, the context is important to how graduateness is interpreted and taught (Jones, 2013). Similarly, considering moral development, Thoma (2014) stated that students in political sciences or philosophy, for example, are more likely to have achieved the postconventional schema than are students in other disciplines. An interesting characteristic of graduateness is thus revealed, namely, that different aspects of graduateness will be highlighted in different disciplines as a result of the nature of the specific discipline. For our model of graduateness, graduateness can thus be achieved through development in different domains. From our definition of graduateness, different interpretations of graduateness follow as study programmes focus more on certain domains than on others. The importance of reflective thinking as an outcome of university learning is emphasised so often (Biggs, 1999; Kember et al., 2000; King & Kitchener, 2004; Pascarella & Terenzini, 2005; Procee, 2001, 2006; Schön, 1983; Van Rossum & Hamer, 2010; Willison & O’Regan, 2008) that we consider reflective thinking to lie at the heart of graduateness. Hence, we consider that performance at the reflective judgement level is a necessary but insufficient condition for graduateness to be achieved. In addition to advanced performance (i.e., at the graduateness level) in the domain of reflective thinking, graduateness presumes advanced performance in at least one other domain (see Figure 1).

Figure 1. Schematic model of graduateness.

Research questions

Can the model of graduateness that is rooted in developmental models be used to adequately describe graduateness in terms of both achievements and growth? If the model successfully measures graduateness, we expect to find that a proportion of students achieve graduateness by the end of the master’s programme. We expect an adequate model to be able to detect growth during the master’s year both in the achievement of graduateness and in the separate domains. Finally, we expect the model to be suitable for different disciplines. We translated these expectations into the following research questions:

1. Do master’s degree graduates achieve graduateness according to their self-reported abilities?

2. Do graduates achieve graduateness more often than non-graduates do?

3. Can growth in students’ perceived abilities be determined for graduateness and in its separate domains (reflective thinking, scholarship, and moral citizenship)?

Method

Data collection

We consider several master’s programmes in this study because we expect a higher proportion of students to achieve graduateness in master’s programmes than students in bachelor’s programmes. Two succeeding cohorts of master’s students in four faculties (social sciences, law, business and economics, and spatial sciences) at a research-intensive university were invited to participate in the study concerning student experiences in master’s programmes. Each cohort was asked to complete an online questionnaire at the beginning and end of the master’s programme.¹ For this study, we used data only from students who participated at both measurement points. Of the cohorts, 38% (N = 485) of the first and 34% (N = 398) of the second completed the questionnaire at the beginning of the programme (T1). At the end of the programme (T2), 32% (N = 157) of the initial response group of Cohort 1 and 37% (N = 147) of the initial response group of Cohort 2 completed the second questionnaire. These two cohorts do not differ from one another concerning the number of domains in which the graduateness level is achieved (T1: chi-square = 1.943, df = 3, p = .584; T2: chi-square = 3.717; df = 3; p = .294). Because no students appeared in both cohorts, the cohorts are considered together in this study. Based on data retrieved from the university’s student administration tool, we were able to identify students who graduated in the year that they participated in the study. Furthermore, the tool offered us the opportunity to compare the response group to the non-response group with respect to several characteristics. Compared with women in the non-response group, women were overrepresented in the response group (approximately 50% of the women responded in contrast to approximately 40% of the men in the population). The results indicated no major age differences between the respondents and non-respondents. Furthermore, the response group obtained, on average, 5 ECs² more than the non-response group did. Such differences between response and non-response groups are commonly found.

Instruments

We investigated the development of three domains of graduateness. Although the questionnaire was developed for quality assurance purposes, these concepts could be identified within the questionnaire. Initially, we had no theoretical reasons to assign different weights to these domains. We, therefore, used average scale scores to correct for the different number of items that the three instruments comprised. For our study, we operationalised reflective thinking by using the deep learning approach scale from the Approaches and Study Skills Inventory for Students (ASSIST; Entwistle, McCune, & Hounsell, 2002). This scale consists of seven items that refer to the ways in which students approach the study material, for example, “Ideas in course books or articles often set me off on long chains of thought of my own”. In this sense, this scale can be interpreted as measuring reflective thinking, which is consistent with the approach of Biggs (1999), who also considered deep learning to be a characteristic of an academic attitude. The answer options range from strongly disagree (1) to strongly agree (4). The average scale score ranges were 1.43 to 4.00 (SD = .38) for T1 and 1.71 to 4.00 (SD = .42) for T2.

The questionnaire includes a section regarding academic skills. For each statement, the respondents needed to judge their ability on a 1 to 10 scale, with 1 indicating poor mastery and 10 indicating excellent mastery. From these skills, a number of items were selected by content to measure scholarship and moral citizenship. For scholarship, five items were identified as referring to higher order mastery of research skills, such as the ability to write a research proposal or to critically evaluate the literature. Average scale scores were calculated, and these scores were between 3.0 and 9.6 (SD = .98) at T1 and between 2.6 and 9.0 (SD = .90) at T2. For moral citizenship, four items were available, largely related to the ability to express one’s opinion and the ability to link knowledge to current events. The average scale score ranges were 2.50 to 9.75 (SD = 1.00) for T1 and 3.50 to 9.25 (SD = .95) for T2 (see Table 4 for the mean scores for each scale at T1 and T2).

For all scales, reliability measures (Cronbach’s alpha) were calculated at both T1 and T2; these values are reported in Table 1. All scales show sufficient reliability coefficients according to Nunnally and Bernstein (1994), who recommended that instruments used in basic research should have reliability coefficients of approximately .70 or above.

Table 1. Results of reliability analyses.

	Cronbach’s alpha
	T1	T2
Reflective thinking Scholarship Moral citizenship	.73 .82 .81	.75 .82 .78

Finally, for the group of students who graduated in the year that they participated in the study, the graduation date should be before the start of the new academic year, which is 1 September in The Netherlands.

Thresholds

Although the items that were selected in this study represent the graduateness level, thresholds must be determined. The scale scores represent both respondents for whom the items do not reflect their particular situation (the lower end of the scale scores) and respondents for whom the items adequately represent their particular situation (the higher end of the scale scores). Within this range of scale scores, a threshold for the graduateness level must be placed at a point where the scale scores reflect the minimum requirements for the graduateness level. We argue that determining these thresholds should be based on theoretical arguments as opposed to empirical data because the basic principle of the theoretical model is that graduateness is characterised by specific behaviour or strategies that are consistent with certain stages from the developmental models. Placing the thresholds this way might be considered arbitrary; however, we considered two criteria before placing the thresholds. First, the threshold should give a reflection of the minimum requirements for the graduateness level as adequately as possible, however, preventing that the thresholds are only met by an elite group of students. An elitist approach would argue against the idea of graduateness, namely, that graduateness should be achievable for all master’s degree students. Second, we acknowledge that the instruments were not specifically designed for the purpose of this study. We, therefore, allowed for less stringent thresholds. At the same time, we do want to set appropriate thresholds because, still, lower scale scores mean that most of the statements do not apply to that student’s situation.

For the domain of reflective thinking, it appears reasonable to determine the threshold for the graduateness level at 3.00, which resembles the answer option “agree” used for this scale. To determine the meaning of this threshold, we sought to find the percentage of respondents for which a scale score of 3.00 and above includes a score below 3 for the separate items, indicating the extent to which the average scale score represents consistent behaviour among the items. The results indicate that for less than 10% of the cases, a scale score of 3.00 or above includes two or three items scored below 3. For the domains of scholarship and moral citizenship, students were asked to rate themselves on a 10-point scale that is similar to the grading system to which they are accustomed. In this regard, determining where an adequate threshold should be placed to represent the graduateness level is slightly more complicated. As in the Dutch grading system, 6 represents the bare minimum to pass, and 7 represents adequate mastery; it seems opportune to assume a minimum scale score of 7 for the domains of scholarship and moral citizenship. In all cases, this threshold results in less than 10% of the cases meeting the threshold for the scale score, with two or three³ items scoring below 7. Obviously, these lower scores for items are compensated with higher scores for other scale items.

Graduateness

The complex construct of graduateness was measured by combining scale scores in each domain using the thresholds. We previously established that meeting the threshold in the domain of reflective thinking is a necessary condition for graduateness. This criterion results in four possible categories of graduateness:

• 0 = graduateness is not achieved in reflective thinking;

• 1 = graduateness is achieved only in reflective thinking;

• 2 = graduateness is achieved in reflective thinking and one other domain;

• 3 = graduateness is achieved in reflective thinking and two other domains.

Only the last two categories are considered to reflect graduateness; the first two categories indicate pre-graduateness (also see Figure 1).

Analyses

To investigate the research questions, several statistical techniques were used. First, the distribution of students and graduates was calculated to determine whether the model fits the master’s level. Paired t tests were used to investigate whether students show growth in perceived abilities in the three domains of graduateness during the master’s year. The scores for the first measurement were paired with the scores for the second measurement.

Results

Of the 304 respondents, 143 (47%) graduated at the end of the master’s year. First, we determined whether these graduates achieved graduateness at the end of the master’s year. A total of 60% of the graduates achieved graduateness, most (41%) by achieving the threshold in the three domains, and the others (19%) by achieving the minimum standard of meeting the threshold in two domains (see Table 2). However, 30% of the graduates failed to achieve graduateness because they did not meet the threshold in reflective thinking.

Table 2. Distribution of graduates and non-graduates across categories of graduateness at T2.

Graduateness		Not graduated yet	Graduated
Pre-graduateness	0	30%	30%
	1	13%	10%
Graduateness	2	16%	19%
	3	40%	41%
Total		100%	100%

Second, we investigated whether the group that graduated in the same academic year in which the study was conducted differed from the group that graduated later. Table 2 presents the distribution of both groups over the different levels of graduateness. Of the respondents who did not graduate at the end of the master’s year, 56% achieved graduateness. Additionally, the same proportion of students in the group of students that graduated in the same year and the group of students that graduated later did not achieve the threshold in the reflective thinking domain. The chi-square results revealed no difference between the two groups (chi-square = .504, df = 3, p =.918).

Third, we investigated the research question regarding whether the proportion of students achieving graduateness differs between T1 and T2 using contingency table analysis (see Table 3). At the beginning of the master’s programmes (T1), 48% of the respondents had already achieved graduateness. At the end of the master’s programmes (T2), this level had increased to 58%. A chi-square test in which the distribution of T2 is compared with the distribution at T1 revealed that more respondents achieved graduateness at T2 (chi-square = 18.90; df = 3; p = .000).

Table 3. Distribution of respondents across categories of graduateness for T1 and T2.

Graduateness		T1	T2
Pre-graduateness	0	33%	30%
	1	19%	12%
Graduateness	2	16%	17%
	3	32%	41%
Total		100%	100%

Fourth, growth in perceived abilities in the three domains was investigated using paired t tests that compared scores at T2 with scores at T1 (see Table 4). The differences were significant at p < .05 for reflective thinking and for scholarship. For moral citizenship, the difference was not significant. The difference between T1 and T2 for scholarship was the largest.

Table 4. Results for paired t test for reflective thinking, scholarship, and moral citizenship.

Domain	Mean T1	Mean T2	T	df	Sig.
Reflective thinking	3.09	3.13	2.117	303	.035
Scholarship	6.84	7.06	4.686	303	.000
Moral citizenship	6.96	7.04	1.945	303	.053

Conclusion and discussion

One of the issues in the graduateness debate is that the level at which students are expected to perform in domains that are associated with graduateness is often unspecified (Davies & Hogarth, 2002). The current study aimed to contribute to this debate by using developmental theories to denote the graduateness level in three domains and to investigate whether master’s programmes contribute to development in these domains. First, we checked whether the model fit with graduates’ perceived capabilities. In fact, 60% of the graduates achieved graduateness at the end of the master’s programme; thus, they achieved the graduateness level in reflective thinking and at least one other domain. Because we considered only two other domains, the proportion that we found is likely to be an underestimation of the real proportion of students who achieved graduateness. However, because 30% of the graduates did not achieve the graduateness level in reflective thinking, only an additional 10% could potentially achieve graduateness through other domains.

Second, our results showed that the model does not discriminate between students who graduated by the end of the master’s year and students who needed more time to graduate; therefore, a difference appears between formal graduateness (i.e., graduation) and actual graduateness (i.e., achieving the specific stage in intellectual development). Moreover, the results suggest that most students who achieve graduateness have already done so before they have completed all course units of the study programme. This could be due to the fact that in this study we used minimum requirements for the graduateness level in each domain. Therefore, graduation is not a prerequisite for students to achieve graduateness. The proportion of students that achieved graduateness at the start of the master’s year would encourage placing the graduateness level at higher stages in the domains considered. Still, the proportion that does not achieve graduateness at the end of the master’s year, according to this model, is substantial.

Finally, we established whether students indicate growth during the master’s year both in achieving graduateness as a general notion and for each domain separately. Regarding graduateness, we found that nearly half of the respondents achieved graduateness in October, and in May this proportion increased to nearly 60% for the entire group of master’s students who completed the questionnaires. Graduateness appears to develop during bachelor’s programmes, but the development does not cease there; rather, it continues in master’s programmes. Regarding growth within the separate domains, the respondents reported stronger capabilities at the end of the master’s programme for the domains of reflective thinking and scholarship than they did at the beginning of the master’s programme. The largest growth was detected in the domain of scholarship, from a mean immediately below the threshold to a mean immediately above the threshold. Although the growth appears to be limited, it is still a relative finding because the relative small sample size of this study would generally make it more difficult to find small significant effects like this. Moreover, the finding is relevant because it indicates that a larger proportion of students meet the threshold, albeit that a slightly higher threshold would still not be met by the students in this study.

The limited growth in graduateness and its separate domains during the master’s year suggests that study programmes can be improved in this matter. In our research, we focused solely on student aspects of graduateness. For universities to further stimulate students’ intellectual development, the efforts of teachers in emphasising graduateness should also be acknowledged. Barrie (2004) found that teachers understand graduateness in different ways that are likely to influence what is taught. For example, when a teacher considers graduateness as an aspect that needs to be developed apart from the disciplinary content, it is highly unlikely that graduateness domains will be addressed explicitly in their lectures and assessments. Furthermore, according to Van Rossum and Hamer (2010), student development depends on the developmental level at which a teacher functions with regard to graduateness. Moreover, De la Harpe and David (2012) found that teachers’ beliefs regarding the importance of aspects of graduateness did not automatically reflect their level of emphasis on graduateness in their teaching. To develop curricula that stimulate the academic intellectual development of students, insights from these studies must be considered to improve both instruction and assessment.

In this study, we place the graduateness level at the minimum required stages in the separate domains. One could argue that in post-graduate education these requirements should be placed at even a higher stage than we did in this study; however, we wanted to prevent an elitist approach of graduateness. Moreover, as the instruments were not specifically designed for this study, we felt that this should be acknowledged in placing the thresholds. We consciously chose a normative approach to determine the thresholds, rather than empirical determination. We used developmental models to determine the graduateness level. It would be difficult, if not impossible, to align these theoretic stages with empirical determination of threshold, for we simply do not know the developmental stage distribution in the research population. A disadvantage of this approach is that the results should be interpreted given these theoretically determined thresholds. When the thresholds would have been placed higher, a smaller proportion of students would have been considered as having achieved graduateness. When placing the thresholds lower, one could question how well the graduateness level is reflected when the proportion of “(strongly) disagree” answers grows in the group of students who meet the thresholds in, for example, reflective thinking. The fact that the mean scores for each domain circle around the threshold values provides some relevance to the chosen thresholds. Moreover, the stages in a developmental approach are characterised by transformation rather than graduality and, therefore, does not align with empirical thresholds.

An advantage of this approach in which the thresholds are determined theoretically is, however, that it offers guidance to explicate the level expected of graduates in the domains of reflective thinking, scholarship, and moral citizenship, and it can easily be expanded to other domains of graduateness. Furthermore, because the graduateness level is determined based on developmental models that distinguish different stages, this approach offers opportunities to study the development of graduateness in greater detail. This approach will in turn offer support for developing tools for assessing graduateness, which will be useful because the lack of suitable assessment tools makes it difficult for academic staff and curriculum designers to explicitly include graduateness in university curricula (Hughes & Barrie, 2010). We are reluctant to determine empirical thresholds for the current study because the manner in which the level of graduateness is determined is based on perceived performances in the current group. This determination would contradict the essence of the theoretical model in which graduateness is a stage based on applied strategies and behaviour rather than a relative standard depending on the scores of others in the group.

All results are based on self-reported estimations of performance and ability. Certain limitations are known when using this type of data. Students tend to overestimate or underestimate their abilities, depending on personality characteristics. We believe that these effects are partly mitigated because we used a repeated-measurement design in which the same group of students report their perceived abilities at two different points in their master’s programmes; in this context, overestimation or underestimation plays a minor role in growth in the domains of graduateness. For the mean differences, it does not matter as much whether a particular student overestimates or underestimates herself or himself: For growth, it is the difference that matters. Furthermore, recent findings of Benton, Duchon, and Pallett (2013) suggested that self-reported ratings of learning can be rather adequate in topics that teachers consider important. Moreover, in the field of competences, Vaatstra and De Vries (2007) showed that students were able to adequately report on their abilities concerning generic and reflective competences. We accepted the restrictions of self-reports because the alternative of developing a test of graduateness also has serious limitations. First, such an approach would be more time consuming to cover all domains adequately. Second, such a test would provide less flexibility in being applied to different disciplines because of the contextual nature of how graduateness is explicated in different disciplines. Although graduateness can be found in all disciplines, its shape differs (Jones, 2013). A test for graduateness would require more details than a questionnaire that leaves the opportunity to formulate general statements that could have different interpretations in different disciplines. For example, in the domain of scholarship, all students need to write a research report; however, the content and underlying skills might differ. Data collection and analyses would be emphasised more in the social sciences than in law. Third, such a test would be far too time consuming to process for large groups; it would be applicable only to small groups. We aimed to provide findings that can be generalised more easily by including a variety of study programmes across different disciplines. Our results are applicable for disciplines ranging from business and economics to law and from social sciences to spatial sciences. The appropriateness of our approach for the natural sciences and humanities should be investigated further.

Overall, the findings of this study suggest that students grow in their academic intellectual development during their master’s year. Although nearly half of the students in our study had already achieved graduateness at the beginning of the master’s programme, the results indicate an increase in graduateness achieved and growth in separate domains. However, a substantial number of master’s students do not meet the thresholds for actual graduateness at the end of their master’s programmes. Before rushing to the conclusion that a substantial proportion of master degree students fail to achieve graduateness by the time they graduate, these results might also reflect the relative importance of graduateness in the study programmes we investigated. It depends on how universities and society value graduateness, and especially in the domains we considered, whether or not this finding is worrying. Bearing in mind the results of Booth et al. (2009) and Glover et al. (2002), these findings might be an expression of the increased value of employability in university education (e.g., Moreau & Leathwood, 2006; Scott, 2003). We can, therefore, only conclude that when the described domains ought to be important aspects of university education, as is argued by some (e.g., Barnett, 2009; Barrie, 2004; Biggs, 1999; Collini, 2012; De la Harpe & David, 2012; Lea, 2011; Peters, 1975; Schuyt, 2011), these aims are not achieved in all post-graduates.

Notes on contributors

Jessica Steur, MsC, worked as a PhD candidate at the University Centre for Teaching and Learning. She currently works as a quality assurance policy worker in the Faculty of Behavioural and Social Sciences at the University of Groningen.

Ellen Jansen holds the position of associate professor in teacher education at the University of Groningen. Her expertise relates to the fields of teaching and learning, curriculum development, factors related to excellence and study success, social (policy) research, and quality assurance in higher and secondary education.

Adriaan Hofman is Professor of Education, especially Higher Education at the University of Groningen in The Netherlands. He specializes in school and teacher effectiveness, higher education, education in developing countries, research methods, urban education, and learning cities.

Notes

1. In The Netherlands, master’s programmes are generally 1-year programmes.

2. European Credits (an EC equals 28 hr of work; most course units awarded 5 ECs).

3. For moral citizenship, no cases met the threshold at the scale score with three lower scoring items.