Manufacturing excellent engineers: skill development in a Masters programme

Abstract

An MPhil programme, delivered by the Engineering Department at the University of Cambridge, claims to be excellent at preparing graduates for manufacturing industry careers. The course uses a combination of different educational experiences, including industry-based assignments, industrial visits and practical exercises.

This research explores how problem solving skills are developed during the first module, Induction, which is designed to enable students to undertake their first industrial assignment.

From the literature, four conditions necessary for skill development were identified:

• Provision of a skill description, making explicit key components

• A number of different experiences with a range of contextual variables

• A teaching process which includes regular feedback and student reflection

• Students motivated to learn.

These were used to construct a skill development framework (SDF).

Using a case study research design, multiple types of evidence were collected to test for the above conditions using both classroom observation and questionnaire methods.

The results confirmed the presence of the SDF conditions at different levels, with reflection aspects considered the weakest. Conflicting results were obtained regarding the students’ self-awareness of skill levels. A plausible explanation is a change in the students’ frame of reference.

This initial study set out to develop a better understanding of the process of skill development. Whilst the SDF appears reasonable, there is a need for further work in three broad areas of defining skills, assessing skills and developing reflection skills.

Introduction

The purpose of this study was to develop a better understanding of skill development in higher education (HE), with particular reference to the more complex skills used in work-related contexts. The aim of this initial work was to construct a skill development framework from the literature and then test it by evaluating the development of problem solving skills in a Masters programme. As skill evaluation can be resource-intensive, student self-assessment was explored as a potential resource-efficient option. Further areas were identified in which to continue research related to skills development.

The Industrial Systems, Manufacture and Management (ISMM) MPhil programme at the University of Cambridge (Institute for Manufacturing, 2011) claims to be excellent at preparing graduates for careers in industry. The programme, initiated 46 years ago, was designed to prepare engineering graduates for operational roles in industry. A study at the 36-year point (Ridgman and Wiggins, 2003) concluded that the programme was very successful but could not determine whether this was a function of the course or, for example, effective selection of candidates.

Four industrial assignments, accounting for 50% of the programme assessment, are a key component of ISMM. Groups of two students spend two weeks working on a real life issue of some significance to a company. Students present their problem definition, analysis and proposed solutions on their last day and submit a written report the following week.

The first module of the programme, Induction, lasts four weeks and is followed by the students’ first industrial assignment. During Induction there is a focus on developing students to undertake industrial assignments. Problem solving skills are considered fundamental and a method of developing these has been established. This starts with a lecture on problem solving and is then followed by five experience-based exercises, with group membership, type of task and group size being varied. Course tutors have found that this preparation has enabled the vast majority of students to successfully attempt the problem faced during their first industrial assignment.

Literature review

This research considers the broad fields of employability, higher education and professional development and how these contribute to skill development. The key bodies of knowledge and how they interrelate are presented in Figure 1.

Employability

Individual skills are a common feature of employability models (Hillage and Pollard, 1998; Knight and Yorke, 2002; Dacre Pool and Sewell, 2007).

The work of Knight and Yorke has been key to developing a definition and model of employability for HE. Graduate employability is defined as ‘a set of achievements — skills, understanding and personal attributes — that makes a graduate more likely to gain employment and be successful in their chosen occupations’ (Yorke, 2006). A key claim (Knight and Yorke, 2004) is this is an academic research-driven view, not driven by the perceived needs of employers or governments.

The USEM Model (Knight and Yorke, 2004) identifies four key components comprising understanding (U), skilled practice (S), efficacy beliefs (E) and metacognition (M). A key feature of the model (see Figure 2) is that the efficacy beliefs provide a foundation to employability and feed the U, S and M components.

The E component represents a persons’ belief that they can make an impact on a situation. It includes a broad range of theoretical contributions (Knight and Yorke, 2004), including Bandura’s work on self-efficacy and work on practical intelligence (Sternberg and Grigorenko, 2003) that has a particular resonance as it closely resembles ISMM problem solving. The term skilled practice (S) was chosen to capture the view that skills are context specific, not easily transferable, and assessed with difficulty.

There are few publications citing this model being used in practice. It is suggested that assessment could be a factor, as broad complex constructs such as employability resist summative assessment (Knight and Yorke, 2003).

Figure 1 Key bodies of knowledge

Figure 2 USEM model of employability (Knight and Yorke, 2002)

Skills

There is no simple definition for skill (Eraut, 1994; Tight, 1996; Moon, 2004). At a high level there is broad consensus that a skill is the ability to do something, for example: ‘skill […] is the ability to do something that has been learnt’ (Moon, 2004). Also agreed is that there is a range of skill types, including physical, practical and cognitive (Moon, 2004).

However, below this level, agreement disappears — with overlapping categories and interpretations of skill, as in the case of “employability” and “professional” skills. Each category is typically labelled and defined for a particular community, causing difficulties in direct and consistent translation.

In HE neither “employability” or “professional” are used in the specification of programme content. Four categories are used (QAA, 2006): knowledge and understanding, intellectual skills, practical skills and transferable skills, with transferable skills appearing to map most closely to employability skills.

There is broad agreement that a skill requires some knowledge “that” combined with some knowledge “how” (Eraut, 1994; Moon, 2004). In terms of professional skills, Eraut states that they ‘require unique combinations of propositional knowledge, situational knowledge and professional judgement’. As judgements are made relative to a particular context it is important that skill development is undertaken during context-specific activities.

Skills are learned by a combination of methods (Eraut, 1994; Moon, 2004; Tether et al., 2005), including education, training and experience.

There are a number of models which propose different levels of skill in a professional or work context, such as the five level Dreyfus Model (Dreyfus and Dreyfus, 1986) and the four level International Project Management Association (IPMA) to describe project management skills (IPMA, 2011). A common thread is the use of multiple experiences to develop higher levels of context-specific knowledge and judgement-making skills.

It was Kolb (1984) who proposed a general model of learning by experience. Whilst all Kolb’s dimensions are likely to contribute to learning, the circular model is perceived as being too simplistic (Coffield et al., 2004; Race, 2010). However, the circle is helpful in reinforcing that a number of related experiences support a person’s learning.

Race (2010) returns to the concerns of Kolb to identify factors that underpin successful learning. Race accepts that there are relationships between these factors but argues that they will vary between people. A key difference with Kolb’s model is the presence of wanting or needing to learn.

In summary, four key aspects of skill development are identified from the above literature: student motivation, experience, feedback and reflection. These concepts will now be explored further for HE.

Student motivation

A common categorisation of student motivation is intrinsic and extrinsic (Harter, 1981). Intrinsically motivated students are curious and want to learn, whilst the extrinsically motivated worry about grades and approval by others.

Skill development could be a challenge for extrinsically motivated students, given that it is problematic (Knight and Yorke, 2003). However, motivation can increase with understanding (Race, 2007) and understanding can be developed by experience and interaction. Principles to promote motivation include discussion of importance and utility, provision of clear and accurate feedback and the provision of stimulating and interesting tasks (Pintrich, 2003).

Experience

Of the experience-based teaching methodologies, problem-based learning (PBL) (Barrett and Moore, 2011) and project-based learning (PjBL) (Graham and Crawley, 2010) would appear to be relevant. However, the best match is consultancy projects used in MBA programmes (Jennings, 2002).

Exercises or simulations are able to provide a wide range of skill development opportunities (Jennings, 2002; Goodhew, 2010). It is noted that they are particularly effective for learning related to complex situations and they tend to be rare in practice due to the time required to develop them (Goodhew, 2010).

Experience can also be used to develop a student’s self efficacy — a key component of the USEM model. Ways of influencing efficacy development are identified as mastery and vicarious experiences (Bandura, 1995).

Feedback

Feedback can be summative in terms of what was or wasn’t achieved and formative in identifying how performance can be improved in future. Some widely recognised indicators of effective feedback include being prompt (Race, 2010) and being positive and constructive (Landsberg, 2003).

Feedback can be relevant to the whole class or to specific individuals or groups, thus different mechanisms are required to ensure this crucial aspect of learning is carried out effectively and efficiently (Race, 2010).

Reflection

Reflection is a process that connects the notions of learning and thinking (Moon, 2004). As well as being key in the learning process (Kolb, 1984), it also plays a significant role in ongoing professional development. The idea of the reflective practitioner was developed by Schön (1987), who describes an outcome as the ability to handle complex problems with confidence, skill and care.

Moon (2004) reports a depth dimension to reflection, with shallow reflection being less likely to be as effective in supporting learning. Reflective learning can be improved by providing a clear purpose and a list of questions linked to associated learning goals. However, students have been found to have limited understanding of reflection (in terms of its value), what it means and how it is undertaken in practice (Moon, 2004).

Framework development

Individual components have been described above and their success in supporting learning could be amplified if they are constructively aligned (Biggs, 2003). This suggests that a skill development framework (SDF), integrating the components identified from the literature, would help inform the design and delivery of skill development activities. To that end, a SDF must be practical and appropriate for a wide range of skills.

The following conditions for the effective development of complex skill sets are identified:

1. The provision of a description of the skill, making explicit key components of knowledge “that” and knowledge “how” and examples of the types of typical judgements associated with the delivery of the skill

2. A number of different experiences with a range of contextual variables and at the level of difficulty to provide mastery and/or vicarious experiences

3. A teaching process which includes student reflection and the provision of feedback after each experience

4. Students motivated to learn, of which indicators are engagement in learning activities and learning outcomes linked to summative assessment.

Indicators are identified for each condition. The number has been limited to a maximum of four to present a framework of practical size. This is shown in Table 1.

The SDF will be tested with regard to problem solving skills, so it is important that these are defined. As with many skills, these have been labelled and defined for a particular community — in this case by the Institute for Manufacturing (IfM) — and defined in relation to industrial problems that students face.

In the absence of a formal definition, a working definition was compiled by extracting key learning points or learning outcomes from course documentation. This identified three key components: problem solving, managing a project and group work.

The first test of the SDF was to evaluate how it relates to the development of problem solving skills, with the objective of determining if the conditions identified are present in practice. In Induction this is carried out in a lecture followed by five exercises and from now on is referred to as the development method (DM).

Further testing of the SDF will require a mechanism to measure levels of problem solving skills. This has been identified as problematic in terms of resource requirements and ability to provide a reliable scaled measurement. With problem solving being much less complex than employability, and a quick method of assessment being very attractive in terms of resource requirements, it was decided to investigate whether a self-assessment instrument could be used.

Methodology

This exploratory research lends itself to the case study research method set out by Yin (2009), using ISMM Induction as the case. To enable results to be unbiased, measures were taken to ensure that the research had a minimal impact on student or facilitator behaviour. The overall research design is shown in Figure 3.

The research requires multiple sources of evidence to be collected using both qualitative and quantitative approaches. The methods used were observation of the DM and questionnaires which were applied at the start and the end of the DM.

This research design does not take into account the student perspective on skill development. As this could have a significant impact this will be investigated via the questionnaire at the end of the DM.

The data collection method for each aspect of the SDF is identified in Table 2. Each cell has a different coding: I denotes indicator, M denotes method and R will be used later to denote result.

At the end of each exercise, each group was given time to discuss their performance and identify three aspects that had gone well and three that could be improved. Results presented to the class were analysed. Four categories were found: problem solving, group working, project management and presentation. These align with the definition of problem solving skills, with the addition of presentations.

Table 1 SDF - indicators

	A. Skill description	B. Experiences	C. Teaching process	D. Motivation
1	Adequate teaching time used to deliver a description of the skill	Number of exercises	Adequate time used to provide feedback on each exercise	Assessment linked to skill development
2	Skill description in terms of knowledge “that”	Range of different exercises appropriate to the skill	Feedback contains content at a general level (i.e. relevant to all)	Explanation of why the skill is important
3	Skill description in terms of knowledge “how”	A sufficient level of challenge to produce mastery-level experiences	Time given for student reflection after each exercise	Level of student engagement
4	Skill description in terms of “common judgements”	Facilitated to provide all with vicarious experience if mastery experience not achieved	Reflection outputs enable learning

Figure 3 Research design

Table 2 SDF — showing research method per indicator

	A. Skill description	B. Experiences	C. Teaching process	D. Motivation
1I	Adequate teaching time	Number of experiences	Adequate time for feedback	Linked assessment
1M	Observe DM and record time	Record the number of experiences where students demonstrate the skill	Observe DM and record time	Review course documentation
2I	In terms of knowledge “that”	Range of different experiences	Feedback relevant to all	Explanation of importance
2M	Observe DM and record what was presented, how and when it happened	Characterise each experience in terms of type and context	Observe DM and record what was presented	Observe DM and record what was presented
3I	In terms of knowledge “how”	A mastery-level challenge	Time for student reflection after each exercise	Level of student engagement
3M	Observe DM and record what was presented, how and when it happened	Observe if any groups were able to achieve a high level outcome	Observe DM and record time	Observe whether students were engaged or not engaged
4I	In terms of “common judgements”	Facilitated to provide all with vicarious experience	Reflection outputs enable learning
4M	Observe DM and record what was presented, how and when it happened	Examples of cohort mastery experiences shared with rest of cohort	Collect outputs of group feedback sessions — see below

Table 3 Judgement levels

Level of presence in Induction	Descriptor — adequacy to support skill development of problem solving in Induction	Descriptor — scope for improvement
1	Fully adequate	Minor/none
2	Adequate	Some
3	Less than adequate	Significant
4	Inadequate	100%

A consolidated view of the DM was undertaken by making a judgement of the adequacy of each component of the SDF using the four levels of judgements described in Table 3.

To understand the student perspectives of skill development, three open questions were asked in order to identify what aspects of Induction had influenced their learning of skills. All reasons were analysed and answers were sorted by those considered to be more supportive of skill development (e.g. problem solving exercises) and less supportive of skill development (e.g. subject-specific lectures).

The data collection for self-assessment was carried out via questionnaires. The start questionnaire was designed to capture self-assessment of skill levels using both a comparison with their previous undergraduate peer groups and self-confidence in their ability to perform skills in an industrial company environment. The final questionnaire was designed to repeat the capture of self-assessment of skill levels described above and explore how students related their current level and understanding of skill to their level at the beginning of the module. From a cohort of 49 students, response rates of 94% and 96% were achieved for the start and end questionnaires. This resulted in 88% complete data sets where comparisons could be made between the start and end. With such high response rates the findings from this research are considered valid for this course.

Ethical issues in relation to the students were considered (Creswell, 2009). All were informed about the research project and reassured that all data would be kept confidential and used only for the purposes of research.

Results

Comparison of SDF to DM

Each element of the SDF is compared to the DM and the results are summarised in Table 4.

Reflection output results

Each output (see Table 5) was shaded to reflect the aspect skill using the key below.

Table

Key
Practical problem solving
Group working
Project management
Presentations

Adequacy of the DM

Given the above results, the levels of judgements described in Table 3 were applied and the results are shown in Table 6.

Student self-assessment

For student self-assessment relative to the module start, the majority of students failed to answer the set of questions as intended. However, it was possible to determine that 57% thought their skill levels had increased and 47% felt that problem solving skills were more complex than they had thought at the beginning of the module.

Student perception of skill development

The percentage of correct answers per question was 75%, 88% and 51%, indicating that understanding of skill development is variable.

Table 4 Results — SDF applied to DM

	A. Skill description	B. Experiences	C. Teaching process	D. Motivation
1I	Adequate teaching time	Number of experiences	Adequate time for feedback	Linked assessment
1R	A 70 minute lecture was used to deliver a skill description Feedback time following each exercise covered some aspects, see C1R	Five	Ex 1 and 2a: <10 minutes was spent presenting a solution Ex 2b: <10 minutes in short bursts responding to student presentations Ex 3: None	50% linked to work on real industrial problems
2I	Knowledge “that” there:	Range of different experiences	Feedback relevant to all	Explanation of importance
2R	is a range of common problems with differing characteristics are a range of tools and techniques is a general process for solving industrial problems	Efficiency improvements for a packing operation Process flow improvements for food manufacturing Cost benefit analysis of packaging options Factory layout improvements for food manufacturing Manual assembly improvements	Ex 1 and 2a: A workable solution was presented Ex 2b: All options were discussed. The option most acceptable to the board was identified Ex 3: Some feedback from other groups that worked on the same problem	It was made explicit that problem solving skills were vital for industrial assignments
3I	Knowledge “how”	A mastery-level challenge	Time for student reflection after each exercise	Level of student engagement
3R	Only the “define problems” aspect was covered in the lecture Some guidance on identifying the nature of the problem given in the feedback sessions for Ex1, 2a and 2b Students were referred to other lectures and resources	For exercise 1, 2a, 2b and 2c some groups demonstrated mastery level It is not known how many groups achieved this for Ex 3	No more than ten minutes in groups	High levels were observed during the lecture and task part of each exercise Lower levels were observed in presentation preparation, feedback and reflection aspects
4I	“Common judgements”	Facilitated to provide all with vicarious experience	Reflection outputs enable learning
4R	Some given (e.g. need to judge the validity of information)	As some groups demonstrated mastery level this provided vicarious experiences for others	See reflection output results that follow

Table 5 Reflective outputs - example results from Exercise 2b

Things done well	Team	Things done not so well
Assigned tasks well	A	Not well organised but better than last time
Good at timing — no mad panics	Rushed too much then had to go back and redo
Communicated well
Assigned tasks and executed well	B	Disorganised
Good involvement	Wasted time doing the same things
Took wide perspective of issue	Presentation
Split tasks up well	C	Time management
Communication, as one person switched between sub groups	Didn’t check results initially
Good answer — only missed out one calculation	Made one slide for presentation four times
Split into groups to analyse main options in detail	D	Made calculation error
Good teamwork and communication	Not enough checking of others work
Good professional PowerPoint presentation	Organisation

Table 6 Skill development framework — level of presence results

A. Skill description	B. Experiences	C. Teaching process	D. Motivation
1 — Adequate teaching time	1 — Number of	2 — Adequate time for feedback	1 — Linked assessment
1 — knowledge “that”	1 — Range of different experiences	2 — Feedback relevant to all	1 — Explanation of importance
2 — Knowledge “how”	1 — Mastery level of challenge	3 — Time for student reflection after each exercise	2 — Level of student engagement
2 — “Common judgements”	1 — Facilitated to provide all with vicarious experience	3 — Reflection outputs enable learning

Table 7 Student self-assessment results

	Mean	Median	Mode	Range
Awareness of skill level in comparison with their undergraduate peers	−0.3%	−2%	−2%	−22% to 36%
Confidence that they could perform specific tasks in an industrial company environment	−2.30%	−2%	−1% and −9%	−21% to 19%

Discussion

Comparison of the SDF to the DM

Reflection aspects were considered to be the weakest. The exercise was problematic as it did not encourage individual, focused or in-depth reflection and students were required to reflect quickly, immediately following an exercise. However, it did provide an effective way to identify issues and prompt discussion on aspects of skill development.

The outputs were judged superficial due to their general nature (e.g. time management) and contradicting statements (e.g. ‘good at timing’ and ‘rushed too much then had to go back and redo’).

Reflective outputs remained superficial throughout the exercises as students demonstrated poor levels of reflective skills and instruction on reflective skills was not provided. The issues identified were mostly project management or team working aspects and, as the students had only had limited descriptions of these elements, there was little basis for reflection.

Some elements of feedback provision, such as prompt feedback of a workable solution and supporting handouts, reflected good practice. However, the workable solution was presented quickly and, as the reflection activity followed immediately, there was little time for digestion.

Levels of student engagement fell during feedback activities. Although this could be due to tiredness on completion of the activity, given that other aspects of skill development have been found to be poorly understood, it would be useful to test understanding of feedback.

Two components of skill description where some weakness was identified was the description of knowledge “how” and “judgement”. These aspects lend themselves to further explanation following an exercise as they are often context-specific. This raises the question of how to incorporate additional teaching time or resources to support this activity.

Strengths and limitations of SDF

The SDF describes skill development as a multi-dimensional construct in which all components should be present, ideally at an adequate or fully adequate level.

As with any framework, when trying to capture complex constructs there is a danger of oversimplification. It is argued that the teaching process condition should be split into separate feedback and reflection conditions. There would then be five conditions which would map closely with the core components of Race’s learning model (Race, 2010). Another issue is that many components are interrelated, a feature poorly represented in the current presentation.

As an indicator, student engagement appears to offer a useful lens, with the ability to highlight whether components are working effectively or not. This correlates with the literature on student motivation which identifies links to many aspects of the learning process.

This framework is very much at the initial stages of development, having been developed as an evaluative tool in one context and only in relation to problem solving skills. A more reliable method of evaluation is required to support further testing. It should also be possible to revise the SDF for use as a design tool at a later stage.

Student self-assessment of skill levels

The importance of student self-assessment lies not only in its contribution to their ability to reflect but also in the practical viewpoint of the resources required to run and provide external assessment for these forms of experiential exercises.

Any self-assessment of skill is made in comparison to a reference point, such as a peer’s or a defined level of competency. Plausible explanations for the conflicting results point to issues of reference point selection and recalibration.

One reference point was a student’s undergraduate peer group. Valid results rely on this reference point remaining fixed for the duration of Induction. It is suggested that for many their reference point changed to be more in line with the current cohort — typically top level undergraduates.

A second reference point was an “industrial company environment”. Given that this was where careers were sought, it might be expected that they had a perception of this. In practice, many students had little exposure to industrial company environments (as determined from work experience data in the start questionnaire).

The final reference point tested was themselves at the beginning of the module. Again, this is problematic as it relies on their ability to keep this point fixed during the module. It was possible to ask more qualitative questions (e.g. is the skill more complex than they thought? Do they know how to improve?) This might form the basis of a reflective exercise towards the end of the module that could help students consolidate their skills learning and identify where they need to improve. These conclusions are similar to those of Knight and Yorke (2007).

Conclusions and further work

Using the SDF to analyse the DM gave the following insights:

• All conditions of the SDF were found to be present in the DM but with different degrees of judged adequacy. Eight of the 15 SDF components were considered fully adequate, five adequate and two less than adequate. Reflection aspects were weakest and exercise aspects the strongest.

• The exercises were judged fully adequate, providing a range of problem types and contexts as well as the level of challenge to produce mastery experiences.

• The knowledge “that” aspect of the skill description was good but there was limited provision of knowledge “how” and “judgement” aspects. These are often context-specific, so lend themselves to further explanation following an exercise.

• There was a lack of formal definition of problem solving skills.

• There was limited provision of formative feedback and opportunities for general feedback were limited by the class time allocated.

• A combination of poor student reflection skills, the use of a general reflective exercise and its timing resulted in superficial reflective outputs.

Since the DM was previously considered to be well developed this suggests that the SDF, even in its current early stage of development, could be a useful evaluation tool.

Student self-assessment proved problematic as they were not able to reliably identify changes to their levels of problem solving skills during Induction and conflicting ratings resulted. Problems arose with identifying appropriate reference points that remain fixed for the duration of Induction.

Whilst the SDF appears reasonable, there is a need for further work to enable future testing and refinement. This involves defining problem solving and developing effective assessment methods, as well as addressing the two further problems discovered that students had a poor understanding of skill development and poor reflective skills.

Three phases of investigation are proposed, defined by the research questions below and shown diagrammatically in Figure 4.

Phase 1

• What is the problem solving skill set?

• How can performance levels be defined and measured?

• How can students’ understanding of skills be developed?

Phase 2

• How can students assess their skill levels?

• How can reflective learning skills be developed?

• What reflective learning tools support skill development?

Phase 3

• How can the DM be improved?

• What implications are there from Phases 1 and 2 for the SDF?

Acknowledgements

The author wishes to thank Dr David Good, Dr Rick Mitchell and Dr Tim Minshall for their support and advice.

Figure 4 Phases of further work