The use of a Delphi consultation to explore the curriculum for sustainable development in engineering

Abstract

Developing a curriculum for sustainable development for engineers, even on a small scale, is a complex task. The University of Manchester has been undertaking a pilot project for an experiential, student-centred approach across engineering and science disciplines, described elsewhere. But that pilot also provided a springboard for the study described in this article. Group techniques for resolving complex issues have been around for half a century but mostly applied to questions of forecasting. Of these, the Delphi technique seemed to best fit the curriculum issues that we wished to explore. This article sets out some of the background to the choice of the Delphi approach, the questions that we sought to answer, the responses that we obtained and some discussion of the appropriateness of the approach to other issues of sustainable development and curriculum design.

Keywords:

• curriculum development
• Delphi technique
• sustainable development
• interdisciplinary
• operations research

1. Introduction

This article reports a study, part‐funded by the Higher Education Academy's Engineering Subject Centre, whose intent was to help develop guidance for Engineering schools to assist in designing modules or threads to embed sustainable development within the curriculum. The work stems partly from a belief that issues of sustainable development are best tackled on a broad front and not in a single‐discipline fashion (Tomkinson et al. 2005). The team had been engaged on a pilot study, part‐funded by the Royal Academy of Engineering (see Tomkinson et al. 2007), whose underlying philosophy towards education for sustainable development was one of encouraging collaboration, both between the various branches of engineering and also between engineers and other professions, in order to help towards the remediation of challenges that generally fall under the category of ‘wicked problems’ (cf. Rittel and Webber 1973). The pilot project proved a useful step along the road, but it is only a first step. The project team saw synergy between the RAEng project and this present project; in particular, the experience gained on the pilot module was used to shape the issues to be tested in the Delphi process. This was a very cost‐effective way of achieving something of great value across the subject area and also, potentially, for other disciplines in UK universities. The article attempts to set the Delphi technique (see e.g. Linstone and Turoff 1975) in context, and to demonstrate the reasons for applying this approach, as well as unfolding the results of the investigation and reflecting on the utility of the approach in other curriculum contexts.

2. The Delphi technique

Before looking at the application to the curriculum for sustainable development, it is necessary first to elucidate the approach applied in this study.

In the period following the Second World War, and before it became absorbed as a branch of applied mathematics, operations research was often conducted by small inter‐disciplinary teams. These would normally include engineers of a variety of disciplines and, in that context, military personnel, as well as mathematicians, statisticians and other scientists. However, these were not confined to the physical sciences and teams might include psychologists, biologists, economists or people from a range of social sciences. The approach of such teams was essentially a scientific paradigm:

1. problem identification;

2. data collection and collation

3. model building;

4. interpretation of results;

5. communication of the answer;

6. maintaining the solution.

Two issues initially arose from this approach. First, the issue of problem definition. Careful analysis and revisiting of the problem identification were necessary: ‘“Finding the problem is half the answer” is a maxim that often holds true in operational research. Sometimes it may be necessary to re‐define the problem after some initial analysis’ (Tomkinson 1972). Second, many engineers and scientists were used to there being an exact answer to a problem, however intractable. However, even in those days, there was an appreciation that some problems were complex and inexact, and Helmer and Rescher (1959), working on the USAF Project Rand, looked at approaches to dealing with some of these issues of ‘inexactness’. They identified two broad groups of approaches to inexact branches of the physical sciences, including engineering:

1. pseudo‐experimentation, including simulation and gaming;

2. expertise, including individual experts and expert groups.

The group approach to expertise was largely associated with the then‐emerging Delphi techniques. Dalkey and Helmer (1963) are reputed to have originated the term ‘Delphi’, though Project Rand had been looking at ways of improving the statistical treatment of individual opinions for some years before (see Kaplan et al. 1950). In this context the Delphi approach was largely seen as predictive and its use became more prevalent in forecasting and in market research. Since that time the Delphi technique has moved into a much wider range of applications and spawned a variety of variants (see, for example, Eskandari et al. 2007).

Dalkey (1969), also with Project Rand, suggests that ‘The Delphi technique is a method of eliciting and refining group judgements. The rationale for the procedures is primarily the age‐old adage “Two heads are better than one”, when the issue is one where exact knowledge is not available. The procedures have three features:

1. Anonymous response – opinions of members of the group are obtained by formal questionnaire.

2. Iteration and controlled feedback – interaction is affected by a systematic exercise conducted in several iterations, with carefully controlled feedback between rounds.

3. Statistical group response – the group opinion is defined as an appropriate aggregate of individual opinions on the final round.

These features are designed to minimise the biasing effects of dominant individuals, of irrelevant communications, and of group pressure towards conformity.’

In examining the efficacy of various methods, Dalkey also looks at the effects of group size. From his figures it can be inferred that the risk of error levels out with group sizes of 30 or more. Van de Ven and Delbecq (1974) suggest that Delphi has a number of advantages over other approaches in terms of the quality of answer, but takes much longer to achieve a consensus and has high administrative costs. Where a number of individual stakeholders can be brought together Van de Ven and Delbecq prefer the structured approach of the nominal group process (see Bartunek and Murningham 1984). Broadly speaking, in this latter technique a target group is brought together – in this case the target group would be a group of experts but the approach can also be used, for example, with groups of students in order to obtain course feedback. The target group may be sub‐divided, so that groups are kept to a size of 8 to ten, and, once the questions have been posed each group works on its own without external intervention. Each individual in the group independently comes up with ideas relating to the issue that the group members are addressing, these are then discussed and group members vote on the priority of the answers. In our case, the prospect of bringing together an appropriate group of experts in a nominal group was a daunting one and the Delphi approach was preferred. This technique was used in a parallel project, to garner student and facilitator feedback and is described in more detail in Tomkinson (2008b).

3. Delphi techniques and sustainable engineering

Although the aims of this project were to apply the technique to issues of curriculum development, the power of the Delphi approach to tackle issues of uncertainty has been used on a number of occasions in studies of sustainability. Appropriately enough, a study from Greece (Manoliadis et al. 2006) uses the Delphi approach in the context of sustainable construction, using an inter‐professional panel of engineers architects and economists. This looked at the drivers for sustainable construction in Greece and concluded that energy conservation was chief amongst these. An earlier Delphi study, also in the field of sustainable construction (Barrett et al. 1999), used two panels, drawn from across the construction industry – a UK one and an international one – to examine differences in results arising from national context. They concluded that results from the two panels were strikingly similar. A paper by Rouch and Stienecker at the 2007 Conference of the American Society for Engineering Education used a Delphi approach to look not only at alternative energy sources but also how this impacts on the curriculum (Rouch and Stienecker 2007).

4. The Delphi consultation project

A number of studies have been undertaken to use Delphi techniques in the process of curriculum design, principally in the field of health and medicine (e.g. McLeod et al. 2001), but also in business and management (e.g. Reeves and Jauch 1978) and, to a lesser extent, in computing, engineering (see Eskandari et al. 2007, Heywood 2005) and other fields. Some of these have focused on the needs of external stakeholders, rather than experts in the field and the use of Delphi techniques in these circumstances has to be questioned. Yrjänheikki and Takala (2001) take a comprehensive view of engineering education in Finland using a type of Delphi approach though with a small sample of key decision‐makers, both from universities and from industry. Some of the key points from that study included an inter‐disciplinary approach and an ability to deal with uncertainty and ambiguity as well as generic and specific technical skills.

This present project undertook a modified Delphi study to bring convergence of the views of experts from a range of engineering disciplines. This focused on a small number of related questions – for example ‘What is a working definition of “Sustainable Development” in the context of engineering education?’, within an overarching question of ‘How may students in a Faculty of Engineering be assisted to develop a set of competences which will enable them to contribute to Sustainable Development related aspects in their professional practice?’

The first step was to decide on the particular approach to be taken. Since the advent of Delphi techniques there have been developments in computing that can support the approach. An online approach was discounted because the software under consideration relied too much on the questions being predetermined by the project team and, hence, constrained the views of the experts. The use of proprietary software to analyse the results was also considered and tried to a limited extent. However with the relatively small numbers in practice, it proved easier to capture the results and feed them forward largely using manual methods: a larger number of respondents might have led to greater use of technology. It was also a major aim to aggregate a rich mixture of responses by using open‐ended questions, an approach that does not sit well with the scoring methods of some Delphi variants. However, in the final rounds it was necessary to consolidate the open‐ended responses and to use statistical approaches to reach convergence.

The second step was to identify a team of experts to participate in the consultation. The initial approaches were to a small core of contacts made during a parallel project sponsored by the Royal Academy of Engineering. These contacts were invited to participate and also to suggest others, in order to build a comprehensive list. All of those on the initial list were contacted to see if they were willing to participate and this reduced the number to 30, about the minimum number for a meaningful result. The consultation was then conducted in four phases:

	In Round I a suggested list of questions was circulated and participants asked both to appraise the wording and intentions of the questions and also to suggest others if they wished.
	In Round II the consultation invited open ended suggestions in relation to the overarching question and the wider list of considerations.
	In Round III the consultation invited participants to review the summary of suggestions from Round II, ranking them and adding further suggestions.
	Round IV reported back to the participants on the outcomes of the consultation. These could then play a central role at an in depth discussion among those interested in the further development of education in sustainable development in the field of Engineering.

Participants were not required to participate in every round or to complete every question in that round: each was able to contribute to the extent that he or she wished. This produced different numbers of responses for the different rounds and, indeed, different respondents. Provision had already been made for an additional consultation round – between III and IV – if necessary, but it was felt that a fair degree of convergence of responses had already been achieved by the end of Round III. At each stage, reminders were issued where appropriate.

5. The consultation questions

The initial round was primed with questions that the project team had asked themselves in the design of a pilot course unit on sustainable development for engineers and scientists (Tomkinson et al. 2007). The 14 initial questions were reviewed by the panel of experts and refined to produce the 10 questions that were used for subsequent rounds:

1. Please suggest what you would see as a useful working definition of education for sustainable development, in engineering.

2. SD calls for wider, interdisciplinary collaboration, but what would you regard as the main challenges of SD that are most frequently encountered in your branch of engineering?

3. What would you regard as the most important responsibilities of a recently graduated engineer in relation to the main challenges of SD?

4. What tasks would you expect a recently graduated engineer in your branch to be able to carry out in relation to these responsibilities?

5. What general or ‘transferable’ abilities and skills would you regard to be important in the successful execution of these tasks?

6. In your opinion how could the students be helped to develop the general abilities and skills?

7. (i) How could such educational activities be organised within the students' curriculum?(ii) How much curriculum time would need to be assigned to your suggestion in (i) above?

8. What evidence would satisfy you that the education in relation to SD had been effective?

9. What incentives might be offered to persuade students in your branch of engineering to participate in SD education?

10. How might academics be persuaded to develop SD courses or components or to include SD in their existing courses?

In the event, the second part of question 7 proved troublesome for respondents and the answers were often in incompatible formats.

6. Results of the consultation

The overall question was ‘How may students in a Faculty of Engineering be assisted to develop a set of competences which will enable them to contribute to SD related aspects in their professional practice?’ and this proved to be fairly complex and difficult to answer, but the answers to the individual sections are summarised below. The findings closely corresponded with the work carried out on a parallel project, sponsored by the Royal Academy of Engineering (Tomkinson 2008a) though in that case some pragmatic considerations had caused us to adopt approaches that fell a little short of the ideal.

Q 1. Definition. The first point of note was the widespread acceptance of the definition (originally that used in our RAEng project) of education for sustainable development, in the context of Engineering, as ‘Education for sustainable development aims to enable the professional engineer to participate with a leading contribution in decisions about the way we do things individually and collectively, both locally and globally, to meet the needs and aspirations of the present generation without compromising the ability of future generations to meet their own needs and aspirations.’

Q 2. Sustainable development challenges for engineers. The main challenges in this area were generally seen to be social and political, rather than technical – being socially and politically skilled as well as technically so. Also identified was a need to confront conservative ideas and reluctance to change.

Q 3. Sustainable development responsibilities for engineers: awareness raising and communication came out as key responsibilities for newly graduated professional engineers, with technical skills only in second place.

Q 4. Sustainable development tasks for engineers. In terms of tasks, the prime concerns were in evaluating complex problems (essentially the wicked problems identified in earlier work) and in systems modelling to try to cope with this complexity. Individual engineering disciplines identified specific tools and techniques that might be expected to be employed (though the number of respondents was too small to otherwise make distinctions) but another major task was seen as that of participating in change management.

Q 5. Sustainable development skills for engineers. Reflecting the weighting given to systems modelling, the dominant generic or transferable skill identified was that of dealing with complexity, but also being able to think ‘outside the box’. Other skills felt to be important were those of communicating and networking – particularly across disciplines.

Q 6 & 7. Designing education for sustainable development. In terms of curriculum, the feeling was largely that sustainable development should be embedded throughout and that student‐centred learning methods, in particular role play and case studies, were most appropriate. Because of the nature of our question, it was not easy to reach a consensus about what percentage of the curriculum this should occupy, but a figure of about 10% seemed to be the median. This is complicated by the conceptions of embedding: for some this suggests that sustainable development then inevitably becomes 100% of the curriculum, at least in terms of underlying philosophy. A significant number of respondents felt that sustainable development should form a compulsory element of the curriculum and others felt that making it compulsory was the only way to get students to take on board issues of sustainable development.

Q 8. Evaluating education for sustainable development. In terms of short‐term evaluation, many favoured the final year project as the way to demonstrate that the lessons of sustainability literacy had been taken on board, but with recognition of the wider perspectives, not just the narrow technical ones. Longer term evaluation would need to review the behaviour and performance of the graduates in professional practice but this appears a more diffuse target.

Q 9 & 10. Embedding education for sustainable development. There was a strong feeling that the engineering bodies should be solidly behind the idea of sustainability literacy if it was to achieve wide recognition by academics in engineering. It was also felt that higher education institutions needed to embrace the ideal: to include this challenge in their mission statements.

7. Discussion

This was a stimulating if, at times, difficult exercise in using the Delphi technique. Perhaps the biggest constraint was the relatively poor response rate. The number of respondents was appropriate to this technique and in other circumstances the response rate would be regarded as good. However, some of those who initially agreed to participate did not contribute at all or contributed only marginally. Those who did take part produced some interesting, and often challenging, responses that led to the conclusion that the exercise had been worthwhile. The key in this type of exercise is in getting the initial set of questions right, so it was vital to treat the first round as a consultation on what questions should be asked, rather than on seeking immediate responses. The Delphi approach should be readily transferable to other curriculum questions, e.g. in the teaching of ethics to engineers. In this case four rounds were formulated but this could have been increased in number if reasonable convergence had not been attained by the end of the fourth round. This, inevitably, has to be a post hoc judgement in the light of responses received.

The approach was perhaps unusual in the choice of experts. In many previous cases in the use of Delphi techniques, the panel has comprised external stakeholders: employers or professional bodies. In this case the panel comprised academic leaders in the field together with Visiting Professors who have been able to span the gulf between industry and academe. Again, manual methods were largely used, primarily because of a desire to allow open‐ended answers, though email was used extensively to communicate responses. The use of open‐ended questions enabled deeper insights to be gained but at the expense of more efficient methods of analysis.

The Delphi approach is capable of more extensive use in curriculum design, and indeed in wider issues of sustainability, possibly incorporating a broader range of stakeholders in order to identify differences of emphasis as well as convergence of views. It can be a protracted approach and one that should not be used lightly if there are opportunities for face‐to‐face methods, such as the nominal group process.

The overall results suggest that the complex issues of sustainable development need to be approached in a systemic fashion that is not appropriate to traditional ‘bolt‐on’ teaching. The approach has to be a broad one, ensuring that social, economic and political aspects are included as well as the more technical ones. Sustainable development has to be embedded in the curriculum as far as is possible and learning has to be approached in a student‐centred, experiential fashion with an emphasis on generic transferable skills as much as on specific knowledge. Above all, sustainable development as part of engineering programmes will only be sustainable if it has the full and enthusiastic support of professional bodies as well as of universities.

8. Conclusions

The Delphi consultation has brought to light some consensus about the design of curricula for sustainable development within engineering courses. Much, but not all, of this is in line with initial expectations. This study should form a useful basis for engineering departments to explore their sustainable development curricula.

The process of using the Delphi consultation was an interesting and challenging one. Online methods were considered but none of them ideally suited this particular approach. This meant that the majority of responses were received by email and analysed manually. As many of the answers were complex, this meant that the collation of responses was equally, if not more, complex. Ninety experts in the field were originally approached, of whom only 30 agreed to participate. This is still a reasonable number for this type of exercise but does point to the need to have a wide field on which to draw. In its way the exercise was a successful one and the approach could readily be applied to other areas of curriculum design, not just in sustainable development or even just in engineering.

The key findings for the development of sustainable development curricula for engineers were that: social and political issues have to be considered, as much as technical ones; communication and advocacy skills need to be developed as much as technical ones; dealing with complexity and systems approaches are core; and student‐centred, experiential approaches to learning are vital.

Acknowledgements

The authors are grateful to the Higher Education Academy Engineering Subject Centre for ‘mini‐project’ funding to support this study; to our Research Assistant, Alvin Lawson, who collated much of the information and to Anna Christie who provided effective secretarial back‐up and support.