Underlying success in open‐ended investigations in science: using qualitative comparative analysis to identify necessary and sufficient conditions

Abstract

Both substantive (i.e. factual knowledge, concepts, laws and theories) and procedural knowledge (understanding and applying concepts such as reliability and validity, measurement and calibration, data collection, measurement error, the ability to interpret evidence and the like) are involved in carrying out an open‐ended science investigation. There is some debate as to whether procedural understanding is of little importance compared to substantive understanding or whether – and this is the view we take – procedural ideas can and should be taught explicitly. We present here findings from a study of undergraduate students who took a module which specifically taught procedural ideas. We employ an innovative method, Charles Ragin’s Qualitative Comparative Analysis (QCA), which involves the analysis of necessary and sufficient conditions and conjunctions of causes. Findings from a comparison of the students’ performance before and after the teaching and from QCA imply that procedural understanding was indeed a necessary condition for carrying out an open‐ended investigation. It was also sufficient when combined with either substantive understanding, prior attainment or both.

Keywords:

• Qualitative Comparative Analysis (QCA)
• open‐ended investigations
• ideas about evidence
• procedural knowledge
• undergraduate students

Introduction

This paper investigates the respective roles of substantive and procedural understanding with regard to students’ ability to carry out an open‐ended science investigation. The role of procedural understanding is often neglected in science education, or rather, it is tacitly assumed that this develops with substantive understanding and with increasing practice of carrying out investigations. We take the view that substantive and procedural understanding are two connected but distinct types of knowledge and that it is possible, and maybe even necessary, to teach procedural ideas separately from substantive content. The research presented in this paper is part of a larger research programme concerned with the role of procedural understanding in investigative work and emerging ideas of scientific literacy. We investigate whether these ideas can be taught and what impact this has on various outcomes: the understanding of the ideas themselves, the ability to carry out open‐ended investigations and the use of those ideas in developing an empowering scientific literacy. This paper concerns itself with open‐ended investigations.

In addition to a descriptive analysis of the data, we are using an innovative method, Qualitative Comparative Analysis (QCA), which was developed by Charles Ragin (Ragin 1987, 2000).

We shall start by giving an overview of the theoretical background, followed by a description of the sample and instruments used. The results are presented in two parts: we give some descriptive results which allow us to draw some tentative conclusions but which also pose new questions. Then, after an introduction to the method, QCA, the results of a QCA analysis are presented which attempt to answer those questions. We end with a discussion of the findings.

Theoretical background and rationale for this research

Our view is constructivist in that understanding requires the learner to construct meaning from ideas. In science the traditional curriculum emphasis in classroom teaching has largely been on the content knowledge ideas of the subject matter, such as mechanics in physics or genetics in biology, to name but two. In other words, we are dealing with the familiar factual knowledge, concepts, laws and theories of science. This is traditionally known as substantive or conceptual understanding.

The facts, concepts, laws and theories that contribute to this substantive understanding are, of course, themselves supported by empirical evidence or are subject to investigation. Science, therefore, encompasses more than just an understanding of the familiar factual knowledge, concepts, laws and theories. Traditional substantive understanding alone is not sufficient to describe the ideas of science. Another key curriculum component is concerned with the procedures of science. Yet this component is often downplayed in curriculum documents, at least in terms of the number of pages devoted to its specification. The procedural component is often represented in curricula in terms of behavioural objectives (Duschl, Schweingruber, and Shouse 2006) and is implicit in text books (Roberts and Gott 2000); a performance emphasis rather than emphasising understanding per se.

The term ‘procedural knowledge’ is found in a number of areas (Star 2000), not just in science, and implies ‘knowing how to proceed’; in effect, in science, a synthesis of manual skills, ideas about evidence, tacit understanding from doing (Polanyi 1966) and the substantive content knowledge ideas relevant to the context. In science education in the UK and in our work the term procedural understanding has been used to describe the understanding of ideas about evidence, which underpin an understanding of how to proceed. The term has been used to distinguish ideas about evidence from other more traditional substantive ideas and we have argued that a lack of these ideas about evidence prevents students from exhibiting an understanding of how to proceed.

These ideas can be applied and synthesised in open‐ended investigations, together, of course, with the traditional substantive ideas of science. (We also consider them to be important in empowered forms of scientific literacy, to enable students to engage with scientists’ claims and scientific argumentation (Gott and Duggan 2007; Roberts and Gott 2007; Tytler, Duggan, and Gott 2001a, 2001b) but that is not the focus of this paper.) As Buffler, Allie, and Lubben (2001) argued:

Procedural knowledge (in the context of experimental work) will inform decisions, for example, when planning experimental investigations, processing data and using data to support conclusions. (1137)

We have referred to these ideas about evidence as ‘the thinking behind the doing’ (not to be confused with meta‐cognitive notions of ‘thinking about one’s own thinking’) and have created a tentative list numbering some 80 or so of them which we have called the concepts of evidence (http://www.dur.ac.uk/rosalyn.roberts/Evidence/cofev.htm). They serve, we argue, as a domain specification of ideas necessary for procedural understanding. From this perspective, the procedural component of a curriculum consists of ideas (in effect, a subset of substantive ideas) that form a knowledge‐base of evidence that can be explicitly taught and assessed, in a similar way to the more traditional substantive elements in the curriculum.

The concepts of evidence include ideas about the uncertainty of data (as taught and researched by Buffler, Allie, and Lubben 2001). They also include ideas important to understanding measurement and data processing, presentation and analysis which may be considered to be part of the mathematics curriculum but which are essential for understanding evidence.

The concepts of evidence are a toolkit of ideas integral to the planning and carrying out of practical investigations with understanding (rather than as a routinised procedure; cf. Roberts and Gott 2003). We have argued that they are necessary but not sufficient for an investigation: the empirical case study reported here explores this.

The procedural component of the curriculum has been conceptualised and articulated differently in the literature by researchers with different research agendas. We will briefly characterise some of these to position the rationale for this research.

Other perspectives

Piagetian schema

We start with extensive research that has been strongly influenced by Piagetian psychology (Inhelder and Piaget 1958; see for instance Adey 1992; Klahr and Nigam 2004; Kuhn, Amsel, and O’Loughlin 1988; Schauble 1996; Toth, Klahr, and Chen 2000). The development of ‘higher order thinking skills’ is thought to be important in helping students better understand science; the development of strategies about control of variables and causal relationships, for example. The focus of such work is on the development of appropriate schemas in pupils so that they can become formal reasoners.

Psychology‐informed research has shown that students can be explicitly taught to develop this understanding; it is not just a ‘skill’ that develops only with practice (cf. the ‘skills’ perspective developed below). Klahr and Nigam (2004) demonstrated the effects of explicit teaching, which seemed to develop understanding which lasted much longer than in students who were just left to practice. Others have shown similar results (see for instance Chen and Klahr 1999; Kuhn and Dean 2005; Shayer and Adey 1992; Toth, Klahr, and Chen 2000).

However, in interventions from this psychological perspective, what is taught, such as the control of variables strategy, is seen not so much as part of science per se, thereby legitimising it within curriculum structures, but as an ‘additional’ psychological component necessary for developing understanding in science (Jones and Gott 1998). In the UK, for instance, in the Piagetian‐based CASE (cognitive acceleration through science education) programme’s interventions, what was taught was selected from the psychological perspective of developing formal reasoning (Shayer and Adey 1992). The interventions, which arguably addressed some of the procedural components of a science curriculum, were seen more as ways of developing underlying cognitive structures so that the learner can better understand substantive science. Jones and Gott (1998) contrasted this with the ‘understanding ideas about evidence’ perspective which underpins this research and is expanded below: the crucial difference is that from the ‘understanding ideas about evidence’ perspective, the ideas of evidence are perceived to be important elements of science, and as such can be selected to construct curricula, can be taught and can be assessed. Thus, although research from a Piagetian perspective has much to inform the procedural component in science education and there is overlap in what is taught, the ‘understanding ideas about evidence’ perspective focuses on ideas integral to science and covers, as a consequence, more ideas from science than does the psychology‐focused development of schemas.

Inquiry

The word ‘inquiry’ is prominent in literature about the procedural component. For instance, an inquiry (or inquiry task) can be similar to an open‐ended investigation, as we have described above. Inquiry is also used as a term for more psychology‐focused processes akin to ‘scientific thinking’. Inquiry‐based curricula employ an inquiry‐based teaching and learning approach. But even as a pedagogical approach the word can imply different things; from simple ‘discovery learning’ (often contrasted with direct instruction) to planned learning progressions which include more explicit teaching in the sequence of lessons (Duschl, Schweingruber, and Shouse 2006). In the UK the inquiry approach was typified by Nuffield curricula (Jenkins 1979) and emphasised learning substantive ideas through practical work, and the US view is not dissimilar (see for instance Duschl, Schweingruber, and Shouse 2006).

Within nearly all the uses of the term inquiry is the view, implicit or explicit, that both the substantive and procedural components of the curriculum are inseparable; the two components are addressed together in inquiry, with the resultant emphasis in practice being largely on the substantive ideas. This is in contrast to the argument we develop in this paper where we have clearly distinguished the substantive and procedural components of the curriculum to structure the discussion. Inevitably there is an overlap between our work and research framed in terms of inquiry, but the focus is different.

The nature of science

Research and curriculum developments concerning epistemology and the nature of science (see for instance Hart et al. 2000; Osborne et al. 2003; Qualifications and Curriculum Authority 2004; Sandoval 2005) address the ways in which science, and more particularly scientists, work. The emphasis is largely epistemological and sociological. We are not concerned with these emphases here, although there is a lot of common ground. The procedural component of the curriculum is important in the nature of science but much of the research comes from a more philosophical perspective. The ideas about evidence (expanded in the section ‘What type of understanding’), we would argue, are a subset of the ideas involved in such an understanding of the nature of science.

Process skills

The skills perspective is characterised by performance, often termed ‘process skills’. The main characteristic of such a perspective is that the procedural component is to be learned by repeated exposure to practical work. The procedural component is largely implicit in teaching and any guidance given to students is through a simple exemplification of the process. An early version of this is typified by the Science: A process approach developed from work by Gagné which identified isolated ‘process skills’ (American Association for the Advancement of Science 1967) which was followed by others, including, in the UK, Warwick Process science (Screen 1986) which emphasised ‘process skills’ such as observing, classifying and interpreting.

Research has shown that ‘children failed to develop meaningful understanding under science‐as‐process instructional programs…but its legacy persists in both policy and practice’ (Duschl, Schweingruber, and Shouse 2006, Ch. 8, 2–3). Elements of that legacy can still be seen in curricula that either have procedural components specified as behavioural objectives, since these may be translated into classroom practice and assessment as just ‘doing’, or in curricula that emphasise using investigations as a pedagogical approach, a way of teaching, for mainly substantive understanding. In such pedagogical approaches the ‘doing’ of science is considered to be sufficient to meet the procedural component of the curriculum; students ‘discover’ the procedural element with practice.

What type of understanding is important?

The evidence with respect to the importance of substantive and procedural understanding in conducting an investigation is not clear cut. Previous research (Erickson et al. 1992; Gott and Murphy 1987; Millar et al. 1994; Ryder and Leach 1999) pointed to both playing a role. Schauble (1996) found that, from a developmental point of view, children’s ability to investigate in ‘knowledge‐rich’, substantively demanding, contexts developed in line with both their understanding about the procedures of science and their substantive knowledge. These changes appear to bootstrap each other, so that appropriate knowledge supports the selection of appropriate experimentation strategies, and systematic and valid experimentation strategies support the development of more accurate and complete knowledge. Gotwals and Songer (2006) used a multidimensional modelling based on Rasch that identified three dimensions in students’ ‘inquiry’: content (substantive) knowledge, creating scientific explanations (which includes aspects such as identifying claims and evidence for them and associated reasoning), and interpreting data (which includes reading a table and graph and drawing conclusions from it). However, none of these findings give any insight into whether either or both substantive or procedural understandings are necessary or sufficient conditions for being able to carry out an investigation.

It is also pertinent to note that this research, along with most other work in this area, is based on post hoc surveys rather than lengthy interventions of the kind we describe here. Within the English school curriculum, it is not straightforward to disentangle the role of substantive and procedural understanding since there is little time devoted to pupils’ conducting their own experiments (House of Commons Science and Technology Committee 2002; Roberts and Gott 2004).

Where students do carry out investigations, these are often fairly standardised and repetitive rather than open‐ended (House of Commons Science and Technology Committee 2002), so that rote learning is required rather than procedural understanding. Open‐ended investigations, by contrast, are those in which students are unaware of any ‘correct’ answer, where there are many different routes to a valid solution and where different sources of uncertainty lead to variations in repeated data so that students reflect and modify their practice in the light of the evidence they have collected. The evidence produced, then, is messy rather than the necessarily cleaned up version common in practical work contrived to illustrate ideas to students. Such open‐ended investigations play an important role in ‘inquiry’‐based approaches to science curricula across the world (Abd‐El‐Khalick et al. 2004). They reflect ‘science as practice’ and are important for understanding not just the practice of scientific investigation but also provide a context for understanding why science needs empirical evidence (Duschl, Schweingruber, and Shouse 2006).

In our research we have taken the view that procedural understanding is a necessary ingredient in an open‐ended investigation, regardless of the kind and amount of substantive understanding required, but is it? And if it is necessary, is it sufficient, in an investigation where there is not a high degree of substantive understanding required? Implicit in the commonly accepted view is that the substantive ideas of science are the necessary conditions to investigate. Judging from the structure of curriculum documents which devote pages to the substantive ideas and relegate ideas associated with investigations to a few ‘process’ words, they also seem to be the sufficient condition to investigate.

Our view here, which we could label an ‘understanding (of procedural ideas) approach’ is that ideas about evidence can be explicitly taught and assessed. This allows us, in this case study, to address whether a procedural understanding is a necessary and/or sufficient condition to investigate. If the underlying ideas about evidence (of reliability, validity, error and uncertainty, etc.) are taught explicitly in the most efficient way, which may or may not involve practical work, then we will arrive at a more efficient curriculum. We here, therefore, developed such a curriculum, in a nine‐week undergraduate module (some 25 hours plus additional directed time) (for details see Roberts and Gott 2007). The advantages, were such an approach seen to be effective, are coverage and efficiency. It is difficult to see, how a series of routinised school‐based practicals, however well planned, necessarily limited in number by exigencies of time, can allow for coverage of all the underlying procedural ideas given the idiosyncrasy of the individual student in their approach to such open‐ended tasks, quite apart from the fact that school‐based practicals tend not to be open‐ended.

More details on the underlying theoretical rationale can be found at http://www.dur.ac.uk/education/research/current_research/maths/msm/understanding_scientific_evidence/.

We are aware that other factors, such as motivation, response to assessment, response to a teacher, etc. may also affect a student’s performance on an investigation. However, we are interested in seeing if the key curriculum components, substantive and procedural ideas, are necessary or sufficient conditions for obtaining at least a basic level of competence in this investigation. To test this, we will make use of the fact that the students were taught substantive and procedural ideas at different times and that we have measures of their performance on the same investigation between the teaching of the two, as well as after. In addition, we conduct an analysis of the necessary and sufficient conditions, using QCA. This is based on the measures we took after the teaching.

Sample and instruments

We use a sample of undergraduate students who carried out such an open‐ended investigation and wrote up their procedure and findings. We do not make any claims as to the generalisability of our findings, rather, we present this as a case study which may serve to give an indication of the respective roles of substantive and procedural understanding in this particular investigation and also to illustrate a possible approach to the problem of identifying necessary and sufficient conditions associated with successfully conducting an investigation.

The sample consisted of 72 students ² in the second year of a B.A.Ed. course in the UK. They are reasonably typical of undergraduate primary education courses, being mainly female with several mature students. They are described in Table 1.

Table 1. The sample.

Age	Mean 22.9 years, 26.4% over 23
Sex	62 (86.1%) female, 10 (13.9%) male
Social background	50 (69.43%) service class (i.e. professional and management)
	22 (30.6%) others (i.e. lower social class)
GCSE	39 (54.2%) at least B or equivalent in science
Access to university	48 (66.7%) at least three A‐levels ^1
	20 (27.7%) other access route
	4 (5.6%) no data

Teaching programme

During the first year of their course, the students took a science module of 22 weeks’ duration which teaches ideas necessary for understanding fundamental ideas in chemistry (substance and chemical change) and physics (force), approximately 11 weeks of each. This module spends far longer on force and motion (substantive ideas relevant to the investigation) than is traditional in schools as our experience is that little real understanding of the ideas remains by the time they reach university, even amongst those with science A‐levels. The course is specifically targeted at developing a deep understanding of a limited number of basic ideas. During the second year, they were taught a module on procedural ideas. There was minimal substantive content involved and the context was kept as simple as possible in order to be able to concentrate on the procedural ideas. The course covered such things as validity and reliability, experimental design, measuring instruments, uncertainty and variation in repeated readings, descriptive statistics and presenting data in tables and graphs. Although there were seminar sessions which included practical work, the students did not have the opportunity of carrying out a complete investigation. Therefore, there was little opportunity for ‘skills’ development.

We should note here that the students were taught and assessed by one of us. The research project had been granted approval by the university’s ethics committee, and the students knew that participation in the research was entirely voluntary and that their data were to be treated confidentially. Nevertheless, we are aware that this situation is not ideal and we cannot rule out that some of them felt compelled to take part in the research because of the overlap with the teaching.

Instruments

Prior attainment: GCSE

The General Certificate of Secondary Education (GCSE) is an examination taken at the age of 16, at the end of compulsory schooling in England. Pass marks range from A* to G, and five or more A*–C grades are usually required in order to continue to A‐level, which offers the opportunity of going on to university.

We have used here the data on students’ GCSE (or equivalent) grades in science as a measure of prior attainment in science.

Science module exam

This was taken approximately four months prior to the start of the evidence module, at the end of the first year science module. The examination attempted to measure understanding rather than depending on the complex recall associated with GCSEs (House of Commons Science and Technology Committee 2002). As might be expected, correlations between university exam marks and GCSE scores are relatively low, as they are separated in time by some distance. The exam did not contain specific questions on friction which forms part of the substantive background for the investigation the students carried out. However, there were questions on forces and motion which also relate to the investigation. The subset of questions on force and motion was highly correlated with the other items (0.67). As we are interested in having a general measure of scientific ability and understanding, and also in view of future analyses with the same group of students, we have decided to use the score for the exam as a whole as a proxy for substantive understanding.

Evidence test

Students’ understanding of the procedural ideas was assessed by means of a written test. The test targets such things as measurement, experimental design and data analysis. Written probes to assess aspects of procedural knowledge have been used by other researchers (see for instance Buffler, Allie, and Lubben 2001; Germann and Aram 1996; Germann, Aram, and Burke 1996; Gotwals and Songer 2006; Lubben and Millar 1996). We have used this test elsewhere (Gott and Roberts 2004; Roberts and Gott 2003, 2006) and it has been refined as a consequence. The pre‐test was taken immediately prior to the teaching of the module on procedural ideas. It took up to an hour for some students to complete, comprising some 17 items spanning the concepts of evidence. Although set within everyday biology, chemistry or physics contexts, the questions required minimal understanding of the substantive ideas. At the end of the module students were asked to complete a post‐test. This was a subset of the pre‐test (time did not allow for a full repeat of the pre‐test), with the items chosen on the basis of a combination of factors.

•	Facility and discrimination: avoiding items with high facilities which would be likely to run out of headroom.
•	Richness: items which, on the pre‐test, gave interesting responses.
•	Spread: across the various concepts of evidence.

The pre‐ and post‐test were scored using the same coding system. A sample of 10 tests was marked independently by the authors and inter‐marker checks showed a consistent application of the mark scheme. In the present paper, we have used the post‐test data only for the analyses since we are interested in the procedural understanding the students had at their disposal after having been taught the module, i.e. at the time of conducting the post‐investigation, and not in the improvement as a result of the module. This test and its coding scheme can be found in Gott, Roberts, and Glaesser (2007).

Investigation

The students were required to carry out the same investigation twice: at the beginning (pre‐) and the end (post‐investigation) of the module on procedural ideas. Their written accounts were used as the basis for assessment. This is considered an acceptable surrogate for an observation of the investigation (Baxter et al. 1992; Gott and Murphy 1987; Welford, Harlen, and Schofield 1985). While Haigh (1999) found that students’ written accounts did not necessarily reflect the subtleties of students’ ideas we encouraged students to write about ‘the thinking behind their doing’ rather than write a more formal ‘apparatus, method, results, conclusion’ account. Martin (1993) argued that a research report is an opportunity to persuade the reader of the reliability of the scientific claim, and we explicitly encouraged the students to justify their decisions and reasoning in a coherent account. This approach to writing has been adopted successfully by Toh and Woolnough (1994).

The task was to find out ‘How much do different surfaces affect how easy it is for a shoe to slide?’

It was selected for this analysis to determine whether substantive and/or procedural ideas are necessary or sufficient conditions for conducting the investigation. Although the investigation is procedurally quite simple, with an independent variable with categoric values, in a context where variables are relatively easy to identify and manipulate and with apparatus with which the students were relatively familiar (newton meters, pulleys), operationalisation of many of the procedural ideas is informed by substantive knowledge. It could be tackled with very little explicit use of substantive ideas; some students did not refer explicitly to the substantive ideas of force or friction and measured ‘slide’ by elevating a ramp until the shoe moved. However, the problem could have called on an interaction of both substantive and procedural ideas. For instance, identification of factors that may affect the amount of slide and their treatment as control variables could be helped by an understanding of force, as would the measurement of the dependent variable, ‘slide’, since a decision was required as to whether, and how to, measure static or sliding friction. We argue here that students with an understanding of force will be advantaged regardless of the method they adopt.

In the marking of the written accounts of the investigations, credit was given to students’ accounts where the concepts of evidence were explicitly and correctly applied. No credit was given to substantive understanding as such, only where it informed the procedure, i.e. in the selection and operationalisation of the variables, etc. That is, we gave no credit to explanations of friction which were not the focus of the task.

Table 2 gives an overview of the measures used and the students’ performance. With the exception of GCSE, the measures are percentages scales, i.e. they range from 0 to 100. The official pass mark is 40. In the case of GCSE, the numeric value was obtained by allocating 10 to the A* grade, 9 to the A grade, etc.

Table 2. Students’ performance on the measures employed (n = 72).

	GCSE	Science	Evidence pre‐test	Evidence post‐test	Pre‐investigation	Post‐investigation
Mean ± SD	7.8 ± 1.0	60.9 ± 13.2	55.5 ± 15.7	67.7 ± 14.5	20.7 ± 8.3	53.7 ± 19.2
Minimum	6	27	9.4	2.4	1	0
Maximum	10	91	82.9	92.9	40	96
1st quartile	7.0	51.0	47.7	62.9	16.0	43.0
Median	8.0	61.0	59.7	69.4	20.0	53.0
3rd quartile	9.0	71.0	65.9	75.9	25.0	65.8

The sequence, then, was as follows. During their first year, the students were taught the science module which was followed by the science exam. Four months later, before any teaching on the module on procedural ideas had started, they took the pre‐evidence test and pre‐investigation. They were then taught the module on procedural ideas and subsequently took the post‐evidence test and post‐investigation. As the module lasted nine weeks, this was the time gap between the pre‐ and post‐tests.

Model to be tested

We test the assumption that the ability to perform the investigation is a function of prior attainment, substantive understanding and procedural understanding. We attempt to determine the extent to which each of these is necessary and/or sufficient. To do so, we first consider the improvement through the teaching. This is followed by an analysis of necessary and sufficient conditions, employing Qualitative Comparative Analysis (QCA), which is explained below.

We do not test the idea that skills acquisition through practice is sufficient. The fact that they had little opportunity to practice such skills means that we do test, by default, the idea that such practice is a necessary condition.

Descriptive results

Pre‐ and post‐evidence test

The evidence test is the written measure of understanding of the procedural ideas taught in the module. On this measure, the students improved considerably after teaching: the mean on the pre‐evidence test was 55.5, with a standard deviation of 15.7, while on the post‐evidence test it was 67.7, with a standard deviation of 14.5. The effect size (Cohen’s d) is 0.8.

Pre‐ and post‐investigation

The improvement was even more marked on this measure. The mean on the pre‐investigation was 20.7, with a standard deviation of 8.3, and on the post‐investigation the mean was 53.7, with standard deviation 19.2. The effect size here was 2.2. It is worth remembering, at this point, that student performance on the pre‐investigation was weak (see Table 2). It was characterised in some cases by mild panic, followed by undirected and uncritical collection of data. We describe this in more detail in Roberts (forthcoming).

Preliminary conclusions

At the time of conducting the pre‐investigation, the students had already been taught the science module. This provided them with the substantive knowledge of forces which was relevant for this particular investigation. Although, of course, there was some variation in the level of that understanding, the students on the whole had understood the main ideas (see Table 2). If substantive knowledge was a sufficient condition for conducting the investigation, they should have been able to conduct the pre‐investigation with a reasonable level of competence. However, they did quite badly. Clearly, substantive understanding alone was not sufficient for reaching even a basic level of competence in the investigation.

They were then taught procedural ideas, but no more substantive content. After this, they were able to carry out the investigation with a reasonable level of competence. This leads us to conclude that an understanding of procedural ideas is a necessary condition for conducting the investigation. We recall that little opportunity for developing proficiency through practice was available, so this cannot have been a deciding factor. Nor did they receive individual feedback on their performance on the pre‐investigation. They did, however, receive feedback on some generic points which was similar in content to the usual teaching of the module on procedural ideas.

On the basis of these findings, though, we cannot be sure whether the understanding of procedural ideas is also a sufficient condition in addition to being a necessary one, and if so, whether it is sufficient on its own, at least among the variables we have access to and that may also have some effect on the outcome. From experience, we do not expect it to be sufficient on its own but rather that substantive knowledge and cognitive ability should also play a part. Since substantive understanding was not sufficient by itself, it might be that both the substantive understanding from the science module and the procedural understanding from the relevant module are jointly important, together with prior attainment (as the best proxy we have for ability).

We could use some form of regression analysis to try and disentangle the respective roles of substantive and procedural understanding. Our findings so far, however, point to the possibility that the joint presence of the two is required. Unless we include interaction effects, this would be difficult to pick up with a regression analysis. Interaction effects are not easy to interpret, but leaving them out of an analysis may not do justice to reality. In addition, our sample is not the ideal size or type for a regression analysis. We have therefore decided to use an alternative method, Charles Ragin’s QCA.

QCA explained

QCA will need to be explained in some detail as it is not likely to be familiar to most readers.

Traditionally, quantitative social scientists use various forms of regression analysis in their research. The rationale behind this approach is the identification of net effects of independent variables on a given dependent variable. It attempts to mirror the nature of some experiments. The underlying model is a linear and additive one, i.e. the effects of the different independent variables are thought to be linearly related to the outcome and to act independently of each other.

This approach has been criticised by various authors for a number of reasons (see for example Abbott 1988; Lieberson 1985; Ragin 1987, 2000, 2006a). ‘Net effects thinking’, as Ragin (2006a) called it, does not adequately reflect social reality. In the social world, the effects of different variables cannot easily be separated from each other. Instead, it is quite common that the effect of one variable depends on the presence or absence of another, or it might only be relevant for one group, but not for another (see also Ragin 1987).

A related issue is the existence of multiple causation. Alternative pathways of varying or equal importance leading to a particular outcome may exist, with neither of them being sufficient or necessary. This situation is difficult to pick up with a regression analysis.

The analysis of sufficient and necessary conditions

As noted above, the QCA approach is an alternative to regression‐based methods. It attempts to identify causal configurations of conditions which are associated with the outcome. The focus is on the case rather than on the relations between variables. Conjunctions of causes and the existence of alternative causal pathways are taken into account. In accordance with this alternative causal model, Charles Ragin developed a method ³ which reflects this configurational thinking about causation (Ragin 1987, 2000).

Originally, QCA was developed for use with small n datasets. Ragin is a political scientist, and one of the research interests in political sciences is the analysis of differences between countries. Researchers in this field often have deep knowledge of the cases, i.e. countries, under study, but only a limited number of cases is available. This situation does not lend itself to traditional quantitative research methods, but QCA is well suited to it, since none of the requirements of regression analysis apply. Although the use of QCA with large n datasets has been explored (see for example Cooper 2005, 2006; Cooper and Glaesser 2007, 2008; Ragin 2003), it is a particularly useful tool for analysing small to medium sized n datasets such as ours. A general introduction to the method and examples of QCA being used to compare countries has been given by Ragin (1987).

Applying QCA involves the identification of necessary and sufficient conditions for a given outcome. The underlying principle is a set theoretic approach which involves determining subset relations. Consider Table 3 where O is the outcome and A is some condition associated with it. This relationship is illustrated by the Venn diagram in the left hand panel of Figure 1.

Figure 1 Sufficiency.

Table 3. Simple implication: sufficiency (‘if A, then O’ expressed in terms of inclusion, sufficient relationship). ⁵

	A	Not A
O	Present	Possible
Not O	Excluded	Possible

In logical terms, condition A is sufficient for outcome O, i.e. whenever A occurs, O will occur, as can be seen from the left‐hand column of Table 3. This does not mean that A is necessary for O to occur, there may well be other conditions ⁴ associated with O, as indicated by the right‐hand column of Table 3. In set relational terms, A constitutes a subset of O.

In the real world, relations are less than perfect. Therefore, it is necessary to consider instances of weaker implication, i.e. the relative frequencies of cases rather than simple presence or absence (Cooper 2006). A diagrammatic representation is given in the right‐hand panel of Figure 1.

This case of near sufficiency can be illustrated by adding some numbers to Table 3 (see Table 4). They represent numbers of cases with the relevant conditions. Out of all the cases with condition A, 90% experience O. This high percentage indicates that A is ‘nearly always sufficient’ to obtain O. This 90% in our example can be referred to as the degree of consistency with sufficiency with which O is obtained given A. This table can also be used to introduce the concept of explanatory coverage, which plays a role in set theoretic analysis analogous to that of variance explained in a regression analysis. In Table 4, for example, 240 cases achieve the outcome. Of these, 90 have condition A. Coverage in this case is the simple proportion of 90 divided by 240, 0.375. It can be seen that this index records the proportion of the outcome set that is covered by the explanatory set. Clearly, in this case, there must be other causal paths to the outcome. It is not necessary to have condition A in order to achieve the outcome, a fact reflected in the low coverage figure.

Table 4. Weaker implication: sufficiency (weaker implication, sufficient relationship: ‘if A, then [nearly always] O’).

	A	Not A
O	90	150
Not O	10	50

We now need to introduce necessity at this point. Consider a variation of Table 3 (Table 5). Here, A constitutes a necessary condition for O, i.e. without A, O cannot occur. In set theoretic terms, A is a superset of O (or, equivalently, O is a subset of A). This is illustrated in the left‐hand panel of Figure 2. ⁶

Figure 2 Necessity.

Table 5. Simple implication: necessity (‘O only if A’, necessary relationship).

	A	Not A
O	Present	Excluded
Not O	Possible	Possible

Again, we have to consider the possibility of less than perfect necessity, illustrated through the right‐hand panel of Figure 2. Adding some numbers to illustrate the necessity relation, we get Table 6.

Table 6. Weaker implication: necessity (weaker implication, necessary relationship: ‘O only if A’).

	A	Not A
O	90	10
Not O	150	50

Here, nearly all of the cases with the outcome O have experienced A. This points to A being a (nearly always) necessary condition for O. The proportion of cases with O who have previously experienced A is 0.90. This 0.90 is a measure of the consistency of these data with a relationship of necessity. ⁷

Note that this does not make any claims about the sufficiency of A: from the left‐hand column, it is clear that even if A is present, O doesn’t usually occur, which points to the possible need for further conditions which have to be present in order for O to usually appear.

So far, in this section we have only considered the case of two variables, one independent and one dependent. However, in the social sciences we usually find more than just one independent variable or causal condition, and we want to consider all the relevant ones simultaneously. This is where causal configurations come into play. In our example, it is possible to add the condition B. This results in Table 7.

Table 7. Two conditions.

	A	Not A
	B	Not B	B	Not B	Total
O	85	5	120	30	240
Not O	5	5	30	20	60
Total	90	10	150	50	300

In order to determine the consistencies of the possible configurations, it is useful to represent the data in what is called a truth table (Table 8). One denotes presence of a condition (or membership in the target set), 0 denotes absence (or non‐membership in the target set).

Table 8. Truth table.

A	B	Number of cases	Proportion obtaining O
1	1	90	0.944
0	1	150	0.800
0	0	50	0.600
1	0	10	0.500

Here, all the four configurations which can be obtained using the conditions A and B are listed and the respective proportions of their members obtaining O are given. These proportions represent the consistencies of the configurations with regard to their sufficiency for achieving O. The rows of the truth table have been sorted into descending order of consistency. The coverage of a particular configuration can be obtained by calculating the proportion of the cases with a given configuration, say A and B (the first row in the truth table, Table 8) out of all the cases with the outcome O, which in this case is 0.354 (85 out of 240 cases; cf. the top left‐hand cell of Table 7). It is conceivable that there is more than one configuration leading to the outcome. In this example, we might argue that the consistency values of 0.944 and 0.8 are both high enough for us to decide that these configurations can be considered to be usually sufficient for the outcomes. In other words, there is a reasonably high proportion of cases with the configurations ‘A and B’ as well as ‘not A and B’ obtaining the outcome.

This leads to the issue of set theoretic notation and minimisation. Set intersection or logical AND is indicated by *. Set union or logical OR is indicated by +. Membership in a set is indicated by upper case notation, non‐membership or logical negation is indicated either by ∼ or by lower case notation, which is what we use here. Looking at the example given above, the solution obtained can be noted as follows: A*B + a*B. This solution can be written in a simpler form, using logical minimisation, i.e. simply B.

Finally, another point can be illustrated using this example. First of all, there is an element of choice in that the researcher decides what level of consistency is still to be considered acceptable when choosing a solution. ⁸ In our example, if someone were to argue that any proportion higher than 0.55 obtaining the outcome indicates (near) sufficiency, ⁹ we would have had the following configurations:

This solution can be further simplified, resulting in a + B. Using the software fs/QCA, it is possible to calculate consistencies for the individual configurations in a solution and for the solution as a whole. In addition, coverage is also given both for the individual configurations and the whole solution. Often, there is some overlap between the configurations found: in our example with the solution a + B, there are cases belonging to both configurations, i.e. all the ones with conditions a*B. In the calculation of the coverage for the configurations contributing to a solution, there are two coverage figures given. Unique coverage refers to the proportion of cases with the outcome accounted for by cases with the configuration without the overlap, raw coverage refers to the proportion of all cases, including those who are also covered by the other configuration given in the solution (Ragin 2006b).

fs/QCA gives the following output (Figure 3) for our example.

Figure 3 fs/QCA output.

The unique coverage figure given for ‘a’ refers to the 30 cases obtaining the outcome (out of 240) who do NOT have B, i.e. which are not covered by the other part of the solution. The raw coverage figure refers to all those cases without the condition A (i.e. ‘a’), regardless of whether B is present or not, i.e. 150 out of 240 who obtain the outcome. In the same way, the unique coverage given for B refers to the 85 out of 240 cases who are not also a (i.e. ‘A’). The overlap, i.e. a*B, can be obtained by subtracting the unique coverage figures from the solution coverage. In our example, the resulting figure is 0.5. This refers to the 120 cases with a*B who experience O (cf. Table 7).

We can see that there is considerable overlap between the two configurations leading to the outcome: the unique coverage is fairly low for both of them compared with the respective figures for raw coverage, indicating that there are many cases in the configuration a*B. This is in line with what can often be observed in the real world: causes or conditions tend to occur together, which can make a regression analysis with its assumption of independence of variables and attempt to determine net effects of questionable value.

So far, we have only been concerned with dichotomous data, i.e. with the simple presence or absence of conditions, membership or non‐membership in a set. It is these so‐called crisp measures we are using in the present paper. We will note here, however, that another version of QCA also exists to allow for the fact that dichotomous data may not always be adequate when one wants to employ more finely graded measurements. Ragin has utilised the concept of fuzzy sets to deal with this issue (Ragin 2000). The underlying idea is to allow for partial membership in a set along with full membership or non‐membership. We are employing this fuzzy set QCA (fs/QCA) in another paper (Glaesser et al., forthcoming) with our present data. Here, we shall just note that the results were very similar to the crisp ones presented in the present paper.

To summarise, fs/QCA allows for the analysis of conjunctions of causes and alternative causal pathways. For example, in the educational context, it might be that high mathematical ability is sufficient to pass mathematics GCSE, but that, without this, very hard work, supplemented by private tuition, could alternatively lead to a pass. Unlike regression analysis, fs/QCA does not attempt to quantify net effects of variables, but treats cases holistically. QCA is particularly suited to sample sizes like ours, i.e. ca. 50–100 cases, which are not small enough to conduct an in‐depth qualitative analysis, but too small for many of the requirements of regression‐based methods and inferential statistics.

After this somewhat lengthy introduction to QCA, we can move on to present the results of the QCA analyses. Their goal is to identify sufficient and necessary conditions associated with performance of the investigation.

Calibration

The measures we are using were originally interval scaled (with the exception of GCSE, which was ordinal). We therefore had to transform them into dichotomous variables in order to perform the crisp QCA analyses. Such a calibration needs to based on a good knowledge of the cases and measures involved (see for example Ragin 2000, 2005). In this case, the transformation was based on detailed knowledge of the relevant tasks themselves, but also based on the extensive experience of teaching these modules over many years. This section gives details of the cut‐off points we used and the reasoning behind them.

The science exam, the evidence test and the investigation were originally marked using percentages, i.e. on a scale from 0 to 100.

GCSE

The GCSE science results provide the basis for the calibration: having obtained at least B constitutes the group we coded as ‘good’. This is on the grounds that in order to get into A‐level science in many schools, a B is usually seen as a minimum.

Science

A score of 65 or above on the percentage scale was coded as being ‘good’. Anybody who achieves 65 or above will have a good understanding of the subject matter, whereas it is possible to obtain a lower grade without a thorough understanding of every aspect.

Evidence test

A score of 70 or above was coded as being ‘good’. The evidence test contains some closed questions which makes it slightly easier for students to answer them correctly. This is why the threshold is higher than for science.

Investigation

As outlined above, we are interested in what it takes to perform an investigation with a reasonable level of competence. This corresponds to a score of 40 or above, which is the official university threshold for passing the module.

We take this, much lower, cut‐off for a number of reasons. We judged that it would be unreasonable to expect excellent performance in an area where students had little prior experience.We are asking a lot of students here: they must select individual ideas from the substantive and procedural modules appropriate to the task, put them together with a plan and then carry it out.

It may be noted here that in the pre‐investigation every single student except for one performed so badly that they would have been coded as not having attained even this moderate level of competence.

QCA results

We present here the results of the QCA analysis. As well as making substantive points, we shall use this analysis to explain further the use of QCA and to give some guidance for the interpretation of QCA results.

The conditions included in the analyses are evidence post‐test, year 1 science exam and GCSE.

The starting point of the analysis is a model such as

where the relationship of the outcome, the investigation (INVEST), with the conditions evidence post‐test (EVIDENCE), year 1 science exam (SCIENCE) and science GCSE (GCSE) is analysed in Boolean terms. We recall that capital letters stand for the presence of a condition.

The first step in the analysis is to produce a truth table (Table 9).

Table 9. Truth table investigation.

Evidence	Science	GCSE	Number	Investigation	Consistency
1	1	1	18		1
1	1	0	3		1
0	1	1	3		1
1	0	1	8		0.875
0	1	0	4		0.75
1	0	0	7		0.714286
0	0	1	10		0.7
0	0	0	19		0.578947

‘Number’ refers to the number of cases contained in each configuration and consistency refers to the proportion of these cases obtaining the outcome, in this case scoring 1 (i.e. achieving at least 40 on the original percentage scale) on the outcome measure, the investigation. The rows have been ordered in descending order of consistency. Before moving on to the actual Boolean solutions, there is some insight to be gained simply from examining the truth table. Ignoring the outcome for a moment, we can see that the cases are not distributed evenly among the rows. Some configurations have only a small number of cases. This shows the extent of the limited diversity in the sample. For example, there aren’t many individuals with the combination evidence*SCIENCE (third and fifth rows, 3 + 4 = 7 cases out of 72, i.e. less than 10% of the whole sample). By contrast, we find 18 cases in the configuration EVIDENCE*SCIENCE*GCSE and 19 cases with evidence*science*gcse: the majority of individuals either do well or do badly on all three measures. This indicates that the causal conditions themselves are strongly interrelated and are not apparently independent of each other. Finally, it is worth noting that SCIENCE appears in the first three rows of the truth table (and only once after that). It is part of all those configurations which have perfect consistency.

The next step is to decide which rows from the truth table are to be entered in the Boolean analysis to produce a solution. Recall that it is necessary for the researcher to decide on a threshold level for consistency which is still considered acceptable, but that it may also be beneficial to run through different levels (cf. footnote 8). There are three obvious places in our truth table at which to put the threshold, all of which are at least 0.70 (which may be considered to be the minimum level). The first three rows in the truth table all have perfect consistency with sufficiency, i.e. all the cases with these combinations of conditions obtain the outcome. There is also a distinct drop in the consistencies from the third to the fourth row. So we can assume that the configurations above and below this jump differ sufficiently for us to treat them differently in the analysis to start with. This produces the following Boolean solution:

Table

	raw coverage	unique coverage	consistency
SCIENCE*EVIDENCE	0.368421	0.052632	1.000000
GCSE*SCIENCE+	0.368421	0.052632	1.000000
Solution coverage: 0.421053Solution consistency: 1.000000

The solution as a whole shows perfect consistency. So does the combination of either being in the set of people with good marks on the evidence test and on the year 1 science exam or being in the set of people with good marks on the year 1 exam and good GCSEs. This means that being in either set is sufficient to obtain the outcome of conducting the investigation at a reasonable level of competence. Although it may not be immediately obvious from the solution, this of course contains the group of people who have scored high on all three conditions, evidence test, year 1 science exam and GCSE, so this combination is also a sufficient condition. There is a considerable difference between the raw and unique coverage figures for both configurations, which indicates that the overlap is substantial, i.e. most cases have indeed scored high on all three conditions (as we already knew from inspecting the truth table).

It is now worth including the next row in the analysis. While it does not have perfect consistency with sufficiency, its consistency is high at over 0.87, and so it is perfectly justified to treat it as consistent with (near) sufficiency. We obtain this result:

Table

	raw coverage	unique coverage	consistency
SCIENCE*EVIDENCE	0.368421	0.052632	1.000000
GCSE*SCIENCE+	0.368421	0.052632	1.000000
GCSE*EVIDENCE+	0.438596	0.122807	0.961538
Solution coverage: 0.543860Solution consistency: 0.968750

The solution as a whole is still very high in consistency, while coverage has increased slightly compared with the previous solution, but it is still not very high. Here, too, we find that the combinations which form part of the solution each have much lower unique than raw coverage, pointing to some considerable overlap. We might summarise this solution by saying that it is sufficient to have any two out of the three conditions (or all three combined) to obtain the outcome.

There is another distinct drop in consistency to be found in the truth table. We have already referred to the one between the third and the fourth row. Another drop is between the second but last and the last row. This indicates that the proportion of individuals who have low marks on all three conditions is considerably lower with regard to obtaining the outcome than that of any other group. So the third obvious place for a threshold for consistency is 0.70. This may still be considered acceptable as being consistent with sufficiency. In effect, this means that all the rows from the truth table except the last one are allowed into the Boolean analysis, producing this result:

Table

	raw coverage	unique coverage	consistency
EVIDENCE+	0.578947	0.087719	0.916667
SCIENCE+	0.473684	0.052632	0.964286
GCSE	0.614035	0.122807	0.897436
Solution coverage: 0.807018Solution consistency: 0.867925

We can see that, at this lower level of consistency, any one of the three conditions analysed is (nearly) sufficient for obtaining the outcome. Taken together, the solution shows that it is also (nearly) necessary to have at least one of them: the coverage is much higher here than in the previous solution. Put differently, we can say that cases who don’t have any of the three conditions will obtain the outcome far less frequently, which also implies that it is necessary to have at least one of them in order to obtain the outcome with some certainty.

Note, however, that we cannot claim that any one single condition is more important, i.e. more necessary, than the others, since the differences in the raw and unique coverage figures of the three conditions are not very large.

To summarise, we have identified two solutions which point to (near) sufficient conditions for obtaining the outcome. They both contain the combination of all three conditions, as well as the combinations of any two of the three. The third solution, which has greatly improved coverage, indicates that it is necessary to have at least one of them, although here too we find very low unique and fairly high raw coverage figures, which implies that the conditions are seldom present on their own. So, even if in principle only the presence of one of them is required, there is only a minority of cases who actually have just one condition but neither of the others.

Discussion

From the descriptive results

Performance on the investigation after the substantive science module, but prior to the procedural ideas module, was very poor. After the teaching of procedural ideas and without any further teaching of substantive ideas, there was substantial improvement on this measure. Taken together, these findings lead us to the tentative conclusion that good attainment on the substantive science exam by itself was clearly not a sufficient condition for showing a reasonable level of competence on the investigation. We do not know, on the basis of these descriptive results alone, whether it was a necessary condition, since it may well be necessary in conjunction with some other factor(s) but not sufficient on its own. As to the procedural ideas taught in the evidence module, it seems that either it was those alone which produced the greatly improved performance on the investigation, or the combination of both these procedural ideas and substantive knowledge. Practice does not seem to have been a necessary condition since the performance on the investigation improved greatly without any practice. It might be argued that the pre‐investigation could have acted as practice, or that the students may have discussed their experience of the pre‐investigation amongst themselves. In our view, this does not constitute skills practice as such, since many more investigations would have to be carried out to provide such practice.

To further clarify the roles of substantive and procedural understanding, we turn to the QCA analysis.

From the QCA analysis

The QCA analysis showed that the joint presence of the three conditions, procedural ideas, substantive ideas as indicated by the science exam results, and prior attainment as measured through GCSE, is sufficient for demonstrating a fair level of competence at doing the investigation. The combination of any two of the three is also (nearly) sufficient, but this finding is based on a smaller number of cases. One of these combinations comprised three cases who obtained the outcome without having done well on the evidence test. As we are particularly interested in the role of procedural understanding, it is worth looking at the latter briefly. They were not remarkable in any way with regard to any of the measures and indicators which we have and have not included in the analyses presented here, i.e. age, access route to university, social background or performance on the pre‐evidence test. If anything, they showed little improvement on the evidence test from the pre‐ to the post‐test.

Setting the threshold for consistency with sufficiency at 0.70 gave us a solution in which having at least one of the three (no matter which) was (nearly) necessary. This also became clear from simply inspecting the truth table: we have seen that the group who do not have any of the three conditions are the least likely to conduct the investigation competently.

It may seem that our claim that having at least one of three is a necessary condition and that science combined with only one of the other two may be a sufficient condition contradicts our earlier conclusion based on the descriptive findings that science is not sufficient. However, we did not make any claims about the necessity of substantive ideas based on the descriptive findings alone since this would not have been possible. As regards sufficiency, we want to stress again that in most cases, science by itself was not sufficient. The apparent contradiction is also due to the difference in the point of view: the descriptive results refer to average findings over all the cases, whereas the QCA results preserve the features of individual cases.

It has become clear that the other important finding from QCA is that, apart from their relationship with the outcome, the three conditions are closely linked with one another. The majority of cases either display all three conditions or they have none of them. This renders any attempt to tear them apart and to try and identify the effect of a single one difficult, if not impossible. We should note that this is not a shortcoming of the method, QCA. Rather, it is one of its strengths, since it reminds us of the fact that conditions are often interlinked and dependent on each other and cannot be separated analytically.

Taken together, the findings from the descriptive analysis and QCA indicate that, in addition to the performance indicators GCSE, science exam and evidence test, the simple fact that the students had all been taught the module on procedural ideas has contributed to their performance on the investigation. Recall that, overall, the performance on the investigation had improved greatly after the teaching. We take this as an indication of the importance of the teaching itself. In addition, QCA helped us identify conditions which are associated with success on the investigation, within this group of people who had all experienced the teaching.

Using ‘TAUGHT’ as shorthand for ‘HAVING BEEN TAUGHT THE MODULE ON PROCEDURAL IDEAS’, we might therefore re‐write the equation which refers to the (near) sufficient solution as:

	TAUGHT*EVIDENCE*SCIENCE +
	TAUGHT*SCIENCE*GCSE +
	TAUGHT*EVIDENCE*GCSE,

and the equation for the necessary solution as:

	TAUGHT*EVIDENCE +
	TAUGHT*SCIENCE +
	TAUGHT*GCSE.

This makes it obvious that having been taught the module was neither sufficient nor necessary on its own, since it is combined with a good performance on the various tasks. Incidentally, this may add to the picture of the three cases discussed above who did well on the investigation without having done well on the evidence test. They, too, had been taught the module, of course, which will have contributed to their performance on the investigation, even though they evidently could not transfer the teaching into a good performance on the evidence test.

Interpretation

Throughout this article, we have used sufficiency and necessity as a means of analysing the skills involved in conducting an investigation. While these concepts intuitively make sense, they are often not applied in correlational or regression‐based research. However, we have found them to be fruitful in thinking about what is required to perform an investigation competently. It has to be borne in mind that while they are rooted in logic, they must not be understood here in a strictly logical sense, where a condition is either sufficient/necessary or it is not. In QCA, this strict assumption is relaxed in order to talk about a condition being ‘usually sufficient/necessary’ to produce the outcome in question. In doing so, the features of each case as a whole are preserved and made apparent, instead of looking at average scores or effects over all the cases.

We have found QCA to be a useful tool for the analysis of necessary and sufficient conditions. Furthermore, it is well suited to analysing conjunctions of conditions rather than net effects of variables and to uncovering limited diversity in the data, i.e. the fact that not all theoretically possible combinations of conditions occur equally frequently empirically. This was clearly relevant to our study: the cases were not spread evenly across the rows of the truth table.

It might be argued that a cross‐tabulation of the various measures we have used would have told us as much as QCA has done. However, we think it is unlikely that the patterns we have found would have been easy to see from such a simple inspection of a descriptive table.

To return to our substantive conclusions: some students may well have conducted a number of open‐ended investigations during their school careers. However, research (House of Commons Science and Technology Committee 2002) suggests that such tasks have not been widely used. Rather, investigation tasks have been used only for assessment purposes in a very limited and routinised way. It is no great surprise, then, to find that the students were unable to conduct an investigation which was fairly simple in terms of substantive content but which involved the application of procedural ideas prior to being taught such ideas explicitly. Our study has shown that this teaching is both possible and necessary. Having substantive knowledge, while also important, is not enough on its own, contrary to traditional views which seem to emphasise the importance of substantive knowledge and neglect procedural understanding. This view can be found, for example, in curriculum documents, where, in effect, it is assumed that substantive knowledge is necessary and also sufficient, although some more recent offerings do go some way to addressing this point (Assessment and Qualifications Alliance 2005; Qualifications and Curriculum Authority 2006).

It seems, at least from this limited sample, that it is possible to develop the ability to carry out such investigations by teaching the underlying substantive and procedural ideas without extensive recourse to ‘skills practice’. If this could be shown to be more generally applicable, a number of consequences would arise.

•	Procedural ideas should be incorporated into the school curriculum at all ages.
•	They should be structured and sequenced appropriately and taught using the most efficient methods (practical or otherwise).
•	Practice to carry out investigations could then be contained in a smaller number of tasks than would be required by a skills approach.

Of course, more research is needed before we can generalise from our findings, both with a wider range of investigation tasks and with school‐aged pupils rather than university students.

Notes

1. A‐level is short for Advanced Level, the highest secondary school qualification on offer in the UK. Three A‐levels are usually the university entry requirement. For the course our students attend, the required A‐level grades are three Bs.

2. The original sample comprised 97 students. Here, we have only included those students for whom we have measures at each point in time on all the variables which form part of the analyses.

3. Together with others, he also has developed the software fs/QCA (for ‘fuzzy set/Qualitative Comparative Analysis’) (Ragin, Drass, and Davey 2006) which performs the required analyses. This is the software we use.

4. We deliberately avoid the use of the terms ‘cause’ or ‘causal condition’ as the relationships described here are patterns of association. Causal statements can only be made based on theoretical considerations.

5. Based on Boudon (1974) as discussed in Cooper (2005, 2006).

6. Note that, in conducting research, temporal order and substantive knowledge need to be used in determining the causal order, i.e. the difference between Figure 1 and Figure 2 lies in what is considered cause and effect. It is conceivable that this may vary or not be clear in a research situation. For our illustrative purposes, however, we have decided that A is the cause and O the outcome. The determination of sufficiency and necessity is based on this decision.

7. Another way of thinking about consistency/sufficiency and coverage/necessity is in terms of inflow and outflow: in a cross‐tabulation such as Table 4, the proportion of cause A in O which we called consistency with sufficiency can be called outflow because it refers to the percentage of people with A who subsequently obtain O. The proportion of O with condition A as described in Table 6 (called consistency with necessity) can also be called inflow because it refers to the percentage of people with O who got there after having also experienced A.

8. Going through various levels of consistency instead of choosing a single one brings out the relative importance of conditions which can be very instructive. Cooper (2005, 2006) makes use of this approach.

9. This would be a rather generous threshold, however, and was chosen only in order to demonstrate a solution with several pathways. It is more common to choose a threshold of at least 0.70.