Large-scale data, “wicked problems”, and “what works” for educational policy making

The current drive for evidence-informed policy making should take little comfort from Rittel and Webber’s (1973) statement in Policy Sciences over forty years ago, that “the search for scientific bases confronting problems of social policy is bound to fail, because of the nature of these problems. … [T]here are no ‘solutions’ in the sense of definitive and objective answers” (p. 155). This could have been written only yesterday, or today.

Look at the “what works” movement. “What works” is a dangerously loose question, prompting answers which are couched in all-too-often simplistic, generalized, context-free, contingency-free, condition-free terms but which attract policy makers like flies to the honeypot (cf. Levin, 1991).

Rittel and Webber’s (1973) celebrated statements of “wicked problems” – those which are not susceptible to simple definition or solution – are characterized thus (pp. 161–167):

1. “There is no definitive formulation of a wicked problem” (there is no clear-cut statement of the problem; formulating the problem is actually the problem).

2. “Wicked problems have no stopping rule” (we never cease searching for solutions to the problem; no solution is complete, as problems are multi-causal and limitless).

3. “Solutions to wicked problems are not true-or-false, but good-or-bad” (i.e., a matter of subjective judgement and values rather than being either correct or incorrect answers that one would find in, for example, a mathematical problem).

4. “There is no immediate and no ultimate test of a solution to a wicked problem” (unanticipated consequences of putative solutions appear over time; one solution creates another problem endlessly).

5. “Every solution to a wicked problem is a ‘one-shot operation’; because there is no opportunity to learn by trial-and-error, every attempt counts significantly” (every solution, including one which attempts to put right an earlier error or problem, leaves traces and has serious consequences for people, that cannot be undone; we cannot turn back the clock; with other problems, e.g., a chess game, trial-and-error can operate without having such serious consequences).

6. “Wicked problems do not have an enumerable (or exhaustively describable) set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan” (ordinary problems have a limited number of solutions; wicked problems do not, as solutions are infinite).

7. “Every wicked problem is essentially unique” (solutions to, or experience of, similar problems may not work in the case of a specific problem).

8. “Every wicked problem can be considered to be a symptom of another problem” (a problem is not self-contained or limited, but stems from, and leads to another problem; there is no single root cause).

9. “The existence of a discrepancy representing a wicked problem can be explained in numerous ways. The choice of explanation determines the nature of the problem’s solution” (different people have different views of what the problem is and what is its solution).

10. “The planner has no right to be wrong” (unlike coolly refuting a scientific hypothesis, the impact of solutions is large and affects many people; problem solvers are not immune to responsibility, accountability, and liability).

Rittel and Webber’s (1973) analysis delivers a broadside to those policy makers and educationists seeking simple solutions to “what works” or even to thinking that it is a legitimate question to ask at a high level of generality or to seek for certainty. There is no simple statement of the problem and no simple solution. The authors’ observations, though commonplace now, are no less pertinent than when they were first written. They are a neat foil to those policy makers seeking short-term, mono-causal analyses and simple solutions to long-term, multi-causal problems.

The four papers in the present issue provide neat examples of wicked problems, which are couched in terms of questions:

• To what extent have male–female differences in educational attainment changed over time in Western countries? What contextual factors might explain over-time and between-country variance of men’s and women’s educational attainment? To what extent does female labour participation and the degree of religiosity in a person’s adolescence affect their educational attainment?

• What are the direct and indirect effects on students’ mathematics and problem-solving achievement of (a) three components of socioeconomic status – family wealth, home educational resources, and parental education – and (b) parental expectations?

• What does it mean to be a “good” citizen, and what is a general model of good citizenship?

• What scientific reasoning skills are relevant for making informed decisions in our everyday lives, and how can these skills be facilitated?

These, and other wicked problems, defy simple solution. Given that attempts to solve educational problems are so easily shipwrecked, how, then, can educationists and researchers proceed?

On the one hand, they might dredge through “big data” and “learning analytics” collected from unobtrusive traces of electronic footprints (Beneito-Montagut, in press; Cope & Kalantzis, 2015), believing that monstrously large numbers speak for themselves because they happen to address four V’s: volume, velocity, variety and veracity (e.g., Ahmadi, Dileepan, & Wheatley, 2016; Halavais, 2015). (Indeed, the four papers here question the value of moves from hypothesis-driven research to “data-driven discovery” typified in “big data” research; Eynon, 2013; Jahanian, 2013). On the other hand, researchers might go to the opposite end – “small data” (boyd & Crawford, 2012) – in seeking to enhance, through contextual detail, the explanatory rather than solely the descriptive potential of data.

Between these two poles lies deliberately planned large-scale research. We make a case here for large-scale research and data in addressing social and educational wicked problems by giving direction and cautious suggestions (cognizant of their limitations) whilst recognizing the defining openness, multi-valency, intractability, and insoluble nature of wicked problems. Such are the four papers in the present issue.

Van Hek, Kraaykamp, and Wolbers use large-scale data on more than 138,000 people in 33 countries, from the European Social Survey and the US General Social Survey, to explain differences in male–female educational attainment over time and across countries. Whilst offering explanations, they recognize that, like wicked problems, their study “presents a challenge for future research concerning the impact of country conditions on differences in educational attainment between men and women. Hopefully, more contextual data on countries and cohorts will become available, broadening the possibilities for comparative research on this topic” for which “our findings may serve as a useful starting point”. In other words, whilst large-scale data can suggest useful lines of exploration of an open-ended problem, simple solutions are out of court.

Long and Pang use data from 5,066 ninth-grade students in China to answer two questions: “[d]o the three components of SES [socioeconomic status] – family wealth, home educational resources, and parental education – and parental expectations directly predict Chinese adolescents’ mathematics and problem-solving achievement?” and “[d]o parental expectations mediate the relationship between these three components of SES and Chinese adolescents’ mathematics and problem-solving achievement?” Their findings suggest that “parental education has smaller indirect effect sizes on both mathematics and problem-solving achievement, while family wealth and home educational resources have reasonable effect sizes on both mathematics and problem-solving achievement”. Then comes the articulation of the wicked problem: the limitations of the study. The authors acknowledge six such limitations: generalizability; causality; levels of data; instrumentation; modelling; and the need for further studies. Large-scale research can usefully suggest ways forward, but typically, like wicked problems, these are open to modification.

The paper by Reichert uses data from 11,695 students in Australia to characterize “empirically identified citizen personalities” and their characteristics, and to test the hypothesis that “positive views toward diversity and Indigenous people would be positively related to the combined group of students who support all norms of citizenship”. He notes that “person-centred research and variable-oriented analysis can be a powerful tool for civic education researchers to understand how different factors exert varying influences on heterogeneous populations”. Then comes the wicked problem: “the person-centred study of students’ attitudes to good citizenship reminds us to take population heterogeneity into account, because educational approaches that are suited for some students may be ineffective among other groups of students with distinct understanding of what it means to be a good citizen”. Wicked problems require recognizing that data and any recommendations stemming from them are open to ongoing modification and reformulation.

Engelmann, Neuhaus, and Fischer use a different kind of large-scale data – meta-analysis and effect size calculations of 30 studies involving 4,333 participants – to investigate “whether interventions targeted at processes of scientific discovery … are equally, more, or less successful than those targeting at other aspects on scientific reasoning” and to identify “factors that moderate the effect of the interventions”. At the end of their paper, having presented their findings, the wicked problem then becomes visible in the caution which the researchers exercise in their wording, for example: “[t]his result could indicate”; “[t]hese results may indicate”; “[t]his finding can be interpreted as …”; “[i]dentifying the conditions for effectively stimulating constructive and interactive learning activities that support the development of scientific reasoning and argumentation is an important area for future primary studies” (italics added to the original). The authors acknowledge that their meta-analysis “cannot exclude the possibility that other intervention studies taught and measured similar skills and knowledge without complying with our inclusion criteria” and that “[f]urther research is needed to test directly the effects found in this meta-analysis. Moreover, other factors that we could not investigate in this meta-analysis … could moderate the success of interventions on scientific reasoning”. In other words, their study, like the other papers in this issue, provides cautious guidance rather than definitive or immutable answers. That’s all we can expect for a wicked problem.

The caution that has to be exercised in addressing wicked problems in education is acknowledged in all of the four papers here. Rittel and Webber (1973) indicated that wicked problems also recognize that social policy, which includes education, operates in a political arena. Education is, as Gallie (1955) observed, an “essentially contested concept” (complex, open to different interpretations, values, and modification in light of changing circumstances). Indeed, Rittel and Webber remark that “the expert is also a player in a political game. … There is no escaping that truism” (p. 169). Wicked problems are wicked because of, in part, their political and policy-challenging ramifications. The four papers here allude to such policy matters.

For example, the paper by Van Hek et al. identifies policy implications: “a large challenge nowadays lies in investigating which factors encourage women to convert their educational credentials into matching labour market outcomes”. Long and Pang’s paper contains policy implications: “students’ achievement is not only related to their family SES, but also to other factors in their family, such as parental involvement, as well as factors in the school, such as curriculum”. Reichert’s paper includes policy implications: “raising the levels of civic knowledge and analytical skills is unlikely to be the most effective way to cultivate citizens that appreciate a broad variety of civic and political behaviours and are thus more likely to participate in respective activities”. Engelmann et al. note policy implications in their comment that other factors should be addressed “such as the meaningful use of technological support”.

The four papers here attest to the inherent and essential – of the essence – openness of research findings, questions, consequences, and uses. Indeed, like wicked problems, they recognize that: (a) one phenomenon might be similar to, but be significantly different from, another; (b) multiple explanations of, and perspectives on, research findings can exist; (c) looking for, or finding, simple causality is elusive, even naïve; and (d) simple solutions are just that – too simple.

Large-scale data, sitting between “big data” and “small data”, as exemplified in the four papers here, can provide guidance and direction in addressing wicked problems in education. Large-scale data retain the human dimension, as they deal with real, sentient people. But also, like wicked problems themselves, such large-scale research and data argue for uncertainty, tentativeness, and provisionality of putative solutions. Like wicked problems, they raise serious questions about serious matters, in contrast to simplistic questions such as “what works?” and simplistic answers for the sake of political expediency.