Back-stage configuring: a textual analysis of formalisation and professional judgement in two systematic reviews

Figures & data

Figure 1. The review process at SIER.

Table 1. The key theoretical resources.

Areas of inquiry	Self-understanding within systematic reviewing	Basic STS concepts	Claims about the historical development of societal and scientific ideals
Concepts	Aggregative and configurative logics	Formalisation and professional judgment	The dominance of mechanical objectivity

Table 2. The theoretical congruity between aggregation/configuration, formalisation/professional judgment, and the levels of analytic progress and abductive steps.

Areas of inquiry	Systematic review	Self-understanding of systematic reviewing	Science and technology studies (STS)
			Collins and others	Porter and others
Key contribution in the analytic process	Texts specifying purposes, questions, methods, results of the reviews	Concepts of “aggregative” and “configurative logics”	Concepts of “formalisation” and “professional judgment”	Claims about the increasing pre-dominance of the norms of mechanical objectivity
Levels of analytic progress (step in abductive process)	Close reading (step 1)	Finding signs of aggregative and configurative logics → (step 2)	Interpreting the use of formalisation and professional judgment → (step 2, ctd)	Conclusions about actually existing norms in systematic review texts and processes (step 3)

Table 3. The abductive process.

Abductive step	Step 1: close reading	Step 2: coding using central distinctions from previous research		Step 3: constructing a model of relationship
Purpose	Understanding the purpose, methods, results and conclusions	Finding signs of configurative and aggregative elements	Interpreting the interplay between formalisation and professional judgment	Relating the results to the critique and rhetoric of formalisation in reviews of educational research
Examples	Reading results about the tripartite distinction of classroom dialogues in disputational talk, cumulative talk and exploratory talk	Exploratory talk is used as an overarching category when understanding all the studies of one review – although only used manifestly in two studies =  configurative logic	In order to pursue such a configurative logic, iterative and interpretive analytic steps need to be taken =  requires professional judgment.	Most elements of the systematic reviews are configurative, requiring professional judgment. In addition to contradicting SIER’s rhetoric, this means the critique of formalisation is exaggerated or even erroneous.
	Questions about effects of digital resources and what might explain why effects appear or not.	Effects are usually aggregative but here without the usual control interventions as comparisons. Explanations of differences are sought without quantitative measures = some aggregative elements but primarily a configurative logic	In order to pursue these configurative aims, constant comparisons of individual details in included studies have to be pursued, going beyond mere aggregation = requires professional judgment

Figure 2. SIER’s example of a forest plot diagram, which differs from conventional forest plot diagrams in that no aggregated effect size is reported (SIER, 2017a, p. 16). The picture is reproduced from SIER, with permission.

SIER. (2017a). Digitala lärresurser i matematikundervisningen. Delrapport förskola., systematisk översikt 2017:02 (2/2). ed. Skolforskningsinstitutet.

 Google Scholar