‘Ordinary science intelligence’: a science-comprehension measure for study of risk and science communication, with notes on evolution and climate change

Abstract

This paper describes the ‘ordinary science intelligence’ scale (OSI_2.0). Designed for use in the empirical study of risk perception and science communication, OSI_2.0 comprises items intended to measure a latent capacity to recognize and make use of valid scientific evidence in everyday decision-making. The derivation of the items, the relationship of them to the knowledge and skills OSI requires, and the psychometric properties of the scale are examined. Evidence of the external validity of OSI_2.0 is also presented. Finally, the utility of OSI_2.0 is briefly illustrated by its use to assess standard survey items on evolution and global warming: when administered to members of a US general population sample, these items are more convincingly viewed as indicators of one or another latent cultural identity than as indicators of science comprehension.

Keywords:

• science comprehension
• risk perception
• global warming
• belief in evolution

1. Introduction: OSI_2.0

This paper (more a technical research note, actually) furnishes information on the ‘ordinary science intelligence’ scale (OSI). OSI is a tool to facilitate empirical investigation of how individual differences in science comprehension contribute to public perceptions of risk and like facts. The paper describes the motivation for OSI, the process used to develop it, and the scale’s essential psychometric properties. It also presents evidence of the external validity of the scale as a predictor of scientific reasoning proficiency. Finally, it illustrates the utility of OSI for testing hypotheses on the relationship between public science comprehension and controversial applications of science.

The current version of OSI is the successor to the science comprehension instrument used in Kahan et al. (2012) and was featured in a study reported in Kahan (2015a). Additional refinements, including creation of a short form, are anticipated. To distinguish it from previous and likely future versions, the scale described in this paper will be referred to as ‘OSI_2.0.’

2. What and why?

The validity of any science-comprehension instrument must be evaluated in relation to its purpose. The quality of the decisions ordinary individuals make in myriad ordinary roles – from consumer to business owner or employee, from parent to citizen – will depend on their ability to recognize and give proper effect to all manner of valid scientific information (Baron 1993; Dewey 1910). It is variance in this form of science comprehension – and not variance in the forms or levels of comprehension distinctive of trained scientists, or the aptitudes of prospective science students – that OSI_2.0 is intended to measure.

This capacity will certainly entail knowledge of certain basic scientific facts or principles. But it will demand as well various forms of mental acuity essential to the acquisition and effective use of additional scientific information. A public science-comprehension instrument cannot be expected to discern proficiency in any one of these reasoning skills with the precision of an instrument dedicated specifically to measuring that particular form of cognition. It must be capable, however, of assessing the facility with which these skills and dispositions are used in combination to enable individuals to successfully incorporate valid scientific knowledge into their everyday decisions.

A valid and reliable measure of such a disposition could be expected to contribute to the advancement of knowledge in numerous ways. For one thing, it would facilitate evaluation of science education across societies and within particular ones over time (National Science Board 2014). It would also enable scholars of public risk perception and science communication to more confidently test competing conjectures about the relevance of public science comprehension to variance in – indeed, persistent conflict over – contested risks, such as climate change (Hamilton 2011; Hamilton, Cutler, and Schaefer 2012), and controversial science issues such as human evolution (Miller, Scott, and Okamoto 2006). Such a measure would also promote ongoing examination of how science comprehension influences public attitudes toward science more generally, including confidence in scientific institutions and support for governmental funding of basic science research (e.g. Allum et al. 2008; Gauchat 2011). These results, in turn, would enable more critical assessments of the sorts of science competencies that are genuinely essential to successful everyday decision-making in various domains – personal, professional, and civic (Toumey 2011).

In fact, it has long been recognized that a valid and reliable public science-comprehension instrument would secure all of these benefits. The motivation for the research reported in this paper is widespread doubt among scholars that prevailing measures of public ‘science literacy’ possess the properties of reliability and validity necessary to attain these ends (e.g. Pardo & Calvo 2004; Guterbock et al. 2011; Roos 2012; Stocklmayer and Bryant 2012). OSI_2.0 was developed to remedy these defects.

The goal of this paper is not only to apprise researchers of OSI_2.0’s desirable characteristics in relation to other measures typically featured in studies of risk and science communication. It is also to stimulate these researchers and others to adapt and refine OSI_2.0, or simply devise a superior alternative from scratch, so that researchers studying how risk perception and science communication interact with science comprehension can ultimately obtain the benefit of a scale more distinctively suited to their substantive interests than are existing ones.¹

3. Item derivation and scale development

OSI_2.0 is a latent variable measurement instrument. Responses to the items that the scale comprises are not understood to certify familiarity with any canonical set of facts or principles. Rather, they are conceptualized as observable (or manifest) indicators of an unobservable (latent) cognitive capacity that enables individuals to acquire and use scientific knowledge. The cognitive capacity is deemed to cause responses to the items, which individually can be seen as imperfect or noisy proxies for the unobserved capacity. When responses to multiple items are appropriately aggregated, their covariance furnishes an even more discerning (reliable) measure of the latent variable insofar as each item’s random variance – that part of the variance unrelated to the latent variable – is offset by the same in the others (DeVellis 2012).

OSI_1.0 – the scale used in Kahan et al. (2012) – consisted of eight items from the National Science Foundation Science Indicators (National Science Board 2014) battery and fourteen numeracy scale items (Lipkus, Samsa, and Rimer 2001; Peters et al. 2006). The goal of combining these measures was to measure a latent science comprehension disposition indicated jointly by knowledge of various basic scientific facts and principles and by quantitative reasoning dispositions essential to giving empirical information proper effect.

Drawn from a more diverse range of sources, and evaluated over multiple different data collections, the 18 items that make up OSI_2.0 reflect the same basic strategy. Although intended to measure a unitary cognitive capacity, the items can for expository purposes be divided into four sets (Table 1).

Table 1. OSI_2.0 items.

Item label	Wording				% correct	Derivation
1. Basic facts
RADIOACTIVE	All radioactivity is man-made. [True or False]				83	National Science Board (2014)
LASERS	Lasers work by focusing sound waves. [True or False]				68	National Science Board (2014)
ELECTRONS	Electrons are smaller than atoms. [True or False]				69	National Science Board (2014)
NITROGEN	Which gas makes up most of the Earth’s atmosphere? [Hydrogen, Nitrogen, Carbon Dioxide, Oxygen]				25	Pew Res. Cntr. (2013)
COPERNICUS	Does the Earth go around the Sun, or does the Sun go around the Earth? [only if ‘earth/around sun’]: How long does it take for the Earth to go around the Sun? (1 day, 1 month, 1 year)				60 (combined)	National Science Board (2014)
ANTIBIOTICS	Antibiotics kill viruses as well as bacteria. [True or False]				65	National Science Board (2014)
2. Methods
VALID	Two scientists want to know if a certain drug is effective against high blood pressure. The first scientist wants to give the drug to 1000 people with high blood pressure and see how many of them experience lower blood pressure levels. The second scientist wants to give the drug to 500 people with high blood pressure and not give the drug to another 500 people with high blood pressure, and see how many in both groups experience lower blood pressure levels. Which is the better way to test this drug? [The first way/The second way]				72	National Science Board (2014)
PROB1	A doctor tells a couple that their genetic makeup means that they’ve got one in four chances of having a child with an inherited illness. Does this mean that if their first child has the illness, the next three will not? (Yes/No)				85	National Science Board (2014)
PROB2	Does this mean that each of the couple’s children will have the same risk of suffering from the illness? (Yes/No)				73	National Science Board (2014)
3. Quantitative Reasoning
DIE	Imagine that we roll a fair, six-sided die 1000 times. Out of 1000 rolls, how many times do you think the die would come up as an even number? [open ended: 50% of or equivalent]				57	Weller et al. (2012).
BUCKS	In the BIG BUCKS LOTTERY, the chances of winning a $10.00 prize are 1%. What is your best guess about how many people would win a $10.00 prize if 1000 people each buy a single ticket from BIG BUCKS? [open ended: 10 or equivalent]				56	Weller et al. (2012)
SWEEP	In the ACME PUBLISHING SWEEPSTAKES, the chance of winning a car is 1 in 1000. What percent of tickets of ACME PUBLISHING SWEEPSTAKES win a car? [open ended: 0.1% or equivalent]				31	Weller et al. (2012)
DISEASE1	If the chance of getting a disease is 20 out of 100, this would be the same as having a _____% chance of getting the disease. [open ended: 20 or equivalent]				75	Weller et al. (2012)
DISEASE2	If the chance of getting a disease is 10%, how many people would be expected to get the disease out of 1000? [open ended: 100 or equivalent]				78	Weller et al. (2012)
CONDITIONAL	Suppose you have a close friend who has a lump in her breast and must have a mammogram. Of 100 women like her, 10 of them actually have a malignant tumor and 90 of them do not. Of the 10 women who actually have a tumor, the mammogram indicates correctly that 9 of them have a tumor and indicates incorrectly that 1 of them does not have a tumor. Of the 90 women who do not have a tumor, the mammogram indicates correctly that 80 of them do not have a tumor and indicates incorrectly that 10 of them do have a tumor. The table below summarizes all of this information.					Weller et al. (2012)
		Tested positive	Tested negative	totals
	Actually has tumor	9	1	10
	Does not have tumor	10	80	90
	Totals	19	81	100
	Imagine that your friend tests positive (as if she had a tumor), what is the likelihood that she actually has a tumor? ___ out of ___ [open ended: 9, 19]
4. Cognitive Reflection
WIDGET	If it takes 5 machines 5 min to make 5 widgets, how long would it take 100 machines to make 100 widgets? ___ minutes [open ended: 5]				27	Frederick (2005)
BATBALL	A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? ___ cents [open ended: 5]				13	Frederick (2005)
LILLYPAD	In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? ___ days [open ended: 47]				23	Frederick (2005)

Notes: N = 2000. Nationally representative sample (US). Correct answer, underlined if multiple choice options, indicated in brackets.

3.1. Scientific ‘fact’ items

The first set of items – six in total – relate to knowledge of certain basic scientific facts. There is indeed no reason for thinking that knowledge of these particular facts is particularly germane to the capacity to recognize and use valid scientific knowledge in everyday life. Nevertheless, they are ones that it is reasonable to believe a person who has acquired such a capacity will be familiar with.

The primary source was the NSF Indicators battery (National Science Board 2014), which is dominated by true–false items relating to basic propositions in the physical and biological sciences (e.g. ‘Antibiotics kill viruses as well as bacteria’; ‘Electrons are smaller than atoms’). The Indicators’ ‘basic fact’ items are known to furnish an extremely undemanding test (Pardo & Calvo 2004). When the battery is administered to a general US population sample, the median number of correct responses to the battery’s nine ‘fact’ questions is six (National Science Board 2014). Over the course of the development of OSI_2.0, certain NSF items (e.g. ‘The center of the Earth is very hot’; ‘It is the father’s gene that decides whether the baby is a boy or a girl’) were discarded as too likely to be answered correctly to furnish information useful for assessing differences in science comprehension.

To supplement the Indicators' 'basic fact' items, comparable ones from the Pew ‘Science and Technology’ battery (2013) were assessed as well. One item, a multiple-choice question (‘which gas makes up most of the Earth’s atmosphere – hydrogen, nitrogen, carbon dioxide, oxygen’), was included in the scale.

3.2. Scientific ‘methods’ items

OSI_2.0 also includes three NSF Indicator items relating to ‘understanding of how science generates and assesses evidence, rather than knowledge of particular facts’ (National Science Board 2014, 7–23). One of these items tests recognition of the contribution that a control condition makes to drawing causal inferences in experiments. Two others assess familiarity with rudimentary principles of probability.

3.3. Quantitative reasoning

Six OSI_2.0 items come from the Lipkus/Peters Numeracy battery (Lipkus, Samsa, and Rimer 2001; Weller et al. 2012). Numeracy items measure proficiency in reasoning with quantitative information. Only modest math skill is needed to answer the problems that the items set forth; solving them depends, first, on fluency with quantitative representations of information (probabilities, percentages, and odds, e.g.) and, second, on consistent recognition of which quantitative reasoning skills (ones involving conditional probabilities, e.g.) are necessary to draw valid inferences from such information.

3.4. Cognitive reflection items

Finally, OSI_2.0 includes the three items that make up the Cognitive Reflection Test (Frederick 2005). The CRT items present word problems, the solution to which requires respondents to resist crediting an answer that is immediately and intuitively appealing but that can be shown by logical analysis to be unsound. Responses to the items are understood to indicate a disposition to critically interrogate existing beliefs on the basis of available information. CRT scores have been shown to predict resistance to cognitive biases associated with over-reliance on heuristic reasoning (Hoppe and Kusterer 2011; Toplak, West, and Stanovich 2011), particularly when used in conjunction with numeracy items (Liberali et al. 2011; Reyna et al. 2009).

4. Psychometric properties

This section reports the psychometric properties of OSI_2.0. The analyses described are based on the responses of a sample of 2000 US adults recruited by YouGov, Inc., a public opinion research firm that conducts online surveys and experiments on behalf of academic and governmental researchers and commercial customers (including political campaigns). The firm’s general population recruitment and stratification methods have been validated in studies comparing the results of YouGov surveys with those conducted for the American National Election Studies (Ansolabehere & Rivers 2013). The sample in this study was 55% female, and the average age of the subjects was 49 years. Seventy-two percent of the subjects were white, and 12% African-American. The median education level was ‘some college’; the median annual income was $40,000–$49,000. The study was administered between 24 April and 5 May 2014.

4.1. Covariance structure

Although it includes a combination of knowledge, skills, and dispositions, ‘OSI’ is posited to be a unitary cognitive capacity. Accordingly, OSI_2.0 should display the properties characteristic of a reliable measure of a single latent variable.

Factor analysis results were consistent with treating it as such (Table 2, I). The analysis, which employed principal factor extraction without rotation, disclosed that a single factor accounted for 87 percent of the variance in OSI_2.0’s 18 items. That factor had an eigenvalue of 7.5, over 12 times as large as the second largest factor (0.6), which explained only 7% of the variance. With the exception of ELECTRON (β = 0.43), every item had a factor-loading coefficient greater than 0.50. Under conventional standards, these results support treating the items as measuring a single latent disposition (DeVellis 2012; Morizot, Ainsworth, and Reise 2007).

Table 2. Factor analyses.

	I		II				III				IV			V			VI
	OSI_2.0		Evolution/bigbang				Evolution/bigbang religiosity				Alternative evolution/bigbang			Alternative plus religiosity			Climate Change
	1	2	1	2	3	4	1	2	3	4	1	2	3	1	2	3	1	2	3
RADIOACTIVE	0.59		0.61		0.36		0.61		0.36		0.59			0.60			0.57
LASERS	0.62		0.57			0.33	0.56				0.55		−0.36	0.56			0.62
ELECTRONS	0.43	0.31	0.44				0.43		−0.37		0.45	0.33		0.47			0.41		−0.39
NITROGEN	0.52	0.33	0.49				0.49		−0.34		0.53			0.52			0.50		−0.37
COPERNICUS1	0.64		0.58				0.57				0.69			0.71			0.64
ANTIBIOTICS	0.57		0.66				0.64				0.50			0.49			0.57
VALID	0.54		0.51				0.51				0.53			0.55			0.52
PROB1	0.66	−0.34	0.66	−0.36	0.39		0.63				0.62			0.62			0.65
PROB2	0.52		0.52			−0.33	0.50				0.41			0.50			0.47
DIE	0.70		0.75				0.74				0.67			0.70			0.70
BUCKS	0.64		0.65				0.64				0.63	−0.30		0.63		−0.37	0.63
SWEEP	0.80		0.81				0.81				0.76			0.76			0.78
DISEASE1	0.66		0.68				0.68				0.61	−0.35		0.66		−0.31	0.64
DISEASE2	0.61		0.63			−0.32	0.61				0.55	−0.36		0.61		−0.38	0.58
CONDITIONAL	0.69		0.69		−0.46		0.69			−0.30	***	***	***	***	***	***	0.74
WIDGET	0.63		0.66				0.65				0.65			0.63			0.62
BATBALL	0.74		0.73				0.71				***	***	***	***	***	***	0.72
LILLYPAD	0.93		***	***	***	***	***	***	***	***	0.93			0.93			0.92
EVOLUTION	NA	NA	0.30	0.74	0.32		0.36	−0.69			0.58	0.48		0.64	0.30		NA	NA	NA
BIGBANG	NA	NA	0.40	0.75			0.46	−0.62			0.55	0.49		0.60	0.37		NA	NA	NA
CHURCH	NA	NA	NA	NA	NA	NA		−0.69			NA	NA	NA		0.80		NA	NA	NA
PRAYER	NA	NA	NA	NA	NA	NA		−0.69			NA	NA	NA		0.82		NA	NA	NA
RELIGIMP	NA	NA	NA	NA	NA	NA	−0.43	−0.80			NA	NA	NA		0.83		NA	NA	NA
AGW	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA		−0.84
GW_RISK	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA		−0.85
LIBCON	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA		0.77
DEM_REPUB	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA		0.78
N	2000		506				505				494			494			1751
Eigenvalue	7.5	0.7	7.1	1.5	1.1	0.7	7.3	2.7	1.2	0.7	6.7	1.2	0.8	7.3	2.5	0.9	7.4	2.7	0.7
Variance explained	0.87	0.07	0.63	0.13	0.10	0.06	0.53	0.20	0.09	0.05	0.64	0.11	0.08	0.55	0.19	0.07	0.64	0.24	.06

Notes: Principal factor method (without rotation) applied to a mixed tetrachoric-polychoric correlation matrix for all analyses (Bartholomew, Knott & Moustaki 2011, 191–93). Factor loadings <0.30 are omitted.

‘NA’ indicates that specified variable was not relevant to the analysis.

*** Indicates that the specified variable was one of a pair of items in which no subject who supplied an incorrect answer to one supplied a correct answer to the other. Because a maximum likelihood estimate of the tetrachoric or polychoric correlation cannot be calculated under these circumstances, the variable was omitted.

The Cronbach’s α was 0.83 for the scale as a whole. This score suggests OSI_2.0 can be expected to furnish a highly reliable measure of the posited ‘OSI’ disposition. But as will be discussed next, the use of item response theory to score responses enables a more informative assessment of scale reliability across the range of the entire latent OSI disposition.

4.2. Item response theory

A two-parameter item response theory model was used to score OSI_2.0 (Table 3). In such a model, the item ‘difficulty’ parameter reflects the level of the latent disposition at which a respondent’s probability of a correct answer is 0.5; the ‘slope’ parameter reflects how dramatically the probability approaches 0 or 1 as the respondent’s disposition level either falls short of or exceeds that level (Figure 1). Weighting items consistently with these parameters enables the measurement precision of the scale (its reliability) to be assessed across the range of the continuous latent variable. This property makes IRT desirable for constructing and scoring scales that a researcher can be confident discriminate among levels of the underlying disposition either uniformly across the range of that disposition or with greater precision within particular ranges of interest (DeMars 2010; Embretson 2010; Embretson and Reise 2000).

Table 3. IRT item parameters.

	Difficulty	Slope
RADIOACTIVE	−1.58	0.81
LASERS	−0.80	0.75
ELECTRONS	−1.18	0.45
NITROGEN	1.37	0.57
COPERNICUS1	−0.45	0.77
ANTIBIOTICS	−0.71	0.66
VALID	−1.11	0.62
PROB1	−1.59	0.94
PROB2	−1.25	0.62
DIE	−0.28	0.89
BUCKS	−0.27	0.76
SWEEP	0.81	1.06
DISEASE1	−1.05	0.89
DISEASE2	−1.28	0.80
CONDITIONAL	2.24	0.96
WIDGET	1.06	0.74
BATBALL	1.64	1.08
LILLYPAD	0.91	1.87

Notes: The ‘difficulty’ parameter (often denoted ‘b’) refers to the point along the continuous ‘ordinary science intelligence’ disposition at which someone with that OSI score would be expected to be as likely to select the right as the wrong answer. ‘Slope’ (often denoted ‘a’) characterizes the degree to which that probability decreases or increases in relation to corresponding changes in OSI and thus how discriminating the item is among individuals’ whose latent disposition level is close to that item’s difficulty-parameter value.

Figure 1. Illustrative item response curves.

Notes: Responsive curves illustrate the relative difficulty and discrimination of OSI_2.0 items. Conditional, with a difficulty parameter of 2.2, is a very difficult item, whereas Valid, with a difficulty parameter of −1.1, is relatively easy. Nitrogen is moderate in difficulty (b = 1.4), but is the least discriminating of the items – that is, has the least steep curve parameter.

Fitting such a model to OSI_2.0 suggests a scoring hierarchy among the different sets of items. The ‘method’ and ‘fact’ items are the easiest, and thus function primarily to rank respondents lowest in ‘OSI.’ Although the most difficult item is CONDITIONAL, a member of the Numeracy set, the CRT items generally are the hardest and thus contribute the most to the ranking of the highest levels of OSI. As a set, the Numeracy items are middling in difficulty, but in fact they contribute to discrimination across the entire OSI range (Table 3 and Figure 2).

Figure 2. Test information functions and reliability.

Notes: The ‘test information function’ indicates the relative precision of an IRT-scored latent variable measurement instrument across the range of the continuous latent disposition. The function can be transformed into a reliability coefficient, which enables reliability to be assessed continuously across the range of the disposition as well (DeMars 2010). The test information functions and variable reliability coefficients for the OSI_2.0 subcomponents and for the assessment instrument as a whole are indicated by separate curves.

The subcomponents of OSI_2.0 combine to form a unitary scale that has high reliability across the entire range of the OSI disposition. The variable IRT reliability coefficient is highest (0.84) at +1 SD but remains above 0.70 even at −2 and +2 SDs.

The reliability of the individual subcomponents, in contrast, can be seen to be concentrated over much smaller portions of the OSI disposition. For example, CRT, the reliability of which peaks at 0.71 at +1 SD, drops below 0.50 at +1.7 SDs. Because of the extreme difficulty of the test, the reliability of CRT drops to 0.0 at the mean, rendering it unable on its own to distinguish varying levels of OSI for approximately half the members of the general population.

5. External validity

Dishearteningly, there is scant research validating the NSF Science Indictor battery or like ‘science literacy’ measures featured in the study of risk perception and science communication (Pardo & Calvo 2004). The dearth of such scholarship furnishes much of the motivation for the development of OSI_2.0.

Evidence of the external validity of OSI_2.0 consists in part in the extensive research examining the power of the items that form its quantitative reasoning and cognitive reflection subcomponents to predict reasoning proficiencies and dispositions essential to recognizing valid scientific knowledge and giving it proper effect. To supplement this evidence, OSI_2.0 scores were used to assess study subjects’ performance on a standard ‘covariance detection’ problem.

A ‘covariance detection’ problem requires respondents to analyze the results of a fictional experiment (Arkes and Harkness 1983). To correctly answer the problem, respondents must contrast the ratio of positive to negative outcomes in a treatment condition with the ratio of such outcomes observed in a control condition, information summarized in a 2 × 2 contingency table or equivalent. Respondents will get the wrong answer if they use either of two common heuristic substitutes for such reasoning: a comparison of the number of positive outcomes to negative outcomes in the treatment condition; or a comparison of the number of positive outcomes in the treatment to the number of positive outcomes in the control (Figure 3). The problem tests not only individuals’ motivation and ability but also their spontaneous recognition of the need to perform an analytical task essential to making valid causal inferences (Stanovich 2009, 2011).

Figure 3. Covariance detection problem.

Notes: The correct response requires subjects to compare the ratio of the values in cells ‘A’ and ‘B’ to the ratio of the values in cells ‘C’ and ‘D’.

The task that the covariance problem features is integral to identifying and giving proper effect to valid scientific information. If OSI_2.0 successfully measures OSI, then it ought to be a strong predictor of performance on this problem.

This turns out to be so. Individuals who score low on the scale are highly unlikely to answer the question correctly. They become progressively more likely to give the right response as their scores progress, and at about the 90th percentile of OSI_2.0 there is approximately a 70% probability a respondent will answer the covariance problem correctly (Figure 4). This relationship furnishes evidence of the external validity of OSI_2.0.

Figure 4. Predictive power of OSI_2.0 for performance on the covariance detection problem.

Notes: N = 372. Predicted probabilities derived by Monte Carlo simulation applied to logistic regression model. Covariance problem is ‘skin rash’ version in Kahan et al. (2013). Colored bars are 0.95 CIs.

The same is true for the relationship between OSI_2.0 and other measures one would expect to play a role in OSI. Unsurprisingly, there is a modest positive correlation (r = 0.40, p < 0.01) between OSI_2.0 scores and level of education. The same degree of association (r = 0.40, p < 0.01) exists between OSI_2.0 and scores on Baron’s ‘actively open minded thinking’ (‘AOT’) scale, which measures the disposition to seek out and fairly evaluate evidence contrary to one’s existing beliefs (Haron, Ritov & Mellers 2013; Baron 2008).

AOT and education are correlated (r = 0.21, p < 0.01), too, but less strongly than either is with OSI_2.0. In addition, neither AOT nor education is as strong a predictor as OSI_2.0 of performance on the covariance detection problem (Figure 5).

Figure 5. Predictive power of education and AOT for performance on the covariance detection problem.

Notes: N = 372. Predicted probabilities derived by Monte Carlo simulation applied to logistic regression models. Covariance problem is ‘skin rash’ version in Kahan et al. (2013). AOT scale scores are normalized; the asymmetry of the values above and below the mean reflect the extreme left-shift in score distributions. Colored bars are 0.95 CIs.

All these relationships supply additional grounds for confidence in the external validity of OSI_2.0. While education and actively open-minded thinking are both integral to OSI, OSI posits a science comprehension capacity that is more specific than, and that does not simply reduce to, either education or AOT. The modest size of the correlations between both education and AOT, on the one hand, and OSI_2.0, on the other, show that the former are failing to account fully for the latent combination of knowledge, skills, and motivations that is being measured by the latter. Further, the superiority of OSI_2.0 in predicting covariance detection supports the inference that the capacities uniquely accounted for by OSI_2.0 are exactly the ones that figure in proficiencies distinctive of OSI.

6. Covariates

Normally distributed in the study sample (Figure 6), OSI_2.0 scores had a small correlation (r = 0.08, p < 0.01) with being male, and only a slightly larger one (r = 0.18, p < 0.01) with being white. The signs of the correlations are consistent with relationships between gender and race on the one hand, and performance on standardized measures of scientific literacy, mathematical skill, and performance on critical reasoning measures, on the other. The size of the effects, however, is smaller than ones typically observed (e.g. Frederick 2005; Kahan 2013; Roth et al. 2001).

Figure 6. Distribution of OSI_2.0 scores.

Note: N = 2000. Sores for 18-item scale scored on the basis of a two-parameter IRT model.

There was no meaningful correlation (r = 0.01, p = 0.59) between OSI_2.0 and political orientation as measured by a scale (α = 0.78) that combines self-reported liberal–conservative ideology and partisan self-identification. This is consistent with findings that show that objective measures of reasoning proficiency, as opposed to various self-report ones that are in fact weaker predictors of forms of behavior associated with critical reasoning (Liberali et al. 2011; Toplak, West, and Stanovich 2011), do not meaningfully vary in relation to political outlooks in general population studies (Baron 2015; Kahan 2013; Kahan et al. 2013).

7. Relationship to ‘belief in evolution’ and ‘belief in the big bang’

The value of a valid and reliable public science comprehension measure ultimately turns on what one can do with it. The relationship of acceptance or non-acceptance of evolution to public science comprehension is an issue of considerable scholarly interest (e.g. Miller, Scott, and Okamoto 2006). Use of OSI_2.0 suggests that for members of a general public sample in the US, at least, variance in ‘belief in’ evolution does not validly convey information about science comprehension. On the contrary, OSI_2.0 reveals that items assessing acceptance of evolution and related facts are biased measures of that disposition.

Among the NSF Indicators ‘basic facts’ items are ones relating to the role of evolution in the natural history of human beings and another relating to a popular rendering of the inflationary theory of the universe in cosmology:

EVOLUTION. Human beings, as we know them today, developed from earlier species of animals (True/false).

BIGBANG. The universe began with a huge explosion (True/false).

‘Differential item function’ is an IRT technique used to assesses whether indicators of a latent variable display the same measurement properties across distinct subpopulations. It is used in the field of standardized testing to identify questions that are ‘culturally biased,’ which refers not to group animus on the part of the test designers or administrators but rather to the systematic misestimation of the aptitude of a group of test takers in whom responses to those questions do not bear the requisite relationship to the aptitude or skill being assessed (Osterlind and Everson 2009).

DIF analysis shows that the Indicators’ EVOLUTION and BIGBANG items are both biased with respect to individuals who display a relatively high degree of religiosity (Figure 7, top two panels). As the form of science comprehension measured by OSI_2.0 increased, the probability of a correct response to the items increased substantially more for relatively non-religious individuals than for relatively religious ones. The impact was especially dramatic for EVOLUTION: in relation to that question, increasing OSI_2.0 scores had no effect on the probability of answering the question correctly for individuals of modestly high religiosity. In other words, for such individuals, the item is simply an invalid indicator of OSI. On BIGBANG, the increasing probability of a correct response that one expects to see in such an indicator was present for both relatively religious and non-religious individuals. Nevertheless, the slope was substantially steeper for the latter (Figure 7).

Figure 7. Differential item function analysis for NSF Indicator EVOLUTION and BIGBANG items.

Notes: N’s from top left to bottom right = 1011, 9888, 999, and 1000. Predicted probabilities derived via Monte Carlo Simulation based on logistic regression. Predicted probabilities for ‘Below’ and ‘Above avg. religiosity’ determined by setting predictor on religiosity scale at −1 and +1 SD, respectively. Colored bars reflect 0.95 confidence intervals.

It bears emphasis that this differential is not a consequence of any relationship between religiosity and science comprehension. There is in fact a small negative correlation (r = −0.18, p < 0.01) between OSI_2.0 and religiosity, which was measured with a scale that combined self-reported church attendance, frequency of prayer, and declared ‘importance’ of religion (α = 0.86). But the DIF analysis examines the discrepancy in the probability of responding correctly among religious and non-religious test takers who have the same OSI level. Accordingly, an otherwise valid science-comprehension test containing these items would systematically underestimate the OSI of non-religious respondents relative to religious ones of the same OSI capacity.

A similar analysis was performed on variants of EVOLUTION and BIGBANG. The alternative items contained introductory clauses that expressly identified the asserted proposition with scientific understandings of the natural history of human beings and the expansionary theory of the universe (National Science Board 2014). One might expect these variants to mitigate the tension between affirmative responses and religious beliefs inconsistent with the asserted proposition. In fact, for both items, the differential between religious and non-religious respondents was substantially reduced (Figure 7).

The conclusion that EVOLUTION and BIGBANG do not validly measure OSI but rather some aspect of personal identity relating to religiosity was also supported by common factor analysis. As indicated in Section 4.1, the covariance structure for the 18 OSI_2.0 items suggests that the scale is reasonably understood as measuring a single, unidimensional latent variable (Table 2, I). When the standard NSF Indicator EVOLUTION and BIGBANG items were included, however, factor analysis suggested that the resulting covariance structure was better explained by positing two factors: one consisting of the 18 OSI_2.0 items, and the other of EVOLUTION and BIGBANG (Table 2, II). In an additional analysis, EVOLUTION and BIGBANG loaded along with the religiosity items on a factor that was separate from the factor explaining the covariance structure of the OSI_2.0 items (Table 2, III).

This analysis suggests that, for US respondents at least, the standard EVOLUTION and BIGBANG items could be viewed along with the religiosity items as indicators of a latent religiosity variable, albeit one with a reliability score (α = 0.85) not materially different from the one formed by the three religiosity items alone.

This result bolsters the conclusion of Roos (2012), who reports that the EVOLUTION and BIGBANG items, along with another on continental drift and another on literal belief in the Bible, cohered as a factor separate from the other NSF Indicator items in a structural equation model. They are likewise consistent with Rissler, Duncan, and Caruso (2014), who report finding that religiosity measures better predict ‘acceptance of evolution’ among university students than did measures of educational attainment.

Similar analyses were performed substituting the alternative EVOLUTION and BIGBANG items. In a factor analysis, the alternative EVOLUTION and BIGBANG variants still loaded more heavily on a second factor distinct from the one formed by the 18 OSI_2.0 item (Table 2, IV). But when the three religiosity items were added (Table 2, V), the variants displayed only modest loading coefficients on the second, religiosity factor (β = 0.30 for EVOLUTION; β = 0.36 for BIGBANG), and the aggregate religiosity scale including these items displayed lower reliability (α = 0.80) than did the three-item scale formed by the church-attendance, frequency-of-prayer, and importance-of-religion items alone. This result supports the conclusion that the reworded items are not valid indicators of either science comprehension or religiosity, although they might well function as indicators of the former if administered to subpopulations of relatively nonreligious individuals only.

The validity of the EVOLUTION and BIGBANG items as indicators of science comprehension when administered to members of the general population in the US has provoked considerable controversy (Mooney 2010). The analysis of these items in relation to OSI_2.0 supports the NSF’s recent decision to exclude these items when assessing performance on the Indicators’ science literacy battery over time and across societies (National Science Board 2014). The OSI_2.0 scale similarly omits them so as to avoid the bias that their administration to a US general population sample would entail.

The proportion of ‘false’ responses to EVOLUTION and BIGBANG is lower in European samples (Miller, Scott, and Okamoto 2006). An appropriately cross-culturally validated version of OSI_2.0 could be used to determine whether these items are in fact free of cultural bias when administered to such samples, and whether they otherwise make a sufficient contribution to discernment of science comprehension to justify including them in the scale when it is administered to residents of those countries. A cross-culturally valid OSI_2.0 could also be used to assure that other items bear the same relationship to the latent science comprehension disposition across and within national samples, an issue that has not been meaningfully investigated in connection with current science-literacy assessments (Pardo and Calvo 2004; Solano-Flores and Nelson-Barber 2001).

8. Relationship to global warming beliefs and risk perceptions

There is also considerable interest in whether political conflict over human-caused global warming is attributable to deficiencies in public science literacy (Miller 2004) or some related form of critical reasoning (Weber 2006). Performance on OSI_2.0 suggests the answer, at least in the US, is No. Professions of belief in human-caused climate change and perceptions of the risks it poses are more convincingly viewed as indicators of an aspect of personal identity associated with individuals’ political outlooks than as indicators of science comprehension.

When study subjects were asked whether human activity is creating global warming, the probability that they would respond affirmatively displayed only a weak relationship to their OSI_2.0 scores. The probability of such a response is 44% (±4%, 0.95 LC) for an individual with a mean OSI_2.0 score. That probability dips down only modestly to 40% (±5%, 0.95 LC), and rises only modestly to 52% (±4%, 0.95 LC), for individuals with scores one standard deviation above and below the mean, respectively (Figure 8). This relative insensitivity to differences in OSI_2.0 scores contrasts dramatically with the sensitivity displayed by valid OSI_2.0 indicators (Figure 1).

Figure 8. Item-response (and DIF) for ‘belief in’ human-caused global warming.

Notes: N = 1769. Predicted probabilities derived via Monte Carlo simulation based on logistic regression. Predicted probabilities for ‘Liberal Democrat’ and ‘Conservative Republican’ determined by setting predictor on Left_right scale – a summated index formed by aggregation of self-reported liberal-conservative ideology and political-party identification (α = 0.78) – at −1 and +1 SD, respectively. Colored bars reflect 0.95 confidence intervals.

More importantly, this item bears a radically different relationship to OSI for individuals of opposing political outlooks. Whereas the probability of belief in human-caused global warming increases slightly for relatively left-leaning individuals, the probability is unaffected for right-leaning ones as OSI_2.0 scores increase. As a result, consistent with previous studies of the relationship between science comprehension and global warming risk perceptions (Hamilton, Cutler, and Schaefer 2012; Kahan et al. 2012), higher OSI_2.0 scores magnify polarization.

Factor analysis also supports the inference that global warming beliefs and risk perceptions are more convincingly treated as indicators of a latent identity that features a left–right political orientation. When the ‘belief in’ climate change item, a global warming risk-rating item, and the two political outlook measures (liberal–conservative ideology and partisan self-identification) are analyzed along with the 18 OSI_2.0 items, the covariance structure is best explained by positing two factors: one consisting of the political outlook measures and the global-warming items, and the other by the OSI_2.0 items (Table 2, VI). Consistent with this analysis, aggregating the global warming items with the political outlook measures forms an even more reliable scale of subjects’ latent political orientation (α = 0.84) than do the two political outlook measures by themselves. In sum, how members of the US general public respond to items assessing their acceptance of human-caused global warming, like ones assessing their acceptance of human evolution, are more convincingly viewed as indicators of who they are, culturally speaking, than of what they know about science.

This conclusion is of obvious significance for assessing what sorts of science communication are likely to be efficacious for dispelling political conflict over climate change in the US (Kahan 2015a). Because the ordinary citizens most able to comprehend scientific evidence are the most politically polarized, it makes little sense to believe disseminating more information will promote convergence of understanding; what is needed are forms of communication, and forms of political deliberation, that disentangle positions on climate change from the cultural meanings that motivate individuals with competing identities to credit such information in opposing patterns (Kahan 2015b). Evidence that similar dynamics of cultural cognition account for conflict over evidence of climate change in other democratic societies (Aasen 2015; Shi, Visschers, and Siegrist 2015) suggests the value of using OSI_2.0 to investigate whether differences in science comprehension are as inconsequential to political conflict in those societies as they are in the US.

9. What next?

The scale-development process that generated OSI_2.0 remains ongoing. Future research will include additional forms of behavioral validation, including assessments of the scale’s power to predict individuals’ capacity to recognize and make appropriate use of valid science in various real-world domains.

Efforts to develop a short-form version of OSI_2.0 and to cross-culturally validate the scale are also anticipated. Item response theory, a scale-development and scoring tool that has been oddly neglected in the study of public science comprehension, is well suited to both of these tasks. The information IRT supplies on the relative contribution that individual items make to measurement precision at various levels of a latent trait can be sued both to economize on items (Weller et al. 2012) and to test for measurement commensurability across different test populations (Richardson and Coates 2014).

10. Conclusion: incremental progress dominates standing still

The scale development exercise that generated OSI_2.0 is offered as an admittedly modest contribution to an objective of grand dimensions. How ordinary citizens come to know what is collectively known by science is simultaneously a mystery that excites deep scholarly curiosity and a practical problem that motivates urgent attention by those charged with assuring democratic societies make effective use of the collective knowledge at their disposal. An appropriately discerning and focused instrument for measuring individual differences in the cognitive capacities essential to recognizing what is known to science is essential to progress in these convergent inquiries.

The claim made on behalf of OSI_2.0 is not that it fully satisfies this need. It is presented instead to show the large degree of progress that can be made toward creating such an instrument, and the likely advances in insight that can be realized in the interim, if scholars studying risk perception and science communication make adapting and refining admittedly imperfect existing measures, rather than passively employing them as they are, a routine component of their ongoing explorations.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. To facilitate such research, the data reported in this study, along with codebook and user guide, can be downloaded from http://www.culturalcognition.net/osi_2/.