Development of the International Ocean Literacy Survey: measuring knowledge across the world.

Abstract

The Ocean Literacy movement began in the U.S. in the early 2000s, and has recently become an international effort. The focus on marine environmental issues and marine education is increasing, and yet it has been difficult to show progress of the ocean literacy movement, in part, because no widely adopted measurement tool exists. The International Ocean Literacy Survey (IOLS) aims to serve as a community-based measurement tool that allows the comparison of levels of ocean knowledge across time and location. The IOLS has already been subjected to two rounds of field testing. The results from the second testing, presented in this paper, provide evidence that the IOLS is psychometrically valid and reliable, and has a single factor structure across 17 languages and 24 countries. The analyses have also guided the construction of a third improved version that will be further tested in 2018.

Keywords:

• Ocean literacy
• measure development
• marine education
• ocean sciences education
• Rasch

1. Introduction

The ocean covers 71% of our planet and holds 97% of the Earth’s water. It is a key ecosystem that encompasses most of the living space on Earth and plays several crucial roles that support the health of the planet and the livelihood of humans. The ocean provides about 15% of the total protein consumed by people across the globe (World Health Organization 2012), drives a substantial portion of the global economy (OECD 2016), regulates the climate and weather, and slows climate change by absorbing about 40% of the carbon dioxide that is being emitted into the atmosphere at an increasing pace by human activities since the beginning of the industrial revolution (DeVries, Holzer, and Primeau 2017). Clearly, the ocean supports life on Earth and provides us with tremendous economic, social, and environmental benefits. Moreover, the ocean is not solely a resource for humans, but has intrinsic value for its own sake and for its inhabitants.

Despite its value, the ocean is showing significant signs of change as a result of human activities. Average sea surface temperatures are rising; the chemistry of the ocean itself is changing, which impacts marine ecosystems and their services (Pörtner et al. 2014); many commercially important fish stocks are fully exploited, overexploited, depleted or recovering from depletion, putting marine biodiversity at risk; and the increasing environmental, social and economic pressures from the exploding human population have led to massive alteration of marine habitats (Rockström et al. 2009). The increasing modification, degradation and contamination of the ocean directly threatens humankind by putting at risk many associated goods, services, and aesthetic and spiritual benefits. Since this has direct impact on communities and nations worldwide, and can be attributed to the lifestyles, decision-making, and choices of individuals, as well as, governments and industry, the involvement of each and every person in understanding the importance of the ocean and the need to protect it are essential (Fauville 2017). For individuals to become thoughtful participants in the debate about solutions to marine environmental issues, they need to be ocean literate.

While the primary meaning of the concept of literacy solely refers to the ability to read and write, this concept has evolved through time. Its meaning has been extended to include the ability to understand a text and be able to make sense of and use it in the world for relevant purposes (Wertsch 1991). More recently, UNESCO expanded the concept of literacy by stating that, “Literacy involves a continuum of learning in enabling individuals to achieve his or her goals, develop his or her knowledge and potential, and participate fully in community and wider society.” (UNESCO 2005, 21)

Various types of literacy, such as science literacy, digital literacy, environmental literacy or ocean literacy point to skills that are essential in our time and that include but go beyond reading and writing in the classical sense.

Cava et al. (2005) defined Ocean Literacy as an understanding of the ocean’s influence on us and our influence on the ocean. Elaborating on this understanding of interdependencies, the authors define an ocean literate person as someone who understands the essential principles and fundamental concepts about the functioning of the ocean, is able to communicate about the ocean in meaningful ways, and is able to make informed and responsible decisions regarding the ocean and its resources.

Ocean Literacy is aligned with the objectives of environmental education as defined by UNESCO in 1975:

•	Awareness: to help social groups and individuals acquire an awareness of and sensitivity to the global environment and its allied problems.
•	Attitude: to help social groups and individuals acquire a set of values and feelings of concern for the environment, as well as, the motivation to actively participate in environmental improvement and protection.
•	Skills: to help social groups and individuals acquire the skills for identifying and solving environmental problems.
•	Participation: to provide social groups and individuals with an opportunity to be actively involved at all levels in working towards resolution of environmental problems. (UNESCO (United Nations of Education Scientific and Cultural Organisation) 1975, 26–27).

Moreover, according to the National Research Council (2010), which reviewed the education programs of the U.S. National Oceanic and Atmospheric Administration, ocean sciences education, as a means to promote Ocean Literacy, is situated at the intersection of environmental education and STEM (Science, Technology, Engineering and Mathematics) education.

Previous research from several countries has shown that citizens have a limited understanding of marine-related phenomena (Brody 1996; Fortner and Mayer 1991; Guest, Lotze, and Wallace 2015), hold misconceptions (Ballantyne 2004), and/or have little understanding of marine environmental issues and protection (Eddy 2014). This lack of familiarity with the ocean can be associated with the fact that ocean concepts are rarely represented in the formal science education curriculum ( Fauville et al. forthcoming; Hoffman, Martos, and Barstow 2007). This omission of ocean related topics triggered grass roots and policy-driven responses aimed at giving the ocean its legitimate central role in science and environmental education.

The grass roots movement for Ocean Literacy started in 2002 in the United States with concerned scientists, and formal and informal educators raising their voices to include ocean sciences in the school curriculum. This resulted in a two-week online workshop and extensive follow-up dialogue between hundreds of U.S. ocean sciences and education stakeholders in 2004 (Cava et al. 2005). This community discussed what knowledge citizens should master by the end of Grade 12 in the U.S. to be considered ocean literate (Schoedinger, Tran, and Whitley 2010), and to build consensus that Ocean Literacy is an essential component of science literacy (Strang, Schoedinger, and de Charon 2007). This process resulted in seven overarching ideas called the essential principles of Ocean Literacy and 44 fundamental concepts (In the 2013 revision, an additional concept was added, resulting in the current total of 45) that elaborate each principle (Figure 1). This ‘ocean literacy framework’ was originally published as, Ocean Literacy: The Essential Principles and Fundamental Concepts of Ocean Sciences Grades K-12 (National Geographic Society et al. 2005), revised in 2013 (National Oceanic and Atmospheric Administration 2013), and supplemented by The Ocean Literacy Scope and Sequence for Grades K-12 (National Marine Educators Association 2010). Rather than serving as a comprehensive ocean sciences curriculum, the Ocean Literacy principles and concepts serve to answer the question, ‘what ideas about the ocean are so fundamental and important that if students did not understand them, they could not be considered science literate?’

Figure 1. The seven Essential Principles of Ocean Sciences.

Since 2004, there has been a growing effort to improve Ocean Literacy around the world (Dupont and Fauville 2017). The U.S. National Science Foundation invested over $40 M over a 12 year period in a network of Centers for Ocean Sciences Education Excellence, and the European Union invested more than 7 M Euro in two large, multi-year international projects, SeaChange and ResponSEAble. The U.S. National Oceanic and Atmospheric Administration is currently collaborating with Canada and the European Union on a Transatlantic Ocean Literacy initiative. New professional organizations and networks, similar to the longstanding U.S. National Marine Educators Association, have emerged, including the International Pacific Marine Educators Network, the European Marine Science Educators Association, the Canadian Network for Ocean Education and the Asia Marine Educators Association. All of these efforts have the stated objective of improving Ocean Literacy. Despite these increased investments in Ocean Literacy, it has been difficult to show progress of the Ocean Literacy movement, in part, because no widely adopted measurement tool exists. Previous researchers on ocean knowledge have used a wide variety of methods, target groups, content goals, and conceptual frameworks. The need for a shared measurement tool has been expressed by members of the Ocean Literacy community around the world to determine the impact of particular interventions, to establish a baseline of Ocean Literacy in particular communities, to detect in change in Ocean Literacy levels in particular communities over time, and to compare differences in levels of Ocean Literacy across communities.

The International Ocean Literacy Survey (IOLS), presented in this paper, aims to serve as a community-based measurement tool that allows the comparison of levels of ocean knowledge across time and location. Community-based in this context refers to two things: (1) The IOLS is being developed on a voluntary, grass roots basis by members of the Ocean Literacy community, and (2) While initial testing of the IOLS is being conducted on a national level for the purpose of validating the instrument in a variety of languages and populations, we anticipate that the finished survey will be most widely used at the community level.

In response to the considerable international need, the lack of funding sources for international collaborations, and the enthusiasm of the Ocean Literacy community, the authors have marshaled the contributions of dozens of organizations, institutions, investigators, and practitioners to engage in a somewhat non-traditional, iterative, community-based research design. We have assembled a survey instrument that has been subjected to two rounds of field testing (the first in English in the U.S., the second in 17 languages in 24 countries), has yielded promising results, and is poised for a third international test. We envision that this paper is the first installment in a series that will tell the story of our efforts to create a nimble yet rigorous process for conducting research that is immediately helpful to both practitioners and investigators. In addition to measuring levels of Ocean Literacy around the world, we also intend to inform other large scale international research-based collaborations.

2. Methodology

Ocean Literacy includes three dimensions: knowledge, communication, and decision-making. These three dimensions represent approximations, stated in lay terms for public and practitioner audiences, of the accepted objectives of environmental education described by UNESCO (1975) and of environmental literacy described by the North American Association of Environmental Education (NAAEE 2011). The IOLS currently focuses on measuring knowledge as a first step toward a more integrated set of measurement tools addressing each aspect of ocean literacy. We are fully aware that levels of knowledge about the ocean do not alone correlate or lead to all three dimensions of ocean literacy. Two signature challenges associated with this project are that (1) its success depends on collaboarion and cooperation among dozens of disparate members of the marine education community from many countries, cultures and linguistic groups, and (2) that the project is as yet largely un-funded. Given these challenges, we made a strategic decision to begin our efforts by focusing on knowedge as the area where there is broad agreement about the content framework (National Oceanic and Atmospheric Administration 2013) that constitutes the foundation of our work across the field.

The IOLS has been comprised of a series of multiple-choice questions addressing all seven principles and most of the 45 concepts of Ocean Literacy (future versions will address all 45). Since these principles and concepts were defined by the Ocean Literacy community as what students should know by the end of high school, the target audience for the IOLS is 16–18 year old students. This specific age range was selected so that we could capture a comparable sample of youth near the end of their compulsory education across variations in science course taking both within and across countries. The Ocean Literacy Scope and Sequence for Grades K-12 (National Marine Educators Association 2010), especially the section related to Grades 9–12 (equivalent to ages 14–18), provides a much more detailed and developmental set of sub-concepts that lead to full understanding of the seven principles and 45 concepts. Assessing understanding of the entire Ocean Literacy Scope and Sequence in a survey such as the IOLS would require significantly more items and would be impractical. For our purposes, it is most important to assess students’ understanding of the terminal principles and concepts, and less important to assess students’ developmental progress toward understanding them. Therefore, the items in the IOLS refer to the Ocean Literacy principles and concepts. There are not an equivalent number of items for each of the seven principles since each principle represents different amounts and depths of knowledge.

As a first step in the community-wide participation in the development of an open-source instrument, researchers contributed previously developed whole surveys or individual multiple-choice items to the IOLS project (COSEE unpublished work; Dromgool, Burke, and Allard 2015; Greely 2008; Guest 2013; Robinson and Sardo 2015). These items were compiled, reviewed, culled for redundancy and/or edited. In addition, many new items were generated by a team led by the authors, as well as, members of the National Marine Educators Association, and several volunteer ocean scientists from several countries. A pilot study was conducted in June 2016 with an initial set of 50 survey items that were administered to 417 U.S. 16 to 18 year old students using the online survey software Qualtrics. Respondents were recruited from existing networks of teachers with a special interest in education about the ocean. This pilot study helped us to identify problematic items that were, for example, outside of the range of appropriate difficulty (too easy or too hard) or appeared to have responses that were driven by something other than Ocean Literacy (e.g., reading comprehension). Based on these results, we revised the items to create a more cohesive instrument that better aligned with the concepts of Ocean Literacy.

The second version of the IOLS is the focus of this study. It included 48 questions aligning with most of the Ocean Literacy principles and concepts (see Appendix 1). For example as presented in Appendix 1, Q6 (How is sea level measured? A. Average depth of the ocean. B. Average height of the ocean relative to the land. C. Level of the ocean at the lowest tide. D. Level of the ocean at the highest tide.) aligns with Concept 1d

Sea level is the average height of the ocean relative to the land, taking into account the differences caused by tides. Sea level changes as plate tectonics cause the volume of ocean basins and the height of the land to change. It changes as ice caps on land melt or grow. It also changes as sea water expands and contracts when ocean water warms and cools.

The IOLS was originally designed in English. Due to enthusiasm in the Ocean Literacy community it was translated by volunteer researchers into 16 languages (Catalan, Croatian, Danish, Dutch, French, German, Greek, Italian, Japanese, Norwegian, Polish, Portuguese, Simplified Chinese, Spanish, Swedish, and Traditional Chinese). This process of translation served two purposes: to create versions of the instrument for use in other languages and to function as a systematic review of the items themselves. In the absence of being able to conduct cognitive interviews with students from each of the countries and representing each of the languages tested, we invited the translators to provide feedback on the items themselves; specifically, the ocean science content, clarity of the wording, and potential complexities introduced by the translation process.

At the end of August 2016, The IOLS V2 was made available on the online software Qualtrics. The authors made use of a wide range of mailing lists, private and professional social media platforms, and newsletters to invite educators to share the survey with their colleagues and to administer it to their 16–18 year old students around the world in the appropriate language for the population. Between August 2016 and October 2016, 6871 individuals agreed to be in the study.

3. Data analyses and results

Since not all questions on the assessment are ‘questions’ (e.g. fill in the blank) we use the term ‘item’ to refer to a combination of a ‘prompt’ and ‘response options.’ Response options refers to the choices from which respondents could select their response to the prompt. The response data are the response options chosen by respondents. Response data for all items were transformed into dichotomously scored data for psychometric analyses, that is, the responses were scored as incorrect (0) or correct (1) for each of the items. In some cases, to correctly respond to the prompt, the respondents needed to select more than one response option for a particular item. In these cases, each response option is scored separately, either responded to correctly or incorrectly and is treated as a separate item. This led to 79 unique items in the IOLS data.

Data were analyzed using the Rasch model (Rasch [1960] 1980) within the Item Response Theory (IRT) modeling framework. Mathematically, the probability of a correct response in the Rasch model can be expressed as

$$P(x/{\theta }_j)=\frac{\exp ({\theta }_j-{\beta }_i)}{1+\exp ({\theta }_j-{\beta }_i)}$$

where a response vector is represented by x = (x1,…, xi). Beta i is the difficulty parameter for item i, and theta j is the ability parameter for a respondent j. P represents the probability of a correct response to item i by a respondent j with ability theta. In this case, this model assumes that a respondent’s ability, ocean literacy knowledge, and the difficulty of the item (i.e. a fixed difficulty relative to the other items, not relative to the ability of the respondent) are the only factors that will influence whether or not the individual gets the item correct.

We performed a series of psychometric analyses to examine the measurement quality of the Ocean Literacy scale. We checked the assumption of the Rasch model that the items measure an underlying unidimensional trait, ocean literacy. Further, we examined the item characteristics, including item difficulty, item characteristic curve [ICC]), reliability, and the quality of the distractors (i.e. incorrect answer options). We also performed fit assessments to detect whether the set of items are consistent with the Rasch model at both the model and item levels; as well as differential item functioning (DIF) to ensure the items are functioning equivalently across subgroups (e.g. gender) in the data. The DIF analyses were implemented in DIF Analysis Software (DIFAS; Penfield 2005) and the rest of the psychometric analyses were implemented in R (Mair and Hatzinger 2007; R Core Team 2017; Revelle 2017; Yves 2012).

3.1. Descriptive statistics

Table 1 displays descriptive statistics of our study sample, showing gender and language for the survey. As can be seen in the table, many of the survey responses were either from Taiwan and completed in the survey in Traditional Chinese or from the United States and completed the survey in English. No other single country or language had a comparable sample size to these two, however, when taken as a whole, the survey responses across the countries and languages of Europe provide sufficient sample size for comparison to the U.S. English and Taiwanese samples. Within each country and language category we had comparable participation of males and females. In no way do we argue that our study sample is representative of the world, nor is the sample representative of the country and linguistic groups they are drawn from. Although effort was given to recruit samples in relatively similar ways across participating locations, the recruitment process was not uniformly systematic across countries, nor were they randomly drawn from a population. That is, these samples of convenience within each country do not reflect the overall population in that country. Therefore observed differences in scores are just as likely, if not more likely, due to variations in recruitment of the sample than variation in levels of Ocean Literacy of the population in those countries. This is an important limitation to possible inferences from these data.

Table 1. Descriptive statistics of the study sample.

	Male	Female	Trans*/Prefer not to answer/Open response	System missing	Valid percent
Chinese Traditional	1022	1078	2	204	33.6%
English	509	706	108	734	29.9%
Portuguese	113	206	57	340	10.4%
Chinese Simplified	186	317	1	1	7.3%
Croatian	42	76	16	174	4.5%
Swedish	43	57	12	47	2.3%
German	22	22	5	76	1.8%
Danish	11	40	7	66	1.8%
French	24	33	1	65	1.8%
Japanese	57	26	23	14	1.7%
Italian	23	19	3	71	1.7%
Greek	16	23	18	25	1.2%
Other European Languages (i.e. Spanish, Dutch, Catalan, Norwegian, Polish)	13	20	2	29	0.9%
	2092	2636	256	1887	6871

3.2. Dimensionality

IRT models, including the Rasch model, assume the items forming the scale are unidimensional, which means only a single latent trait (Hambleton, Swaminathan, and Rogers 1991), Ocean Literacy, drives the responses to the set of items. In this study, we tested this unidimensionality assumption using a Confirmatory Factor Analysis (CFA) with weighted least squares estimator, which is a robust estimator and allows for modeling categorical or ordinal data.

Unidimensionality assumption was evaluated via CFA. Rules of thumb (see Brown 2015; Hu and Bentler 1999) were a cutoff value close to 0.95 for Tucker-Lewis Index (TLI) and Comparative Fit Index (CFI), a cutoff value less than 0.08 for Standardized Root Mean Square Residual (SRMR), and a cutoff value less than 0.06 for Root Mean Square of Approximation (RMSEA); or using a combination of SRMR less than 0.09 and RMSEA less than 0.06; would generate lower Type II error rates with acceptable Type I error rates. Results of all CFA model fit indicators met the criteria (see Hu and Bentler 1999) indicating the IOLS scale did not violate the assumption of unidimensionality. The results indicate that the set of knowledge data fit the one-factor model well (i.e. χ² = 21923.61, df = 3002, p < .05; Comparative Fit Index [CFI]=0.919; Tucker-Lewis Index [TLI]=0.917; Root Mean Square of Approximation [RMSEA]=0.036, Standardized Root Mean Square Residual [SRMR]=0.038). All knowledge items, except for two (i.e. Q40, Q26_4, see Appendix 1) generated significant factor loadings to the single factor. This implies that all but these two items were psychometrically relevant to measuring Ocean Literacy. We have since revised both of these items for version 3 of this assessment (see Appendix 1 for revisions).

3.3. Reliability

The traditional way to determine reliability is to use Cronbach’s alpha. However, Cronbach’s alpha assumes that the underlying data are continuous variables, and in our case the data are coded as dichotomous (correct or incorrect). To account for the non-continuous underlying data we used the polychoric matrix for the internal consistency estimation and computed the ordinal reliability (Zumbo, Gadermann, and Zeisser 2007). The ordinal reliability of the knowledge scale was excellent (alpha = 0.94), indicating that the knowledge scale was a well-defined construct – Ocean Literacy. Equally, all items positively contributed to the overall scale reliability.

3.4. Model fit and item fit

Model fit and item fit statistics include INFIT and OUTFIT indices. OUTFIT detects unexpected responses to items with a difficulty distant from a person’s ability level (Linacre 2002); for example, OUTFIT is high when several low ability respondents correctly answer a difficult item or when high ability respondents get a relatively easy item incorrect. INFIT, on the other hand, detects responses to items that are so aligned to a person’s ability level that the item provides redundant information to the other items on the scale. In this study, we calculated Mean-square statistics (MNSQ) and, aligning with convention, considered items as potentially misfitting if their MNSQ values were smaller than 0.5 or larger than 1.5 (Linacre 2002).

The results of model and item fit statistics are listed in Table 2. Overall, the average values of INFIT and OUTFIT statistics were nearly perfect: 0.99 and 0.97, respectively. This means that our data fit the Rasch Model very well. Among the set of items, one item (i.e. Q47_1) had OUTFIT MNSQ value as 1.51, just above our threshold of fit value. This item was flagged as potentially misfitting and requiring further review.

Table 2. Psychometric properties of each scored response in the same order they were presented to survey respondents.

Item	Outfit MSNQ	Infit MSNQ	Item difficulty	Item	Outfit MSNQ	Infit MSNQ	Item difficulty	Item	Outfit MSNQ	Infit MSNQ	Item difficulty
Q1	0.97	0.98	−0.24	Q36	1.03	1.02	1.39	Q46_4	1.18	1.14	−0.58
Q2	0.59	0.92	−2.77	Q38	0.91	0.92	0.76	Q47_1	1.51	1.35	0.29
Q10	0.72	0.91	−1.62	Q39	1.03	0.98	1.43	Q47_2	0.90	0.91	−0.50
Q3	1.03	1.02	−0.53	Q40	1.32	1.20	1.59	Q47_3	0.93	1.00	−0.96
Q5	1.12	1.08	−0.09	Q41	0.92	0.93	1.45	Q47_4	0.81	0.86	−0.38
Q6	1.07	1.04	0.10	Q43	0.74	0.86	−0.82	Q37_1	0.77	0.84	−0.02
Q7	1.08	1.07	0.80	Q44	1.00	0.98	1.24	Q37_2	0.87	0.91	−0.25
Q8	0.87	0.93	−0.73	Q48	1.12	1.09	1.31	Q37_3	1.00	1.01	−0.26
Q9	0.95	0.98	0.30	Q49	0.99	0.98	1.65	Q37_4	0.76	0.83	−0.12
Q12	0.98	0.99	0.54	Q32	0.62	0.84	−1.27	Q13_1	0.89	0.93	0.15
Q14	0.84	0.88	−0.18	Q50	1.05	1.05	0.57	Q13_2	0.99	1.00	0.17
Q15	0.80	0.92	−0.98	Q4_1	1.47	1.12	−0.88	Q13_3	0.81	0.88	−0.33
Q17	1.03	1.01	−0.43	Q4_2	0.97	0.98	−0.43	Q13_4	0.76	0.84	−0.18
Q20	0.93	0.95	0.31	Q4_3	0.91	0.92	−0.35	Q18_1	1.15	1.08	0.41
Q21	0.80	0.89	−0.90	Q4_4	1.27	1.25	0.05	Q18_2	0.74	0.86	−0.64
Q22	0.88	0.93	−0.65	Q11_1	1.29	1.21	−0.19	Q18_3	1.18	1.14	0.96
Q23	0.97	0.99	0.06	Q11_2	0.94	0.92	−0.59	Q18_4	1.41	1.34	0.53
Q24	1.01	1.01	0.50	Q11_3	0.92	0.95	−0.33	Q19_1	0.95	0.94	−0.11
Q25	1.08	1.05	1.10	Q11_4	1.26	1.13	−0.59	Q19_2	1.31	1.28	0.27
Q27	0.57	0.83	−1.55	Q26_1	1.32	1.27	0.71	Q19_3	0.96	0.97	−0.18
Q28	0.96	0.97	0.00	Q26_2	1.18	1.12	1.30	Q19_4	0.88	0.91	0.21
Q29	0.89	0.93	−0.53	Q26_3	0.88	0.91	−0.32	Q42_1	0.80	0.87	−0.29
Q30	1.20	1.04	2.35	Q26_4	1.21	1.20	0.05	Q42_2	0.85	0.91	−0.87
Q31	1.26	1.13	1.66	Q46_1	0.57	0.81	−1.25	Q42_3	0.80	0.88	−0.43
Q33	0.98	1.00	−0.06	Q46_2	0.91	0.93	0.06	Q42_4	0.71	0.82	−0.40
Q34	0.76	0.87	−0.58	Q46_3	0.98	0.99	0.27	Q42_5	0.91	0.91	0.74
Q35	1.08	1.06	0.12

3.5. Item characteristics

In the Rasch model, item and ability parameters are aligned on the same latent trait continuum or scale. The set of IOLS knowledge items had Rasch item difficulty ranging from -2.77 to 2.35, with a mean difficulty value of zero (see Table 2). Among these 79 items, 35 of them had Rasch difficulty measures above zero while 44 of them had difficulty measures below zero. Figure 2 is an example of ICC plots. In ICC the probability as a function of ability forms a logistic S-shaped curve, in which the vertical axis is the probability of getting an item correctly, and the horizontal axis is the scaled units of the latent trait. A respondent with higher ability levels on the latent trait (i.e. Ocean Literacy) would have higher probability of getting a correct response, hence the vertical axis increases as the horizontal axis increases. Using Rasch, ICCs across all items have the same slope but vary by their locations (i.e. difficulties) on the latent trait (i.e. Ocean Literacy). In ICC, ‘location describes the extent to which items differ in probabilities across trait levels’ (Embreston and Reise 2000). Figure 3 is a person-item map, which compares the distribution of ability for the respondents with the item difficulty for the scales. The person-ability distribution is shifted to the right of the center of the item-difficulty distribution. This implies that abilities were higher than the item difficulties. Said another way, the items were easy for respondents in our study sample. The least difficult item in the instrument was Q2 (also seen in the ICC plot below). This question asked:

Which statement is true: Q2_1 The ocean covers 70% of the Earth’s surface; Q2_2 The land covers 70% of the Earth’s surface; Q2_3 The ocean and the land each cover 50% of the Earth’s surface; Q2_4 The ocean covers 10% of the Earth’s surface

Over 90% of respondents answered this item correctly (Q2_1), indicating a very easy item. The revision of this item is described in the discussion section, and listed in Appendix 1. The most difficult item in the instrument was Q30. This question asked: The accumulation of oxygen in Earth’s atmosphere was necessary for life to develop and be sustained on land. Where did this oxygen originate? Q30_1 Oxygen was already there when the Earth was formed. Q30_2 All oxygen originated from photosynthetic organisms on land. Q30_3 All oxygen originated from photosynthetic organisms both on land and in the ocean. Q30_4 All oxygen originated from photosynthetic organisms in the ocean. This item has been modified to reduce the reading demand and improve the overall clarity of the item (See Appendix 1 for the changes to the item, and see Appendix 2 for Item difficulty estimates).

Figure 2. Examples of some ICC plots.

Figure 3. Person-Item Map.

3.6. Distractor analysis

We conducted the distractor analysis to determine whether item options function effectively. Desirable distractors should attract respondents to choose them when respondents are unsure of the correct answer but should not be so attractive that respondents choose them more often than the correct answer.

In this study, we examined the frequency distribution of item options chosen by respondents; any item option that was chosen less than 5% of the time was flagged for follow up discussion and potential revision. Results of distractor analyses indicate that some of the items (identified in Appendix 1 with the symbol ‘Ψ’) had options that were not sufficiently attractive to respondents. For example, for item Q2, only 1.9% of respondents chose the second option, 1% chose the third option, and 0.5% chose the fourth option; meaning that these distractors were not providing useful information to differentiate low and high proficient respondents; 96.6% of respondents selected the correct answer (i.e. the first option) to this item. These item options would benefit from a thoughtful revision. Appendix 1 provides the complete list of item options that were suggested for content expert review.

3.7. Differential item functioning

Differential item functioning (DIF) means that items function differently across sub-groups in the sample. One example of DIF is when a boy and a girl that have the same ability estimates, but have different probabilities of getting the item correct because the item is easier for one gender than it is for another. DIF analysis is essential in the development of a scale in order to determine if the test is fair across respondents.

In this study, DIF detection was implemented with nonparametric analyses for dichotomously scored items. This study used the Mantel-Haenszel chi-square (Holland and Thayer 1988; Mantel and Haenszel 1959) and Educational Testing Services (ETS) classification scheme for evaluating DIF (i.e. A = negligible DIF; B = moderate DIF; C = large DIF; Zieky 1993). The criteria to diagnose a DIF item in this study is the presence of both statistical significance (i.e. Mantel-Haenszel chi-square value exceeding 3.84, or p < 0.05) and practical significance (i.e. the presence of moderate or large levels of the ETS DIF classification scheme) (Chen and Jiao 2014).

In implementation, this study used the summated test score of the items selected for the DIF analyses as the stratification variable. The grouping variables included (1) boy and girl; (2) Taiwan and non-Taiwan; and (3) Europe and USA. These comparisons were chosen because each had sufficient sample size and each provided insight on test characteristics. Examining DIF across gender allows us to use all the data collected from around the world to look for differences in gender responses. Taiwan and non-Taiwan is important to examine since the Taiwan data make up such a large portion of the total data, we wanted to ensure that their responses are not skewing the overall results. Europe and the USA allows for vetting the instrument across these cultural and linguistic differences. Table 3 summarizes the flagged potential DIF items. Results show that three items were flagged as potentially showing DIF across gender (i.e. Q3, Q8, Q4_1). Further, 40 items were flagged as having potential DIF effects between Taiwan and non-Taiwan respondent groups. A common characteristic across many of the items that favor Taiwanese students is that these items contained words such as ‘never’ or ‘always’ in some of the incorrect response options. These items were easier to answer correctly for respondents in Taiwan compared to non-Taiwanese students. One possible explanation for this DIF effect is that Tawainese students learn specific test-taking strategies; for example, eliminating response options with these words. Revising these items to eliminate these words is in alignment with best practices for assessment construction and may eliminate much of the DIF observed between Taiwan and non-Taiwan students. Also, further screening from content experts would be needed to see whether there is any translation issue that could contribute to these differences. Results show that 22 of the items potentially had DIF effects between Europe and the USA. While the sheer number of items that demonstrated DIF was high, that does not mean that all of these items were differentially functioning. The analysis merely points to items that function differently across different sub-groups, but doesn’t tell us why. While much of this difference maybe due to construct irrelevant differences (e.g., better test takers in Taiwan than in the USA), some of this difference is construct relevant, meaning that the difference is related to the subject of study. For example, item Q18 which was flagged for DIF asked about changing sea levels; respondents in Taiwan, an island nation along the tropic of cancer, had more difficulty selecting ice caps melting and growing as having an influence on sea levels than respondents outside of Taiwan (mostly in the U.S. and Europe). This difference in difficulty may be due to different emphases in the curriculum across these locales given their relative proximity to polar ice caps. This may be the type of difference across countries that the survey is aiming to uncover.

Table 3. Summary of flagged potential DIF items in the same order they were presented to survey respondents.

Item	Gender		TW vs. NonTW		EURO vs. USA		Item	Gender		TW vs. NonTW		EURO vs. USA
	MH	Favor	MH	Favor	MH	Favor		MH	Favor	MH	Favor	MH	Favor
Q1			160.5^C	TW			Q4_1	34.1^B	F
Q3	30.6^B	M	48.6^B	TW	30.9^B	EURO	Q4_4			71.0^B	NonTW	22.3^B	EURO
Q5			93.7^C	TW			Q11_1			59.5^B	NonTW
Q6			37.8^B	TW	44.1^C	EURO	Q11_2			142.8^C	TW
Q7			78.3^B	NonTW			Q11_4					45.0^C	EURO
Q8	31.2^B	M					Q26_1			122.3^C	NonTW
Q9			68.5^B	NonTW			Q26_2					35.4^B	USA
Q12			57.4^B	NonTW			Q46_2			42.2^B	TW	22.0^B	USA
Q14			381.2^C	TW			Q47_2			27.5^B	TW
Q21			20.1^B	TW			Q47_4			21.3^B	TW
Q22			152.2^C	TW			Q37_1					28.2^B	USA
Q25			337.9^C	TW	57.4^C	EURO	Q37_2			93.5^C	TW	33.3^B	USA
Q27					9.1^B	EURO	Q37_3			106.9^C	NonTW
Q29					13.6^B	EURO	Q37_4			270.1^C	TW	51.3^C	USA
Q30			272.7^C	NonTW			Q13_1					45.3^C	USA
Q31			295.1^C	NonTW	40.7^C	USA	Q13_3			38.3^B	TW
Q34					27.8^C	EURO	Q13_4			126.7^C	TW
Q35			179.4^C	NonTW			Q18_1			41.3^B	TW	21.8^B	USA
Q38					57.5^C	EURO	Q18_2			27.6^B	NonTW
Q39			234.1^C	NonTW			Q18_3			88.5^B	TW
Q40			190.8^C	NonTW			Q18_4					58.3^C	EURO
Q48			254.7^C	NonTW			Q19_2			316.8^C	NonTW
Q49			83.6^C	NonTW	29.8^B	USA	Q19_4					16.9^B	EURO
Q50			56.0^B	NonTW			Q42_5			196.6^C	TW	96.5^C	USA

^B moderate DIF

^C large DIF.

3.8. Modifications and preparations for version 3

Developing a single instrument that functions equally across linguistic and cultural differences is exceedingly difficult. The results from the second trial are informing the construction of the next version of the survey. Content experts are currently revisiting the items to review them for clarity, content alignment, and explore ways to modify the items to perform better across participating countries and languages. The results of this process can be seen in Appendix 1. Additional items have been added to the assessment to measure concepts that had previously been left out of the survey.

3.9. Summary of the findings

Our analyses indicate that the survey indeed assesses one factor, that we are referring to as ‘Ocean Literacy.’ Given the challenges associated with the community-development of a survey in 17 languages, this provides encouragement to continue development of this international collaborative effort. Because the survey was initially developed by gathering items from existing instruments, and only a few previous studies existed, we were limited in the types of items included. Many of the items, for instance, assessed only declarative knowledge or factual recall. Others had inconsistent construction of distractors, with spurious words, inconsistent distractor length, or contain words like ‘never’ or ‘always’ that often indicate that these are not the correct answer. Appendix 1 shows how we have modified many questions from V2, discussed in this paper, to create V3, to minimize these issues and which will be administered for a new round of testing.

Some modifications are intended to simply make small improvements to an item based directly on analysis of results from V2, i.e., the item did not test well either because one or more of the distractors were infrequently selected, or the item favored a particular population for reasons we think are unrelated to understanding of Ocean Literacy. For example, in Q1 ‘Which statement is the most accurate,’ some distractors are negative statements while some are positive, and distractors Q1_1 (The water in the Pacific Ocean will never reach the Indian Ocean) and Q1_4 (The water in the Gulf of Mexico can never reach the Pacific Ocean) both contain the word, ‘never.’ The item favored Taiwanese respondents, who may learn to avoid distractors that include words such as always and never. We have revised the item for V3 so that the correct response and all distractors are positive statements, and we eliminated the use of the words, ‘never.’

Other modifications are intended to reframe items from representing declarative knowledge to more conceptual understanding. For example, Q2 ‘Which statement is true: Q2_1 The ocean covers 70% of the Earth’s surface; Q2_2 The land covers 70% of the Earth’s surface; Q2_3 The ocean and the land each cover 50% of the Earth’s surface; Q2_4 The ocean covers 10% of the Earth’s surface’ asks respondents only to recall an important fact about the ocean. In V2, 96.6% of respondents answered this question correctly. The question did not provide useful information to differentiate low and high respondents. The concept that most of Earth’s surface is covered by the ocean is a defining idea in Ocean Literacy, but respondents’ ability to recall of the percentage does not indicate understanding of why this idea is so important to earth systems. Q2 has been revised for V3 to be more conceptual in nature:

Because the ocean covers most of the Earth (select the best answer): Q2_1 It controls our weather, climate and oxygen production; Q2_2 Most living things are concentrated on the continents; Q2_3 Lots of the Earth is not very useful for humans; Q2_4 It generates most of the Earth’s greenhouse gases.

3.10. Limitations

The samples used in this study are not representative of the countries from which they are drawn. So, we are careful to not draw conclusions about the level of Ocean Literacy across these countries. However, it is possible that there was systematic bias in the way the data were collected across all countries that lead to a poor representation of students around the world. The systematic bias that would be most harmful for our work would be overly represented knowledge about the ocean. It is possible that our estimates of item difficulty are biased downward because teachers who teach about the ocean were more likely to administer the survey. However, for our purposes the items are being evaluated in a relative sense to each other, not against the sample itself. Therefore, the analyses conducted are relatively robust to any systematic bias in the sample.

4. Discussion

The ocean is an important part of our world, something we all share, we all benefit from and we all have an impact on. Understanding our connection to the ocean and being an informed participant in the discussion of the future of the ocean requires a degree of Ocean Literacy. There have been many considerable investments made over the last 15 years by governments and non-governmental organizations for the purpose of increasing ocean literacy. There have been few attempts, however, to understand the influence of these efforts on learners or the general public. Since the work to improve Ocean Literacy is relatively new, especially outside of the United States, it has focused mainly on programmatic activities and interventions. These interventions until recently, have not attracted much attention from the education research community. The International Ocean Literacy Survey is among the first tools intended to support the efforts of those aiming to educate our citizenry about the ocean that has been translated widely, pilot tested multiple times and subjected to rigorous psychometric scrutiny.

The work presented here provides evidence that the survey instrument is psychometrically valid and reliable, and has a single factor structure across 17 languages and 24 countries. Further, we argue that there is still much work to be done to produce an instrument that can be used equivalently across these different cultural and linguistic contexts. The authors are continuing these efforts. We have made several changes to individual items as a result of our findings, and we will be further testing the Survey in early 2018.

Further, this paper attempts to demonstrate that a small group of people can: lead an international effort on a limited budget with contributions from dozens of researchers and practitioners around the world; maintain the integrity of the research design despite using somewhat non-traditional methods; and make a useful and practical impact on efforts to understand and improve education efforts around the world. We are experimenting with new methods of grassroots, stone soup-style, community-based instrument development, using a process similar to the collective impact framework (Kania and Kramer 2011), that relies on a network of committed individuals and organizations, with a common goal and common measures, and led by a trusted backbone organization. The community is willing to forego some traditional academic needs for ownership and authorship in order to achieve practical collective results that mark steady progress toward achieving the goal. Dozens of researchers, evaluators, scientists and educators freely contributed original instruments or individual items, edited or adapted items, reviewed or administered the survey, and advised on the process. The large number of contributors posed some challenges to the research design, but in the end, all contributors have a single goal, to assist in the development of a new, universal instrument that allows those in the community to measure progress and compare results across user groups. The goal is to create an instrument that represents a whole that is greater than the sum of its parts. We invite additional investigators interested in either the methods and technical aspects of the effort or in the advancement of understanding about Ocean Literacy to participate in the ongoing development and administration of the International Ocean Literacy Survey.

The International Ocean Literacy Survey is designed to detect progress toward, and so, to support the improvement of, international and potentially global efforts to build public understanding of the importance of the ocean. Use of the IOLS is a key strategy for justifying and promoting efforts to increase the public’s capacity to understand, communicate about, manage, sustain and protect ocean resources and ocean ecosystems.

We recognize the essential need to move beyond measuring only ocean knowledge to really understand levels of Ocean Literacy among our 16–18 year old audience. Ocean Literacy is defined as also including the ability to communicate about the ocean in a meaningful way, and to be able to make informed and responsible decisions regarding the ocean and its resources. Knowledge of the essential principles and fundamental concepts about the ocean is the dimension of Ocean Literacy most well defined, so we have chosen to begin our measurement efforts there. We intend to expand our efforts over time to include measures of communication and decision-making.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Géraldine Fauville is a researcher at the Department of Education, Communication and Learning at the University of Gothenburg, Sweden. Her studies focus on the role and potential of digital technologies for contributing to the process of learning about marine environmental issues. She also has a background in marine biology. She has also worked for 10 years as manager of a long-lasting project between Stanford University and University of Gothenburg developing digital learning resources for marine environmental education. She co-founded the European Marine Science Educators Association (EMSEA) in 2011 where she currently serves as Director. She also serves on the Board of Directors of the U.S. National Marine Educators Association (NMEA).

Craig Strang is an associate director, Lawrence Hall of Science at the University of California, Berkeley. He leads the Learning and Teaching Group, and is principal investigator of BaySci: The Bay Area Partnership for K-12 Science; an NSF-funded research study on science professional development, and is PI of a project to improve instruction in outdoor science programs. He is founding director of MARE: Marine Activities, Resources & Education, a K-8 professional learning and curriculum program. He was PI of the NSF-funded Center for Ocean Sciences Education Excellence--California. He was the president of the National Marine Educators Association in 2012, and co-leads the Ocean Literacy Campaign and development of the Ocean Literacy Framework. He is co-author of three sets of science curriculum materials for grades K-8, and is PI of the Ocean Sciences Curriculum Sequences for Grades 3-5 and 6-8. Before turning to science education he did research on elephant seals, humpback whales and California sea lions.

Matthew A. Cannady is the director of Quantitative Studies for the Research and Impact Group at the Lawrence Hall of Science at the University of California, Berkeley. He has been involved in science research and education for more than 15 years. After spending a year as a product design engineer, and planetary scientist at NASA Ames Research Center, he taught high school physics for five years before earning his doctorate degree in Educational Research, Measurement and Evaluation from Boston College. Since then, he has collaborated on numerous large-scale research and evaluation studies in the field of education, including evaluations of teacher professional development, design-based research for after-school science programs, and construction of research measurement instruments.

Ying-Fang Chen is a research specialist at the Research and Impact Group at the Lawrence Hall of Science at the University of California, Berkeley. She brings her expertise in psychometrics, statistics, and evaluation to STEM research projects within the Research and Impact Group. She has many years of professional experience in item response theory modeling, statistical methodology, large-scale survey data systems, as well as educational psychology research and institutional research. She received her doctorate in Measurement, Statistics and Evaluation from University of Maryland, College Park. Her doctoral and postdoctoral research focused on developing innovative psychometric modeling to improve the measurement of large-scale assessments.

Acknowledgment

We would like to acknowledge the essential contributions of the IOLS partner organizations. The following organizations participated in and/or support the development and administration of the IOLS: Asia Marine Educators Association (AMEA), Canadian Network for Ocean Education (CaNOE), National Marine Educators Association (NMEA), Marine Conservation Society, Partnership for Observation of the Global Oceans (POGO), and Surfline. In addition, we thank and acknowledge the following individuals who volunteered countless hours to translate the IOLS into 16 languages:

•	Chinese Trad: Liang-Ting Tsai, Jack Chang
•	Croatian: Melita Mokos
•	Danish: Lene Friis Møller
•	Dutch: Evy Copejans, Pieter Demuynck
•	French: Géraldine Fauville
•	German: Mirjam Glessmer, Maike Nicolai
•	Greek: Yolanda Koulouri, Maria Cheimonopoulou, Athanasios Mogias, Theodoros Kevrekidis, Theodora Boubonari
•	Italian: Martina Milanese
•	Japanese: Tsuyoshi Sasaki
•	Norwegian: Theodora Sam
•	Polish: Grażyna Niedoszytko, Weronika Podlesinska, Dominika Wojcieszek
•	Portuguese: Raquel Costa, Diogo Geraldes, Fernanda Silva
•	Simplified Chinese: Mo Chen, Guoguang Yang
•	Spanish and Catalan: Manel Gazo, Carol Campillo Campbell
•	Swedish: Susan Gotensparre, Björn Källström

Finally, we thank and acknowledge all of the dozens of individuals around the world who contributed previously developed surveys, participated in the development of new items, reviewed and edited the survey for accuracy, and who assisted in the administration of the IOLS to over 6000 students in 24 countries.

Appendix 1

^{Table A1} 

Table A1. List of the questions from Version 2 and their modification in V3. Note there is no Q16 and 45 as they were deleted after V1 and the numbers were kept unchanged in order to make it easier to track the questions from one version to the next. Column C. corresponds to the specific ‘fundamental concept’ from the Ocean Literacy Framework that is addressed by each specific question. The items in bold are the correct answers^. The symbol ‘Ψ’ represent distractors that were not sufficiently attractive to respondents. The items are placed in numerical order.

Q.	C.	V2	V3
Q1	1a	Which statement is the most accurate:	Which statement best explains how ocean water moves?
Q1_1		The water in the Pacific Ocean will never reach the Indian Ocean	Water in each ocean basin (e.g. Atlantic, Pacific, Indian) circulates only within that basin
Q1_2		Water in Pacific Ocean will eventually reach all other parts of the world ocean	Water in the Pacific ocean basin will eventually circulate to all other ocean basins
Q1_3		The water in the North Atlantic Ocean will eventually move throughout the Northern Hemisphere, but cannot move to the Southern Hemisphere	Ocean water circulates throughout the northern hemisphere but will not cross over to the southern hemisphere
Q1_4		The water in the Gulf of Mexico can never reach the Pacific Ocean	Water in the Gulf of Mexico stays there, and does not reach other ocean basins.
Q2	3a	Which statement is true:	Because the ocean covers most of the Earth (select the best answer):
Q2_1		The ocean covers 70% of the Earth’s surface	It controls our weather, climate, and oxygen production
Q2_2		The land covers 70% of the Earth’s surface	Most living things are concentrated on the continents
Q2_3		The ocean and the land each cover 50% of the Earth’s surface	Lots of the Earth is not very useful for humans
Q2_4 ^Ψ		The ocean covers 10% of the Earth’s surface	It generates most of the Earth’s greenhouse gases
Q3	1b	How deep is the deepest part of the ocean?	Which statement best describes the ocean floor compared to the land?
Q3_1 ^Ψ		500 meters	The land has mountains, valleys and plains, but the ocean floor only has trenches
Q3_2 ^Ψ		1000 meters	The ocean floor has small hills but no mountains
Q3_3		6000 meters	The ocean floor has mountains and valleys that are larger than those on land
Q3_4		11,000 meters	The land has mountains and valleys, but the ocean floor is mostly flat
Q4	1c	Ocean circulation is influenced by (select all that apply):	SAME
Q4_1		Satellites	Global ship traffic
Q4_2		The shapes of ocean basins	SAME
Q4_3		Adjacent land masses	SAME
Q4_4		Acidity of the ocean	SAME
Q5		By which process does the ocean lose heat that it absorbs from solar radiation?	DELETED
Q5_1		Precipitation	DELETED
Q5_2		Condensation	DELETED
Q5_3		Evaporation	DELETED
Q5_4		Sublimation	DELETED
Q6	1d	How is sea level measured?	SAME
Q6_1		Average depth of the ocean	SAME
Q6_2		Average height of the ocean relative to the land	SAME
Q6_3		Level of the ocean at the lowest tide	SAME
Q6_4		Level of the ocean at the highest tide	SAME
Q7	1e	Which of these statements is TRUE?	SAME
Q7_1		Seawater freezes at a lower temperature than freshwater	SAME
Q7_2		Seawater freezes at the same temperature as freshwater	SAME
Q7_3		Seawater freezes at a higher temperature than freshwater	SAME
Q7_4		Seawater cannot freeze	SAME
Q8	1e	Approximately how much of the Earth’s water is in the ocean?	Where is most of the water on Earth?
Q8_1 ^Ψ		Very little	In the ocean
Q8_2 ^Ψ		A small amount	Frozen in the polar ice caps
Q8_3		About half of it	Trapped in underground aquifers
Q8_4		Almost all of it	Circulating in the atmosphere
Q9	1f	Which is the most accurate statement about the water in the Earth’s water cycle?	SAME
Q9_1		Much of the same water has been traveling through the water cycle for millions of years	SAME
Q9_2		Water joins the water cycle when new water is made through condensation	SAME
Q9_3		Water leaves the water cycle through evaporation	SAME
Q9_4 ^Ψ		All of the water in the water cycle is liquid	Water leaves the water cycle when humans drink it
Q10	1f	Water moves from the ocean to the atmosphere to the land and back again to the ocean by a process called:	Because of the global water cycle:
Q10_1 ^Ψ		Watershed	When water evaporates from the ocean the rain that forms is a little bit salty
Q10_2 ^Ψ		Hurricane	Water goes from the ocean to the atmosphere to the land
Q10_3 ^Ψ		Cyclone	Climate change could cause more tsunamis
Q10_4 ^Ψ		Tsunami	Water that evaporates from the ocean will never return to the ocean
Q10_5		Water cycle	DELETED
Q11	1f	What connects the ocean to all of Earth’s water reservoirs? Select all that apply	SAME
Q11_1		Sublimation	Ionization
Q11_2		Precipitation	SAME
Q11_3		Evaporation	SAME
Q11_4		Deposition	SAME
Q12	1 g	Rivers supply most of the salt to the ocean. The salt comes from:	SAME
Q12_1		Seafloor reactions	SAME
Q12_2		Eroding land	SAME
Q12_3		Volcanic emissions	SAME
Q12_4		The atmosphere	SAME
Q13	1 g	Which of the following are transported by rivers from watersheds to estuaries and to the ocean? Select all that apply	SAME
Q13_1		Nutrients	SAME
Q13_2		Salts	SAME
Q13_3		Sediments	SAME
Q13_4		Pollutants	SAME
Q14	1 h	Which statement is the most accurate?	Which is the most accurate description of the ocean?
Q14_1		There is one ocean, it is large, and it has enough resources to support the growing human population	SAME
Q14_2		When resources are depleted from one ocean, we can always find them in another ocean	SAME
Q14_3		There are many oceans and they can each replenish themselves	SAME
Q14_4		There is one ocean, it is large, it is finite, and the resources are limited	SAME
Q15	1 h	What is one example of an ocean resource at risk of being depleted?	SAME
Q15_1		Fishes and invertebrates.	Seafood
Q15_2		Wave energy	SAME
Q15_3		Sand	SAME
Q15_4 ^Ψ		Salt	Carbon dioxide
Q17	2c	Which statement is the most accurate?	SAME
Q17_1		The sand present on most beaches has probably been there for 100 years	Sand on the shoreline is mostly stable and is likely to stay on the same beach for about 100 years
Q17_2		Sand on the shoreline is constantly moving and being redistributed by waves and currents	SAME
Q17_3		Sand on the shoreline is constantly moving and being redistributed by animals that live there	Sand on the shoreline is constantly moving and being redistributed mostly by activities of animals that live there
Q17_4		The sand present on most beaches has probably been there for 10,000 years	Sand on the shoreline is mostly stable and is likely to stay on the same beach for several 100 years
Q18	1d	What processes cause changes to sea level? Select all that apply	SAME
Q18_1		Movement of Earth’s crust	SAME
Q18_2		Ice caps melt and grow	Melting and growing of ice caps on land
Q18_3		Seawater expands and contracts when it warms and cools.	Warming and cooling of ocean water
Q18_4		Coastal erosion	SAME
Q19	2e	What naturally influences the physical structure and landforms of the coast? Select all that apply	Which of the following can lead to changes in the physical structure and landforms of the coast? Select all that apply
Q19_1		Sea level changes	SAME
Q19_2		Changing salinity of the seawater	SAME
Q19_3		Tectonic activity	SAME
Q19_4		Forces of waves	SAME
Q20	3a,b	Imagine there are two cities of the same size that are at the same latitude (same distance north or south of the equator). One is on the coast, and the other is 200 km inland. On average, which of the following conditions would you expect? Choose the best answer	Look at the image. If both cities are at the same elevation, which is the best explanation of the average temperature in cities A and B:
Q20_1		The coastal city will have hotter summer temperatures and cooler winters	City A will have warmer summers and cooler winters than city B
Q20_2		The inland city will have cooler summer temperatures and warmer winters	City B will have cooler summers and warmer winters than city A
Q20_3		The coastal city will have cooler summer temperatures and warmer winters	City A will have cooler summers and warmer winters than city B
Q20_4		Temperature is not affected by distance from the ocean.	SAME
Q21	3a,b	What is the difference between weather and climate? Choose the best answer	SAME
Q21_1 ^Ψ		Weather and climate are the same	Weather is what is happening right now, and climate is what happened last year
Q21_2		Weather is what is happening right now, and climate is what happens over many years	Weather is what is happening now, and climate is what happens over many years
Q21_3		Climate is what is happening right now, and weather is what happens over many years	SAME
Q21_4		Weather is everywhere while climate is local	SAME
Q22	3b	If Earth did not have an ocean, what would the surface temperatures on Earth be like? Would surface temperatures be:	If Earth did not have an ocean, what would the surface temperatures on Earth be like?
Q22_1		More extreme than they are now	There would be more extreme high and low temperatures than there are now
Q22_2		More uniform around the globe	Temperatures would be more uniform around the globe
Q22_3 ^Ψ		About the same as today	There would be cooler temperatures in the summer and warmer temperatures in the winter
Q22_4		We don’t have enough information to know what would happen	Scientisits don’t have enough information to know what would happen
Q23	3c	What is the most common impact of an El Niño year?	SAME
Q23_1		The salinity of the ocean water changes	The salinity of the global ocean water changes.
Q23_2		The temperature of the ocean gets colder	The temperature of the global ocean gets colder
Q23_3		There are significant temporary changes in global weather.	SAME
Q23_4		There are significant permanent changes in global weather.	SAME
Q24	3c	El Niño originates from which ocean basin?	El Nino is a complex weather pattern associated with changes in water temperature in which ocean basin?
Q24_1		Atlantic	SAME
Q24_2		Indian	SAME
Q24_3		Arctic	SAME
Q24_4		Pacific	SAME
Q24_5 ^Ψ		Southern	DELETED
Q25	3d	Most rain that falls on land originally evaporated from:	SAME
Q25_1		The tropical ocean	The tropical region of the ocean
Q25_2 ^Ψ		The polar ocean	DELETED
Q25_3		The temperate ocean	The temperate region of the ocean.
Q25_4		The Great Lakes	The nearby lakes and rivers
Q25_5		The ocean nearest the land where it fell	SAME
Q26	3f	The ocean has a significant influence on climate change by absorbing, storing, and moving what? Select all that apply	SAME
Q26_1		Salts	SAME
Q26_2		Carbon	SAME
Q26_3		Heat	SAME
Q26_4		Fresh water	Trash
Q27	3 g	In what way is global warming impacting the Arctic?	SAME
Q27_1 ^Ψ		Humpback whales populations are decreasing	The impact on the Arctic is the same as on the rest of the planet
Q27_2		Polar ice is decreasing	The Arctic is warming faster than the rest of the planet
Q27_3 ^Ψ		Mountain glaciers are growing larger	Glaciers are melting in some parts of the Arctic and growing in other parts
Q27_4 ^Ψ		Arctic fish populations are increasing	Populations of warm water fishes are migrating to the Arctic
Q28		What is one possible impact of a warming Arctic?	DELETED
Q28_1		Less snow and ice, causing more solar energy to be absorbed by the Earth’s surface	DELETED
Q28_2		Less snow and ice, causing less solar energy to be absorbed by the Earth’s surface	DELETED
Q28_3		Decrease in sea level	DELETED
Q28_4		Less fresh water available to coastal communities	DELETED
Q29		The uneven heating of Earth’s surface causes the temperature of the ocean to vary with latitude. Which of the following maps is correct if 1 represents the warmest ocean water and 3 the coldest ocean water?	DELETED
Q29_1		Image 1	DELETED
Q29_2		Image 2	DELETED
Q29_3		Image 3	DELETED
Q29_4		Image 4	DELETED
Q30	4a	The accumulation of oxygen in Earth’s atmosphere was necessary for life to develop and be sustained on land. Where did this oxygen originate?	Where did most of the oxygen in the atmosphere originally come from?
Q30_1		Oxygen was already there when the Earth was formed	The same processes that formed the Earth
Q30_2		All oxygen originated from photosynthetic organisms on land	Photosynthesis by plants on land
Q30_3		All oxygen originated from photosynthetic organisms both on land and in the ocean	Respiration of animals
Q30_4		All oxygen originated from photosynthetic organisms in the ocean	Photosynthesis by organisms in the ocean
Q31		What produces most of Earth’s oxygen?	DELETED
Q31_1		Tropical Rain Forests	DELETED
Q31_2		Photosynthetic organisms in the ocean	DELETED
Q31_3 ^Ψ		Respiration from marine animals	DELETED
Q31_4 ^Ψ		Decomposition of dead plants and animals	DELETED
Q32	4b	Where did the first life on Earth evolve?	SAME
Q32_1 ^Ψ		In the desert	On land
Q32_2		In the ocean	In the ocean
Q32_3		Under rocks on high mountains	Deep under the surface of Earth
Q32_4 ^Ψ		In the atmosphere	In fresh water
Q33	5a	What is the largest animal ever to live on Earth?	SAME
Q33_1		Giant squid	SAME
Q33_2 ^Ψ		Elephant	Orca (killer whale)
Q33_3		Blue whale	SAME
Q33_4		Tyrannasaurus rex	SAME
Q34	5b	Which types of living things are there the most of in the ocean?	Most of the biomass in the ocean resides in:
Q34_1		Fish	Fishes
Q34_2		Plankton	SAME
Q34_3 ^Ψ		Animals with shells	Mammals
Q34_4 ^Ψ		Whales and seals	Molluscs
Q34_5 ^Ψ		Sharks	DELETED
Q35	5c	How large is the variety of living things in the ocean compared to other environments?	There are 35 major groups of organisms (vertebrates, arthropods, molluscs, etc.) Where are most of the major groups found?
Q35_1		More in the ocean than in forests	Almost all are found exclusively in the ocean.
Q35_2		Equally in the ocean and in the jungle	Most are found equally on land and in the ocean
Q35_3		Less in the ocean than in the forests	Slightly more than half are found exclusively in the ocean
Q35_4 ^Ψ		Equally in the ocean and in the desert	Most are found exclusively in the tropical rainforests
Q36	5e	Both land and ocean provide space for animals and other organisms to live. How much of Earth’s living space is found in the ocean?	SAME
Q36_1		Only a little bit	SAME
Q36_2		About half	SAME
Q36_3		A little more than half	SAME
Q36_4		Nearly all	SAME
Q37	5e	In the ocean living organisms are found (select all that apply):	SAME
Q37_1		At the surface	SAME
Q37_2		In the water column	SAME
Q37_3		On the seafloor	SAME
Q37_4		In the tidal zone	SAME
Q38	5f	Which of the following influences the vertical distribution of organisms in the open ocean?	SAME
Q38_1		Amount of time exposed to air	SAME
Q38_2		Crashing waves	SAME
Q38_3		Light levels	SAME
Q38_4		Human activity	SAME
Q39	5 g	What is the source of energy for primary productivity in ocean ecosystems where there is no sunlight?	SAME
Q39_1		Chemical energy	SAME
Q39_2		Wave energy	SAME
Q39_3		Nuclear energy	SAME
Q39_4		Cold fusion	SAME
Q40	5 h	What is the main cause of vertical zonation patterns along the shore that influence the distribution and diversity of organisms?	All of the options below have an influence on the vertical distribution of living organisims along the shoreline. Select the option that has the largest influence on shorelines around the world
Q40_1		Sunlight	SAME
Q40_2		Salinity	SAME
Q40_3		Tides	SAME
Q40_4		Trampling by people	SAME
Q41	5i	The marine habitat that provides the most important and productive nursery areas for many marine and aquatic species are:	SAME
Q41_1		Regional seas	SAME
Q41_2		The deep sea	SAME
Q41_3		Rivers	DELETED
Q41_4		Open ocean	SAME
Q41_5		Estuaries	SAME
Q42	6b	Which of the following statements are true about how humans depend on the ocean? Select all that apply	SAME
Q42_1		It provides us with food and medicine	SAME
Q42_2		It provides us with mineral and energy resources	SAME
Q42_3		It provides us with transportation and jobs	SAME
Q42_4		It benefits our economy	SAME
Q42_5		It is important to our national security	DELETED
Q43	6b	Which statement about eating animals from the ocean is true?	SAME
Q43_1		All kinds of ocean animals are endangered, so no one should eat any ocean animals	SAME
Q43_2		Some populations of ocean animals are declining, so people should choose carefully what to eat	SAME
Q43_3		In the ocean, only whales and dolphins are declining so it is OK to eat fish	SAME
Q43_4		There are plenty of all the kinds of ocean animals that people normally eat	SAME
Q44	6d	What statement about ocean acidification is the most accurate?	Which statement is the best explanation of ocean acidification?
Q44_1		Burning fossil fuels adds carbon dioxide to the atmosphere, which is then absorbed by the ocean and increases its acidity	SAME
Q44_2		Human caused pollution adds toxic chemicals to the ocean that increases its acidity	SAME
Q44_3		Fertilizers used in agriculture are washed into the ocean by rainfall and this increases the acidity of seawater	SAME
Q44_4		Ocean currents and other natural cycles are constantly changing the acidity of the ocean around the world	SAME
Q46	6d	Humans affect the ocean in a variety of ways. What does human development and activity often lead to? Select all that apply	SAME
Q46_1		Pollution.	SAME
Q46_2		Physical changes to beaches	SAME
Q46_3		Changes to ocean chemistry	SAME
Q46_4		Increased frequency of tsunamis	SAME
Q47	6e	What will be the most immediate effects of climate change on the ocean? Select all that apply	Recent human activities have changed ocean temperatures and pH. Which of the following have happened because of these changes? Select all that apply
Q47_1		Changes to ocean chemistry	Ocean salinity is increasing
Q47_2		Changes to sea level	Many corals reefs are dying
Q47_3		More oil spills	The frequency of oil spills is increasing
Q47_4		Changes in sea surface temperatures	Biodiversity in the ocean is decreasing
Q48	6f	Most humans live:	SAME
Q48_1		Near rivers	SAME
Q48_2		In rural areas	SAME
Q48_3		In coastal areas	SAME
Q48_4 ^Ψ		In inland areas	SAME
Q49	7a	About what percentage of the ocean has been explored to date?	SAME
Q49_1		5%	SAME
Q49_2		25%	SAME
Q49_3		50%	SAME
Q49_4		75%	SAME
Q49_5		90%	DELETED
Q50	7d	Fewer ocean scientists go to sea to conduct their research than in the last century. They rely more on satellites, buoys and unmanned submersibles. What impact is that having on our understanding of the ocean: Select the best answer	Ocean scientists increasingly rely on satellites, buoys and remote-operated submersibles. What impact is that having on our understanding of the ocean: Select the best answer
Q50_1		It improves our understanding because the new technology can collect vastly more data than scientists on ships can	It improves our understanding because the new technology can collect much more data than scientists on ships can
Q50_2		It decreases our understanding because scientists don’t collect data with their own hands	SAME
Q50_3		It decreases our understanding because the technology isn’t very reliable	SAME
Q50_4		It improves our understanding because it eliminates human error	SAME

Appendix 2

^{Table B1} 

Table B1. Item difficulty estimates listed in the same order they were presented to survey respondents (0.95 Confidence Interval).

	Estimate	Std. Error	Lower CI	Upper CI
Q1	−0.25	0.03	0.18	0.31
Q2	−2.77	0.08	−2.93	−2.61
Q10	−1.62	0.05	−1.72	−1.52
Q3	−0.53	0.04	−0.60	−0.46
Q5	−0.09	0.03	−0.16	−0.03
Q6	0.10	0.03	0.04	0.16
Q7	0.80	0.03	0.74	0.86
Q8	−0.74	0.04	−0.81	−0.66
Q9	0.30	0.03	0.24	0.37
Q12	0.54	0.03	0.48	0.60
Q14	−0.18	0.03	−0.24	−0.11
Q15	−0.98	0.04	−1.06	−0.90
Q17	−0.43	0.04	−0.50	−0.36
Q20	0.31	0.03	0.25	0.37
Q21	−0.90	0.04	−0.98	−0.83
Q22	−0.65	0.04	−0.72	−0.58
Q23	0.06	0.03	0.00	0.12
Q24	0.50	0.03	0.44	0.56
Q25	1.10	0.03	1.04	1.16
Q27	−1.55	0.05	−1.65	−1.46
Q28	0.00	0.03	−0.07	0.06
Q29	−0.53	0.04	−0.60	−0.46
Q30	2.35	0.04	2.27	2.42
Q31	1.66	0.03	1.60	1.73
Q33	−0.06	0.03	−0.12	0.00
Q34	−0.58	0.04	−0.65	−0.51
Q35	0.12	0.03	0.06	0.18
Q36	1.39	0.03	1.33	1.45
Q38	0.76	0.03	0.70	0.82
Q39	1.43	0.03	1.36	1.49
Q40	1.59	0.03	1.53	1.65
Q41	1.45	0.03	1.39	1.51
Q43	−0.82	0.04	−0.90	−0.75
Q44	1.24	0.03	1.18	1.30
Q48	1.31	0.03	1.25	1.37
Q49	1.65	0.03	1.59	1.71
Q32	−1.27	0.04	−1.36	−1.19
Q50	0.57	0.03	0.51	0.63
Q4_1	−0.88	0.04	−0.95	−0.80
Q4_2	−0.43	0.04	−0.50	−0.36
Q4_3	−0.35	0.03	−0.42	−0.29
Q4_4	0.05	0.03	−0.01	0.11
Q11_1	−0.20	0.03	−0.26	−0.13
Q11_2	−0.59	0.04	−0.66	−0.52
Q11_3	−0.33	0.03	−0.39	−0.26
Q11_4	−0.59	0.04	−0.66	−0.52
Q26_1	0.71	0.03	0.65	0.76
Q26_2	1.30	0.03	1.24	1.36
Q26_3	−0.32	0.03	−0.39	−0.26
Q26_4	0.05	0.03	−0.01	0.11
Q46_1	−1.25	0.04	−1.34	−1.17
Q46_2	0.06	0.03	0.00	0.13
Q46_3	0.27	0.03	0.21	0.33
Q46_4	−0.58	0.04	−0.65	−0.51
Q47_1	0.29	0.03	0.23	0.35
Q47_2	−0.50	0.04	−0.57	−0.43
Q47_3	−0.96	0.04	−1.04	−0.88
Q47_4	−0.38	0.04	−0.45	−0.32
Q37_1	−0.02	0.03	−0.09	0.04
Q37_2	−0.25	0.03	−0.32	−0.18
Q37_3	−0.26	0.03	−0.33	−0.20
Q37_4	−0.12	0.03	−0.18	−0.05
Q13_1	0.15	0.03	0.09	0.21
Q13_2	0.17	0.03	0.11	0.23
Q13_3	−0.33	0.03	−0.40	−0.27
Q13_4	−0.18	0.03	−0.24	−0.11
Q18_1	0.41	0.03	0.35	0.48
Q18_2	−0.64	0.04	−0.71	−0.57
Q18_3	0.96	0.03	0.90	1.02
Q18_4	0.53	0.03	0.47	0.59
Q19_1	−0.11	0.03	−0.17	−0.04
Q19_2	0.27	0.03	0.21	0.33
Q19_3	−0.18	0.03	−0.25	−0.12
Q19_4	0.22	0.03	0.15	0.28
Q42_1	−0.29	0.03	−0.36	−0.23
Q42_2	−0.87	0.04	−0.95	−0.79
Q42_3	−0.43	0.04	−0.50	−0.36
Q42_4	−0.40	0.04	−0.47	−0.33
Q42_5	0.74	0.03	0.68	0.80