Interest in STEM among children with a low socio-economic status: further support for the STEM-CIS-instrument through the adapted Dutch STEM-LIT measuring instrument

Abstract

When in 2014 the STEM Career Interest Survey (STEM-CIS) was developed, the researchers could check this instrument with other target audiences. Also, the question remained if the instrument was applicable for both boys and girls. This article describes the development and validation of a Dutch language version of it, called the STEM-LIT instrument, an instrument to measure the interest of children aged between ten and 12 years in Science, Technology, Engineering and Mathematics (STEM), focusing specifically on children from families with low socio-economic status (SES). The instrument has been adapted and developed in five stages and tested among Dutch primary school pupils in groups seven and eight (ages 10–11 and 11–12). The instrument was first tested in two pilot studies, after which some amendments were made. The final version of the instrument was tested with 212 pupils, with a Cronbach’s Alpha of.91, adding supporting evidence to the reliability of the original STEM-CIS instrument and showing that the test is as reliable for both girls and boys.

Keywords:

• Science communication
• education
• STEM education
• instrument
• socio-economic status
• equality
• tutoring
• primary education
• The Netherlands

PUBLIC INTEREST STATEMENT

We are missing out on talent in the STEM (science, technology, engineering and math) sectors, as social-economic status and gender important indicators to choose to study or work in STEM. All children should be able to engage with STEM subjects, but how can this be done in an effective way? In this research, we adapted the original STEM-CIS instrument to the first Dutch instrument to measure STEM-interest with children at primary schools in disadvantaged neighbourhoods. We then checked if it was as reliable for younger children and for both boys and girls. It is. Such a test can be broadly implemented to measure the impact of interventions to enhance interest and attitude of children towards STEM and to determine the difference in STEM-interest between children from different societal groups, both for boys and girls.

1. Introduction

There is a growing interest in encouraging children to engage with STEM subjects (Jones et al., 2018). Time, money and energy are being invested in well-intentioned activities, but the effectiveness and reliability of these interventions is disputed, particularly for target groups that are difficult to reach such as children from families with lower socio-economic status (SES) and girls (Ado, 2019; Freeman et al., 2019). Validated and reliable instruments are essential to be able to properly measure the effectiveness of these projects. To date, validated instruments are scarce. The STEM Career Interest Survey (STEM-CIS) was developed by Kier et al. (2014), but was originally tested with children from 11 to 14 years old. We wanted to broaden this age in the Netherlands by validation with younger children from lower SES and in the meantime also validate whether the test is equally reliable for boys and girls. Therefore, we adapted the original STEM-CIS test into a Dutch version, the Science Technology Engineering Mathematics Leiden Interest Test (STEM-LIT) measuring instrument (see Appendix A).

We wanted to broaden the age of the original instrument since in several educational systems such as the Netherlands, Austria, Belgium, etc. pupils have to make important decisions about their future career as early as at the end of group 8 (age 12), in contrast to countries with a comprehensive education system, such as Sweden and Canada, where pupils follow a common curriculum for a longer period of time (Van de Werfhorst, 2019). If this instrument can also be used with younger children, it allows also the use of experiments in non-comprehensive educational systems. Also, if the instrument is as reliable for boys and girls, the instrument allows to be used to check the effects of experiments on gender.

2. Theoretical framework

Children aged between ten and twelve tend to choose professions that are part of their social class (DeWitt et al., 2019; Kuijpers, 2012). It is important to take this into account if we want to motivate children from difficult-to-reach target groups, such as families with a low socio-economic status, to become interested in a STEM field. To improve the future prospects of children and young people who experience barriers within their environment, it is important to improve pupils’ self-confidence and to make it possible to discuss the restrictions they experience, consciously or not, in their environment (Kuijpers, 2012).

In 1994 Lent, Brown and Hackett developed the Social Cognitive Career Theory (SCCT) to explain differences in study and career choices. The SCCT is based in part on Social Cognitive Theory (SCT) (Bandura, 1986, 1989; see, for example, Yazilitas et al., 2013). There are a number of key concepts within the SCCT that explain the differences in study choices among pupils. These are:

1. individual self-efficacy or self-confidence in one’s own behaviour

2. outcome expectations,

3. personal contribution, and

4. supporting factors and barriers in the environment.

The SCCT model has served as the basis for different evaluation studies on STEM choices among pupils in secondary and higher education (see, for example,Tytler et al., 2008; Wang, 2013), but, as Kier et al. (2014) remark, most evaluations are focused on only some aspects of the model or they concentrate on a few domains of STEM.

Kier and colleagues, therefore, developed their own instrument that combines the different aspects of the SCCT model and measures these aspects for the four different domains within STEM. This instrument, the STEM Career Interest Survey (STEM-CIS), comprises 44 items that represent the main aspects, such as self-efficacy, outcome expectations, personal input and relevant supporting factors and barriers. The underlying concepts of the STEM-CIS based on Lent et al. (1994) are:

• 2 questions for each field on self-efficacy

• 2 questions for each field on personal goals

• 2 questions for each field on output expectations

• 2 questions for each field on the interest in the given field

• 2 questions for each field on contextual support

• 1 question for each field on personal input.

The 44 items are also divided into four scales, each representing the STEM fields. The original STEM-CIS has been tested among 1061 pupils, aged on average from eleven to 14 years, from seven schools in parts of the south-eastern United States characterised by generally high levels of poverty. Cronbach’s Alpha for the STEM disciplines ranged between 0.77 and 0.89 (Kier et al., 2014). An important addition to the original study is that our present study will report if the scale is equally appropriate for both girls and boys in our target group.

3. Methodology and findings

Here we describe the development of the STEM-LIT, suitable for Dutch-speaking children from groups seven and eight (ages 10–11 and 11–12) with a low SES background. The instrument was developed in several phases, each with its own respondents and methodology.

For each round, permission was sought from the teacher of the pupils concerned. The research was carried out during class time under the supervision of the teacher. Parents were informed in advance by letter; both parents and children were allowed to refuse the child’s participation.

3.1. Defining the target group

In order to make the instrument suitable for our specific younger target group, namely primary school pupils with a low SES, a selection was made of the primary schools to be approached for the research. In order to select the schools, we looked at such factors as the neighbourhood in which a school is located. We focused on neighbourhoods with generally low SES. Schools in Amsterdam West, Amsterdam Zuidoost, the Schilderswijk in The Hague, Schalkwijk in Haarlem, Rotterdam Delfshaven and Rotterdam-Zuid were approached to participate in the study. In addition, the outflow levels of the schools were taken into account. The www.scholenopdekaart.nl website was used for this purpose. The majority of the pupils of the schools that took part in this study were recommended to follow intermediate preparatory vocational education (VMBO). The level of the voluntary parental contribution was also looked at, as this can be indicative for the SES of pupils.

3.1.1. Phase 1: preliminary study of the language used by pupils

Dutch children at primary school are not yet confronted with STEM as a subject, thus these children are often still unfamiliar with the individual terms referred to. In the first phase of the study, we, therefore, mapped out which Dutch words pupils from groups seven and eight use to communicate about STEM. For this purpose, a written questionnaire with open questions was conducted among 97 pupils from two different primary schools and one weekend school. The latter is a school that offers supplementary education on Sundays for children aged from ten to 12 years, from socially and economically disadvantaged neighbourhoods. In addition to the questionnaire, eleven group discussions were held among these pupils.

The questionnaire consisted of questions such as: “What do you think about when you think of arithmetic?” and “Explain briefly what science is.” In addition, the questionnaire also contained three questions to classify the SES of the pupils: the number of books in the pupil’s home, the level of education of their mother and the profession of the parents. Because many pupils did not know what kind of work their parents did, this question was not included in the pilot version of the instrument.

Eleven Group discussions with 3–5 children were used to check whether the pupils use other words for STEM in conversations compared to the words they had written down in the questionnaire. The conversations were held in groups of three to five pupils, lasted about three to 10 min and were recorded using a voice recorder. For example, in the conversations the pupils were asked what science is, what a scientist does and should be able to do, and whether there is a difference between a scientist and a researcher. Another question was what the pupils thought about in terms of technical subjects and technology.

3.1.2. Phase 2: developing the first version of the STEM-LIT adaptation

The answers given by the pupils to the questionnaire and in the group discussions from the first phase were used to retranslate the STEM-CIS instrument into language accessible for Dutch pupils. Draft versions of this translation were presented to three experts (an expert with knowledge of the target group, an expert on STEM and an expert in the field of survey development) in order to arrive at a first test version.

3.1.3. Phase 3: pilot 1

The first pilot study was conducted during school hours at three different schools. The session lasted 20 min. After applying the instrument, there was a short group discussion in each class with four to six pupils to find out what they thought of the instrument and whether they understood the questions. The first version was presented to 56 pupils of grades seven and eight from three different schools. The first group of respondents consisted of 36 boys and 19 girls in total (1 respondent did not indicate gender), see Table 1.

Table 1. No of respondents phase 1

	Boys	Girls	X	Total
Younger than 10	2	0	0	2
10 of 11	32	16	1	49
12 or 13	2	3	0	5
Total	36	19	1	56

3.2. Reliability

In terms of reliability, the Pilot 1 instrument scored a Cronbach’s Alpha of .875 across the entire scale, i.e. all items for Science, Technology, Engineering and Mathematics, with Alpha results for the subscales.

1. Science: .712 where the omission of no single item would raise the Alpha,

2. Technology: .876 where the omission of no single item would raise the Alpha,

3. Engineering: .833 where the omission of no single item would raise the Alpha,

4. Mathematics: .584 where omitting item 2 (I am able to complete my arithmetic homework) could raise the Alpha to .704.

This analysis showed that item 2 for Mathematics reduced the reliability, which is why we modified this item. The choice to modify the item rather than omit it was made to ensure consistency of content between the four scales and to remain as close as possible to the original STEM-CIS instrument.

3.3. Revision

On the basis of the first pilot study, a number of items were reformulated, partly on the basis of the analyses, partly on the basis of the interviews and partly on the basis of findings while administering the instrument.

While the instrument was being administered, a number of pupils asked what “looking up to someone” meant. A number of pupils also indicated that they did not understand the items in which this occurred. For this reason, the items “I look up to someone who uses science/arithmetic/technology/design in his or her job.” have been changed to “I admire someone who uses science/arithmetic/technology/design in his or her job.” In addition, it turned out that the pupils from one school were not given homework because of extra teaching time at school and the pupils of the other two schools were not given science homework. This explains why the item “I am able to complete my homework using science” has been replaced by “I can complete activities using science, at home or at school”. Also, the item “I can finish my arithmetic homework” has been changed to “I can finish my arithmetic tasks/assignments”.

3.3.1. Phase 4: pilot 2

The revised version was presented to 39 pupils from one grade seven class and one grade eight class at two different schools. Again, after applying the instrument in both classes, a short group discussion was held with four to six pupils to find out what they thought of the instrument and whether they understood the questions. The number of respondents by gender and broken down by age group can be found in Table 2. Nine pupils did not fill in their age, nor their gender.

Table 2. No of respondents phase 4

	Boys	Girls	X	Total
Younger than 10	0	0	0	0
10 or 11	10	16	1	27
12 or 13	1	2	0	3
X	0	0	9	9
Total	11	18	10	39

3.4. Reliability

1. In terms of reliability, the instrument scored a Cronbach’s Alpha of .924 for the entire scale, The Alpha results for the different sub-scales:

2. Science: .736 where the omission of items 10 and 11 could raise the Alpha to .740,

3. Technology: .806 where the omission of item 11 could raise the Alpha to .807,

4. Engineering: .859 where the omission of no single item would raise the Alpha,

5. Mathematics: .823 where the omission of no single item would raise the Alpha.

The discussions did not reveal any further major uncertainties and the instrument was retained for the final validation.

3.4.1. Phase 5: final validation

The final instrument was presented to 212 pupils from grades seven and eight from four different schools. The number of respondents by gender and by age group can be found in Table 3. Four pupils did not fill in either age or gender.

Table 3. No of respondents phase 5

	Boys	Girls	X	Total
Younger than 10	0	1	0	1
10 or 11	75	91	0	166
12 or 13	18	22	1	41
x	0	0	4	4
Total	93	114	5	212

3.5. Reliability

The instrument scored a Cronbach’s Alpha of 0.910 for the entire scale in terms of reliability.

The Alpha results for the different sub-scales:

1. Science: .782 where the omission of item 11 could raise the Alpha to.806,

2. Technology: .862 where the omission of item 11 could raise the Alpha to .868,

3. Engineering: .879 where the omission of item 11 could raise the Alpha to .894,

4. Mathematics: .830 where the omission of item 11 could raise the Alpha to.834.

These results indicate a solid instrument in which the choice can be retained whether or not the eleventh item is omitted from each of the four sub-scales. The questions for item 11 are:

1. I know someone in my family whose work involves using science.

2. I know someone in my family whose work involves using technology.

3. I know someone in my family whose work involves designing things.

4. I know someone in my family whose work involves using arithmetic.

In order to guarantee the validity of the content of the instrument and to remain as close as possible to the original STEM-CIS instrument, it seems more appropriate to use the eleventh item, especially since the reliability is already very high and only increased slightly when omitted.

Essentially to our goals, we checked whether the instrument had any bias with respect to gender or age, but no significant correlation could be found here. Table 4 shows the reliability split based on gender.

Table 4. Alphas for the different subscales and the total scale split based on gender

	Girls (n = 106)	Boys (n = 85)
Science	.77	.79
Technology	.85	.84
Engineering	.87	.88
Mathematics	.83	.83
Total	.90	.91

4. Conclusion: the final version of the instrument and recommendations for use

Our adapted STEM-LIT version of the STEM-CIS instrument by Kier et al. (2014) adds support to the original instrument. the results of our STEM-LIT instrument are remarkably similar to the original findings of the STEM-CIS instrument, CFR Table 5.

Table 5. Overview Cronbach alphas STEM-CIS and STEM-LIT

	STEM-CIS	STEM-LIT
Science	.77	.78
Technology	.85	.86
Engeneering	.89	.88
Mathematics	.86	.83

Our results indicate that our STEM-LIT instrument can be used for children age 9, younger than the original STEM-CIS instruments as a mean for, e.g., pre- and posttests in RCT’s working on attitudes towards STEM-education.

Both the STEM-CIS and STEM-LIT instruments consists of four scales who each are composed of eleven items relating to the pupils’ interest in the four STEM subject areas: science, technology, engineering and mathematics. Above the scales we have included an explanation of the field of study, also using the pupils’ words from qualitative research (phase 1). The items represent the most important aspects of the SCCT, such as self-efficacy, outcome expectations, personal input and related supporting factors and barriers. The items can be answered on a five-point scale ranging from 1: Totally disagree to 5: Totally agree.

The instruments takes 20 min to administer and offers researchers the opportunity to measure in a relatively short time the attitude of children aged 9 to 14 in relation to STEM, whereby a distinction can be made between the various sub-aspects of STEM. The questionnaire is to be used in a written and individual manner and therefore does not require a scientist with background knowledge to be present when completing the questionnaire. Both the STEM-CIS and STEM-LIST can be used in their entirety, or in part if a single subject area is being investigated, for primary school pupils in the upper years, but will also be useful in the lower years of secondary education. We expect that the test can provide insight into baseline measurements of interest in STEM in different groups, but it can also be used to measure the impact of an intervention by applying the instrument before and after the intervention. This makes the instrument widely applicable and gives hope for a new dimension in the move towards attracting more STEM graduates. The tools have been developed for children from families with a low SES background, and will therefore also be accessible to children who enjoy a more linguistically rich upbringing.

5. Limitations and suggestions for further research

As our research was executed with a translation of the original instrument, we cannot prove that the original STEM-CIS instrument can be used for younger children, although our results do suggest this could be the case. This is worth further research. Further validation may be of interest for other target groups, in other languages, in other regions and other socio-economic groups. Our result suggests that this could well be worth the investment.

Additional information

Funding

This research project is fully funded by the NWO grants within the Dutch Research Agenda. [Dossier number: 400.17.608].

Notes on contributors

J. Cramer

The research group Science Communication and Society at Leiden University studies science communication in an evidence-based manner. Our research program is aimed at improving science communication with as central themes “Authenticity in informal learning” and “Bridging the gap between experts and the general public”. The presented work aims to bridge between students and the children from disadvantaged neighbourhoods. This work prepares for the evaluation and research within a large project for evidence-based outreach to understand and increase participation in STEM learning of youth with backgrounds. This project is part of the Dutch National Research Agenda, focussed on interactions with society.