Validation of an e-instrument for assessing five-year-old children’s development in Estonia: a comparison of children’s skills and teachers’ evaluations

ABSTRACT

Children’s development during early childhood affects their well-being and educational success, but there are few reliable assessment instruments available. The aim of the study was to develop, pilot and validate an e-assessment instrument for assessing five-year-old children’s development in cognitive processes, learning, language and mathematical skills. The tablet-based instrument was piloted in Estonia in 11 kindergartens with 122 children. The teachers guided the children one by one over several days. The instrument was validated using teachers’ evaluations of the sampled children’s skills. The results showed positive associations in expected skill areas supporting the claim of the instrument being valid for assessing young children’s development.

GRAPHICAL ABSTRACT

KEYWORDS:

• Children’s development
• assessment instrument
• cognitive processes
• learning skills
• language skills
• mathematical skills

1. Introduction

There is strong evidence that early learning matters and affects children’s development and coping in later life (Bartik 2014; Brownell and Drummond 2020; Schoon et al. 2015). The first five years of children’s lives are critical to their development (Shuey and Kankaraš 2018), with high-quality preschool education affecting academic achievement in later school levels (Fuller et al. 2017; Karoly and Whitaker 2016; Bakken, Brown, and Downing 2017). Even though it is well-acknowledged that early childhood education and care matters, children’s experiences with it vary largely due to their individual characteristics, family background, access and participation in early childhood education (Working Group on Early Childhood Education and Care under the auspices of the European Commission 2014).

Monitoring development in early childhood helps to make children’s early learning visible (Dunphy 2010). It is important to get feedback at the local and national levels (Working Group on Early Childhood Education and Care under the auspices of the European Commission 2014) as well at the programme and individual level (National Research Council 2008) in order to understand how to support children’s development effectively. In Estonia, various methods are used to assess young children’s skills, including observations in settings where one or two teachers have to usually look after 20 or even 24 children at a time (Kikas and Lerkkanen 2010). Observation as a formative assessment method is common in early childhood education. However, it is also crucial that the observers have extensive knowledge about children’s development and the assessment is manageable in terms of time and resources (Dunphy 2010). E-assessment allows teachers to save time on assessment and/or the subsequent scoring and gives them instant feedback. Compared to other assessment methods (e.g. observations or interviews), e-assessment can be more engaging for the children by including visual, audio, and video materials (Adkins 2021).

E-assessment instruments are often standardised, making them reliable instruments due to their extensive development period and quality. For a test to be of value to teachers, it should give a clear indication of the child’s strengths and weaknesses which teachers can use to make decisions to support children’s development (Brown and Hattie 2012). Although there has been much research on young children’s skills, most studies have focused on a specific developmental domain (see Best et al. 2011, for cognitive processes; Collie et al. 2019, for social skills; Morgan et al. 2019, for cognitive processes; Watts et al. 2014, for mathematical skills; Duncan et al. 2020, for literacy). Some studies involving more than one developmental area (Fuhs et al. 2014; Janus and Offord 2007; OECD 2020; Pisani, Borisova, and Dowd 2018; Fuller et al. 2017) have shown the benefits of taking a holistic approach to understanding young children’s development. In addition to traditionally emphasised developmental areas (literacy and numeracy), assessments should include non-academic skills affecting children’s development and well-being, e.g. cognitive, self-regulation, social and emotional skills (National Research Council 2008; Shuey and Kankaraš 2018).

Even though in Estonia, the curriculum of preschool childcare institutions states that assessment of children’s development is part of the daily teaching and learning process (The Government of the Republic 2008/2011/2008/2011), it is not specified what assessment instruments teachers can use and how they interpret the gathered data. Although studies have shown that tablet-based assessment involving storytelling is appropriate for children as young as four or five (Adkins 2021; OECD 2020), there are no widely available assessment instruments which cover various developmental areas (Pisani, Borisova, and Dowd 2018; Reid et al. 2014). However, e-assessment instruments should not be merely paper-based tests transferred to digital format but innovative assessment instruments employing the advantages of technology (Day et al. 2019). For this reason, the Estonian Education and Youth Board started developing a new electronic assessment instrument for assessing young children’s skills. The instrument does not claim to offer a complete overview of children’s development; rather, it offers information in areas where teachers might need assistance and e-assessment has advantages. It should be kept in mind that a single assessment cannot provide a definitive evaluation of children’s development. Instead, rather various tools should complement each other (Boekaerts and Corno 2005). In order to ensure that the results of an assessment instrument are trustworthy, the assessment instrument should be validated (Messick 1994). Sireci (2007) has written that validity should be considered when using a test for a particular purpose, but not as a property of the test in question. Evidence that could support or discredit the test scores should be collected to validate a test (Camargo, Herrera, and Traynor 2018).

The current work presents the developed tool and describes its validation process involving comparing children’s direct assessments with teachers’ evaluations of the same skills. Other assessment instruments (Alloway et al. 2008) or parent questionnaires have been previously used to validate new assessment instruments (Pisani, Borisova, and Dowd 2018). Previous research has shown that teachers are quite accurate in evaluating children’s skills (Kowalski et al. 2018; OECD 2020; Cabell et al. 2009) and their evaluations align with actual skills more than parents’ opinions (OECD 2020).

1.1. Developing e-assessment instrument for assessing five-year-olds’ development

The development of the e-assessment instrument for five-year-old children was commissioned by the Estonian Ministry of Education. Estonian Education and Youth Board, together with researchers from Tallinn University, started the development of the instrument at the beginning of 2019. The team brought together experts from psychology, early childhood education, and special education. The instrument has been developed drawing on the experience and knowledge gained from the IELS study (Phair 2021) and from previous research in Estonia (Männamaa and Kikas 2011). The instrument is intended to be used on tablets with teachers guiding the children one-on-one basis. As the instrument is entirely computer-based and includes audio instructions, teachers are only expected to guide the children on using the tablet’s functionality. The instrument is developed to give kindergarten teachers an indication of children’s skill levels, including areas and topics where they outperform expected outcomes or where they might need further support. The tool will be freely available to all kindergarten teachers in Estonia who have attended a training course.

An assessment framework and prototype were created during the first stage of the instrument’s developmental process. The assessment framework includes the theoretical basis for assessing and supporting young children’s development and the areas covered with the instrument. The theoretical background of the assessment instrument draws on the social constructivist approach of Vygotsky (1978), retaining the notion that children construct their knowledge together with parents and peers while the surrounding environment, culture, and society also play a crucial role. In order to support children in this process, their knowledge and skills need to be assessed using appropriate methods (Boekaerts and Cascallar 2006).

The developmental areas included in the assessment instrument were chosen based on the national curriculum of preschool childcare institutions (The Government of the Republic 2008/2011/2008/2011) and previous studies which indicated the competencies needed to become an autonomous self-regulated learner (Boekaerts and Cascallar 2006; OECD 2020). There are no expected outcomes for five-year-old children in the Estonian preschool curriculum, only for six-seven-year-old children (The Government of the Republic 2008/2011/2008/2011). Therefore, in addition to taking the curriculum as a basis, the appropriate level of the instrument was set based on previous similar research (OECD 2020). According to the framework, the assessment instrument includes five areas: cognitive processes, mathematical, language, learning, and social skills. Each area is divided into topics and/or constructs to be assessed. The instrument’s prototype includes a continuous thematic story and ideas for assessment items (Kikas et al. 2020). The areas of cognitive processes, language and mathematical skills were separated to be able to assess cognitive processes, which are a basis for other developmental areas (Best et al. 2011). Learning skills are assessed as previous research has shown the importance of learning and social skills in contributing to a child’s later academic success (McClelland, Acock, and Morrison 2006). In the current Estonian curriculum, the cognitive processes and learning skills are presented together as ‘cognitive and learning skills’ and while components of cognitive skills have been specified, components of learning skills have not. As the curriculum is currently under revision to make it more consistent with the curriculum for primary schools, where learning skills have been explicitly brought out (The Government of the Republic 2011/2022/2011/2022), the cognitive processes and learning skills were included separately in the instrument. Social skills will be included in the later stage of the development of the instrument.

1.2. The present study

The aim of the present research was to pilot and validate a new electronic instrument for assessing five-year-old children’s development in Estonia. The current study involves the second stage of the instrument’s development, namely, the development of two tests based on the prototype and piloting of the assessment tool in a sample of kindergartens. The tests include four developmental areas: cognitive processes, mathematical, language, and learning skills. The developmental areas and topics included in the present study are shown in Figure 1. In the areas of cognitive processes and subject competencies, the topics are further divided into sub-topics (constructs), but in learning skills, the topics correspond to the constructs. The four areas are integrated into the tests, and both tests include a continuous story that gives a guide to solving the items (see below in measures).

Figure 1. Developmental areas and topics included in the assessment instrument and present study.

The present study focuses on the following research question: Are the results obtained with the children’s assessment instrument consistent with the results of the teachers’ questionnaire? Teachers’ evaluations were chosen to validate the instrument as there were no other suitable measures freely available, and previous research has shown that teachers’ evaluations of children’s skills broadly align with children’s actual skills (Kowalski et al. 2018; OECD 2020; Cabell et al. 2009). Teachers do not tend to overestimate results as much as parents (OECD 2020). We have set the following hypotheses:

(H1) The assessed developmental areas and topics have moderate positive associations. Previous research has shown that children’s outcomes in different topics in one developmental area as well as across different developmental areas are positively correlated (Reid et al. 2014; Best et al. 2011; Vitiello and Williford 2021; McClelland, Acock, and Morrison 2006). Previous research has shown that the associations between topics are somewhat lower for learning skills (Yen, Konold, and McDermott 2004);

(H2) The results of the children’s assessment and teachers’ evaluations have moderate positive associations. Previous studies have shown that teachers are quite accurate in evaluating young children’s skills, with teachers’ evaluations being moderately correlated with children’s actual skills (Reid et al. 2014; Cabell et al. 2009; Kowalski et al. 2018; Vitiello and Williford 2021) but also based on previous research, we assume that there are some discrepancies between teachers’ ratings and children’s actual skills; this discrepancy could be attributed to rater bias (Hinnant, O’Brien, and Ghazarian 2009; Mashburn and Henry 2004; OECD 2020; Waterman et al. 2012);

(H3) The correlations between children’s direct assessment and teachers’ evaluations are stronger in academic skills (language, mathematics) than in cognitive processes and learning skills. As teachers cannot fully rely on the pre-school curriculum which only states outcomes of six-seven-year-old children and studies have concentrated more frequently on academic skills and seldom on cognitive processes and learning skills (McClelland, Acock, and Morrison 2006; Kesäläinen et al. 2022), teachers have less information about these areas. Also, we expect weaker associations in more complex topics which are on a level not yet expected from five-year-olds and harder for teachers to assess (Mashburn and Henry 2004).

1.3. Children’s e-assessment in Estonia

Various e-assessment instruments have been developed in Estonia following the Estonian Lifelong Learning Strategy 2020 to focus on a digital turn in education (Ministry of Education and Research 2019). The general goal of the strategy was to consider everyone as a learner whose specific needs and abilities should be considered while providing learning opportunities. Five strategic goals were set for 2020: (1) the change in approach to learning considering each learner’s individual and social development, (2) competent and motivated teachers and school leaders, (3) concordance of lifelong learning opportunities with the needs of the labour market, (4) having a digital focus on lifelong learning which supports the use of digital technology in learning and teaching, (5) equal opportunities and increased participation in lifelong learning (Ministry of Education and Research 2014). The first and fourth goals are the most relevant in terms of this work as their principles guide the development of the new e-instrument. The goals are continued in a follow-up Education Strategy for 2021-2035, which also emphasises the importance of developing general competencies and digital pedagogy in all educational levels and forms (Ministry of Education and Research 2021). Even though both strategies emphasise the importance of continuity through education levels, until now, e-assessment instruments developed in Estonia have only been used in primary and secondary education (Eurydice 2020).

Estonia participated in the International Early Learning and Child Well-being Study (IELS) in 2018 to get information about various aspects of children’s development and the environmental factors affecting it. The study involved assessing five-year-old children’s development using a specially developed e-assessment instrument. The study showed the possibility and suitability of tablet-based assessments for young children (OECD 2020). Estonian teachers were highly motivated to participate in IELS as study administrators. They expressed their opinion that similar tablet-based tools were needed in their work, making it evident that teachers are positively minded towards e-assessment (Meesak 2019).

2. Methodology

The aim of the current study was to pilot and validate an electronic instrument which can be used for assessing five-year-old children’s development. The study involves quantitative data collection and analysis methods. Previous research has shown that information regarding young children’s skills can be gathered quantitatively with an e-assessment instrument (OECD 2020) and teachers evaluations can be used for validating those results obtained (Cabell et al. 2009; Kowalski et al. 2018). Quantitative approach allows the researcher to generalise the results to a larger population (Carr 1994) and can reduce the burden put on participants without compromising the quality of the research (Rhemtulla and Hancock 2016).

2.1. Sample and procedure

The assessment instrument for assessing five-year-old children’s skills is developed by the Estonian Education and Youth Board on behalf of the Estonian Ministry of Education and Research. The approval from ethics committee is not required for nationally developed studies in Estonia. However, the study followed the American Psychological Association’s ethical guidelines (American Psychological Association 2017). Consent for the study was given by all parties concerned – kindergartens, teachers, parents and children.

The chairwoman of the board of the Estonian Association of Pre-School Education Leaders (MTÜ Eesti Alushariduse Juhtide Ühendus) was asked to invite the association’s members to participate in the item trial of a new assessment instrument for assessing five-year-old children’s development. The first eight kindergartens who responded by e-mail were able to take part. In addition, two kindergartens from which a teacher and head of studies were involved in developing the items and two kindergartens where Tallinn University Preschool Education Programme’s Bachelor’s-level students worked joined the study. Due to the time-consuming nature of the assessments, only 10 children from each kindergarten could participate. The representatives of the kindergartens gave their written consent for the kindergartens and their teacher’s participation. The children’s parents received a letter informing them about the study’s aims, procedure, and possibilities of participating in the study, as well as the conditions regarding withdrawing their consent. Parents signed an informed consent form for their child’s participation, and each child was also asked for verbal consent to participate before each assessment session. Due to the children’s age, they were asked if they wanted to ‘play a game on a tablet which they can stop at any time’. Only children whose parents had given written consent and who had agreed to the games participated in the study.

The assessments were carried out on a one-on-one basis with the child and the administrator. A tablet computer was used for instrument administration. All the administrators received a manual for carrying out the assessments, and online training was provided. The manual included an introduction to the instrument and instructions for carrying out the assessments. Screenshots of the introductory items, along with specific suggestions, were included. The administrators were instructed to encourage the child to get to know the assessment instrument during the introductory items, while the administrators were allowed to guide and help the child in any way, including explaining the meaning of each button. During the assessment, the administrators were allowed only to answer technical questions (e.g. ‘should I push this button to continue?’).

Each child took part in the assessment for more than one day. There was no maximum time limit, but administrators were instructed not to assess one child longer than 30-minutes at a time. Taking breaks and continuing later the same day or another day was encouraged. Going through the introductory items took children, on average, about five and half minutes (M = 5.33 min, SD = 2.95 min), solving the Home test took on average 38 min (M = 38.25, SD = 8.73) and solving the Shop test took on average 67 min (M = 66.84, SD = 17.87).

The teacher who knew the child the best filled out an online questionnaire about the child’s skills before the assessments. The questionnaire was administered through LimeSurvey and filled out for each child separately. The questionnaire administration period was 2nd November – 9th November 2020. The children’s assessment period was 9th November – 15th December 2020.

In total, 122 children from 11 kindergartens from five different counties participated in the study. Data from 119 children were used for analysis as one child was excluded due to a special need (colour-blindness), and two children did not finish the assessments. The children’s age range was 4 years 10 months to 6 years 1 month (M = 5.54; SD = 0.28). 55% (N = 65) were boys and 45% (N = 54) were girls. The teachers were all women (N = 14) with an average age of 44 (SD = 8.90). The age of the teachers ranged from 25 to 62.

2.2. Measures

2.2.1. Children’s assessment instrument

The children’s assessment instrument currently includes items to assess cognitive processes, language, mathematical and learning skills, which are divided into two thematic tests. The theme of the first test is ‘children at home’ (Home test), and the second theme is ‘children in the shop’ (Shop test). The two tests have different difficulty levels, with the Shop test including more advanced items. The Home test includes 77 items, and the Shop test includes 101 items. Also, there are 12 introductory items which have to be solved before the tests, the results of which will not be assessed. Each item from the mentioned two tests corresponds to at least one topic and construct, the construct being the smallest component assessed. The instrument included 16 topics, 46 constructs and multiple items corresponding to each topic and construct. In the present work, the sums of the scores for topics (in language and mathematical skills) or skill areas (in cognitive processes and learning skills) were used. All items include only audio instructions and are entirely computer-assessed. Examples of the children’s assessment items for each developmental area are included in Supplementary Materials.

2.2.2. Teacher’s questionnaire

A teachers’ questionnaire was developed based on the concept of assessing five-year-old children’s development (Kikas et al. 2020) and the constructs included in the assessment instrument. The questionnaire included five statements on cognitive processes, eleven on language skills, eight on mathematical skills and five on learning skills (see Appendix A, Table A1). The statements on language and mathematical skills were divided into three sub-topics. All the statements had a 3-point scale with an additional option of omitting an answer (1 = ‘not at all’, 2 = ‘partially’, 3 = ‘very well’, 4 = ‘don’t know’). If the fourth option was chosen, the respondent was asked to elaborate on why they could not respond. The option ‘partially’ included specifying information on the statement (e.g. ‘depending on the task’). There was an open-ended option after each skill area to leave comments. The comments were either general (e.g. ‘The child is smart, sharp and broad-minded’, or noted in which situations the child demonstrates their skills (e.g. ‘The child excels in smaller groups of children’; ‘The child might need some extra time in bigger groups’) or indicated if the topic is included in the kindergarten’s curriculum (e.g. ‘We are still in the process of learning letters and sounds’; ‘So large numbers are not used or taught in kindergarten’). As none of the comments affected the accuracy of the teachers’ questionnaire, nor were relevant while considering the children’s assessment results, they were not further analysed.

3. Analysis strategy

Data analysis was conducted with R version 4.2.1 (R Core Team 2022). The reliability of the children’s items used for assessing each construct was first tested with Cronbach’s alpha or Kuder – Richardson Formula 20 from [psych] package v[2.2.5] (Revelle 2022). The reliability coefficients were calculated for items or questions for each construct. The constructs with a high number of items or questions and low coefficient α < 0.5 (Castro 2016; Nunnally 1967) or very high success percentage were modified by decreasing items or questions or entirely leaving them out of the construct. Some constructs with a low number of items and low coefficients were still included based on the experts’ decisions and previous research showing that acceptable values used in research differ and decisions should not only rely on the size of the value rather than theoretical and logical conclusions (Kopalle and Lehmann 1997; Taber 2018). Values as low as α = 0.3 can be considered acceptable, when the number of items is limited and the diverse and engaging breadth of the constructs is more important than achieving high coefficients (Ryff and Keyes 1995), as is the case with developing an assessment instrument for young children. Similarly, the Cronbach’s alpha was calculated for initial questionnaire items based on intended factors.

To determine if the children’s assessment and teachers’ questionnaire data were Missing Completely At Random (MCAR) the Little (1988) MCAR test from [naniar] package v[0.6.1] was used (Tierney et al. 2021). For the children’s assessment, the data were divided into topics as the test can be performed with a maximum of 50 variables (Yanagida 2022). The children’s assessment and teachers’ questionnaire data were then imputed using classification and regression trees from [mice] package v[3.14.0] (van Buuren and Groothuis-Oudshoorn 2011).

Confirmatory factor analysis (CFA) from [lavaan] package v[0.6-12] with the Maximum Likelihood Estimation Method (Rosseel 2012) was used to examine the structure of the teachers’ questionnaire. First, the option ‘don’t know’ was recorded as a missing value (−99). According to Kline (2016), a CFI (comparative fit index) and TLI (Tucker-Lewis index) values above 0.90 indicate a good model fit, while an SRMR (standardised root mean square residual) value below 0.08 (Hu and Bentler 1999) and an RMSEA (root mean square error of approximation) value below 0.07 (Steiger 2007) can be considered acceptable.

Spearman correlations between children’s assessment results and teachers’ evaluations were then calculated using [Hmisc] package v[4.7-0] (Harrell and Dupont 2022) to validate the children’s assessment instrument. Correlations of children’s assessment instruments and teachers’ questionnaires were visualised using [corrplot] package v[0.92] (Wei et al. 2021). According to Ellis (2010), correlations below 0.30 are considered low, 0.30–0.50 are considered moderate, and correlations exceeding 0.50 can be considered large.

4. Results and discussion

4.1. Reliability of constructs

The number of items for the children’s assessment instrument in each developmental area and topic, descriptive statistics and reliability indicators are included in Appendix A (Table A2). The reliability coefficients of the constructs ranged from 0.56–0.91 for cognitive processes, 0.32–0.87 for language skills, 0.49–0.92 for mathematical skills and 0.57–0.64 for learning skills. Only topics having an equivalent in the teachers’ questionnaire were included for further analysis. Similarly, the descriptive statistics and reliability coefficients for teachers’ questionnaire are included in the Appendix A (Table A3).

4.2. Data imputation

Before further analysis, the data were imputed using classification and regression trees (van Buuren and Groothuis-Oudshoorn 2011). The Little (1988) MCAR test showed that the questionnaire data were not MCAR (MCAR test: χ2 (860) = 1044, p < 0.001). The data for cognitive processes (MCAR test: χ2 (276) = 351, p < 0.001), text comprehension (MCAR test: χ2 (94) = 144, p < 0.001), emerging reading and writing (MCAR test: χ2 (769) = 995, p < 0.001), counting (MCAR test: χ2 (110) = 215, p < 0.001), numbers and figures (MCAR test: χ2 (73) = 147, p < 0.001) were not MCAR, whereas the data for vocabulary (MCAR test: χ2 (11) = 7,94, p = 0.718), calculation (MCAR test: χ2 (173) = 218, p = 0.011) and learning (MCAR test: χ2 (88) = 99.0, p = 0.198) were MCAR.

4.3. Latent structure of teachers’ questionnaire

The latent structure of the teachers’ questionnaire was then confirmed using CFA. The model has eight factors corresponding to the initial intended factors. The model fit indices showed good fit between the hypothesised model and the data with CFI = 0.942, TLI = 0.924, RMSEA = 0.063, SRMR = 0.059. The model with standardised coefficients and covariates is shown in Figure 2. There are significant covariations between all factors of the model.

Figure 2. Questionnaire latent variable model with standardised path coefficients and covariates. Note: COG = cognitive processes, TEXT = text comprehension, PHONO = emerging reading and writing (phonology), VOC = vocabulary, NUM = numbers and figures, CALC = calculation, COUNT = counting, LEARN = learning skills, all covariations are significant, * p < 0.05, ** p < 0.01, *** p < 0.001.

4.4. Construct validity

For determining the construct validity of the e-assessment instrument, Spearman correlations were first calculated separately between children’s assessment topics, then teachers’ questionnaire factors and finally between the corresponding topics and factors. The p-values were adjusted with Holm’s method.

The significant Spearman correlations between children’s assessment instrument topics are shown in the upper part of Figure 3 and correlations between teachers’ questionnaire factors in the lower part. The figure shows that all the topics of the children’s assessment are positively and significantly interrelated (except learning skills), with the correlations ranging from moderate (0.30 < ρ < 0.50) to large (ρ > 0.50). The analysis included all the topics of cognitive processes, three out of four topics of language skills, three out of six topics of mathematical skills and all the remaining topics of learning skills. In line with a number of previous studies (Reid et al. 2014; Best et al. 2011; Vitiello and Williford 2021; McClelland, Acock, and Morrison 2006) and our H1, children’s outcomes in assessed developmental areas are positively related. Also, similar to previous research showing moderate relations between learning skills and other developmental areas (Yen, Konold, and McDermott 2004), correlations for learning skills were low and not statistically significant. This shows that five-year-old children’s interest and perceived ability in an area or topic are not associated with their actual skills indicating that children are happy to show an interest in topics they do not excel in. However, in the case of the teachers’ questionnaire, learning skills were moderately related to other topics, suggesting that teachers perceive higher-performing students to have a higher interest and perceived ability as well. This shows the need for an instrument that allows the children to express their level of interest, compared to teachers evaluating it indirectly. For the questionnaire, all the factors except vocabulary and calculation are significantly positively correlated, with the effect ranging from moderate (0.30 < ρ < 0.50) to large (ρ > 0.50).

Figure 3. Spearman correlations for children’s assessment topics (upper right part of matrix) and teachers’ questionnaire factors (lower left part of matrix). Note: COG = cognitive processes, TEXT = text comprehension, PHONO = emerging reading and writing (phonology), VOC = vocabulary, NUM = numbers and figures, CALC = calculation, COUNT = counting, LEARN = learning skills, *** p < 0.001, ** p < 0.01, * p < 0.05, p-values are adjusted with Holm’s method.

The Spearman correlations between factors identified in CFA and the sums of the children’s assessment items (topics) corresponding to the factors are shown in Table 1. As expected in H2, the results of the children’s assessment and teachers’ evaluations were moderately positively associated. The findings are similar to several previous studies (Reid et al. 2014; Cabell et al. 2009; Kowalski et al. 2018; Vitiello and Williford 2021). In line with H3, the relations were stronger in academic areas of language and mathematics and moderate in cognitive processes. This indicates that teachers can more accurately assess areas explicitly brought out in the preschool curriculum, making it more important to give them an accurate tool for assessing other skills.

Table 1. Descriptive statistics and Spearman correlations between children’s assessment topics and teachers’ questionnaire factors.

Topic/factor	M (SD) children	Min-Max children	M (SD) teachers	Min-Max teachers	ρ
Cognitive processes	89.42 (10.30)	55–106	10.38 (1.62)	4–12	0.352**
Text comprehension	20.14 (5.16)	6–26	6.47 (1.80)	3–9	0.550***
Emerging reading and writing	27.67 (7.23)	10–38	4.78 (1.10)	2–6	0.598***
Vocabulary	3.82 (0.52)	1–4	8.03 (1.07)	6–9	0.484***
Numbers and figures	27.58 (6.25)	6–32	6.82 (1.67)	4–9	0.772***
Calculation	11.19 (3.58)	3–17	3.56 (1.18)	2–6	0.491***
Counting	20.07 (4.19)	1–24	5.44 (0.76)	3–6	0.345*
Learning skills	32.08 (3.96)	16–36	4.95 (0.95)	3–6	0.012

Note: M = mean, SD = standard deviation, ρ = Spearman’s rho, *** p < 0.001, ** p < 0.01, * p < 0.05, p-values were adjusted with Holm’s method.

The correlations between children’s results and teachers’ evaluations were highest in topics such as numbers and figures, text comprehension, and emerging reading and writing, while the more difficult topics of counting, calculations, and vocabulary were less strongly correlated. This might indicate that teachers tend to under – or overestimate children’s skills (Begeny and Buchanan 2010) or are not confident in evaluating skills which exceed the expected level (Mashburn and Henry 2004). As previous research has shown that five-year-old children have high mathematical skills (OECD 2020), despite the tasks being more difficult than expected in the curriculum, it is important to provide teachers with an instrument that, in addition to assessing the skill level expected from this age group, can give information on topics exceeding it.

There were no significant correlations between children’s assessed learning skills and teachers’ evaluations of those skills, underlining the notion that teachers associate higher interest and perceived ability with children’s skills in other areas, which might not be the case and therefore, teachers are not able to evaluate children’s learning skills accurately. We should provide teachers with an instrument that gives them a more accurate overview of children’s opinions. Previous research has indicated that motivated, confident and attentive children tend to have higher academic results (Yen, Konold, and McDermott 2004). For this reason, it is very important for teachers to get information about this area to encourage children to continue learning in the areas that motivate them.

Several limitations should be considered while interpreting the results. First, the kindergartens participating in the study did it voluntarily with a first-come, first-served policy. As the study’s timeframe was limited, only 10 children from each kindergarten could participate. Despite the sample size being rather small, it can still be considered sufficient considering the time-consuming process of assessing each child. Second, there were some discrepancies between the children’s direct assessment results and teachers’ ratings, particularly in learning skills which might suggest that teachers were not able to accurately assess those skills. Previous studies have also shown that the accuracy of kindergarten teachers’ evaluations is moderate (Begeny and Buchanan 2010; Kowalski et al. 2018; Vitiello and Williford 2021), and teachers are not very accurate in identifying children at risk (Cabell et al. 2009). However, it could also indicate problems with questions. Thus, further studies are needed to assess children’s learning skills with different methods (observations, interviews) and validate the adequacy of the questions used in this instrument.

5. Conclusion

The results of the current study confirm the reliability and construct validity of the new e-assessment instrument for assessing young children’s skills in four areas. The present work contributes to the area of early learning studies, showing the relations between children’s developmental areas as well as offering a practical outcome of a validated assessment instrument. The research showed that e-assessment can be used for assessing young children’s skills as five-year-olds were successfully able to solve the items on a tablet and their results broadly aligned with teachers’ evaluations making way for similar assessments to be carried out in other countries. At the same time, the results showed that using common methods such as observations, teachers might not be able to accurately assess all skill areas, thus showing the need for varied methods and instruments to be available to kindergarten teachers.

The instrument provides information on emergent language and mathematical skills that teachers are more accustomed to evaluating, as well as on cognitive processes and learning skills that teachers might not feel as confident to assess or might assess inaccurately. Items for assessing social skills will be added to the instrument. The use of the e-assessment tool helps teachers assess the child’s development in certain areas, which also supports the implementation of the curriculum at the level of both the kindergarten group and the institution. The instrument is tablet-based, entirely computer-assessed, includes audio instructions and will be freely accessible to all teachers in Estonia. The instrument helps teachers get an overview of children’s development in areas and topics where direct e-assessment has advantages compared to other measures used.

As some items were further developed or added to the instrument, their properties should be analysed as well. In the next stage of the development, a study with a larger and more representative sample should be carried out. Also, parents should be involved in a study alongside teachers to get information about other aspects of children’s development and from all people involved in children’s lives. In the future, a study could be carried out to determine the consistency of the children’s direct assessment results over time.

Acknowledgements

We thank the kindergartens, teachers, study administrators, children and their families who participated in the study and as well other experts involved in developing the instrument, namely Tiiu Tammemäe, Maire Tuul, Astra Schults and Natalia Tšuikina.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The current research is based on the Estonian Education and Youth Board’s project of developing an e-assessment instrument for assessing five-year-old children’s skills. This article was supported by the Research Fund of the School of Educational Sciences at Tallinn University through project ‘Research group for studying the kindergarten learning environment and teachers’ teaching practices’ under grant [number 120_TA3221 ] and by the European Regional Development Fund through an ASTRA measure project ‘TLU TEE or Tallinn University as the promoter of intelligent lifestyle’ under grant [number 2014-2020.4.01.16-0033].

Appendix A

Table A1. Questionnaire items before analysis.

Item	Content	Scale	Intended factor
Q1	Knows how to group objects by shape, size or colour	1 = not at all, 2 = partially (depending on the situation), 3 = very well, 4 = don’t know	cognitive processes
Q2	Can follow complex instructions		cognitive processes
Q3	Can remember five displayed objects		cognitive processes
Q4	Can transition from one activity to another		cognitive processes
Q5	Can sequence activities from experience or by logic		cognitive processes
Q6	Knows time concepts (before, after, yesterday, today)	1 = not at all, 2 = partially (knows some), 3 = very well, 4 = don’t know	vocabulary
Q7	Knows position concepts (up, in the middle, down, next to)		vocabulary
Q8	Knows antonyms (big-small, high-low, tall-short)		vocabulary
Q9	Knows letters		text comprehension
Q10	Can write short words	1 = not at all, 2 = partially (can do some), 3 = very well, 4 = don’t know	text comprehension
Q11	Can find the first and last vowel (vocal or consonant)		reading and writing
Q12	Can locate the position of a vowel (e.g. whether in the word ‘kass’, ‘k’ is inside, in the beginning, or at the end of the word		reading and writing
Q13	Can estimate the length of a vowel in a word		reading and writing
Q14	Can read short words		text comprehension
Q15	Can use appropriate case endings		vocabulary
Q16	Can answer questions based on a short text		text comprehension
Q17	Knows numbers and respective notations of numbers 1–9	1 = not at all, 2 = partially (knows some), 3 = very well, 4 = don’t know	know math
Q18	Knows numbers from 10 to 100		know math
Q19	Knows geometric shapes (circle, square, rectangular, triangle, pentagon)		know math
Q20	Can count up to 10	1 = not at all, 2 = partially (less than within 10), 3 = very well, 4 = don’t know	counting
Q21	Can add and subtract numbers up to 10		calculation
Q22	Can count to 20	1 = not at all, 2 = partially (more than within 10, less than within 20), 3 = very well, 4 = don’t know	counting
Q23	Knows the order of numbers up to 20		know math
Q24	Can add and subtract numbers up to 20		calculation
Q25	Is interested in exercises related to mathematics	1 = not at all, 2 = partially (depending on the task), 3 = very well, 4 = don’t know	learning skills
Q26	Is interested in exercises related to letters, vowels, or words		learning skills
Q27	Believes that they can solve exercises related to mathematics		learning skills
Q28	Believes that they can solve exercises related to letters, vowels, or words		learning skills
Q29	Can self-assess how well they have managed to solve the exercises		learning skills

Table A2. Descriptive statistics and number of items for each topic after initial analysis.

Competences and topics	No. of items after analysis	M (SD)	Min	Max	α
Cognitive processes	32	91.29 (8.49)	67.5	106
Attention and perception	18	41.55 (4.75)	23	48.5	0.56 – 0.91
Thinking	7	18.41 (4.37)	4	25	0.64 – 0.88
Working memory	7	30.15 (4.36)	14	35	0.59 – 0.84
Language skills	42	60.50 (10.67)	25	74
Vocabulary	7	3.81 (0.53)	1	4	0.43 – 0.7
Emerging reading and writing	27	29.48 (6.49)	11	38	0.69 – 0.87
Text comprehension	6	21.67 (4.42)	9	26	0.51 – 0.56
Grammatical forms*	4	2.36 (1.19)	0	4	0.32–0.65
Mathematical skills	68	127.33 (12.10)	94.5	140.5
Calculations	11	11.65 (3.60)	3	17	0.63 – 0.8
Counting	13	20.56 (3.69)	3	24	0.64 – 0.92
Quantities*	12	5.49 (1.28)	1	7	0.49 – 0.52
Geometric shapes*	12	36.7 (2.77)	16.5	40	0.50 – 0.90
Numbers and figures	10	28.40 (5.59)	6	32	0.65 – 0.9
Sizes and measurements*	10	20.38 (4.00)	7	25	0.81- 0.88
Learning skills	12	32.19 (3.93)	16	36
Interest	6	16.26 (2.09)	8	18	0.64
Perceived ability	4	10.46 (1.73)	6	12	0.64
Valuation of solution	2	5.45 (0.86)	2	6	0.57
Rationale of solution**	0
Checking results**	0
Total	154

Note: No of items = number of items after initial analysis, M = mean, SD = standard deviation, α = range of values of Cronbach’s alpha or Kuder-Richardson’s formula 20 for constructs in the topic, ** Topics left out of the instrument and not included in further analysis, * topics not included in further analysis.

Table A3. Descriptive statistics for initial questionnaire items based on intended factors.

Factor	No. of items	M (SD)	Min	Max	α
Cognitive processes	5	12.91 (2.03)	5	15	0.82
Text comprehension	4	9.11 (2.03)	5	12	0.80
Emerging reading and writing	3	4.78 (1.10)	2	6	0.80
Vocabulary	4	10.60 (1.40)	8	12	0.73
Numbers and figures	3	6.82 (1.67)	4	9	0.80
Calculation	2	3.56 (1.18)	2	6	0.82
Counting	2	5.44 (0.76)	3	6	0.26
Learning skills	5	12.17 (2.23)	7	15	0.87

Note: No of items = number of items in the initial questionnaire, M = mean, SD = standard deviation, α = Cronbach’s alpha for the factor.