Measuring senior high school students’ self-induced self-reflective thinking

ABSTRACT

Theoretically, reflection is known to be an essential skill for improving learning on a metacognitive level. In practice, students may not use it of their own accord to improve this kind of learning because it can be mentally demanding. The author reports on the legitimation of an instrument measuring self-induced self-reflective thinking, which is reflection of one's own accord focused on improving general knowledge of the learning process. In 2 studies, the psychometric properties and nomological validity of open-ended self-induced self-reflective thinking questions were examined. Senior high school students responded to these questions and several measures of general knowledge of the learning process. Results showed statistically significant relationships between self-induced self-reflective thinking and general knowledge of the learning process. Implications for educational research are discussed.

KEYWORDS:

• Learning to learn
• metacognitive knowledge
• self-reflection
• upper level secondary education

Educational researchers of different theoretical perspectives have emphasized that students should become accomplished in using reflection with respect to their learning process (Baird, Fensham, Gunstone, & White, 1991; Boud, Keogh, & Walker, 1985; Bransford, Brown, & Cocking, 2000; Ertmer & Newby, 1996; Zimmerman, 1989). When students use reflection, they do so to improve how they learn by critically evaluating their learning experiences (Dewey, 1910; Mezirow, 1990). However, it is also generally known that it is difficult to measure students’ reflection due to two problems.

The first problem involves the data collection method. Often self-report rating-scale questions are used to measure the frequency with which students consider reflection to be useful, effective, important, and etcetera (e.g., Kember et al., 2000; Mair, 2012; Phan, 2009). An advantage of this data selection method is that quantitative data is obtained, which provides for consistency in interpretation and it enables inferential and advanced statistical analyses. However, a disadvantage of this method is that students have little opportunity to explain their selected response. Moreover, circling a rating-scale question can be done quickly and may provide for ill-considered responses. These problems regarding rating-scale questions can be overcome by using qualitative data-collection methods (e.g., Bell, Kelton, McDonagh, Mladenovic, & Morrison, 2011), such as open-ended questions, interviews, observations, and reflective writing tasks (e.g., journals, portfolios, notebooks, diaries, rubrics, and online measures). Although these qualitative data-collection methods can provide for insights on students’ understanding of reflection, a difficulty is the obtaining of consistent grading of the constructed responses and the difficulties students can encounter when they have to provide many constructed responses (Cronbach, 1951).

The second problem in the measurement of reflection is that students can differ with regard to their actual use of reflection. Generally, research on reflection showed that students find it difficult to make use of reflection (cf., Grossman, 2009; Rogers, 2001; Van Velzen, 2012). In this respect, Brown (1987) and Ertmer and Newby (1996) argued that reflection can be a mentally demanding skill that requires practicing. For example, White and Frederiksen (1998) found that seventh-grade students needed explicit instruction and practice of reflection. They also found that for the students to find reflection acceptable and enjoyable, to a point, they needed to understand that it was the performance and not the student that needed to be reconsidered. Other research pointed out that students can lack the motivation to engage in reflection (De Bruin, Van der Schaaf, Oosterbaan, & Prins, 2012).

Therefore, it can be expected that students may refrain from being engaged in reflection of their own accord to improve how they learn, because it can be mentally demanding. In addition, students who have been informed about reflection through instruction may be able to provide appropriate responses to questions inquiring about reflection, even though they do not actually make use of it of their own accord to improve their learning. For example, McNamara (2011) referred to this, in the context of metacognitive strategies, as to whether students tend to reflect as a naturally occurring strategy as distinct from being induced or prompted by others to use reflection.

In this study, these problems regarding the measurement of students’ understanding of reflection were addressed via two separate studies. The purpose of these two studies was to examine the legitimation of a relatively short measurement instrument that consisted of three open-ended questions. The focus was on students’ reflection of their own accord to improve general knowledge of the learning process. This was called self-induced self-reflective thinking (SISRT). I begin with an overview of the research literature on how reflection is measured to explain the characteristics of the SISRT questions. Next, I explain and define the characteristics of SISRT through an overview of the research literature.

Measuring reflection

In educational research, reflection is measured via different instruments such as questions and prompts (Bannert, 2006), journal writing (Boud, 2001), and computer-based tasks (Downing & Chim, 2004). Generally, reflection can be measured in two ways: as an awareness and attitude, by focusing on the relatively stable beliefs and perceptions of reflection, and as a skill, by focusing on students’ understanding of reflection. Instruments that measure reflection as an awareness and attitude usually consist of closed-ended self-report and rating-scale questions. An example is the Questionnaire for Reflective Thinking (Kember et al., 2000) that was designed to measure individual differences in perceived reflection. This instrument specifically differentiates between nonreflection, reflection, and critical reflection (Thorpe, 2004). That is, nonreflection involves habitual and thoughtful action, reflection involves the reconsideration of a situation to understand that situation, and critical reflection involves a reconsideration of the situation to obtain improvements by making a critical evaluation.

Particularly, the distinction between reflection and critical reflection was found by exploring undergraduate students’ journal writing (Kember et al., 1999). That is, a qualitative method was used to discover this distinction. Generally, qualitative methods are useful to explore to obtain an understanding of phenomena. Recently, this qualitatively developed coding scheme by Kember et al., (1999) was used to measure the levels of reflection in students’ written journals (Bell et al., 2011). In this case, reflection was measured as a skill in that the coding scheme provided insights into what students reflect upon.

However, a problem with qualitatively measuring students’ reflection via writing tasks is that the written texts do not have to include reflections. Reviews showed that reflective writing tasks often consist of descriptive accounts of learning events without mentioning reflections (Cowan, 2014; Dyment & O'Connell, 2011). For instance, portfolios that also included a justification of the selected student work consisted of a description of what has been learned instead of mentioning how learning can be improved (Calfee & Perfumo, 1996). Therefore, because students are often reluctant to reflect, it seems that prompting (Black & Wiliam, 1998; Cacciamani, Cesareni, Martini, Ferrini, & Fujita, 2012) and instructor feedback (Ash, Clayton, & Atkinson, 2005; Elshout-Mohr, 1994) have to be included regarding reflective writing tasks.

For example, students can be asked to write an essay for their portfolio by reflecting on their progress, areas of weakness, and plans for improvement (O'Sullivan et al., 2012). Other researchers also reported about the importance of prompting in online reflection tasks (Chen, Wei, Wu, & Uden, 2009). Particularly the type of online reflection tasks that aims to develop and demonstrate reflection on the process of learning, can support students in tracking their developmental changes over a period of time and encourage them to engage in self-directed learning (McDonald, 2012).

The point is that the use of questions in qualitative data-collection methods seems essential to ensure that students provide reflective responses. For example, McCrindle and Christensen (1995) found that university students who were keeping a learning journal on their learning process that included questions, performed significantly better on the final exam than the control-group students who did not keep such a learning journal. Furthermore, Berthold, Nückles, and Renkl (2007) found that students who were asked to reflect in learning journals improved their learning outcomes when they also received cognitive or a combination of cognitive and metacognitive prompts. Accordingly, it can also be important to quantitize the obtained qualitative data to examine its merits. For example, Lew and Schmidt (2011), using coding software to analyze the content of 690 university students’ reflections on their learning goals, found positively weak and nonsignificant correlations between reflection and knowledge acquisition.

In the studies in this article, three open-ended SISRT questions were used to enable the students to respond comprehensively to, what was assumed for them to be, difficult questions. In this way, qualitative data was obtained of which the content could be analyzed relatively quickly because there were only three questions.

Defining self-induced self-reflective thinking

So far, the word reflection was employed as this is commonly used to refer to the evaluation of experiences with the aim to obtain improvements. However, in educational research, reflection is defined in varied ways and used in different contexts (Denton, 2011; Moon, 1999). In most definitions of reflection, a reference is made to Dewey (1910), who described reflection as a systematic way of thinking through one's experiences to attain additional or new information. Mezirow (1990) elaborated this definition by distinguishing between reflection and critical reflection, of which the latter refers to a person focused on meaningful changes. Furthermore, Von Wright (1992) added the prefix self to reflection to emphasize that self-reflection involves seeing oneself as the active agent with different alternatives. In the studies in this article, self-reflection was defined as critically evaluating experiences to obtain improvements while taking into account oneself as the active agent with different alternative. In addition, this self-reflection was retrospective or reflection-on-action (Schön, 1983).

Research on reflection also includes different contexts (Baird, 1986; Grossman, 2009). For example, reflection has been studied regarding primary and secondary education (Brown, 1997), higher education (Rogers, 2001), and work-based education (Warhurst, 2008). In the studies in this article, the context consisted of senior high school students because the focus was on the improvement of metacognitive knowledge. This knowledge is essential for students to direct their cognitive processes, while relatively few educational research studies investigate it (Cotterall & Murray, 2009; Efklides & Misailidi, 2010). Generally, metacognition is divided into metacognitive knowledge and the executive processes. Metacognitive knowledge is known to include an awareness and understanding of one's cognitive processes (Brown, 1987; Flavell, 1979; Pintrich, 2002). To develop metacognitive knowledge, some researchers have focused specifically on declarative metacognitive knowledge or knowing what is needed to direct cognitive processes as distinct from knowing how to regulate cognitive processes (Brown, 1987; Kluwe, 1982; Kuhn, 2000).

In the context of learning, this declarative metacognitive knowledge refers to general knowledge of the learning process or knowing what is needed to learn effectively (Van Velzen, in press). However, self-reflective thinking and general knowledge of the learning process are barely studied concepts and little is known about their relationship. Presently, these are the concepts of a theoretical model. Therefore, theoretically it was expected that the kind of reflections needed to improve general knowledge of the learning process would be different from the kind of reflections that are focused on regulating cognitive processes. In the studies in this article, the term thinking was added to self-reflection to point out that self-reflective thinking aims to improve knowledge regarding the thinking through of what is needed to learn effectively. In other words, self-reflective thinking was defined as the retrospective reconsiderations of learning experiences to improve general knowledge of the learning process in that critical evaluations were made while taking into account oneself as an active agent with alternatives.

Furthermore, it was expected that students’ self-reflective thinking could occur either consciously or unconsciously. Conscious self-reflective thinking is intentional and can lead to an improved understanding of general knowledge of the learning process. However, this can be initiated either because others ask students to reflect or because students do so of their own accord. Therefore, to include the deliberate initiative of a student to use self-reflective thinking, in the studies in this article the term self-induced was added. The term self-induced with regard to reflection was also used by Dunlap and Grabinger (2003), who argued that people engage in lifelong learning do so of their own accord because they want to improve the quality of their lives. In the same vein, other researchers have pointed out that to prepare people for lifelong learning, educational opportunities to develop the self-induced use of metacognition and self-reflection are essential (cf., Wiersema & Licklider, 2007).

Although research studies exist on reflection to improve metacognitive knowledge in the context of learning (Brown, 1987; Cotterall & Murray, 2009; Pintrich, 2002; White & Frederiksen, 2005), no studies were found that focused on measuring students’ use of SISRT. Therefore, the following research questions were stated:

• Research Question 1. Do the SISRT questions provide for interpretability?

• Research Question 2. Do the SISRT questions provide for nomological validity?

General method

In this article, two separate studies are presented that investigated the legitimacy of the measurement of SISRT. These two studies were similar with respect to the target group of participants, the SISRT questions, the procedure in collecting the data, and the data analysis of the SISRT questions. This is described in more detail in this introductory paragraph.

Target group of participants

Because self-reflection is known to be an active and effortful enterprise, adolescent students can be expected to be better in reflecting on their learning processes than younger students (Brown, 1987; Ertmer & Newby, 1996). Senior high school students (i.e., Grade 9–12 university-preparatory education) were selected as the target group because these students were expected to have developed metacognitive knowledge (Schneider, 2008). In addition, these students are also still developing their metacognitive knowledge (Weil et al., 2013), which makes it worthwhile to measure their SISRT. Therefore, convenience sampling was used in that the senior high schools students who were available for the study were recruited as participants.

The participants all came from high schools in the surroundings of Amsterdam in the Netherlands. The invited participants all participated and were included in the data analyses because all had filled in, or partly filled in, the SISRT questions. The participants followed the schools’ obligatory study-skill courses regarding study techniques, planning, reflection, and school-life in general, which take place from Grades 7 to 12. In these obligatory study-skill courses the participants had been informed about what reflection is and for which purposes it can be used. Because all high school students in the Netherlands follow these obligatory study-skill courses, they can be expected to know what reflection means. However, the participants were specifically trained in neither SISRT nor in developing general knowledge of the learning process.

Data-collection procedure

The schools informed the parents about the research investigation that would take place and gave the parents the opportunity to exclude their child or children from participating. The data were collected in one session that provided the participants with enough time to respond to the questions. The data-collection session was kept during a lesson and was chaired by a familiar teacher or mentor. The teachers and mentors received strict instructions to inform the participants that their responses would be used only for the research purpose of obtaining an understanding about reflection. Next, the teachers and mentors distributed the questions. The questions were administered in a paper-and-pencil format in which the participants were specifically asked to respond to the questions as comprehensively as possible. The participants first answered the questions regarding general knowledge of the learning process and then the SISRT questions. Finally, the teachers were expressly instructed not to react to and answer the participants’ questions.

SISRT questions

Three open-ended SISRT questions were used to enable the participants to write down a response in their own words by stating what they knew about SISRT. The three questions followed the process of reflection (Boud et al., 1985; Moon, 1999), which begins with reconsidering one's understanding by obtaining an overview of the situation. Accordingly, the focus is on the essential features of the situation. Finally, critical evaluation of the situation is taking place to establish improvements.

In other words (see the Appendix), the first SISRT question referred to analyzing or the obtaining of a general understanding and interpretation of learning experiences. The second SISRT question referred to the evaluation of essential features or outcomes of learning experiences. Finally, the third SISRT question referred to the critical evaluation or the synthesizing of information from learning experiences. The questions were stated in this order to provide support to the participants in responding, because it was assumed that these questions would be difficult for them.

Data analysis regarding the SISRT questions

The responses of the participants were carefully interpreted in line with the operationalization of SISRT that was established in a previous and preliminary study (Van Velzen, 2015). That is, the operationalization of senior high school students’ SISRT provided for three hierarchical categories, of which the highest level (i.e., self-induced) was defined as indicating an understanding of the process of SISRT, which was differently interpreted per SISRT question (see Table 1). For this operationalization, intercoder reliability was established through another person who was informed (i.e., not trained) about the categories, which resulted in a reliability of κ = .74 that was considered good (Robson, 1993).

Table 1. Operationalization of the SISRT questions.

Question	Operationalization
SISRT1. When you retrospectively reconsider if you have understood the material, then you think about …	The response indicates what it is that someone will do to check if the material is understood, such as stating and asking oneself if one would be able to remember it, should reread it, should ask the teacher, could relate it to prior knowledge or experiences, could detect inaccuracies and could make tests and exams.
SISRT2. When you retrospectively reconsider why the material that was taught is relevant to you, then you think about …	The response indicates what the learning of the material has yielded in terms of having obtained an understanding or doing something that one did not understand or was not able to do before – indicating growth in learning and personal development.
SISRT3. When you retrospectively reconsider how you can improve the learning of the material, then you think about …	The response indicates (a) the knowing of that one can do better—also explaining why one can do it better, (b) the stating or questioning of what it is that one could have done better and (c) the knowing of reasons for applying a particular method of learning.

Note. SISRT1 = first self-induced self-reflective thinking question; SISRT2 = second self-induced self-reflective thinking question; SISRT3 = third self-induced self-reflective thinking question.

This operationalization included that relevant responses not referring to the process of self-reflective thinking were interpreted as providing general statements and examples. Contrarily, relevant responses including a reference to the process of self-reflective thinking were interpreted as providing an indication of self-induced, because the use of self-reflective thinking of one's accord will lead to an understanding of what the process of self-reflective thinking encompasses. To clarify this difference, examples of both kinds of responses are presented per SISRT question. Examples regarding the first SISRT question were the following: “That I will have to work on it” and “That I will check if I need more or additional information and explanations.” The latter response indicated the process of self-reflective thinking in that it included checking and this was done for a specific purpose, suggesting that one knows why and how to undertake the activity.

Examples regarding the second SISRT question were “That I ask myself if it useful to me” and “That I can see how it helps me to understand things that I encounter in my life.” Here the latter example indicated the process of self-reflective thinking in that it stated the actual usefulness of having this kind of reconsideration, suggesting a focus on growth in learning and personal development. Finally, examples regarding the third SISRT question were the following: “That I will do it differently the next time” and “That I will need to repeat subject matter to prevent that I make mistakes.” The latter example indicated the process of self-reflective thinking in that it stated the usefulness of a particular improvement, suggesting that one can be specific about why the application of a study technique could be effective.

In line with the operationalization obtained in the previous study, in the two studies presented in this article the responses of the participants to the SISRT questions were scored as (a) blank responses and responses unrelated to the question were scored as absent self-reflective thinking, (b) responses that were related to the question though that did not include a reference to the process of self-reflective thinking were scored as self-reflective thinking, and (c) responses that were related to the question and also included a reference to the process of self-reflective thinking were scored as SISRT. In the case that multiple responses were given per question, the highest score was noted down.

Next, to examine the reliability of the SISRT questions, Cronbach's alpha was calculated. Also, interitem correlations and principal component analysis were calculated to examine the interrelatedness of the questions (Cronbach & Shavelson, 2004). The principal component analysis were based on an extraction with eigenvalues, which was used to examine interrelationships among the items to identify if the items formed a principal (i.e., first) dimension (Hair, Black, Babin, & Anderson, 2010). More specifically, because the interpretability of a test depends on its homogeneity, the level of the factor loadings was examined (Cronbach, 1951).

Nomological validity was established because the relationship between SISRT and general knowledge of the learning process involves concepts of a theoretically based model. That is, theoretically general knowledge of the learning process is related to SISRT; however, evidence for this relationship is not yet available. Moreover, because the instruments that measure general knowledge of the learning process are under development, principal component analyses were calculated to establish suitable items. That is, to obtain a set of items that is reliable and not capable of division into discrete clusters (Cronbach, 1951); only the items that loaded statistically significant on the main component (Hair et al., 2010) were included in further analyses.

Nonparametric analyses were used because the data were of an ordinal level. Spearman rank-order correlation was calculated to examine the statistical significance of the obtained correlations because the participants were not specifically trained in SISRT and general knowledge of the learning process. Therefore, a relationship with SISRT was expected to be present in terms of whether or not this relationship would occur by chance.

Study 1

Study 1 considers the interpretability of the SISRT questions and the nomological validity with general knowledge of the learning process.

Method

Participants

The participants were 108 eleventh-grade students (66 women, 41 men, one unknown; M_age = 17 years). The participants came from six high schools that had reacted to a call for research participants that had been sent to 20 high schools. The data were collected in the beginning of the school year. Because relatively few men were included in Study 1, the SISRT data were examined for sex differences, showing no statistical significant differences (p = .94; Mann-Whitney U test at the specified .05 level).

Materials

To examine nomological validity, 14 open-ended questions were used that measured students’ understanding of general knowledge of developing cognitive knowledge (KDCK). The KDCK was recently developed (Van Velzen, in press) and provided ordinal data or three hierarchical scores (i.e., absent, tacit, and explicit KDCK) in line with the theory of Schraw and Moshman (1995). The development of the KDCK showed a reliability of α = .85 and preliminary construct validation (r_s = .24, p = .05). The KDCK included three kinds of items: declarative, procedural, and conditional general knowledge of the learning process in line with Jacobs and Paris (1987) and Schraw and Moshman (1995). Examples of the KDCK questions were regarding declarative “I know if I understand the essence of subject matter, because I focus on…” regarding procedural “I know if I can summarize subject matter, because I focus on…,” and regarding conditional “I know when a learning technique is needed, because I focus on…”

Results and discussion

The reliability of the SISRT questions showed α = .60, which was acceptable when taking into account the amount and type of the questions (Kehoe, 1995; Nunnally, 1978). Examination of the interrelationships between the SISRT questions through interitem correlations showed positive homogeneity (see Table 2). That is, although one correlation was below the required .30, the mean correlation was .32 (Robinson, Shaver, & Wrightsman, 1991). Homogeneity below .30 can in this case be the consequence of that each SISRT question referred to a different element (i.e., analyze, evaluate, and synthesize) of the self-reflective thinking process.

Table 2. Interitem correlation for the SISRT questions in the first study.

	Interitem correlation
Question	SISRT 1	SISRT 2	SISRT 3
SISRT1	1.00	.39	.36
SISRT2		1.00	.20
SISRT3			1.00

Note. SISRT1 = first self-induced self-reflective thinking question; SISRT2 = second self-induced self-reflective thinking question; SISRT3 = third self-induced self-reflective thinking question.

Also, principal component analysis showed one component and the factor loadings for the SISRT questions 1–3 were .82, .71, and .69, respectively. When taking into account the sample size, the level of these factor loadings were assumed statistically significant on a .05 level with a power level of 80% and standard errors assumed to be twice those of conventional correlation coefficients (Hair et al., 2010). Therefore, it was concluded that the interpretability of the SISRT was acceptable in that it was interpretable while all students were able to respond to the questions (Cronbach, 1951).

Descriptive statistics of the SISRT data showed an average mean score (M = 1.91, SD = 0.43), and the percentages were the following: absent self-reflective thinking = 15%, self-reflective thinking = 78%, and SISRT = 7%.

Regarding the KDCK, Table 3 shows the results of the principal component analysis based on an extraction with eigenvalues to obtain items with factor loadings on the first component that were > .55. That is, these factor loadings were assumed statistically significant on a .05 level with a power level of 80% and standard errors assumed twice those of conventional correlation coefficients when taking into account the sample size (Hair et al., 2010). Examination of the interpretability of the KDCK with 14 items showed a reliability of α = .88 (M = 1.94, SD = 0.41) and a mean interitem correlation of .36 (see Table 4). It was concluded that the KDCK was acceptable.

Table 3. Factor loadings for principal component analysis based on an extraction with eigenvalues for KDCK.

	Component
Item	1	2	3	4
P1	.68	.02	−.03	−.17
P6	.68	.01	−.39	−.18
C3	.67	−.39	.06	−.02
P3	.67	−.05	−.06	−.30
D4	.66	.25	−.12	−.07
P5	.65	.25	−.37	.06
D5	.62	.20	.30	.39
C4	.62	−.61	.18	−.11
C2	.61	−.31	.31	.29
P2	.61	−.04	.30	−.43
C6	.60	−.33	−.18	.22
C1	.60	−.42	.07	−.02
C5	.56	−.02	−.14	.34
D6	.55	.44	.21	.02

Note. Factor loadings > .55 for the first component are in boldface (Hair et al., 2010). C = conditional knowledge of developing cognitive knowledge; D = declarative knowledge of developing cognitive knowledge; KDCK =knowledge of developing cognitive knowledge; P = procedural knowledge of developing cognitive knowledge.

Table 4. Interitem correlation for the GKLP in the first study.

Item	D4	D5	D6	P1	P2	P3	P5	P6	C1	C2	C3	C4	C5	C6
D4	1.00	.30	.35	.44	.34	.41	.41	.44	.34	.32	.27	.25	.36	.34
D5		1.00	.48	.33	.29	.28	.41	.34	.33	.46	.34	.26	.30	.29
D6			1.00	.41	.28	.39	.43	.29	.15	.21	.30	.09	.23	.21
P1				1.00	.49	.40	.36	.44	.33	.32	.38	.38	.36	.42
P2					1.00	.41	.31	.37	.34	.34	.36	.45	.21	.24
P3						1.00	.45	.42	.39	.32	.49	.41	.26	.36
P5							1.00	.58	.26	.22	.36	.20	.35	.34
P6								1.00	.37	.20	.42	.33	.35	.33
C1									1.00	.42	.42	.63	.32	.30
C2										1.00	.51	.51	.32	.40
C3											1.00	.62	.31	.48
C4												1.00	.25	.48
C5													1.00	.47
C6														1.00

Note. C = conditional knowledge of developing cognitive knowledge; D = declarative knowledge of developing cognitive knowledge; GKLP = general knowledge of the learning process; P = procedural knowledge of developing cognitive knowledge.

The relationship between the KDCK and the SISRT was statistically significant (r_s = .33, p = .01), showing that the relationship was not a result of chance. This raised the question of whether the same results could be obtained with another measures of general knowledge of the learning process.

Study 2

Study 2 considers the interpretability of the SISRT questions and the nomological validity with two other instruments measuring general knowledge of the learning process.

Method

Participants

The participants were 125 ninth-grade students (81 women, 44 men; M_age = 15 years). The participants came from one high school that had reacted to a call for research participants that was sent to 6 high schools. The data were collected in June at the end of the school year. Examination for sex differences by using the SISRT data showed no statistical significant differences (p = .23; Mann-Whitney U test at the specified .05 level).

Materials

To examine nomological validity, the Metacognitive questionnaire for General Knowledge of the Learning process (MGKL) was used, to measure students’ understanding of general knowledge of the learning process, together with a rating-scale instrument that measured the frequency with which students used STudy Techniques questions (STT; Van Velzen, 2013). The MGKL had shown to be reliable (α = .57 for seven items), which is acceptable for a multiple-choice instrument (Kehoe, 1995). However, only four items provided the required > .55 factor loadings (i.e., principal component analysis based on an extraction with eigenvalues). Therefore, the STT was included (α = .58 for 6 items) due to its strong correlation with the MGKL (r_s = .79, p = .01). Both instruments have five-point scoring. An example of the MGKL was “I understand the essence of subject matter: (a) when I understand it; (b) when subject matter is repeated often; (c) when I can explain it to myself; (d) when the teacher says that it is important; and (e) when I learn it again.” An example of the SST questions was “How often do you make summaries?: (a) never; (b) sometimes; (c) neutral; (d) often; and (e) always.”

A secondary measure to examine nomological validity consisted of two questions developed to measure general knowledge of the learning process (GKLP; Van Velzen, 2013). The GKLP included two choices regarding unconsidered (i.e., habitual) versus considered learning. That is, the students had to choose between “I always learn in the same way” versus “Beforehand I consider the best way to learn” and between “Whenever my marks are low I reconsider how to improve my learning” versus “Whenever my marks are low I will learn for a longer period of time.” The GKLP was scored as (a) blank or no response provided, (b) low level or one blank and one habitual or unconsidered learning response, (c) medium level or one habitual or unconsidered and one considered learning response, and (d) high level or considered learning responses.

Results and discussion

The reliability of the SISRT questions showed α = .62, which is acceptable for this kind of question (Kehoe, 1995; Nunnally, 1978). The interitem correlations showed positive homogeneity (see Table 5) in that none was below the .30 (Robinson et al., 1991). Principal component analysis showed one component and the factor loadings for the SISRT questions 1–3 were .81, .74, and .73, respectively. These factor loadings were statistically significant on a .05 level (with a power level of 80% and standard errors, assuming twice those of conventional correlation coefficients when taking into account the sample size; Hair et al., 2010). Therefore, it was concluded that the interpretability of the SISRT questions was acceptable in that it was interpretable and all students were able to respond to the questions (Cronbach, 1951).

Table 5. Interitem correlation for the SISRT questions in the second study.

	Interitem correlation
Question	SISRT1	SISRT2	SISRT3
SISRT1	1.00	.40	.41
SISRT2		1.00	.30
SISRT3			1.00

Note. SISRT1 = first self-induced self-reflective thinking question; SISRT2 = second self-induced self-reflective thinking question; SISRT3 = third self-induced self-reflective thinking question.

Descriptive statistics of the SISRT data showed an average mean score (M = 1.97, SD = 0.37) and the percentages were 12% for absent self-reflective thinking, 83% for self-reflective thinking, and 5% for SISRT.

Regarding the MGKL/STT, Table 6 shows the results of the principal component analysis based on an extraction with eigenvalues to obtain items with factor loadings on the first component that were > .50. These factor loadings were statistically significant on a .05 level, with a power level of 80% and standard errors assuming twice those of conventional correlation coefficients when taking into account the sample size (Hair et al., 2010). The MGKL/STT provided 5 items with an acceptable reliability of α = .60 for an instrument that also includes multiple-choice questions (Kehoe, 1995). Although the mean interitem correlation of .22 was below the required .30 (see Table 7), which may have been the result of the mixing of two instruments, it was decided to use the MGKL/STT based on its factor loadings and reliability.

Table 6. Factor loadings for principal component analysis based on an extraction with eigenvalues for MGKL/STT.

	Component
Item	1	2	3	4	5
1	.67	−.25	−.22	−.30	−.07
2	.58	.08	−.23	−.39	.09
3	.55	−.36	−.01	−.04	.48
4	.54	−.17	.07	.33	−.12
5	.50	.39	.08	.41	.11

Note. Factor loadings > .50 for the first component are in boldface (Hair et al., 2010). MGKL = Metacognitive questionnaire for General Knowledge of the Learning process; STT = study techniques.

Table 7. Interitem correlation for the MGKL/STT questions in the second study.

Item	1	2	3	4	5
1	1.00	.25	.14	.41	.16
2		1.00	.23	.31	.15
3			1.00	.27	.19
4				1.00	.11
5					1.00

Note. MGKL = Metacognitive questionnaire for General Knowledge of the Learning process; STT = study techniques.

Descriptive results for the MGKL/STT showed M = 2.82 (SD = 0.80) and the percentages of the responses were 4% for wrong or never, 36% for weak or sometimes, 39% for moderate or neutral, 21% for good or often, and 0% for explanatory or always. The percentages for the GKLP were 9% blank and 21% low-level, 37% medium-level, and 33% high-level responses.

Table 8 shows that the examination of nomological validity provided for statistically significant results for MGKL/STT (p = .01) and GKLP (p = .05), though the positive correlations were small (r_s = .23 and r_s = .21, respectively). When taking into account the results of the interpretability of the validation instruments, the statistical significant results suggest that the SISRT questions were validated. However, the small correlations indicated that more validation instruments are needed to include other contributing variables (Campbell, 1988).

Table 8. Psychometric properties and nomological validity for the major study variables in the second study.

Variable	M	SD	α	rs
SISRT	1.97	0.37	.62	—
MGKL/STT	2.82	0.80	.60	.23^**
GKLP	3.38	1.36		.21^*

Note. GKLP = general knowledge of the learning process questions; MGKL = Metacognitive questionnaire for General Knowledge of the Learning process; SISRT = self-induced self-reflective thinking questions.

* p = .05;

** p = .01.

Summary and concluding discussion

The results regarding the measurement of SISRT that were reported in this article seem encouraging in that the interpretability of the SISRT questions appeared to be acceptable when the uniqueness and complexity of the questions were taken into account. In addition, the lengthening of the SISRT questions by developing more items may increase reliability though it may also have the disadvantage that when students have to respond to more questions, interpretability can decrease (Cronbach, 1951).

Furthermore, a limitation of the studies in this article was that the results regarding validation seem incomplete. That is, nomological validity was partly established in that statistically significant relationships between SISRT and the measures of general knowledge of the learning process were obtained; however, the correlations were positive but small. This suggests that more variables are involved in the theoretically based model (Robinson et al., 1991).

When taking into account these limitations of the studies in this article, the results suggested that a relatively low percentage of the participants appeared to be self-induced self-reflective thinkers. That is, 7% of the responses in Study 1, and 5% in Study 2, were scored as SISRT. This result further emphasizes the importance of measuring senior high school students’ SISRT accurately in educational research.

This raises the question of which further research studies could be conducted when using the SISRT questions to obtain a better understanding of students’ SISRT. First, it seems likely that senior high school students can provide information about the reasons for being or not being engaged in SISRT. In this respect, research on reflection indicated that students often encounter barriers toward reflection in higher education (Davis, 2003) and professional education (Gunn, 2010; Holloway & Gouthro, 2011). Therefore, in future studies, senior high school students could be asked to provide further insights about the factors that hinder and encourage them to use SISRT.

Second, senior high school students may also provide information regarding the influence of the teaching of reflection. That is, the low percentage of SISRT could be related to the teaching of reflection. Possibly, teaching may not always support senior high school students to such a degree that they become self-induced in their self-reflective thinking. For example, to become self-induced in self-reflective thinking senior high school students are likely to need both structured and unstructured self-reflective thinking tasks, though the latter kind task is not often included in schools (cf., Dean, Sykes, Agostinho, & Clements, 2012). Furthermore, it also seems likely that senior high school students need to come to understand the value of putting effort in using SISRT through teaching. Finally, it is also well known that self-reflective thinking is a cognitively demanding activity for senior high school students and, it may very well be that the mental efforts required in performing self-reflective thinking may refrain them from becoming self-induced in self-reflective thinking. That is, senior high school students may be taught about cognitive overload reduction techniques (Van Merriënboer & Sweller, 2005) with regard to self-reflective thinking. Therefore, in future studies, the influence of the teaching of self-reflective thinking could be taken into account when studying senior high school students’ SISRT.

Third, senior high school students’ abilities to analyze, evaluate, and synthesize their learning experiences might need to be taken into account in future studies. For example, it is known that students who reflect often appear to report about this in the form of descriptions instead of evaluations (Dyment & O'Connell, 2011). Research studies on this topic often mention the training of self-assessment (cf., Ash et al., 2005) and critical evaluation (Val Velzen, 2016) to support the learning of self-reflection and self-reflective thinking, respectively.

In conclusion, the results of the studies presented in this article suggest that the SISRT questions can measure senior high school students’ SISRT. However, a nomological network needs to be developed that can further explain the theoretical relationships between SISRT and other constructs.

Appendix: The self-induced self-reflective thinking questions

1. When you retrospectively reconsider if you have understood the material, then you think about …

2. When you retrospectively reconsider why the material that was taught is relevant to you, then you think about …

3. When you retrospectively reconsider how you can improve the learning of the material, then you think about …