Perceived relevance of university physics problems by pre-service physics teachers: personal constructs

ABSTRACT

Pre-service physics teachers often do not recognise the relevance for their future career in their university content knowledge courses. A lower perceived relevance can, however, have a negative effect on their motivation and on their academic success. Several intervention studies have been undertaken with the goal to increase this perceived relevance. A previous study shows that conceptual physics problems used in university physics courses are perceived by pre-service physics teachers as more relevant for their future career than regular, quantitative problems. It is however not clear, what the students’ meaning of the construct ‘relevance’ is: what makes a problem more relevant to them than another problem? To answer this question, N = 7 pre-service teachers were interviewed using the repertory grid technique, based on the personal construct theory. Nine physics problems were discussed with regards to their perceived relevance and with regards to problem properties that distinguish these problems from each other. We are able to identify six problem properties that have a positive influence on the perceived relevance. Physics problems that are based on these properties should therefore potentially have a higher perceived relevance, which can have a positive effect on the motivation of the pre-service teachers who solve these problems.

KEYWORDS:

• Motivation
• physics education
• pre-service teachers
• repertory grid

Introduction

Perceived relevance and lack of motivation

In German universities, physics major students and pre-service secondary school physics teachers generally attend the Experimental Physics courses – which make up a large part of the first semesters of their bachelor programme – together. These courses are however mostly aimed at the physics major students. Many pre-service physics teachers do not perceive these courses as relevant for their future career (AG Studienqualität, 2011; Merzyn, 2004). This problem extends beyond Germany and beyond the subject of physics: Koponen, Asikainen, Viholainen, and Hirvonen (2016) report the same issues in mathematics courses at Finnish universities.

Relevance can be defined as a ‘personally meaningful connection to the individual’ (Priniski, Hecht, & Harackiewicz, 2018, p. 12). Priniski et al. (2018) conceptualise relevance along a continuum of personal meaningfulness, starting at a personal association level and reaching to identification. Their relevance framework is based on the four-phase model of interest development – where relevance serves as a trigger for situational interest (Hidi & Renninger, 2006) – the expectancy-value model of achievement-related choices (Eccles et al., 1983) and the self-determination theory by Deci and Ryan (1985) – where relevance is seen as part of the internalisation process. Many studies show that a lower perceived relevance can have a negative effect on academic achievement and on the retention of knowledge (Harris, Heneghan, & McKay, 2003; Malau-Aduli et al., 2013; Petty & Cacioppo, 1981, 1986). It has also been connected to a lower learning motivation (Frymier & Shulman, 1995; Kember, Ho, & Hong, 2008; Sass, 1989; Weaver & Cottrell, 1988). A result of a low perceived relevance of university courses by students can be, next to the possible results described above, the discontinuation of their study (Albrecht, 2011; Heublein et al., 2017).

Improving perceived relevance

It is therefore important for the content in a course to be perceived as relevant by the students, since perceived relevance is likely to have a positive effect on educational outcomes like an increase in subjective task value, interest development and autonomous motivation (Priniski et al., 2018). What is not directly clear is how to improve the perceived relevance of course content. Frymier and Houser (1998) studied the interaction between immediacy and relevance in a 2 × 2 experimental design (high/low immediacy and high/low relevance) since they found that immediacy and relevance both were linked to a higher motivation, but were also associated with one another (Frymier & Shulman, 1995). They find that only immediacy has an impact on motivation and learning; relevance does not. This could be explained by not having sufficiently changed the relevance in the experimental design, but also by the fact that the relevance was determined by coders and not by the students. They suggest further research into factors that enhance relevance.

Personalising the learning experience, as described by Walkington and Bernacki (2014), is one of these possible factors. They theorise that a personalised learning experience can lead to students discovering more personally meaningful types of relevance. Another possible factor that enhances relevance is making a link between the course content and applications in the later profession of students. Owen (2017) linked the course content in a research methods course to the personal or professional goals of students. The content was perceived as more relevant when it was explicitly linked to an application in their possible later profession. Similar studies have been conducted for the specific group of pre-service teachers. In these studies, the course content was related to the content knowledge category that described the teacher-specific knowledge (Dreher, Lindmeier, Heinze, & Niemand, 2018; Riese et al., 2015; Woehlecke et al., 2017). Stäcker, Ropohl, Steffensky, and Friedrichs (2018) developed learning opportunities for pre-service chemistry teachers where, among others, a content analysis of an everyday phenomenon is reduced for teaching purposes using knowledge maps. They find positive effects on the perceived relevance for some of the themes of the course. Massolt and Borowski (2018) used physics problems that were based on the model of school-related content knowledge (SRCK, Woehlecke et al., 2017) to increase the perceived relevance of problem sets for experimental physics tutorial groups by pre-service physics teachers. The SRCK model describes the content knowledge that is specific for teachers. The problems that are based on the facets described in this model are supposed to have a higher perceived relevance by pre-service physics teachers because they make a connection to their future career. They find that conceptual problems based on this model do have a higher perceived relevance by pre-service physics teachers than regular, quantitative problems. However, conceptual problems that are not based on school-related content knowledge are also found to have a higher perceived relevance than the regular problems. This suggests that there are more influences on the perceived relevance of these problems than just the problem properties derived from the facets of SRCK.

Relevance from the perspective of the receiver

The question that remains is how students determine how relevant for their future career the content is to them. An answer to this question would provide us with indications on what exactly influences the perceived relevance, which can tell us how to adapt learning material in order to increase the perceived relevance of this material by students.

Many strategies that enhance the perceived relevance are theoretically determined (Keller, 1983; Sass, 1989; Weaver & Cottrell, 1988). In his ARCS model for motivational design (Attention, Relevance, Confidence, Satisfaction), Keller (2010) for instance mentions relating to goals, matching interests and tying to experiences as strategies to enhance relevance – strategies that all increase the personal meaningfulness (Priniski et al., 2018). Even though relevance can be seen as a ‘personally meaningful connection’ (Priniski et al., 2018, p. 12), these strategies are mostly developed from the perspective of the instructor (Muddiman & Frymier, 2009). According to Frymier and Houser, relevance:

is a receiver based construct that varies from receiver to receiver, and it only exists in the mind of the receiver (1998, p. 127)

It is therefore also important to research these receiver-based constructs. To focus more on the student perspective, Muddiman and Frymier (2009) asked students what teaching behaviour and strategies teachers use to increase the perceived relevance. Most of the categories that were identified are in line with the strategies described in the model by Keller (1983). What is still unclear is whether the teacher behaviours that are listed by students make the content more relevant, or that the perceived relevance of the content is a result of being more motivated and engaged, also while some of the categories suggest that perceived relevance is seen as an outcome rather than a component of effective teaching (Muddiman & Frymier, 2009). Kember et al. (2008) asked students what typical approaches to teaching in their programme were, and what their effect upon their motivation was. Textbooks were for instance seen as more relevant when they showed that theory also applied to local issues and not just to issues that did not play a role locally. Relevance to everyday applications and current topics were also mentioned. Although the textbook example relates to teaching material, the other examples can also relate to teaching behaviours, which still leaves us with the question of whether perceived relevance is an outcome, or rather a component, of effective teaching.

Personal construct theory

It is important to distinguish between explicit and implicit knowledge when asking students for their perspective on what influences relevance. In their study, Muddiman and Frymier (2009) directly asked the students to describe behaviour that has a positive influence on relevance. Kember et al. (2008) also used direct questions in his research on teaching approaches that influenced the motivation of students. The categories that were determined therefore only represent the students’ explicit constructs.

Implicit constructs are determined by the tacit knowledge of students, which can be understood as a ‘gut feeling’ or intuition (Haldin-Herrgard, 2004). An appropriate tool to make tacit knowledge explicit is the use of the personal construct theory developed by Kelly (1955), which states that the way how future events are anticipated is guided by psychological processes. With this theory, he tried to explain why different people have different attitudes and views towards events in their world. One of his claims is that people use personal criteria – or constructs – to structure a meaningful world. Based on this theory, Kelly (1955) developed the role construct repertory test, now known as the repertory grid technique. This interview technique makes it possible to elicit personal constructs and thereby capture the subjective views of students in a structured way; it offers the possibility to describe their implicit knowledge (Fromm, 1995; Hillier, 1998; Jankowicz, 2004).

Research questions

Massolt and Borowski (2018) find that students perceive conceptual physics problems as more relevant for their future career than calculational problems. The problems that they use in this study have however many additional properties in addition to them being conceptual or calculational. In this study, we want to find out which of these properties have a positive influence on the perceived relevance by pre-service physics teachers. The results will help us to create problems that are perceived as more relevant by students, but it will also give us more insight into the personal constructs that students use to determine how relevant a problem is to them. Our research question is therefore:

Which problem properties have a positive influence on the perceived relevance of physics problems by pre-service physics teachers?

Design and methods

Research design and sample

In order to answer the research question, we have conducted interviews with N = 7 (6 male, 1 female) pre-service physics teachers at the University of Potsdam in the winter semester of the 2017/2018 academic year. Two participants were second-semester students, five participants were in the fourth semester of their study. All the participants received a small financial compensation.

In these interviews, nine problems that came from problem sets used in the first-semester course Experimental Physics 1 from that same year were discussed. Three of these problems were quantitative problems without explicit school relevance, three were conceptual problems that are based on the SRCK model and three problems were conceptual problems without explicit school relevance. The problems can be found in Appendix 1; for a detailed description of the problem types, see Massolt and Borowski (2018). The problems were randomly selected within their problem type from the full set of problems used in the course. The second-semester students were familiar with these problems since they attended Experimental Physics 1 in that year. Since some of the problems were unknown to the fourth-semester students, they were asked to prepare the interview by solving the problems. The interviews were recorded and later transcribed.

Interview technique

Every problem was printed on a card and was randomly assigned a number (1–9). Each interview was structured according to the following scheme (see also Figure 1), derived from Fromm (1995) and Jankowicz (2004):

1. The students were given the full set of nine questions and were asked to rate the problems with regards to their perceived relevance with the question ‘to what extent do the problems prepare you for your future career?’ (Massolt & Borowski, 2018). A scale from 1 to 6 is hereby used, where 1 refers to ‘relevant’ and 6 refers to ‘not relevant’.

2. In the second step, the students were given a set of three problems (for instance, problem numbers 1, 4 and 7). The students were asked ‘Which two of these problems are the same in some important way, and different from the third?’ They then had to explain what the two problems had in common, as opposed to the third. A possible answer could be, that two of the problems were ‘easy’ and the other ‘difficult’. The goal is to obtain a bipolar expression: here ‘easy’ is one of the poles of the expression and ‘difficult’ the contrast of this pole. This bipolar expression is the person’s construct he or she used to describe the difference between the problems. If the student could not find a way to describe the differences between the problems or could not find any differences, a new set of three problems was presented.

3. The newly defined construct was then presented as a rating scale, where 1 represents one pole of the construct (‘easy’), and 6 the contrast to this pole (‘difficult’). The students were then asked to rate all the nine problems on this new rating scale.

4. Steps 2 and 3 were then repeated with a new set of three problems, until no new constructs could be elicited or until the interview lasted more than 90 min.

Figure 1. Flow chart of the interviews using the repertory grid technique. The numbered cards represent the problems used in the interview.

The same set of nine problems was used in every interview. The numbering of the problems was also identical for all the interviews and the same pattern for presenting the set of three problems was used.

Analysis

Although the repertory grid technique is a way to find the personal constructs of participants, it is possible to get shared group constructs by combining the repertory grids that were obtained in single interviews (Honey, 1979; Jankowicz, 2004; Rojon, McDowall, & Saunders, 2018). In order to do this, the constructs have to be grouped together into inductively generated construct-categories. Next to that, a similarity score (Honey, 1979) has to be calculated for every construct in order to find out how much every construct is connected to relevance.

Similarity score

The similarity score reflects the extent to which the ratings on a construct matches the ratings on an overall construct (Honey, 1979; Jankowicz, 2004). In our study, this means that we are interested in the extent to which the ratings on the elicited constructs scales match the ratings on the relevance scale. For this reason, the following procedure was used to find the similarity score for each construct:

1. Calculate the sum of differences for each construct. This is defined as the sum of the absolute differences between the relevance rating for a problem and the rating on the construct’s scale for that problem over all the problems, see Equation (1). A lower sum of differences means that the construct is more similar to our overall construct, relevance.(1) $\mathrm{s}\mathrm{u}\mathrm{m}\mathrm{o}\mathrm{f}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{s}={\sum }_{i=1}^N\left|{\left(\mathrm{R}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{v}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{g}\right)}_i-\left(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{c}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{n}{\mathrm{g}}_i\right)\right|$(1)

2. It is up to the participant to define the description of the pole and the contrast to the pole in a construct. This means that one student could come up with a construct where 1 means ‘easy’ and 6 means ‘difficult’, whereas another student could reverse both meanings, so 1 means ‘difficult’ and 6 means ‘easy’. Both constructs have the same meaning, but a calculation of the sum of differences would lead to different values. A way to correct for this is to also calculate the sum of differences for a situation where one of the constructs is reversed (in our example: 1 – easy and 6 – difficult versus 1 – difficult and 6 – easy). Out of practical reasons, this calculation is done instead with the relevance rating reversed, which gives the same result.

3. The similarity score is then defined as the absolute difference between the sums of differences calculated in step 1 and 2 (Rojon et al., 2018). A higher similarity score refers to the construct having a higher similarity to relevance.

For each participant, the constructs are then labelled with three indices H, I and L according to their similarity score. Constructs with similarity scores in the highest tercile for that participant are labelled ‘H’, in the lowest tercile with ‘L’ and in the intermediate tercile with ‘I’.

When the sum of differences for the reversed construct (step 2) is smaller than that for the regular construct (step 1), then the construct’s poles need to be reversed. So, if the reversed ‘1-difficult and 6-easy’ leads to a lower sum of differences than ‘1 – easy and 6 – difficult’, the poles that are used in the further analysis of this construct become ‘1 – difficult and 6 – easy’, because in the remainder of this work we focus on the positive effect of a construct on perceived relevance.

Combining grids – construct categories

All the constructs, independent on their similarity score and their tercile, are subsequently sorted into inductively generated construct-categories. Non-categorisable constructs go into ‘miscellaneous’. These construct-categories describe the constructs within them and are also bipolar: they describe a pole and a contrast to this pole. The generation of these construct-categories has been done by two experts, independent of each other. Agreement on categories is then reached in discussion.

With the agreed construct-categories, the constructs are assigned to the construct-categories by two raters. Depending on the agreement between both raters, another adaption of the construct-categories can be carried out after which the agreement is calculated again.

Finally, the constructs that form the construct-categories are examined. Construct-categories that mainly have high- and intermediate level constructs (the H and I terciles) are retained whereas construct-categories with mainly low- and intermediate level constructs are discarded. The construct-categories that are retained then describe the personal constructs with a high to intermediate similarity to relevance; they give us information about the problem properties that are correlated with the perceived relevance of physics problems.

Results

Implicit constructs

An example of the results of a repertory grid interview can be found in Figure 2. The seven students generated a total of 56 constructs in the interviews (M = 8.0; SD = 1.4). The constructs with their sum of differences score, the reversed sum of differences score, the similarity score and their tercile are listed in the table in Appendix 2.

Figure 2. Repertory grid of participant #4. On the left the poles of every construct (1 on the scale), on the right the contrasts to these poles (6 on the scale). The indexes on the top indicate the problem number. The figure was made using the OpenRepGrid R package (Heckmann, 2016).

Construct categories

The constructs were, not depending on their similarity score, independently sorted into inductively created construct-categories by two experts. A percentage agreement of 59% was reached between the two construct-categories systems. After discussion, a full agreement was reached. A total of 15 construct-categories were identified, including a ‘miscellaneous’ category. One construct-category with only one construct was created because of a theoretical interest into this construct-category; the other construct-categories had at least two constructs.

Another rater repeated the assignment of constructs to construct-categories. After an iterative and discursive process, a percentage agreement for the 56 constructs of 84% between both raters was reached. The Cohen’s Kappa coefficient was calculated as $\kappa =0.83$, which is considered ‘almost perfect’ (Landis & Koch, 1977).

The construct-categories shown in Table 1 have constructs with predominantly high- and medium-tercile constructs in them. These constructs have a high similarity to relevance and are retained for further analysis. A table with all construct-categories, their constructs and their respective terciles can be found in Appendix 3.

Table 1. Construct-categories with predominantly high- and intermediate tercile constructs with their constructs and their respective terciles.

Construct-category	Constructs (number)	Tercile
Conceptual vs calculational	Thinking problem/calculational problem (1)	H
	Comprehension problem/calculational problem (50)	H
	Explanational problem/applied mathematics (22)	H
	Thinking/plug in (31)	H
	Explain/calculate (33)	I
	Calculational problem/thinking problem (40)	I
	Calculational problem/conceptual question (10)	L
Close to everyday life (on continuum scientific/technological/personal)	Possible to experience problem personally/phenomenon rather in technical area (52)	H
	Close to reality/theoretical (25)	H
	Reference to real life/no reference to real life (46)	H
	Everyday/not everyday (12)	I
	Close to everyday life/distant to everyday life (3)	L
Mathematical requirements	Rather scalar/rather vectoral (15)	H
	Lower mathematical effort/higher mathematical effort (17)	H
	Vectoral approach not necessary/vectoral approach necessary (27)	I
Enjoyment (perceived personal enjoyment)	Fun/no fun (7)	H
	Fun solving/no fun solving (45)	H
	Enjoyed solving/did not enjoy solving (28)	I
Curricular order – school-relevant content	School relevant/not school relevant (related to content) (6)	H
	More a school problem/more a university problem (47)	H
Perceived learning gain	Leads to deeper understanding/merely executing (29)	H
	Increase in knowledge/no increase in knowledge (39)	H
Analytical	Analytical/not analytical (a given situation has to be analysed) (23)	H

Notes: The constructs used in this table have, when necessary, first been reversed. The number behind the constructs refers to the construct number in Appendix 2.

Discussion

For categorising the 56 constructs, a total of 16 construct-categories were needed. This shows that these constructs clearly vary between students, they are really considered personal constructs. Not only the personal constructs vary between students, but also their relation to perceived relevance. One student, for instance, sees a calculational problem as less relevant (participant 1, construct 1), but for another student, a calculational problem has a higher perceived relevance (participant 2, construct 10). Both results can be seen as evidence for the theory that relevance is a receiver-based construct (Frymier & Houser, 1998). The definition of relevance by Priniski et al. (2018), where relevance is a ‘personally meaningful connection to the individual’, also fits these results: relevance is a personal connection, it differs from person to person.

Construct-categories that influence relevance

Although relevance can be seen as a personal connection and varies from person to person, we managed to find categories that summarise these personal connections. The seven construct-categories in Table 1 summarise the personal constructs that are contained in each construct-category. An important step in analysing these construct-categories is determining their influence on the perceived relevance, especially in what direction they correlate. This is done by looking at the description of both poles in each personal construct.

Conceptual vs. calculational: Five out of seven constructs show that a problem that is rather conceptual leads to a higher perceived relevance. There are two remaining constructs (40 and 10) that work in the other direction: a calculational problem here has a higher relevance than a conceptual problem. Since these two constructs however only belong to an intermediate and low tercile, their influence on the construct-category as a whole can be neglected: these students do not see a strong relation with perceived relevance. Generally speaking, a problem that is more conceptual than calculational therefore has a higher perceived relevance. This is in line with the results from Massolt and Borowski (2018): they found that conceptual problems are considered more relevant than calculational problems.

Close to everyday life (on continuum scientific/technological/personal): All the constructs in this construct-category work in the same direction, which here means that a problem that is closer to everyday life is considered more relevant. This can also be found in the ARCS model for motivational design (Keller, 2010). ‘Connection to previous experience’ is seen here as a strategy for establishing relevance. ‘Relevance to everyday applications and current topics’ was also mentioned in the interviews by Kember et al. (2008) as having a positive influence on relevance.

Mathematical requirements: A vectoral approach to a problem can be considered a higher mathematical requirement than a mere scalar approach: both direction and quantity have to be considered. We can therefore conclude that all constructs point in the same direction: a problem with a lower mathematical requirement is considered more relevant. From the perspective of pre-service physics teachers, this can be understood in such a way that they feel that the higher mathematical requirements do not help to prepare them for their future career: as a future secondary school teacher, they do not use physics problems with very high mathematical requirements.

Enjoyment (perceived personal enjoyment): When the students enjoy solving the problem, they perceive the problem as more relevant. However, there are two reasons for not further considering this construct-category: it is unclear, whether more enjoyment leads to a more relevant problem or the other way around: a more relevant problem might lead to more enjoyment. The other reason for discarding this construct-category is that we are interested in possible problem properties that have an influence on the perceived relevance in order to modify physics problems. Perceived personal enjoyment cannot be seen as a property of a problem that can be tweaked easily, it is rather the result of other problem properties.

Curricular order – school-relevant content: When the content of a problem is more school relevant, the problem is perceived to be more relevant. Both constructs clearly work in this direction. It is a result that can of course be expected: school-relevant content has high relevance for the future career of pre-service physics teachers. Previous research has also shown that pre-service physics teachers wish for more school-relevant content in their studies to make it more relevant to them (AG Studienqualität, 2011; Riese, 2009).

Perceived learning gain: If students learn more when solving a physics problem, they consider the problem more relevant for their future career. Such a problem is, with regards to perceived relevance, preferred over a problem where a mere executing (‘plug and chug’) is sufficient for solving a problem. It is important to point out, that this is a perceived learning gain: the students have the idea that they learn more from a particular problem. There is however only a moderate relation between self-assessment of knowledge and cognitive learning, as pointed out by Sitzmann, Ely, Brown, and Bauer (2010). Self-assessments of knowledge are strongly related to affective evaluation outcomes (Sitzmann et al., 2010), which leads to the question whether the perceived learning gain is an outcome of the perceived relevance or the other way around.

Analytical: Even though it contains only one construct, this construct-category was created because of a theoretical interest in the construct; an analysis of a given situation (for instance with regards to the nature of misconceptions or approximations made by students) is a typical problem based on the facets of SRCK (Woehlecke et al., 2017) used in the study by Massolt and Borowski (2018). Such a problem, an analytical problem, is not typically used in physics problem sets. Research suggests that worked example problems can be beneficial for novice problem solvers (e.g. Sweller & Cooper, 1985; van Gog, Kester, & Paas, 2011). A variation on these problems, where students have to analyse an incorrect worked example, might have the same benefits. It is therefore also interesting to notice that these problems are maybe not just beneficial to, but that they are also seen as more relevant by pre-service physics students.

Out of the seven construct-categories that are described above, only the last one can be connected to the problems based on the facets of SRCK (Massolt & Borowski, 2018), the other ones are not described there. This is also what was expected since Massolt and Borowski (2018) found no statistically significant differences between the perceived relevance of the conceptual problems based on SRCK and the conceptual problems that are not based on the facets of SRCK. This suggests that there are more problem properties that influence perceived relevance than just the ones described in SRCK. At the same time, some of the problem descriptions based on the facets of SRCK do not show up in the personal constructs mentioned in this study. This can be explained by the fact that only three randomly selected problems based on SRCK were used in this study: not all the problem descriptions were covered in this set of problems.

What is interesting to note is that difficulty is not seen as a construct-category that correlates with perceived relevance: as can be seen in Appendix 3, ‘Difficulty’ shows up in one high, one low and three intermediate terciles and is therefore not seen as having an effect on perceived relevance. Massolt and Borowski (2018) show that conceptual problems have a higher perceived relevance by pre-service physics teachers than calculational problems. At the same time, these problems are also often considered easier. One would, therefore, expect the difficulty to have an influence on perceived relevance. The simple relation ‘easier means more relevant’ is therefore not valid.

Limitations

Repertory grid interviews are very time-intensive. Because of this, the number of problems that can be discussed is limited. As said before, this can explain the fact that not all the descriptions of the problems used by Massolt and Borowski (2018) show up in the results. Using more problems in the interviews would therefore probably lead to more construct-categories. A different set of randomly selected problems could also lead to different construct-categories. Potentially the list of construct-categories and therefore the list of problem properties that have an influence on the perceived relevance of physics problems can be expanded.

Some of the construct-categories that were created only contain a few constructs. Because relevance is personal, but also because the constructs that were used to distinguish the problems from each other are personal constructs, there is a large variance in the personal constructs that were elicited. A different sample of students could, therefore, have an influence on the results. A larger sample size could improve the results and would probably lead to construct-categories with more constructs. The proportion of female to male in our sample (1–6) is somewhat low compared to the proportion in the population of pre-service physics students in both semesters (1–3.1 for the 2017/2018 cohort; 1–1.4 for the 2016/2017 cohort). It is however unclear what the influence of this unrepresentative proportion of female to male in the student sample is. In earlier research on perceived relevance by university students (e.g. Frymier & Shulman, 1995; Kember et al., 2008; Muddiman & Frymier, 2009) the results are not distinguished between female and male.

Implications

Physics problems are not always seen as relevant by pre-service physics teachers. Since perceived relevance has an influence on motivation, it is important to increase the perceived relevance of these problems. It is possible to do this by using problems that

• are conceptual problems,

• are close to everyday life,

• have lower mathematical requirements,

• are based on school-relevant content,

• require more than merely ‘executing’, where students have the feeling they learn something,

• contain a situation that has to be analysed, like a student’s misconception.

We explicitly do not say that physics problems for pre-service physics students should for instance only have a low level of mathematical requirements or be only conceptual. It is part of a physics study to use advanced mathematics to solve quantitative problems. In Germany, it would also lead to other issues, since pre-service physics teachers often attend the same courses as physics major students, that require a higher level of mathematics. However, courses with a healthy mixture of, for instance, conceptual problems and calculational problems, of problems that are theoretical and those that are close to everyday life and of problems that require a high mathematical level and problems that require a lower mathematical level will already have a positive effect on the perceived relevance of these courses by pre-service physics teachers.

Acknowledgments

This project is part of the ‘Qualitätsoffensive Lehrerbildung’, a joint initiative of the Federal Government and the Länder which aims to improve the quality of teacher training. The authors are responsible for the content of this publication. The authors would like to thank Frank Jaiser for help with some of the physics problems.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Joost Massolt http://orcid.org/0000-0001-6131-5389

Andreas Borowski http://orcid.org/0000-0002-9502-0420

Additional information

Funding

The programme is funded by the German Federal Ministry of Education and Research.

Appendices

Appendix 1

Figure A1. Problem 1 (after Mazur, 2014).

Figure A2. Problem 2 (after King & Regev, 1997).

Figure A3. Problem 3 (after Giancoli, 2008).

Figure A4. Problem 4.

Figure A5. Problem 5 (‘nullphasenwinkel’ is translated as ‘zero phase angle’).

Figure A6. Problem 6 (after O’Kuma, Maloney, & Hieggelke, 2004).

Figure A7. Problem 7.

Figure A8. Problem 8 (after Lindner, 2003).

Figure A9. Problem 9.

Appendix 2

Table A1. Constructs with poles (poles/contrast) for every participant.

Participant	Construct number	Pole/contrast	Sum of differences	Sum of differences – reversed	Similarity score	Tercile
1	1	Thinking problem/Calculational problem	9	24	15	H
1	2	Easy/Difficult	12	23	11	H
1	3	Close to everyday life/distant to everyday life	13	20	7	L
1	4	Basics/Application	14	21	7	L
1	5	Long problem text/short problem text	12	19	7	L
1	6	School relevant/not school relevant (related to content)	7	20	13	H
1	7	Fun/no fun	9	26	17	H
1	8	A lot of time needed/not a lot of time needed*	20	13	7	L
2	9	Open problem/closed problem*	25	16	9	I
2	10	Calculational problem/conceptual question	19	20	1	L
2	11	High diversity in requirements/low diversity in requirements*	26	17	9	I
2	12	Everyday/not everyday	15	24	9	I
2	13	Short problem definition/long problem definition	13	32	19	H
2	14	Complex/not complex	19	22	3	L
3	15	Rather vectoral/rather scalar*	25	10	15	H
3	16	Intuitive knowledge/expert knowledge (Problem is solvable with …)	19	20	1	L
3	17	Higher mathematical effort/lower mathematical effort*	25	16	9	H
3	18	Approximation necessary/approximation not necessary	18	21	3	I
3	19	Long problem/short problem	13	14	1	L
3	20	Motivating design/non-motivating design	14	21	7	I
3	21	Possible to guess subject/not possible to guess subject (… by looking at appearance of problem)	12	21	9	H
4	22	Applied mathematics/explanational problem*	30	7	23	H
4	23	Analytical/not analytical (a given situation has to be analysed)	7	30	23	H
4	24	Sketch necessary/sketch not necessary*	24	17	7	L
4	25	Close to reality/theoretical	7	32	25	H
4	26	Personal/distant	14	27	13	L
4	27	Vectoral approach necessary/vectoral approach not necessary*	30	11	19	I
4	28	Enjoyed solving/did not enjoy solving	10	27	17	I
4	29	Leads to deeper understanding/merely executing	6	29	23	H
4	30	Visually appealing/visually not appealing	11	20	9	L
5	31	Thinking/plug in	12	23	11	H
5	32	Difficult/Easy*	21	16	5	I
5	33	Explain/calculate	15	22	7	I
5	34	Solved rapidly/big effort	16	17	1	L
5	35	Open problem/closed, unambiguous	17	18	1	L
5	36	Descriptive/not descriptive (concerning images)	11	22	11	H
5	37	Only one subject/connection of multiple subjects	18	23	5	L
5	38	Complex problem definition/problem definition understandable*	24	15	9	I
5	39	Increase in knowledge/no increase in knowledge	7	26	19	H
6	40	Calculational problem/Thinking problem	13	16	3	I
6	41	Closed problem/open problem*	17	16	1	L
6	42	Problem with numerical values/generally valid problem*	19	18	1	L
6	43	Eager to discuss/not eager to discuss	14	21	7	I
6	44	High time investment/low time investment*	20	17	3	L
6	45	Fun solving/no fun solving	8	23	15	H
6	46	Reference to real life/no reference to real life	9	22	13	H
6	47	More a school problem/more a university problem	12	21	9	H
6	48	Not satisfied with performance/satisfied with performance*	20	11	9	H
6	49	Imagination necessary/no imagination necessary*	21	12	9	H
7	50	Calculational problem/comprehension problem*	26	9	17	H
7	51	More from own perspective/more from third person’s perspective	7	24	17	H
7	52	possible to experience problem personally/phenomenon rather in technical area	6	21	15	H
7	53	Problem definition easy to understand/problem definition difficult to understand*	21	14	7	L
7	54	Easier to solve/more difficult to solve	9	24	15	I
7	55	Descriptively stated/less descriptively stated*	22	17	5	L
7	56	High degree of hints/lower degree of hints*	24	11	13	I

Notes: The sum of differences refers to the sum of the differences between the construct score for every problem and the score on the relevance scale. For the ‘sum of differences – reversed’ the same calculation was used, but with the relevance scale reversed. The similarity score is calculated from both sums of differences. An asterisk* indicates that the poles need to be reversed. The constructs are divided into top (H), medium (I) and lower (L) terciles for every participant.

Appendix 3

Table A2. Construct-categories with their constructs and their respective terciles.

Construct-category	Construct	Terciles
Conceptual vs calculational*	1,50,22,31,33,40,10	H,H,H,H,I,I,L
Difficulty	2,32,38,54,53	H,I,I,I,L
Looks and descriptiveness of problem	36,21,20,30,55	H,H,I,L,L
Close to everyday life (on continuum scientific/technological/personal)*	52,25,46,12,3	H,H,H,I,L
Mathematical requirements*	15,17,27	H,H,I
Openness of problem	9,35,41	I,L,L
Superficial properties – text length	13,5,19	H,L,L
Difficulty – time investment	8,34,44	L,L,L
Enjoyment (perceived personal enjoyment)*	7,45,28	H,H,I
Curricular order – school-relevant content*	6,47	H,H
Curricular order – necessary knowledge for solving problem	37,16	L,L
Developing mental problem representation (sketching/imagining situation)	49,24	H,L
Perceived learning gain*	29,39	H,H
Complexity	4,14	L,L
Proximity to person (persons are actors)	51,26	H,L
Analytical*	23	H
Miscellaneous	11,18,42,43,48,56	I,I,L,I,H,I

Note: The construct numbers refer to the table in Appendix 2. The construct-categories marked with an asterisk are retained for further analysis.