Applying a fuzzy questionnaire in a peer review process

Abstract

This paper introduces the application and the results of a fuzzy number based questionnaire used for evaluating lecturers’ performance in a newly launched peer review programme in order to enhance the reliability of peer evaluations. In our approach, the membership function of the fuzzy number is composed of an increasing and a decreasing sigmoid membership function conjuncted by Dombi’s intersection operator. Compared to the traditional Likert scale-based evaluation, the proposed fuzzy number based methodology allows peer reviewers to express their uncertainty and the variability of the reviewed lecturer’s performance within a single lecture and during a semester in a quantitative way. By utilising the principles of Dombi’s Pliant Arithmetic, fuzzy Likert scale based evaluations given by peer reviewers can be cumulated by aggregating the parameters of the left- and right-hand sides of the fuzzy numbers separately. This property results in a much simpler statistical analysis of the gathered data than the methods previously proposed in the literature. A demonstrative case study illustrates our main contributions in peer evaluation.

Keywords:

• Likert scale
• fuzzy rating scale
• pliant arithmetic
• peer review
• higher education
• service quality

1. Introduction

How can we assess the quality of educational services offered by a higher education (HE) institution? How can we know reliably whether the expectations of customers and other stakeholders, primarily of students are met? In order to answer these questions, there is a need to design suitable and reliable methodologies to evaluate performance and to identify appropriate measure units to highlight the achieved service performance level (Lupo, 2013; Battisti, Nicolini, & Salini, 2005, 2010).

Measuring service quality and the satisfaction of stakeholders in HE is mainly realised through the application of Likert scales (see e.g. Brochado, 2009; Teeroovengadum, Kamalanabhan, & Seebaluck, 2016; Nadiri, Kandampully, & Hussain, 2009; Lalla, Facchinetti, & Mastroleo, 2005). Chen (2001) argues that the use of these scales reduces the human perception on one concrete number. Others have also claimed that individuals cannot use an exact number to express their opinion about a given situation. Instead of that, linguistic assessment is preferred to represent that specific numerical value (Herrera & Herrera-Viedma, 2000; Herrera, López, Mendana, & Rodríguez, 1999; Kacprzyk, 1986; Andayani, Hartati, Wardoyo, & Mardapi, 2017; Carrasco, Villar, Hornos, & Herrera-Viedma, 2011; Chang & Wang, 2016; Cabrerizo, López-Gijón, Martínez, Morente-Molinera, & Herrera-Viedma, 2017). In order to consider human perceptions, the fuzzy set theory is increasingly applied in these situations as they improve successfully the reliability of service process measurements and evaluations (Li, 2013; Lupo, 2013; Lin, 2010a; Deng, 2008).

This paper focuses on a challenging problem which is related to how to handle properly the inherent uncertainty of human perceptions. The current study aims to demonstrate the application of fuzzy Likert scales for evaluating lecturers’ performance in the peer review programme launched at the Faculty of Economic and Social Sciences. The proposed approach can help to deal with the vagueness arising either from the uncertainty of the reviewers or from the fluctuation of the observed lecturers’ performance. By providing a fuzzy Likert scale to evaluate lecturers’ performance, reviewers can express their uncertainty, their contrasting perceptions and the variability of the observed lecturers’ performance in a quantitative way.

Despite the fact that fuzzy numbers are able to model human judgement more precisely, the lack of simple methods to carry out statistical analyses has been turned out to be an obstacle in their utilisation. Several methods proposed in the literature (see e.g. Frühwirth-Schnatter, 1992 or Amini & Jochem, 2011) require both expertise in fuzzy set theory and high calculation efforts, which could hinder their utilisation. By following Dombi’s Pliant Inequality Model (Dombi, 2009), the aggregate evaluation can be computed in a convenient way which can serve the purposes of statistical analysis as well and allows the draw of more reliable managerial conclusions.

The paper is structured as follows. The next section highlights the advantages of fuzzy Likert scales compared to traditional Likert scales and provides our problem statement in the light of the peer review programme available at our faculty. Then the fuzzy set theory based methodology is presented. Finally, a case study demonstrates the advantages of the use of fuzzy Likert scales in peer evaluations. At the end we sum up the main findings and outline future research directions.

2. Literature review: traditional rating vs. fuzzy rating

The evaluation of educational activities may be carried out in several ways and shows great diversity. Berk (2005) gives an overview of the possible varieties of evaluation applied in educational context in case of which commonly employed scales are Likert ones (Hartley, 2014; Murray, 2013). It is a discrete scale by choosing the most appropriate ‘values’ within a class according to the raters’ judgement, opinion, valuation (Gil & González-Rodríguez, 2012) and leads to ordinal data from a set of pre-fixed categories. When applying these Likert-type questionnaires and rating scales, responses are averaged in order to express the mean of a specific teaching performance attribute (De Witte & Rogge, 2011) which is then utilised as a kind of index and applied as a part of formative and/or summative evaluation. Another possible approach is summing the ratings and expressing them as a percentage to the maximal attainable overall rating (Liaw & Goh, 2003). It is also possible to ask students, peer reviewers or other stakeholders to rate the overall, semester-long performance of the lecturer on a single scale (Ellis, Burke, Lomire, & McCormack, 2003).

Rating items in a questionnaire can be considered as a complex task as raters make multiple decisions under uncertainty. The number of ‘values’ to choose from is small (Gil & González-Rodríguez, 2012) which means that the variability, diversity and subjectivity associated with an accurate rating is usually lost. Another disadvantage originates from the fact that when values are encoded by their relative position in accordance with a certain ranking, differences between codes cannot be interpreted as differences in their magnitude. It means that only statistical conclusions addressed to ordinal data can be reliable and relevant information can be lost (Lubiano, de Sáa, Montenegro, Sinova, & Gil, 2016). Another major issue related to the application of Likert scales is the weighting of aspects as they are usually not equally important for the raters. An additional concern arising with expressing overall evaluations is the fact that raters’ attitudes towards the rated item are not homogeneous as time goes on (see e.g. Tóth Zs, Surman, & Árva, 2017b). Third, if raters’ preferences are heterogeneous, it matters how and to what extent it influences the overall evaluation of a rated item. Average scores are supposed to hide the real situation, namely, the performance of the rated item (Kuzmanovic, Savic, Popovic, & Martic, 2013). Moreover, when Likert-type data are analysed for statistical purposes, the techniques to analyse them are quite limited (Lubiano et al., 2016). Different studies have been carried out to discuss the reliability of the analysis of these responses pointing out that increasing the number of responses results in an increase of information and reliability (Lozano, García-Cueto, & Muñiz, 2008; de Sáa, Gil, González-Rodríguez, López, & Lubiano, 2015). However, it cannot be achieved by using a natural language (Sowa, 2013). To manage these disadvantages there is an alternate approach which takes into account that the nature of most attributes related to evaluations, judgements involve subjectivity and certain imprecision (Lubiano et al., 2016; Quirós, Alonso, & Pancho, 2016).

Hesketh, Pryor, Gleitzman, and Hesketh (1988) proposed the fuzzy rating scale without raters being constrained to choose among a few pre-specified categories. It is expressive enough to find a value in it fitting appropriately the valuation, opinion, judgement involving subjective perceptions in most real life situations (Gil, Lubiano, De Sáa, & Sinova, 2015). This kind of scale has the ability to model the imprecision of human rating evaluations, formalise them mathematically, to ‘precisiate’ them in a continuous way, and to develop mathematical computation with them (Gil et al., 2015; Calcagnì & Lombardi, 2014; Gil & González-Rodríguez, 2012). This approach leads to a fuzzy-valued response format enabling a level of variability and accuracy which would not be captured when using a Likert scale.

In the service quality literature several initiatives confirm the recent shift towards the utilisation of fuzzy ratings (Lin, 2010b). Liou and Chen (2006) demonstrate that fuzzy linguistic assessment of service quality is much closer to human thinking than methods based on crispy numbers. The recent literature provides a number of modifications of the widely applied service quality models (Liu et al., 2015; Mashhadiabdol, Sajadi, & Talebi, 2014; Lupo, 2016; Zhang, Lin, & Ren, 2010).

Fuzzy rating scales have been intensively applied recently in HE context as well to measure quality related issues. Başaran, Kalaycı, and Atay (2011), Lalla et al. (2005) propose a method to implement fuzzy logic in student evaluations of teaching performance. Büyüközkan, Ruan, and Feyzioğlu (2007) evaluate the quality of e-learning websites using fuzzy logic. Yu, Tsang, and Chen (2016) develop a fuzzy linguistic scale to establish a satisfaction index in case of e-learning systems. Lupo (2013) introduces a fuzzy SERVQUAL based method for reliable measurements of education quality. Rouyendegh and Erkan (2013) apply a fuzzy logic based method for academic staff selection, Hammed (2011) introduces one for a student evaluation system.

The membership functions of fuzzy sets can have various shapes (see Figure 1). The aforementioned literatures in fuzzy rating of service quality mainly employ triangular or trapezoidal membership functions, however, they are not always consistent with human thinking and judgement, since in these cases the membership functions have the same slope on the whole interval (see e.g. Hammed, 2011). In reality, the judgement of the respondent changes only slightly around points that represent the worst and the best possible judgement on the service quality, and the same property holds for the evaluation which is most likely to express the rater’s judgement. Hammed (2011) also pointed out the disadvantages of triangular membership functions and proposed a Gaussian one to enhance the reliability of student evaluations. In our paper, we work with fuzzy numbers the membership function of which is composed of an increasing and a decreasing sigmoid membership function based on its favourable property that it increases or decreases around the above mentioned points less sharply (Dombi, 2008, 2009). That is, the slope of the sigmoid function is not constant, therefore, applying sigmoid functions to establish fuzzy numbers results in a more precise reflection of human thinking and judgement.

Figure 1. Triangular, trapezoid, Gaussian and sigmoid membership functions.

By following Dombi’s Pliant Inequality Model (Dombi, 2009), the aggregate evaluation based on different raters’ judgement can be computed in a convenient way since the parameters of our membership functions are aggregated according to the arithmetic and harmonic mean. This can serve the purposes of statistical analysis as well and allows the draw of more reliable managerial conclusions.

2.1. Peer review framework at the Budapest University of Technology and Economics

Based on the relevant literature and taking international practices into consideration (Samson & McCrea, 2008; Ihsan et al., 2012; Washer, 2006; Blackmore, 2005; Courneya, Pratt, & Collins, 2008; Brent & Felder, 2004), a semester-long peer review of teaching process has been launched at our Faculty. The questionnaires applied to evaluate lecturers’ performance include the observation of lectures, the observation of midterm tests and/or exams, and the overall evaluation of the lecturers’ semester-long performance in case of which most of the aspects of the lecturer’s performance are evaluated using a traditional 5-point Likert scale (Tóth Zs, Andor, & Árva, 2017a). Peer evaluations include both formative and summative components, since the applied questionnaires consist of two main parts: a numerical scale assessment of specific skills and of performance attributes and a second part standing for obligatory narrative comments.

For the purposes of this study, we focus on the questionnaire applied during classroom teaching observation. In this questionnaire the following performance dimensions have been evaluated on a traditional Likert scale: (1) Volume, intonation; (2) Grammar, intelligibility, speech rate; (3) Learning, explaining the terminology; (4) Explanatory capability; (5) Maintaining students’ attention; (6) Choosing proper presentation techniques; (7) Quality of the slide show; (8) Consonance of the slide show with verbal communication; (9) Logical structure of the lecture and presentation. This classroom questionnaire includes a second part as well allowing answers for elaborative questions.

The first results of the peer review programme have been already presented by Tóth Zs, Andor, and Árva (2017a). During the evaluation process, we have come across several major constraints arising from the application of traditional Likert scales. Three main problems of feedbacks have been found that can affect the reliability of results originating from the process of peer reviewing, namely, the uncertainty inherent among evaluations, the variation of lecturers’ performance during the semester and the lack of methods available to compare different reviewers’ narrative evaluations (Tóth Zs, Surman, and Árva, 2017b). These issues addressed the research problem, namely, the enhancement of the reliability of measurements and evaluations executed in HE context by utilising fuzzy logic.

The evaluations conducted by either peer reviewers, students or other stakeholders can be considered as a complex combination of verbal statements. Evaluators have personal subjective preference or judgment depending on their individual knowledge or experiences. It is suitable for evaluators to use linguistic variables to express their subjective opinions in the assessment process. Researchers usually turn originally verbal data into numerical data by using Likert-type scale questionnaires when they are trying to get the opinions, judgements or valuations of the students, peer reviewers or other stakeholders (Başaran, Kalaycı, and Atay, 2011).

2.2. Drawbacks of peer evaluation

The aspects and evaluation dimensions of the questionnaire based on a traditional Likert scale are not able to cover the fluctuation of the observed lecturers’ performance within the same lecture, since evaluations are given at the end of the lecture reflecting an average evaluation in each dimension. Moreover, the retrospective statistics including the mean, the range and the standard deviation of the overall evaluations given at the end of the semester are more likely to reflect the differences between the various reviewers’ judgement than the variability of the lecturers’ performance.

Narrative comments in the questionnaires are of high importance and kindly welcomed from the reviewees’ points of view. They can reflect the variability of the observed lecturer’s performance as well, but there is a lack of simple methods to deal with linguistic feedbacks. This leads to an insufficient elaboration of this kind of information gained during the evaluation process. It means that by comparing and evaluating different lecturers’ performance based solely on numerical assessment, a remarkable part of the information gained is either lost or is not taken into account, which cannot serve the purposes of the faculty as a whole.

A fuzzy Likert scale can help to overcome the aforementioned difficulties. A fuzzy number depicts not only the differences between different colleagues’ judgements, but also the contrasting perceptions and impressions of the same reviewer. The more the uncertainty associated with the judgement and the more unbalanced the observed lecturer’s performance are, the more spread out the fuzzy number is.

3. Methodology

3.1. Fuzzy numbers as intersections of two soft inequalities

In our approach, the values on a Likert scale are represented by fuzzy numbers; that is, instead of expressing an opinion by selecting a particular crisp value on the scale, we allow the evaluator to select an ‘approximately x’ value that is given by a fuzzy number. We will use sigmoid functions to compose the membership functions of fuzzy numbers.

Definition 1

 The sigmoid function with parameter and is given by where and is nonzero.

The main properties of the sigmoid function are as follows.

• Range. The range of is the interval

• Continuity. is continuous in

• Monotony.

  • o If , then is strictly monotonously increasing

  • o If , then is strictly monotonously decreasing

• Limits.

• Role of parameters.

  • o Parameter is the locus at which has the value 0.5

  • o The slope of at is ; that is, the parameter determines the gradient of function curve at

Figure 2 shows examples of sigmoid function graphs. Following Dombi’s Pliant Inequality Model (Dombi, 2009), we will use the following definitions where the indexes and stand for ‘left’ and ‘right’, respectively, and will be used to denote left hand side and right hand side components of fuzzy numbers.

Figure 2. Intersection of two fuzzy sets given by an increasing and a decreasing sigmoid membership function.

Definition 2

 The soft inequality is given by the sigmoid function where , , .

The soft inequality represents the truth of the inequality . This truth value is in the interval (0, 1) and it depends on . Since is positive, the greater the variable is, the higher the truth of the inequality is. Notice that the parameter is responsible for the ‘sharpness’ of the soft inequality . It can be shown that if , then ; that is, the truth of the relation tends to 1, which means that stands in the traditional manner. This result tells us that the soft inequality is a generalisation of the crisp inequality .

Definition 3

 The soft inequality is given by the sigmoid function where , , .

The soft inequalities given by Definitions 1 and 2 allow us to express how much we consider a particular value of being greater than or less than . The introduced soft inequalities represent fuzzy sets with the membership functions and . That is, for any , and are the membership values of in the fuzzy sets which contain the numbers that are greater than and less than , respectively. Intersection of these two fuzzy sets represents the soft interval . We will use the Dombi conjunction operator (Dombi intersection) to implement intersection of two fuzzy sets (Dombi, 2008).

Definition 4

 The Dombi intersection of the fuzzy sets and that are given by the membership functions and , respectively, is the fuzzy set with membership function : where , , , and denotes the Dombi intersection operator.

If we apply the Dombi intersection to and with , we get

Utilising Definition 2 and Definition 3:

Figure 2 shows the intersection of two fuzzy sets given by an increasing and a decreasing sigmoid membership function. The following theorem demonstrates a key property of the soft inequalities given by sigmoid functions.

Theorem 1

 If are fuzzy sets with the membership functions , respectively, , and the fuzzy set is given by the linear combination where , then is also sigmoid-shaped with the membership function , where

Proof:

See Dombi (2009). ▪

Theorem 1 allows us to separately aggregate the left hand sides and right hand sides of fuzzy intervals that are given by like inequalities.

The parameters and of the sigmoid function can be unambiguously given by determining two points of the function curve. The sigmoid function neither takes the value 0, nor the value 1, these are its limits. In practical applications, it may be useful if the function is given by two points which have vertical coordinates close to 0 and 1. Let be a small positive value, for example , and

If we wish to take the values of and at and , respectively, the parameters and need to be set as follows: where . Let be three (crisp) numbers so that If we require to take the values of and at and , respectively, and we also require to take the values of and at and , respectively, then need to be set as follows:

Next, if we generate the Dombi intersection of the increasing sigmoid fuzzy membership function and the decreasing sigmoid fuzzy membership function with the above parameters, then we get the membership function

It can be proven that the function has its maximum at

Furthermore, ; that is, the function may be viewed as the membership function of the fuzzy number ‘approximately ’. Moreover,that is, if is close to zero, then . Since the parameter is a small positive number (e.g. ), the membership function approximates quite well the fuzzy number ‘approximately ’. Figure 3 shows an example of the fuzzy number ‘approximately ’ given by the membership function .

Figure 3. The cut of the fuzzy number ‘approximately m’.

3.2. A possibilistic representation of the perceived performance

We have shown that the function , for any , approximates the truth of the soft equality quite well, so we can represent by the function . Since for any , and is strictly monotonously increasing, if ; and it is strictly monotonously decreasing, if , there are exactly two places, , and , at which the membership function takes the value of , where . Thus, the cut of is the interval ; that is,Owing to the construction of the function , the approximate values of , and can be computed from the left-hand side and right-hand side sigmoid components of : When ‘approximately ’ is utilised for describing a perceived performance, and the fuzzy number ‘approximately ’ is given by the membership function , then the left- and right ends of the cut interval of , respectively, , and , can be interpreted as follows. If the perceived performance is between and , then the truth level of the statement that the performance is ‘approximately ’ is at least .

Figure 3 shows a graphical example of the above-mentioned interpretation of , and .

4. Case study

4.1. Fuzzy Likert scale based evaluation of teaching performance

Based on the favourable properties of fuzzy Likert scales, that can help to overcome the difficulties associated with the peer review process, some aspects of the applied questionnaire have been evaluated by using a fuzzy scale parallel to the traditional Likert scale during the fall semester 2016. Based on various feedbacks, the evaluation dimensions related to the delivery of the lectures have been selected for this purpose, since these dimensions have turned out to be the most difficult to evaluate on a traditional scale. In case of five evaluated lecturers, three reviewers have been asked to express their judgement on a fuzzy Likert scale regarding the dimensions mentioned previously. The scale has been applied with a division of 0.25 units in order to allow the reviewers a more precise reflection of their judgement. Altogether 135 fuzzy evaluations have been collected representing three reviewers’ opinion of performance of five selected reviewees in nine evaluation dimensions listed above. It can be concluded that the reviewers have not experienced any difficulties when making their evaluation; Gil, Lubiano, De Sáa, and Sinova (2015) have come to the same conclusion.

The following figure (Figure 4) illustrates the fuzzy numbers (represented by blue curves) used to evaluate Lecturer 1’s performance as well as the aggregate fuzzy evaluation (denoted by the red curve) in the dimension entitled Learning, explaining the terminology.

Figure 4. Fuzzy numbers based evaluation of dimension Learning, explaining the terminology in case of Lecturer 1.

The aggregate evaluations of the other dimensions in case of Lecturer 1 have been computed similarly. Figure 5 represents the aggregated fuzzy evaluations in each dimension as well as the average evaluation of the dimensions, which represents the average evaluation of Lecturer 1’s performance.

Figure 5. Fuzzy numbers based evaluations of Lecturer 1.

The average evaluations for all involved lecturers are presented in Figure 6. Based on the evaluations shown in Figure 6, Lecturer 1, 2 and 5 has approximately the same performance, however, in case of Lecturer 1 and 2, the performance or the judgement of the reviewers is more unbalanced. Based on the traditional Likert scale-based evaluation, however, it was almost impossible to make a distinction between their performance levels.

Figure 6. Aggregate evaluations of the 5 selected lecturers by fuzzy numbers.

Aggregate evaluations can be computed not only for a given lecturer, but for a specific dimension as well. Based on the evaluation of the five involved lecturers, the average judgements in each dimension are presented and listed in Figure 7. Table 1 contains the parameters of fuzzy numbers and the average of crisp evaluations given on a traditional Likert scale by the same reviewers in each dimension. It can be concluded that the means of crisp evaluations and the centres of fuzzy numbers do not differ significantly.

Figure 7. The average evaluations of five involved lecturers’ performance in each dimension.

Table 1. Parameters of fuzzy numbers, means and 95% confidence intervals of crisp evaluations representing the average evaluations in each dimension used to evaluate lecturers’ performance.

Evaluation dimension	Fuzzy number			Crisp evaluation
	Left	Centre	Right	Mean	St. deviation
Volume, intonation	4.127	4.441	4.642	4.568	0.545
Grammar, intelligibility, speech rate	4.268	4.583	4.771	4.636	0.532
Learning, explaining the terminology	4.004	4.646	4.717	4.682	0.471
Explanatory capability	3.997	4.244	4.657	4.295	0.553
Maintaining students’ attention	3.797	4.418	4.587	4.432	0.625
Choosing the proper presentation techniques	3.557	4.198	4.447	4.273	0.949
Quality of the slide show	3.716	3.974	4.356	4.119	0.633
Consonance of the slide show with verbal communication	4.667	4.833	4.917	4.762	0.431
Logical structure of the lecture and presentation	4.663	4.917	4.988	4.818	0.446
Average evaluation	4.088	4.473	4.676	4.510	0.632

4.2. Discussion of findings

The main benefit of applying fuzzy numbers to evaluate lecturers’ performance is their ability to model the imprecision and the uncertainty inherent in reviewers’ evaluation as well the representation of the variation of performance. Based on Figures 4–7 it can be concluded that the performances of the observed lecturers are unbalanced and reviewers have different perceptions on their performance as well. Utilising traditional Likert scale based evaluation, it is almost impossible to capture accurately this variation of the performance (see Table 1 and Figure 6). As the case study demonstrates, the centres of fuzzy numbers do not differ significantly from the means of crisp evaluations. Despite the fact that fuzzy evaluations do not differ more than half a unit, the information encoded in fuzzy evaluations supports well the identification of areas need to be improved. Moreover, the evaluation dimensions, which turned out to be more unbalanced, were also downgraded by students, further emphasising the need for improvement in these areas. Having investigated the evaluation dimensions in case of which fuzzy numbers indicate a greater diversity and taking the narrative feedbacks of previous evaluations into account, it can be concluded that the variation of the observed lecturer’s performance could be expressed on a fuzzy scale, since the fuzzy numbers are spread out in those dimensions in which the previous narrative comments indicated a more unbalanced performance. That is, by offering the opportunity to use a fuzzy Likert scale to evaluate the observed lecture, reviewers can express their uncertainty and the variability of the observed colleague’s performance during a single lecture in a quantitative way.

5. Conclusions and managerial implications

Based on the first results it can be concluded that the presented fuzzy judgement can help the university staff to overcome several difficulties associated with peer evaluations. It is beneficial from the faculty’s point of view, namely, it allows comparisons between different lecturers’ performance and the identification of best practices and weaknesses as well. Based on the encouraging results, applying fuzzy evaluation in the peer review process may be considered as a viable technique.

5.1. Practical implications

By applying fuzzy numbers for peer reviewing, the faculty can decrease the amount of narrative comments as well, which is beneficial from the statistical point of view of the analyses and allows more information to be taken into account when comparing different lecturers’ performance and identifying best practices.

Besides peer evaluation, the application of the presented methodology may arise in evaluations where Likert-type scales are applied traditionally. The most desirable way is to extend the application of fuzzy scales to student satisfaction measurement, since students face similar difficulties when evaluating courses, which results not only in a low response rate, but also in less reliable information. Moreover, students usually have various perceptions on teaching quality which cannot be measured by a single number on a traditional scale. By applying fuzzy numbers, students can choose more than one single value which represents their opinion, for example, their optimistic and pessimistic judgement on the teachers’ performance.

5.2. Limitations and further research plans

Besides feeding back the results to the peer reviewed lecturers, student satisfaction or course evaluation results should be continually monitored and compared to the results of the peer review programme. In order to evaluate the ultimate impact of peer reviewing on teaching performance and the effectiveness of fuzzy Likert scales used for this purpose, relationships between the results of peer reviewing and student satisfaction surveys should be further investigated. Extending fuzzy evaluations to student satisfaction may result in a more detailed investigation of cause and effect relationships as well. Fuzzy regression (Alfonso, López, de Hierro, & Roldán, 2016) or fuzzy inference systems (e.g. Hammed, 2011; Jónás & Árva, 2016) would be a suitable tool to deal with this challenge.

Disclosure statement

No potential conflict of interest was reported by the authors.