Is it necessary to evaluate university research performance instrument? Evidence from Indonesia

Figures & data

Figure 1. Comparison of publications of several ASEAN countries.Note. From Scopus, March 2021.

Table 1. Condition of several national ranking system

COUNTRY AND NAME OF SYSTEM	CONDITION
Argentina, Consejo Nacional de Evaluation y Acreditation de las Universidades	Lack of detailed information
Australia, Good Universities Guides	No overall ranking
Australia, Excellence in Research for Australia	No overall ranking
China, Education18 Ranking	Lack of detailed information
China, Netbig	Lack of detailed information
Germany, CHE	No overall ranking
India, India’s best colleges	No overall ranking
Italy, Vision	Lack of information about sustainability
Korea, Korean Council for University Education Ranking	Outdated
Korea, Joongang daily ranking	Lack of detailed information
New Zealand, Performance-Based Research Fund	No overall ranking
Romania, Ad-Astra Ranking	No overall ranking
Ukraine, Top-200 Ukraine	Lack of detailed information
USA, Washingtonmonthly	Lack of detailed information
Brazil, Folha’s University Ranking	Ranking Methodology is clear. Sustainability of ranking is more than 2 years.
Chile, EI Mercurio	Ranking Methodology is clear (9 indicators) Sustainability of ranking since 2011
China, Greater China Ranking	Ranking Methodology is clear (22 indicators) Sustainability of ranking since 2011
Kazakhstan, Ranking of Higher Education Institutions in Kazakhstan	Ranking Methodology is clear (18 indicators) Sustainability of ranking since 2008
Lithuania, Veidas	Ranking Methodology is clear (12 indicators) Sustainability of ranking since 1997
Macedonia, Macedonian HEIs Ranking	Ranking Methodology is clear (35 indicators) Sustainability of ranking 2011
Malaysia, Setera	Ranking Methodology is clear (32 indicators) Sustainability of ranking 2007
Pakistan, HEC Higher Education Commission	Ranking Methodology is clear (9 indicators) Sustainability of ranking 2010
Poland, Perspectywy	Ranking Methodology is clear (19 indicators) Sustainability of ranking since 2008
UK, Complete University Guide	Ranking Methodology is clear (42 indicators) Sustainability of ranking since 2007
The USA, U.S. News College Rankings	Ranking Methodology is clear. Sustainability of ranking is more than 2 years.
Turkey, URAP	Ranking Methodology is clear (9 indicators) Sustainability of ranking since 2011

Note. Adapted from Cakır (Çakır et al., 2015).

Table 2. Several performance appraisal model

No	Model	Explanation
1	Essay Method	This method uses the formulation of the assessment results in the form of an essay that contains an explanation of the strengths and weaknesses of previous performance, positions, and suggestions for improvement.
2	Critical Incident	This method uses an instrument for recording positive and negative performance on critical, special, or important events focusing on the key factors that make a difference in carrying out activities efficiently.
3	Checklist	This model uses an instrument that lists indicators of work results, work behavior, or traits needed in internal performance appraisal. The results can be quantitative and give weight to characteristics. The answer to the checklist is often “Yes” or “No”
4	Graphic Rating Scales	This method uses an instrument that is presented in the form of a scale (number) in a brief description to describe reliability, consistency over time, usability, and practicality.
5	Forced Distribution	This model requires the evaluator to classify the assessed individual into several categories so that the manager has a disproportionate number of employees in the “superior” category
6	Forced Choice Scale	This model developed for military service forces the Evaluator to choose a behavior that is better than others in several sets of choices.
7	Model Behaviorally Anchor Rating Scale (BARS)	This method combines elements of acritical incident and outdated assessments, in observing valued behavior based on available behavioral anchors, then checking the existing level of performance level description scale.
8	Management by Objectives (MBO)	This model measures the outputs quantitatively in the form of strategic planning and object hierarchy, goal setting, action planning, MBO implementation, control and assessment, subsystems, and organization and management development.
9	360 Degree	The model requires more than one assessment consisting of your immediate supervisor, subordinates, coworkers, customers, customers, clients, and yourself. The information obtained is the impact of actions on other units that provide ideas about behavioral changes that may be caused through the process of increasing self-awareness.
10	Balanced Scorecard (BSC)	Kaplan and Norton (1992) introduced the BSC to evaluate whether an institution objectives can be achieved from four different perspectives: financial, customer, internal business processes, and learning and growth, where this method continues to envolve to this day.
11	Business Advantage Model (BEM).	The European Foundation of Quality Management (EFQM) developed BEM in 1990, which is related to 9 aspects: results—financial, customer satisfaction, people satisfaction, impact on society, leadership, policy and strategy, people management, resources, and processes management (Kim et al., 2008).
12	Key Performance Indicators (KPI)	KPI is a performance appraisal model that focuses on critical aspects of the outputs or outcomes. Performance indicators can be defined as components of information collected and evaluated periodically so thata system according to its objectives as an aspect of quality control (Meng & Minogue, 2011).

Note. Adapted mostly from (Aggarwal et al., 2013; Jafari et al., 2009).

Table 3. The cluster of Indonesian university on research performance

No	Period (Year)	Mandiri	Utama	Madya	Binaan	Participant
1	2007–2009	10	22	71	291	394
2	2010–2012	14	36	79	772	901
3	2013–2015	25	73	160	1,219	1,447
4	2016–2018	47	146	479	1,305	1,977

Note. Reprinted from Higher education research performance report (2019), Ministry of Research and Technology/BRIN.

Table 4. FGD participant criteria and code

PHASE	PARTICIPANT CRITERIA	CLUSTER	RANKING IN LAST EVALUATION	CODE
I	Six heads of LPPM with the status of state universities (PTN) from six PTNs	MANDIRI	7; 5; 1; 14; 13; 9	P1-A; P1-B; P1-C; P1-D; P1-E; P1-F
II	Four heads and one secretary of research institutes with the status of private universities (PTS) from five PTS	MANDIRI	39; 44; 32; 22; 34	P2-A; P2-B; P2-C; P2-D; P2-E
III	One head of PTN LPPM and five PTS LPPM from 6 PT	UTAMA	54; 27; 56; 83; 61; 74	P3-A; P3-B; P3-C; P3-D; P3-E; P3-F
IV	Four heads of PTN LMI, three heads of PTS LMI, and two PTN LMI officers from 7 PT	-	8; 3; 3; 15; 11; 4; 240; 10; 1	P4-A; P4-B; P4-C; P4-D; P4-E; P4-F; P4-G; P4-H; P4-I

Note. In the fourth FGD, there were four informants from the same two universities.

Figure 2. Policy and general issues.

Figure 3. Issues in implementation of performance.

Figure 4. Issues in performance evaluation component.

Figure 5. Issues in performance evaluation component.

Çakır, M. P., Acartürk, C., Alaşehir, O., & Çilingir, C. (2015). A comparative analysis of global and National University ranking systems. Scientometrics, 103(3), 813–848. https://doi.org/10.1007/s11192-015-1586-6

 Web of Science ®Google Scholar

Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard – measures that drive performance. Harvard Business Review, 70(1), 71–79.

 PubMed Web of Science ®Google Scholar

Kim, D.-Y., Kumar, V., & Murphy, S. A. (2008). European foundation for quality management (fEFQM) business excellence model: A literature review and future research agenda. Asac, 1, 1–48. http://ojs.acadiau.ca/index.php/ASAC/article/view/839/728

 Google Scholar

Meng, X., & Minogue, M. (2011). Performance measurement models in facility management: A comparative study. Facilities, 29(11), 472–484. https://doi.org/10.1108/02632771111157141

 Google Scholar

Aggarwal, A., Sundar, G., & Thakur, M. (2013). Techniques of performance appraisal - A review. International Journal of Engineering and Advanced Technology, 2(3), 2249–8958. http://www/ijeat.org/wp-content/uploads/papers/v2i3/C1188022313.pdf

 Google Scholar

Jafari, M., Bourouni, A., & Amiri, R. H. (2009). A new framework for selection of the best performance appraisal method. European Journal of Social Sciences, 7(3), 92–100.

 Google Scholar