Is it necessary to evaluate university research performance instrument? Evidence from Indonesia

Abstract

Evaluation of research management is essential in higher education The Indonesian government regularly evaluates the research performance, as part of improvement the research ecosystem in higher education system. This study aims to analyze the problems in implementing research performance evaluations in Indonesian higher education system This study used a qualitative method with Focus Group Discussion (FGD). The participants invited were leaders of research institutes at universities that represent the best college clusters. We conducted a four-stage FGD to obtain the required information. The results are answered with recommendations to rearrange the assessment and implementation model that is adapted to conditions in Indonesia. The author believes that this research will not only contribute to the scientific development but also provide assistance to international readers with insight into the research evaluation system in Indonesia, as well as policy-makers in implementing a better research evaluation system.

Keywords:

• research performance evaluation
• higher education
• measurement
• performance indicator

PUBLIC INTEREST STATEMENT

Evaluation of research implementation and management is essential in structuring research in higher education. The Indonesian government has used the results of performance evaluations of higher education research institutions as the basis for determining the provision of incentives, grants, and program funding to universities, including research and development contracts in the internal budgeting process. Performance evaluation is not only a part of universities’ promotion, but can also help improve their performance and direct them to become world-class universities that support the improvements of the quality of life and the nation’s competitiveness. Therefore, it is necessary to rearrange the assessment and implementation model that is adapted to policy and conditions in Indonesia.

1. Introduction

The current policy-based measurement method is a vital factor in supporting the achievement of national research policy targets. Different objectives and policies of stakeholders, who adopt a unique research performance evaluation perspective, affect the dimensions of performance evaluation (Abramo & D’Angelo, 2015), including factors that prove significant for stimulating innovative activity in one country, which may not be at all effective in countries with other economic and cultural determinants (Karpińska, 2020). It is closely related to the term performance that Armstrong (1994) conveyed as a result of work that has a strong relationship with the organization’s strategic goals, customer satisfaction, and contribution to the economy.

The Indonesian government has made efforts to increase the productivity and relevance of research both in universities and research institutions (Ristekbrin, 2021), which is in line with the definition of performance according to Kamble and Wankhade (2017) as one of the main factors that affect productivity. Another effort of the Indonesian government is to increase the contribution of science and technology development to national development, including in solving social problems in society. It is encouraged through collaboration between universities, R&D institutions, the business sector, and other stakeholders so that research can be utilized by industry and contribute to the economy and national development. Based on this policy, research output is directed not only to academic purposes or to produce inventions but also to economic and social benefits in the form of commercial products or other outputs that can help solve societal problems.

Recently, the ecosystem that supports the research and innovation sector in Indonesia has become conducive, so that the output produced has significantly increased (Ristekbrin, 2020b) amid differences in the situation among Southeast Asian countries in the ease of conducting research, especially in field research (Morgenbesser & Weiss, 2018). One of the increases in output achievement is presented in the figure 1, comparing the achievement of publication output among several ASEAN countries. The improvement of the number of Indonesian article publications at the ASEAN country level is the fastest growing. However, some of the publications were still published in the proceedings, which shows that the quality still needs improvement. This condition is inversely proportional to other ASEAN countries which have a higher percentage of articles in journals than proceedings. For Indonesia, this is important, in which publication in reputable journals is one of the assessment variables for lecturer careers listed in the Minister of Education and Culture of the Republic of Indonesia Number 92 of 2014 regarding guidelines for performance assessment implementation for lecturer career. However, the quality of research outputs in Indonesia still needs to be improved, with one of the strategies by encouraging competition between universities through a performance appraisal system that is per the global rating assessment system.

Figure 1. Comparison of publications of several ASEAN countries.Note. From Scopus, March 2021.

Higher education, is one of the research institutions with the largest science and technology human resources in Indonesia and has big contributed to research outputs target achievement in Indonesia over the last five years. Higher education is one of the research institutions with the largest science and technology human resources in Indonesia and has big contributed to research output targets achievement in Indonesia for the last five years. According to the data from the Forlap PDIKTI system—the lecturer data information system—in February 2020, there were 4,667 with 920 academies, 325 polytechnics, 2,517 high schools, 237 institutes, 634 universities, and 34 community academies (Dikbud, 2020). There are 296.040 lecturers from all higher education, consisting of 207.586 lecturers with master’s degree backgrounds and 42.825 lecturers with doctoral degrees.

With this very significant contribution, the priority of improving the quality of research in higher education should be the government’s attention. Performance Evaluation is one of the supporting data not only for the government in preparing evidence base policies, but also for officials within the institution. It is important for an institution to think about its mission and goals, how to measure its internal performance and determine its performance indicators that need to be adjusted (Ball & Wilkinson, 1994). This information has also expected to support research institutions in making the right decisions regarding policies, programs, and funding. Therefore it can accurately allocate funds according to their needs.

This significance underlies the author’s curiosity to evaluate the implementation of research performance measurement in universities in Indonesia.

2. Performance evaluation measurement

Evaluation of research implementation and management is essential in structuring research in higher education. The benefit for the university being in a high ranking is the government’s trust that views the competitive status of the university (Hazelkorn, 2008).

The government prefers higher-performance universities in an education or research project, as evidenced by the government’s support through funding to improve the quality of teaching and research. Higher education ranking results are a source of comparative information for various stakeholder groups, which can influence policies in higher education and motivate them to be better in all aspects (Feranecová & Krigovská, 2016), thus improving higher education institutions’ performance (Söderlind & Geschwind, 2019).

Çakır et al. (2015) compared several university performance evaluation systems, stating that the selected indicators are based on their respective interests. An overview of the measurement systems can be seen in Table 1.

Table 1. Condition of several national ranking system

COUNTRY AND NAME OF SYSTEM	CONDITION
Argentina, Consejo Nacional de Evaluation y Acreditation de las Universidades	Lack of detailed information
Australia, Good Universities Guides	No overall ranking
Australia, Excellence in Research for Australia	No overall ranking
China, Education18 Ranking	Lack of detailed information
China, Netbig	Lack of detailed information
Germany, CHE	No overall ranking
India, India’s best colleges	No overall ranking
Italy, Vision	Lack of information about sustainability
Korea, Korean Council for University Education Ranking	Outdated
Korea, Joongang daily ranking	Lack of detailed information
New Zealand, Performance-Based Research Fund	No overall ranking
Romania, Ad-Astra Ranking	No overall ranking
Ukraine, Top-200 Ukraine	Lack of detailed information
USA, Washingtonmonthly	Lack of detailed information
Brazil, Folha’s University Ranking	Ranking Methodology is clear. Sustainability of ranking is more than 2 years.
Chile, EI Mercurio	Ranking Methodology is clear (9 indicators) Sustainability of ranking since 2011
China, Greater China Ranking	Ranking Methodology is clear (22 indicators) Sustainability of ranking since 2011
Kazakhstan, Ranking of Higher Education Institutions in Kazakhstan	Ranking Methodology is clear (18 indicators) Sustainability of ranking since 2008
Lithuania, Veidas	Ranking Methodology is clear (12 indicators) Sustainability of ranking since 1997
Macedonia, Macedonian HEIs Ranking	Ranking Methodology is clear (35 indicators) Sustainability of ranking 2011
Malaysia, Setera	Ranking Methodology is clear (32 indicators) Sustainability of ranking 2007
Pakistan, HEC Higher Education Commission	Ranking Methodology is clear (9 indicators) Sustainability of ranking 2010
Poland, Perspectywy	Ranking Methodology is clear (19 indicators) Sustainability of ranking since 2008
UK, Complete University Guide	Ranking Methodology is clear (42 indicators) Sustainability of ranking since 2007
The USA, U.S. News College Rankings	Ranking Methodology is clear. Sustainability of ranking is more than 2 years.
Turkey, URAP	Ranking Methodology is clear (9 indicators) Sustainability of ranking since 2011

Note. Adapted from Cakır (Çakır et al., 2015).

The Indonesian government has used the results of performance evaluations of higher education research institutions as the basis for determining the provision of incentives, grants, and program funding to universities, including research and development contracts in the internal budgeting process (Dikbud, 2013). Additionally, research fund management institutions, such as the Indonesian Endowment Fund (LPDP), also use university rankings to provide research funds to universities with better performance, through the invitation of RISPRO scheme (LPDP, 2018). Previous studies have shown that the United Kingdom (Ball & Wilkinson, 1994), Sweden (Söderlind & Geschwind, 2019), Malaysia (Tahira et al., 2015), and Thailand (Siripitakchai & Miyazaki, 2015) carried out similar policies.

To acquire good measurement results (Dixon & Hood, 2016) recommend the following three conditions: the ranking can discern the institution being evaluated based on the indicators used; Stable rating instrument; the evaluation is not only beneficial for the Institute but also for the wider community. While Sizer et al. (1992) explained that there are five performance indicators in the measurement: Monitoring aspect, which measures the continuous assessment of a program, institution, or system; Evaluation aspect, which looks at the achievement of goals and objectives; Dialogue aspect, as a basis for communicating concepts and goals, so that we have the same perception of policy; Rationalization aspect, where performance indicators can encourage rational and coherent policy-making; and The aspect of resource allocation, which becomes the basis for resource mapping for resource strengthening.

We summarized some of the performance assessments presented in Table 2, while Meng and Minogue (2011) stated that there are several performance evaluation models that are more widely accepted and more effective than others, namely Key Performance Indicators (KPI), Balanced Scorecard (BSC), and Business Advantage Model (BEM).

Table 2. Several performance appraisal model

No	Model	Explanation
1	Essay Method	This method uses the formulation of the assessment results in the form of an essay that contains an explanation of the strengths and weaknesses of previous performance, positions, and suggestions for improvement.
2	Critical Incident	This method uses an instrument for recording positive and negative performance on critical, special, or important events focusing on the key factors that make a difference in carrying out activities efficiently.
3	Checklist	This model uses an instrument that lists indicators of work results, work behavior, or traits needed in internal performance appraisal. The results can be quantitative and give weight to characteristics. The answer to the checklist is often “Yes” or “No”
4	Graphic Rating Scales	This method uses an instrument that is presented in the form of a scale (number) in a brief description to describe reliability, consistency over time, usability, and practicality.
5	Forced Distribution	This model requires the evaluator to classify the assessed individual into several categories so that the manager has a disproportionate number of employees in the “superior” category
6	Forced Choice Scale	This model developed for military service forces the Evaluator to choose a behavior that is better than others in several sets of choices.
7	Model Behaviorally Anchor Rating Scale (BARS)	This method combines elements of acritical incident and outdated assessments, in observing valued behavior based on available behavioral anchors, then checking the existing level of performance level description scale.
8	Management by Objectives (MBO)	This model measures the outputs quantitatively in the form of strategic planning and object hierarchy, goal setting, action planning, MBO implementation, control and assessment, subsystems, and organization and management development.
9	360 Degree	The model requires more than one assessment consisting of your immediate supervisor, subordinates, coworkers, customers, customers, clients, and yourself. The information obtained is the impact of actions on other units that provide ideas about behavioral changes that may be caused through the process of increasing self-awareness.
10	Balanced Scorecard (BSC)	Kaplan and Norton (1992) introduced the BSC to evaluate whether an institution objectives can be achieved from four different perspectives: financial, customer, internal business processes, and learning and growth, where this method continues to envolve to this day.
11	Business Advantage Model (BEM).	The European Foundation of Quality Management (EFQM) developed BEM in 1990, which is related to 9 aspects: results—financial, customer satisfaction, people satisfaction, impact on society, leadership, policy and strategy, people management, resources, and processes management (Kim et al., 2008).
12	Key Performance Indicators (KPI)	KPI is a performance appraisal model that focuses on critical aspects of the outputs or outcomes. Performance indicators can be defined as components of information collected and evaluated periodically so thata system according to its objectives as an aspect of quality control (Meng & Minogue, 2011).

Note. Adapted mostly from (Aggarwal et al., 2013; Jafari et al., 2009).

KPI is widely used in global and national performance evaluation models, and is a guideline to divide for universities (Azma, 2010). Azma stated that in the 1970s, the US Board of Education had started designing national indicators to compare universities, including the system for determining PT ratings by the US Board of Education.

The indicators of the higher education performance measurement system used in the previous research were: changing environment, development of the nature of university activities, competitiveness, and changes in the management system (Balabonienė & Večerskienė, 2014). Meanwhile, Tijssen and Winnink (2022) stated that Developing quality criteria and performance indicators of local excellence can encourage researchers to contribute to socio-economic development and innovation.

This means that indicators are crucial in measuring performance, and the indicators in each national measurement system are unique (Çakır et al., 2015). The performance measurement model used is based on the measurements of research performance indicators that comply with the aims and objectives of the performance measuring institution. The performance measurement process is regarded as a key element of strategic management and can identify gaps between the current situation in the organization and the level of excellence with the proposed objectives of strategic planning and the use of indicators.

3. Evaluation of research performance in Indonesia

The Ministry regularly evaluates the performance of the Research Management Institution, to improve the research ecosystem in higher education. Through a system called simlitabmas, the evaluation produces outputs in the form of four university groupings, namely the Mandiri, Utama, Madya, and Binaan clusters (Dikbud, 2013).

The Mandiri Cluster is a group of universities with the best performance. This group is considered to have implemented a quality assurance system in research management so that it has superior research resources and very high research productivity with an international reputation. Meanwhile, the Binaan cluster has the lowest performance, and just starting to arrange resources in conducting research. Utama and Madya clusters are between Mandiri and Binaan clusters, in which the Utama cluster already has a good research management system but has not produced many research outputs with an international reputation, and the Madya is a group consisting of universities that necessary to increase the capacity of resources and research management.

Statement of the determination of universities in the group based on the results of the evaluation. The evaluation begins with data input from each university in the period determined by the Ministry. Experts assigned by the Ministry verify all data collected in the research performance database and declare it valid if the data is complete and the supporting documents required are under the provisions

Four aspects were used as the basis of appraisal of college research, including research resources, management, output, and revenue-generating (Dikbud, 2013). The indicators of the four aspects include the number of researchers, research funding, facilities, institutional, standards, publications, procedures, intellectual property, textbooks, and research contracts. The indicators used in the evaluation process have not changed since the first evaluation conducted in 2010 until the last evaluation in 2019. So 2022 is a good time to adapt and adjust the indicators assessed. The measurement of the performance of research institutions in higher education is carried out every three years, beginning in 2010 with a performance assessment in period 2007–2009. The last evaluation was in 2019, assessing the period 2016–2018. The Table 3 shows the results of performance measurement in four periods.

Table 3. The cluster of Indonesian university on research performance

No	Period (Year)	Mandiri	Utama	Madya	Binaan	Participant
1	2007–2009	10	22	71	291	394
2	2010–2012	14	36	79	772	901
3	2013–2015	25	73	160	1,219	1,447
4	2016–2018	47	146	479	1,305	1,977

Note. Reprinted from Higher education research performance report (2019), Ministry of Research and Technology/BRIN.

The latest ministry policy is different from the existing measurements evaluating the performance of Higher Education Innovation, where there are four components to be evaluated on the aspects of policies, institutions, networks, resources, and results of innovation. This measurement determines which universities have not led toward research and innovation, so that the Ministry can take the policy to accelerate downstream research (Ristekbrin, 2020a).

4. Research design and methods

The purpose of this study is to investigate the problems that arise during research performance evaluation in Indonesian higher education institutions. The research questions are:

1. What are the general problems and the current issues related to the implementation of performance measurement in research aspects?

2. What are the current issues related to the implementation of the measurement?

3. What indicators may need to be used as a basis for measuring the current policies and conditions?

4. What aspects of innovation might affect the evaluation of research performance of institutions in accordance with government policies?

This research employs a qualitative research method based on the efforts to obtain in-depth information on data, with a thematic approach. Problems that arise during the evaluation of research performance are interpreted using content analysis (Sekaran & Bougie, 2016; Weber, 1990).

The method chosen is focus group discussion (FGD) with many people to obtain in-depth information from several individuals simultaneously (Macdonald & Headlam, 2008). We conducted four stages of FGD to obtain information on the objectives of this study. Each group consists of 6–10 participants (Cooper & Schindler, 2014) from higher education research and innovation institutions with different characteristics based on the results of measuring higher education performance in the latest period. The selection of the set of higher education institutions is based on the following criteria: (1) it is located in the same country, (2) it has presented a diversity of universities, and (3) universities are in the development phase (Bosch & Taylor, 2011; Diezmann, 2018). We invited participants from Mandiri and Utama clusters (Table 4). Participants were leaders of research and innovation units at public or private universities, and we also involve polytechnics.

Table 4. FGD participant criteria and code

PHASE	PARTICIPANT CRITERIA	CLUSTER	RANKING IN LAST EVALUATION	CODE
I	Six heads of LPPM with the status of state universities (PTN) from six PTNs	MANDIRI	7; 5; 1; 14; 13; 9	P1-A; P1-B; P1-C; P1-D; P1-E; P1-F
II	Four heads and one secretary of research institutes with the status of private universities (PTS) from five PTS	MANDIRI	39; 44; 32; 22; 34	P2-A; P2-B; P2-C; P2-D; P2-E
III	One head of PTN LPPM and five PTS LPPM from 6 PT	UTAMA	54; 27; 56; 83; 61; 74	P3-A; P3-B; P3-C; P3-D; P3-E; P3-F
IV	Four heads of PTN LMI, three heads of PTS LMI, and two PTN LMI officers from 7 PT	-	8; 3; 3; 15; 11; 4; 240; 10; 1	P4-A; P4-B; P4-C; P4-D; P4-E; P4-F; P4-G; P4-H; P4-I

Note. In the fourth FGD, there were four informants from the same two universities.

The procedures carried out at each stage of the FGD are as follows (Cooper & Schindler, 2014; Creswell, 2014; Macdonald & Headlam, 2008):

• Meetings have been held virtually due to the COVID-19 pandemic;

• The discussion in the FGD has been recorded;

• The FGD began with an introduction of the facilitator and the informants;

• The facilitator starts the discussion by delivering an introductory material for the discussion, in the form of the background and objectives;

• Informants convey their experiences or opinions; and

• The discussion has been recorded for transcripts.

We analyzed The FGD results using content analysis,, by systematically evaluating the symbolic content of the recorded and transcribed results. The transcribed FGD results were encoded into similar content categories, and then relational and conceptual analyses were conducted by examining the relationship between concepts in the text (Sekaran & Bougie, 2016).

As a means for addressing ethical challenges in FGD and analysis, where there are three main issues: consent; confidentiality and anonymity; and the risk of harm (Sim & Waterfield, 2019), we take several steps. We adopted a strategy by conducting an approval process for participants at the briefing session. Then we also coded the participants for data presentation and analysis to maintain confidentiality. While the third issue is not related to the topic in this study

5. Result and discussion

5.1. Policies and general issues

There are seven issues related to the policies and general problems that are often raised in the four stages of the FGD, as shown in figure 2. These issues are the indicators used, the heterogeneity of universities in Indonesia, unclear guidelines and their socialization, the verification and validation process for data use, the career path of lecturers, the benefits of evaluation, and the use of assessment instruments.

Figure 2. Policy and general issues.

Most participants stated that the performance measuring must be adjusted to the measurement of the main performance indicators of universities, which have been set by the government. The assessment components currently used in simlitabmas still use the old system, so it is necessary to adjust to the new indicators that are adapted to the relevant policies and issues, as well as diverse universities in Indonesia (Ball & Wilkinson, 1994; Tognolini et al., 1994; Vernon et al., 2018).

The issue of heterogeneity in higher education becomes important. Chigudu and Toerien (2018) stated that diversity must be regulated to produce better performance. For example, the role of foundations in private universities related to is very significant in achieving the research output. The results of previous studies show the importance of the support of private university owners in improving the quality of research (Javed et al., 2020). Existing performance appraisals do not take this issue into account.

The issue of socialization of performance measurement implementation was also raised at the four stages of discussion. To prepare performance measurement, it is necessary to socialize the details of indicators and the effect of each point (Tognolini et al., 1994). Informative technical guidance is clearly needed since it is based on performance evaluation. A lot of data were invalid, maybe due to a lack of clear information. Data entry should be done accurately and completely, accompanied by the necessary evidence. If the information on how the filling process is not obtained by the operator correctly through technical guidelines, the possibility of invalid data will be higher. So the improvement and dissemination of the guideline are very necessary.

Another issue is the review process, which must be more thorough and precise, including how to capture the ethics behind the numbers so that the qualitative measurement is clear. Additionally, the results of the validation of performance assessments carried out in the previous period can be submitted to the assessed universities so they can determine the shortcomings and treat them as a basis for future improvement (Sizer et al., 1992; Söderlind & Geschwind, 2019).

On the other hand, there are differences in the perspectives of lecturers, university management, and the government. From the lecturer’s perspective, innovation is related to the change in the teaching workload, adding research, or more innovation. It shows that there is injustice in the assessment of individual performance in advancing the career of lecturers because the current regulations do not place a large portion of the innovation aspect in the career development so that there is a statement that innovation is not supported by lecturers but only the requests of the leaders of higher education institutions. The evaluation of research performance at research institutions in universities in Indonesia had a significant positive impact, including motivating researchers to produce better outputs (Feranecová & Krigovská, 2016). However, an evaluation will be more contributive if it was associated with performance-based funding. Therefore, evaluation quality should be maximized (Favero & Rutherford, 2019; Hazelkorn, 2008; Layzell, 1999; Sizer et al., 1992).

A topic of the discussion is that it is necessary to see how supportive the performance measurement is in improving the quality of higher education, especially by considering how relevant the national indicators are to the parameters used by global performance-measuring organizations. The performance evaluation that culminates in a national ranking is expected to be able to direct world-class universities that support the improvement of the quality of life of the community and national competitiveness. This is mainly to improve the quality of higher education, which can be measured from the results of the assessment. Many activities that have a competitive euphoria culminate in how to focus on having a better ranking or reputation than others. And in fact universities are competing to improve their performance to become a world-class universities while providing information to the public regarding their quality and position. Therefore, it is necessary to be transformed into an attitude pattern of how the universities’ ranking is more contributive in providing solutions to the nation’s problems through, research and innovation in representing realistic things.

Performance evaluation is also a part of college promotion (Feranecová & Krigovská, 2016). However, evaluation could be biased, because, before the mapping of conditions in universities, there were already universities with a high public trust, so those with good quality would be affected by the evaluation results. However, in principle, the evaluation results are essential in conveying the position and quality of higher education institutions to the public.

However, some problems can create confusion among universities due to the new policy for measuring innovation performance. Some of these instruments can be carried out in an instrument, which consists of aspects of supporting research, innovation, and community service (Tognolini et al., 1994).

5.2. Implementation of performance evaluation

figure 3 presents the most frequently discussed issues in the implementation of performance evaluation. FGD participants discussed many things related to data input time, validation process, system reliability, and data integration.

Figure 3. Issues in implementation of performance.

Data integration for similar variables from multiple evaluation systems will eliminate duplication of work and make it more effective. Synchronization is needed from the evaluation unit manager in the ministry discussing this issue. Data from institutions need to be collected at the same time and then used separately. One solution is to do a single entry, a multi-purpose internal university that emphasizes increasing data consolidation. Data interconnection is also one of the issues raised. Another problem related to data integration is that many criteria in the system are irrelevant to various policies related to lecturers.

The more parameters used, the more data are needed so that they are more representative of valid research performance (Dixon & Hood, 2016). This will be supported if it is associated with the capabilities of the information system required by the user. This means that the data entered are very important to be transferred into information and decision-making through a reliable system. One of the solutions is a web-to-web-based system that allows the system to be integrated. If assessment parameters are added to the evaluation instrument, there are consequences for improving the system to be able to facilitate this. It is difficult to determine a platform that can be accepted by all institutions, especially if it is connected to other related systems.

The performance evaluation must be accurate in providing maximum performance support (Meek et al., 2005). The current evaluation uses a data input mechanism that takes more time, while the filling period is limited so that operators in institutions focus on the filling process. Thus, the operator’s role can be different from the actual conditions. This includes clarity of the charging mechanism scheduling. This mechanism needs to be improved to increase data validity by no longer limiting data entry at certain times as if there was a data entry competition—currently, research performance is determined by physical evidence uploaded to the system, so some of data are declared invalid. Participants recommended that the input data are updated at any time, thus simplifying the data management. Filling in data entry and regularly and continuously will map the performance conditions of the university to be more realistic.

Despite the many invalidities due to input errors, the results of the validity process are valuable information for universities as input and improvements in the future. The information must also be shared with clear criteria standards so as not to raise questions from the university.

5.3. Performance evaluation component

Regarding the third research question on the performance appraisal component, several issues were raised in the FGD, such as productivity assessment, impact/benefit, weighing indicators, interdisciplinary, output quality, policy relevance, and the type of higher education, which is illustrated in figure 4.

Figure 4. Issues in performance evaluation component.

Currently, performance evaluation is measured by the cardinal size of the number of specified data indicators. It is unlikely that small universities will have better evaluation results than larger ones. It makes the results of the performance appraisal less suitable for the conditions. It is necessary to arrange parameters or weighting related to productivity aspects (Avanesova & Shamliyan, 2018; Dundar & Lewis, 1998; Marisha et al., 2017; Ramsden, 1999) because many small universities are dissatisfied with the assessment result. Productivity can be measured through the ratio, for example, the ratio of the number of lecturers with a doctorate to the total lecturers or the number of funds allocated to research, compared to the total funds of the university or the ratio of citation count to the number of publications (Diem & Wolter, 2013). Weighting the difference in the size of a university can also be an alternative to measuring performance.

The FGD participants agreed that the existing measurement instruments are good, but it was necessary to review the suitability of the indicators with the objectives and needs. In developing future indicators, it is necessary to consider aspect of long-term impact of a research as an assessment indicator, so to the instrument can measure the contribution of universities in providing solutions to the community’s problems. Many kinds of research and innovations have been carried out, such as those applied to the community through community service, which produce social impacts, and those used by the industry (economic impacts), which produce policy recommendations (Avanesova & Shamliyan, 2018; Balabonienė & Večerskienė, 2014; Zhang & Wang, 2017).

Performance evaluation can encourage higher education institutions that have not entered the QS list or other international measurements to adjust their indicators. As well as how the adjustment towards The Tridharma aspect (teaching, research, and service society) which is the primary mandate given to lecturers.

Research results can be optimized not only by managing the number of publication indicators but also by students involved, administrative officers utilized, the number of assets, and the optimization of research spending, as well as the current and future collaborative partnerships (Wood, 1990). Indicators of intellectual property need to clarify the definition used. Meanwhile, revenue-generating parameters require measurements of 1) work contracts; 2) research results from business unit; 3) business unit revenue. As for the research management parameters, the definition of variables in a study program needs to be clarified.

Participants also conveyed other issues related to the weighting of indicators in the measurement criteria related to the components in the related parameters. By categoring these indicators into several groups, the weighting become crucial. The assessment can be used as a reference for universities to improve quality, so universities can be considered as organizations with an open system that converts inputs into outputs. In this case, the categorization of indicators in the assessment can be in the aspects of input, process, output, outcome, and research impact (Abramo & D’Angelo, 2014; Brown & Svenson, 1988; Çakır et al., 2015; D’Abate et al., 2009; Dill & Soo, 2005; Hermanu et al., 2021; Savithri & Prathap, 2015; Schroeder & Goldstein, 2016).

Research collaboration is a parameter that should be considered (Bonaccorsi & Secondi, 2017; Eduan & Yuanqun, 2019; Hill, 1995; Sabah et al., 2019), not only from the aspect of the number of collaborations but also regarding the heterogeneity of research fields. Assessment of the collaboration aspect is also something that is missing in the existing performance evaluation process, so it needs to be deliberated as part of the research performance assessment The strength of this multidisciplinary network can accelerate innovation (Aldieri et al., 2019; Kobarg et al., 2017; Okamuro & Nishimura, 2012; Tseng et al., 2018). It is impossible to match the productivity between the fields of the Faculty of Mathematics and Natural Sciences, Faculty of Engineering, compared to the Faculty of Social Sciences (Abramo et al., 2014; Prathap, 2016; Sabharwal, 2013), with a coordinator who is able to communicate and connect various fields of science, education, and the role of academic institutions in increasing interdisciplinary cooperation (Parti et al., 2021). Thus, it is necessary to rearrange assessment instruments that consider the differences across scientific disciplines.

The output quality is also an aspect that has not been used in an existing assessment indicator. The system does not distinguish the quality of published in various journals, or other quality output quality measures like patents. There is no difference between patents that have been granted or those that are still registered. It needs to be formulated in comprehensive guidelines, including how to consider research results used by the industry as assessment indicator.

The indicator component is different from the enumerator. It does not mean if, for example, an assessment indicator is the number of citation publications, people compete to cite each other. Indicators become intermediate data showing something else. With the two ministries supervising the university, it becomes challenging to compile an instrument containing indicators that are in line with the policies of the two ministries.

Moreover, with a large number of universities in Indonesia, varying both in terms of capacity and trust, it is possible to find discrepancies. There could be indicators that are not even a function of a university. It is necessary to consider the division of roles in higher education, either as a teaching or a research university. Performance appraisals are tailored to their respective roles.

5.4. Innovation

The existing assessment instruments is not per the latest government policies, especially in term of measuring the impact of research that can contribute to the society. So it need a breakthrough in including these aspects, For example, the impact on the society and the economic development. figure 5 shows that issues in the innovation aspect are regional and environmental, economic, social-developmental, and related to capacity-building for lecturers.

Figure 5. Issues in performance evaluation component.

Universities need to contribute to the community and the surrounding environment. The long-term impact of research is measured by the outputs and outcomes of research that has been used by society, industry, or other stakeholders (Avanesova & Shamliyan, 2018; Balabonienė & Večerskienė, 2014; Zhang & Wang, 2017). This includes transforming research results into something challenging when applied to the world of teaching, local government, MSMEs, and the business world.

Issues that are raised in the innovation aspect include standards for evaluating innovation performance. In the current measurement of innovation performance, there is a wedge in the measurement of research performance even though the perspective is different so that these parameters can still be entered. However, what needs to be emphasized in the context of the innovation.

Research performance must be able to measure the impact of innovation on the economy and society, as well as on how universities can package patents into patents that are granted in performance appraisals. Indicators related to research results or breakthroughs that generate large amounts of money (economic impact) need to be included in the assessment (Ahmad et al., 2015; Alstete, 2014; Hannover Research, 2020). Although many innovations have been applied either directly to the community through community service or to some research results used by industry, it is necessary to emphasize the context of the extent to which downstream can be categorized as an innovation.

FGD participants have raised the issue of the importance of university involvement in research-based regional development planning. This is an opportunity that universities must take to focus on the potential of their respective regions and thus contribute to regional development planning. It needs to be classified differently in performance evaluation (Javed et al., 2020; Reichert, 2019).

Technology transfer can be designed to drive more valuable innovation in a holistic sense, including creating economic and social value, particularly in achieving the core objectives of the missions of the three universities (Madl et al., 2021). It has been argued that innovation has been accepted by the industry or society and does not have to have patented or commercialized results. Not all innovators want to commercialize their innovations, but some are dedicated to the benefit of society. As social research lags in the context of innovation, the term innovation has become more dominant in engineering, medicine, and the natural sciences. How the term innovation can become a holistic unit, between social sciences and humanities and others, is yet to be seen, which is why it is necessary to approach the humanities social research community so that they can enter various other fields. Especially to encourage technological progress, not only aspects of the natural sciences but also social sciences that have not been touched much

The downstream process is not only related to commercialization but also to how it can be carried out upstream. This is especially true with the national policy regarding “Merdeka Belajar Kampus Merdeka”, an emancipated learning program where students master various disciplines useful for improving their capacity (Dikti, 2020). This includes developing strategies so that publication results are implemented in the learning process, or developing networks, such as inviting industry to science technology parks in universities so that the downstream process can involve students. Another form of downstream linkage is incorporating research findings into community service. A new measurement model is needed regarding the aspects of lecturer self-development. The perspective is that the lecturer conducts research, which results in a role in the company, then returns and provides feedback to the campus (D’Abate et al., 2009).

6. Conclusion

This paper investigated the problems that arise during research performance evaluations in Indonesian higher education institutions. This includes analyzing general issues in the implementation of previous performance evaluations to technical matters such as the measurement process, indicators, and other problems that have arisen due to the incompetence of research management institution officials in conducting evaluations.

We designed this study to obtain input on whether the recent performance evaluation of research institutions is in line with the government policy and expectations of higher education institutions. The answers to the research questions related to aspects of general issues, implementation of the measurement, indicators used, and aspects of innovation are as follows.

The measurement of existing institutional performance evaluation model is based on measurement based on indicators of research/innovation performance indicators that are determined in accordance by the aims and objectives of the performance measuring institution. Therefore, it is highly expectation that through a good system and the right indicators, performance measurement evaluation process can identify gaps between the current condition and ideal ones to develop strategies for achieving higher education goals. Moreover, it is recommended to adapt the performance appraisal instruments to the current conditions through further research to prove the effectiveness of combined instruments. In the implementation aspect of performance evaluation, follow-up discussions are needed to integrate data and improve information systems and implementation mechanisms. Adjustment of the existing performance appraisal indicators with the policies and expectations of universities is another consideration, both in terms of the type of indicators, weighting, and factors related to the aspects of innovation, such as economic, social, regional, and environmental development impacts, as well as the impacts on capacity development.

Performance evaluation is not only a part of universities’ promotion (Feranecová & Krigovská, 2016) but can also help improve their performance and direct them to become world-class universities that support the improvements in the quality of life and the nation’s competitiveness. Therefore, the author’s curiosity regarding whether it is necessary to revamp Research Performance Evaluation Instrument of Research Institutes at Indonesian universities is answered with a recommendation to rearrange the assessment and implementation model that is adapted to conditions in Indonesia.

We believe that this study will not only contribute to the scientific development. But also assist international readers with insights regarding the research evaluation system in Indonesia, as well as policy-makers in implementing a better research evaluation system.

Some aspects that need to be targeted by them in needs analysis to remodel research evaluation systems include government policies, users, business processes and plans, problems and solutions, and equipment, as well as university goals. These are to support the process of research performance evaluation, starting from planning, technical and personnel preparation, data collection, analysis and evaluation, result feedback, and use of evaluation results.

With its various shortcomings, we also hope that there will be further comparative studies, as well as more developed performance measurement models in accordance with the Indonesian conditions. Such as an inventory of parameters that affect research performance as the basis for developing research evaluation models, deeper exploration of technical problems in the field, and even comparative studies of models applied to other countries with similar characteristics.

Correction

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was funded by the Indonesian Ministry of Research and Technology/National Agency of Research and Innovation ministry of research/ BRIN

Notes on contributors

Diana Sari

Adhi Indra Hermanu is a Ph.D student at the Department of Management and Business, Faculty of Economics and Business, Padjadjaran University, Indonesia and works at the Ministry of Research and Technology/National Research and Innovation Agency. He received a master’s degree from University of Indonesia. He is Interested in research management and policy and higher education management and policy. Orcid Id 0000-0002-4034-5906

Diana Sari, SE., M.Mgt, PhD. is an associate professor in the Faculty of Economics and Business, Padjadjaran University. She received her Ph.D degree from Monash University, Australia with an interest in Management. He is also the Director of Innovation at Padjadjaran University. Orcid id: 0000-0002-2984-3605.

Dr. Mery Citra Sondari is an Associate Professor in Department of Management and Business, Faculty of Economics and Business, Universitas Padjajaran, Indonesia. She received a Doctoral degree from Bandung Institute of Technology and a master degree from Padjadjaran University. She is interested in Human Resources and Organizational Management. Orcid Id 0000-0003-0503-3605

Dr. Muhammad Dimyati is an Associate Professor in Department of Geography, Faculty of Mathematic and Natural Sciences, Universitas of Indonesia, Indonesia. He was a Director General of Research and Technology Development strengthening in Ministry of Research and Higher Education. He obtained his Doctoral and Master degree from Kyoto University. He interested in Research Policy, Remote Sensing, GIS and Modelling. Orcid Id 0000-0003-4703-4227.