Academic research evaluation in artistic disciplines: the case of Poland

Abstract

This study focuses on the evaluation of artistic disciplines (visual and performing arts) within performance-based university research funding systems. It offers an analysis of the Polish academic research evaluation system and investigates its effects on the scholarly productivity of artists-academics. Poland has adopted a performance-based research funding system that includes evaluation of artistic productions (without considering them as ‘research’) and developed a comprehensive quantification of artistic achievements. We analyse data from the Polish Current Research Information System on 136,894 outputs submitted by 93 art faculties for two evaluation exercises (in 2013 and 2017). Our findings demonstrate a change in scholarly productivity of artists-academics towards publication practices characteristic for science-oriented disciplines. We also find that art departments located in universities exhibit a greater tendency for changing publication practices than departments located in independent art schools. The findings from this study demonstrate that both: (1) the designs of national research evaluation systems and (2) the structures of higher education may influence scholarly outcomes in practice-based disciplines.

Keywords:

• Research evaluation
• artistic disciplines
• higher art education
• Poland

Introduction

Countries developing university research evaluation systems face the challenge of accommodating practice-based academic fields in evaluative frameworks. As one of those fields, the arts (including artistic disciplines such as visual and performing arts) are difficult to assess using conventional research criteria like citations or publications, since their communication modes and outcomes are different from research in other fields. This article demonstrates how the challenge of research evaluation in the field of art has been approached in Poland. The Polish research evaluation model, which differs considerably from other country models described in the literature, is a specific case that may provide useful considerations for other higher education structures and practice-based disciplines.

Research funding systems are country-specific (Hicks 2012) and evaluation of research in the arts is closely linked with local politics of higher education and university structures (Kälvemark 2010). At the same time, a cross-country analysis reveals two general approaches to evaluation of the arts underlying national systems of research funding. On the one hand, there is the approach that certain artistic practices can be considered as research. In consequence, artistic outputs that qualify as research – as they make a contribution to knowledge through a process of investigation (Borgdorff 2012) – are deemed eligible for research funding, whereas outputs of ‘regular’ professional artistic practice are not. On the other hand, there is the ‘conservative approach’ (Strand 1998, 40) that artistic work cannot be considered, or evaluated, as research because art and research are two separate domains with distinct interests, goals and quality criteria.

The first approach underlies evaluation systems in Australia, the UK, Sweden, Norway, Finland and Belgium (Flanders). In those countries, evaluation of the arts has been influenced by national and international higher education reforms that reorganised university systems. In Australia, the Unified National System (UNS) introduced in the late 1980s brought higher art education institutions into the university system through institutional mergers between Colleges of Advanced Education and universities (Wilson 2011; Schippers, Tomlinson, and Draper 2017). In the UK, the merger of specialist art colleges into larger units and the transformation of polytechnics into universities (Rust, Mottram, and Till 2007; Butt 2017) led to the introduction of Arts & Design departments into university structures. In continental Europe, the integration of the arts into wider university systems was a result of national reforms and the widespread restructuring of higher education in the aftermath of the Bologna Process. In Sweden, Finland, Norway and Flanders, many art colleges became subject to university regulations and the harmonisation of higher education structures (Arlander 2013; Lundén and Sundén 2015; Vanlee and Ysebaert 2019).

Broadly speaking, all those reforms have resulted in the positioning of the arts in the university system and the ‘academisation’ of artistic disciplines (Georgii-Hemming, Johansson, and Moberg 2020) – that is, the adjustment of art faculties to the standards of academic research. The compliance of the arts to research standards was pushed further in countries that introduced artistic disciplines into performance-based university research funding systems (PRFSs) (Hicks 2012) – namely, in Australia and the UK. Both the Excellence in Research in Australia (ERA) system and the British Research Excellence Framework (REF) require that artistic outputs meet the official research definition (REF) or are accompanied by a convincing 250-word statement that ‘identifies the research component’ of the artistic work (ERA 2018, 21). In those systems, creative work must fit into the category of research to be eligible for funding. In countries like Sweden, Finland and Norway, where the arts are not part of PRFSs, evaluation models are driven by a more balanced, ‘artistic research’ approach, which seeks parallels between research and art rather than elevating art to the status of academic research (Biggs and Karlsson 2010).

The second, ‘conservative’ approach, which assumes separation of art and research, shapes evaluation systems in the USA, Germany, France and Poland (Kälvemark 2010; Riley 2013; Schwab and Borgdorff 2014). The conservative perspective has not sparked nearly as much interest in the last two decades as the ‘art as research’ approach, and therefore there are very few studies that investigate the evaluation of artistic disciplines adopted in those countries. As a result, we have fragmentary knowledge about evaluation of the arts in the higher education context. This article fills in the gap in the literature through an analysis of the Polish evaluation of artistic disciplines.

In this article, we provide a detailed analysis of the Polish system and demonstrate how such a designed evaluation model affects the research productivity of artists-academics. Whereas research evaluation in artistic disciplines can take different forms, including assessments of practice-led doctorates (Nilsson, Dunin-Woyseth, and Janssens 2017), or research grant applications (Hellström 2010), this study focuses on the ex-post evaluation of academic outputs within the Polish PRFS, which is the foundation of the country’s research evaluation system (Lewandowska and Stano 2018; Korytkowski and Kulczycki 2019). Very few studies have discussed the impact of PRFSs on artistic disciplines (Vanlee and Ysebaert 2019) and the Polish model provides an opportunity to explore how academic staff react to the incentives of those systems.

The arts in the Polish academia

Like in other countries, in Poland the issues of research evaluation in artistic disciplines have been closely connected with large-scale higher education reforms. Over the last three decades, four major reforms took place (Dziedziczak-Foltyn 2018) leading to the restoration of institutional autonomy to higher education institutions after the communist era (1990), Europeanisation of higher education in response to the Bologna Process (2005) and to the strengthening of the role of the performance-based research funding system and its key element – research evaluation (2011 and 2018).

The impact of those reforms on higher art education institutions has been related with the gradual assimilation of the latter to the higher education system. These processes date back to 1962 when the communist government introduced the Act on Higher Art Education emphasising that art schools (‘academies of art’) are subject to the same higher education regulations as universities (Millati 1972). The incorporation of academies of art in the higher education system was put forward after Poland’s transition to democracy. The Act on Higher Education, introduced in 1990, restored autonomy to Polish universities, liberated them from the power of the communist party and opened the academic sector to both the free market and to research evaluation of higher education institutions. In one of the opening paragraphs, the act explicitly states that the terms ‘science’ and ‘research’ used in the document also concern ‘art’ and ‘artistic production’, respectively. In this way, the document emphasised that academies of art belonged to the higher education sector both in terms of teaching and research activity.

In the first decade after the transition, the equal research status of universities and academies of art was more declarative than real. Although doctoral and postdoctoral (habilitation) degrees could be formally awarded in artistic disciplines, it was impossible to gain a doctoral degree for creative arts practice since only PhDs by written thesis were eligible. Academies of art developed their own promotion system, in which a candidate for the position of assistant or associate professor had to pass the qualification process conducted by an art faculty council. This procedure led to first-degree qualifications (an equivalent of PhD) or second-degree qualifications (an equivalent of habilitation), which were valid for life and formally gave artists-academics the same privileges as scientific degrees give university professors. However, the academic community did not perceive artistic qualifications to be truly equivalent to scientific degrees; for example, qualifications were not recognised internationally, which impacted art schools’ international collaborations. Artists-academics also felt that they were denied doctoral titles because their work was perceived as less valuable than the work of university researchers (Krasowska-Marczuk 2002).

In 2003, the government enacted the Law on Academic Degrees and Title and Degrees and Title in the Arts (Wojtczak 2019). The act introduced new academic titles – ‘doctor of art’ and ‘habilitated doctor of art’ awarded in artistic disciplines and allowed doctoral candidates to submit an ‘original artistic achievement presented in the public domain’ instead of a written dissertation (although an artistic achievement has been typically accompanied by a short written exegesis highlighting its aims and context). The legal act distinguished academic degrees in the arts from ‘scientific degrees’ awarded in all other disciplines (Zieliński and Izdebski 2015). Therefore, while the introduction of a doctoral degree in the arts has strengthened the academic profile and research standing of higher art education institutions, it has also emphasised that art schools and universities have been ‘equal but distinct’ – equal in terms of formal powers but incomparable when it came to academic practices and standards.

The distinction between art and other institutions of higher art education has been closely linked with the overall structure of higher art education. Academies of art (N = 18), which are the largest component of the higher art education sector and also include the majority of art departments (Table 1), are independent entities (operate beyond university structures). Unlike in other countries, in Poland there have been no large-scale mergers between universities and colleges of art education. In addition to academies of art, artistic training is offered by art departments operating at some Polish universities (N = 12); a comparably smaller part of the higher art education system (Table 1).

Table 1. The structure of higher art education in Poland.

Art faculties at academies of art				Art faculties at universities
Music	Fine arts	Theatre and film	Interdisciplinary	Music	Fine arts	Theatre and film	Interdisciplinary
34	35	13	4	2	3	1	7
86	13

The ‘equal but distinct’ position of higher art education institutions has been reflected in the design of the PRFS – the most widespread and systematic form of research evaluation in Poland. On the one hand, art faculties are included in the evaluation of research achievements, along with higher education institutions and research institutes in all other fields. The inclusion of the arts within the remit of PRFS secures financial stability for art faculties, as the results of research evaluation have played an increasing role in the funding of higher education institutions. On the other hand, special regulations and criteria have been designed to evaluate achievements in artistic disciplines.

The arts in the Polish PRFS

Poland is one of several European countries that developed a performance-based research funding system. The Polish PRFS was introduced in 1991 and has been used to distribute research funding among institutions of higher education. Evaluation exercises are performed every four years; the last exercise took place in 2017, covering the period 2013–2016. Since the beginning up to the 2017 exercise, the so-called scientific unit (faculty or research institute) was a unit of assessment. In the upcoming evaluation, a (scientific/artistic) discipline within the institution will be the unit of assessment. Here, we analyse the evaluation system and the results of the last two evaluation exercises conducted in 2013 and 2017. Both exercises were conducted by the Committee for the Evaluation of Scientific Units which operated from 2010 to 2019. Regulations of the 2017 exercise were a slightly modified version of the 2013 exercise but the key criteria and parameters did not change.

Whereas the criteria of evaluation within the Polish PRFS concern different aspects of academic staff’s scientific activity – including the social influence of scientific achievements and research funding obtained from external sources – final assessments of institutions depend predominantly on the evaluation of publications and artistic outputs (in artistic disciplines it accounts for 80 per cent of the final score). The evaluation of outputs is based on the ‘commensuration’ logic (Espeland and Stevens 1998), which means that faculties obtain a certain number of points for each evaluation item. The points obtained by scientific units are used to build rankings of units within a particular field of science (e.g. faculties of visual arts are compared with other faculties of visual arts) and assign scientific categories: A+ (excellent), A (very good), B (average) and C (unsatisfactory). The categories attached to units play a major role in the distribution of public subsidies to academic institutions and tend to affect other parts of the research funding system, such as external funding opportunities (e.g. research grants from foundations or research councils).

The design of the Polish system stems from several assumptions about what constitutes excellence in research and academic publishing. One of these assumptions is that journals are the most fundamental form of scholarly communication in contemporary science. Despite many authors highlighting differences between disciplinary publishing cultures and arguing that books are more renowned than journal articles in many fields of the humanities and arts (Archambault et al. 2006; Nederhof 2006; Ferrara and Bonaccorsi 2016), the Polish system awards publishing in academic journals in all academic disciplines. For example, the number of books and chapters that can be submitted for the evaluation process is limited (in the analysed period, the number of books and chapters submitted by a faculty in the humanities field could not exceed 40 percent of the total number of faculty’s submitted publications). As a result, scientific units need to submit more articles than books and chapters to increase their evaluation results. In addition, articles in prestigious, international journals receive much more points than Polish and locally oriented periodicals.

The assessment of outputs within the Polish system is not limited to written publications. Similarly to the REF in the UK and the ERA in Australia, the Polish system recognises artistic works as eligible PRFS outputs. When the first version of the Polish PRFS was established, art academies were not evaluated within the system. In 1999, the system was significantly modified and transformed into a parametric model. In 1999–2001, art academies were assigned to the committee responsible for the evaluation of the humanities and only in 2007 the arts – as an independent field – were fully incorporated in the Polish PRFS and research evaluation model for the 2009 evaluation.

The system assumes that, if research is the basis of teaching and training in universities, artistic production plays the same role in academies of art. In consequence, the Polish PRFS equates artistic outputs with scholarly publications (e.g. an original artwork disseminated abroad is equivalent to a mid-range journal article). The equivalence approach is not unique to the Polish system. In Australia, for example, the equivalence between artistic research outputs and publications was established in the late 1990s (Strand 1998), but was considered inappropriate by the artistic community and replaced by a more qualitative (peer review) approach in the mid-2000s (Schippers, Tomlinson, and Draper 2017). In Poland, where a comprehensive adoption of peer review in the PRFS is problematic due to the lack of trust in public experts characteristic of post-Communist countries (Kulczycki 2017), the quantitative approach, although perceived as an imperfect solution, has become generally accepted by Polish artists-academics.

At the same time, unlike the systems introduced in the UK and Australia, the Polish PRFS does not consider artistic outputs as research achievements. The Regulation of the Ministry of Science and Higher Education – a document which specifies the rules of research assessment and provides submission guidelines – makes a distinction between ‘research’ and ‘creative’ achievements as well as ‘scientific’ and ‘artistic’ activity. Artistic works, such as an exhibition or a musical performance, are recognised as legitimate academic outcomes and no additional written documentary about their ‘research qualities’ is required. Types of eligible outputs and their point value are listed in the Regulation (Table 2).

Table 2. Types of artistic outputs evaluated within the PRFS (regulations of the 2017 evaluation).

Group	Output description	Points
I	1) An original (major) music composition for a large ensemble (a symphony orchestra, including leading performers, choirs or electronic compositions), at least 50-min duration (disseminated through a performance, recording or published score);2) First public presentation of a musical composition for a large ensemble (by a conductor, leading performer, excluding solo parts performed by an orchestra or choir members);3) Original choreographic work of at least 40 min;4) Leading part in an opera, operetta, musical or ballet performance;5) Original artwork, multimedia work, a collection of visual art creations, conservation work including projects and documentation, disseminated in the public domain;6) Original major design of sold products or work awarded at a prestigious exhibition or in a competition;7) Input into a film production (by a director, scenographer, cinematographer, film editor or leading actor/actress), at least 70-min duration, presented in the public domain (cinema, television or gallery circulation);8) Input into a theatre production (by a director, scenographer or leading actor/actress), at least 70-min duration.	Disseminated in Poland: 15 pointsDisseminated internationally or presented at an artistic event of major public prominence: 20 pointsExcellent work (awarded in a prestigious competition): 50 points
	II	1) An original (substantial or minor) music composition (for a symphony or chamber orchestra, choir ensemble, a solo performer or an electronic composition – disseminated through a performance, recording or a published score);2) Original soundtrack of a film presented in the public domain (cinema, television or gallery circulation);3) Original (substantial or minor) choreographic work for a small ensemble or a solo performer, at least 20-min duration;4) First public presentation of a substantial or minor musical composition (by a conductor, leading performer, a team of 2–4 performers, excluding solo parts performed by an orchestra or choir members);5) A music recital (first performance);6) A symphony orchestra concert, opera or a ballet performance – by a conductor (first performance);7) First recording by a renowned record label: CD, DVD, BLU-RAY (a conductor, leading performer, team of 2–4 performers, sound director);8) Original sound component of an audiovisual work (e.g. a film, theatre performance, multimedia composition);9) Original screenplay of a dance or eurythmics spectacle;10) Supporting part in an opera, operetta, musical or ballet performance;11) Original artwork presented through gallery circulation;12) Solo exhibition in an art venue of major public prominence;13) Original major design of sold products or work presented at an exhibition or in a competition;14) Original (minor) conservation work15) A conservation project;16) Conservation or reconstruction of a historic musical instrument;17) Input into a film production (by a director, scenographer, cinematographer, film editor or leading actor/actress), 3- to 70-min duration, presented in the public domain (cinema, television or gallery circulation, excluding television shows);18) Input into a theatre production (by a director, scenographer or leading actor/actress), less than 70-min duration.19) Supporting part in a film or theatre production (excluding extras and episodes).	Disseminated in Poland: 5 pointsDisseminated internationally: 10 points
		III	1) Other artistic works disseminated through publication (e.g. exhibition catalogue), recording or presented in the public domain;	Disseminated in Poland: 2 points
			2) Production management – a film of at least 3 min, presented in the public domain (cinema, television or gallery circulation, excluding television shows);	Disseminated internationally: 5 points
			3) Curation of an exhibition.

Therefore, unlike other country systems which expect artists-academics to adjust to university research conventions and position their work as research (Bennett, Wright, and Blom 2009; Wilson 2016), the Polish system establishes separate criteria for artistic work, which is expected to meet the standards of the art world rather than the criteria of scholarly research. However, the system motivates art faculties to balance their practice orientation with the production of written publications. In the analysed period, the number of artistic outputs could not exceed 50% of the faculties’ total number of evaluated outputs; the other 50% had to be written publications. In addition, the number of books and chapters could not exceed 25% of the faculty’s written publications. Artists-academics were not directly obliged to produce scholarly publications, but artistic units were required to produce a certain amount of text-based outputs to maintain their research position within university disciplinary structures. Therefore, whereas the design of the PRFS has reflected the distinction between art and other institutions of higher education and upheld the division between art and research, it has also encouraged art faculties to render their creative practices more ‘scientific’ by producing more written outputs, preferably published in scholarly journals.

Research evaluation and scholarly productivity in the arts

Thus far, this study has focused on the accommodation of the arts in the Polish university research evaluation system. In this section, we analyse the scholarly productivity of art faculties to explore the influence of the system on the artistic and research activity of artists-academics. First, we examine scholarly productivity trends in artistic disciplines. Second, we explore factors of publication trends. We conduct an exploratory data analysis using data concerning outputs of art faculties submitted for two evaluation exercises (2013 and 2017). Information on the items submitted for evaluation was derived from the Integrated System of Information on Science and Higher Education (POL-on) which is the national Current Research Information System (CRIS) (Sivertsen 2019). In the Polish system, scientific units representing the same discipline are assigned to Joint Evaluation Groups. We extracted data concerning units assigned to the ‘Artistic Production’ group represented by faculties of music, fine arts (including design and conservation), theatre and film. Our sample included a total of 93 art faculties: 81 faculties in academies of art and 12 university-based faculties. To be able to compare two evaluation periods, we excluded faculties that were created, dissolved, divided or otherwise transformed between the evaluation exercises. Our dataset contained 136,894 outputs: artistic productions and publications (journal articles and book publications: monographs, chapters and edited volumes), registered in the Polish national database in the period 2009–2016 (Table 3).

Table 3. Number of outputs submitted by art faculties for evaluations in 2013 and 2017.

Evaluation year	Artistic outputs (N)	Publications – All (N)	Articles (N)	Articles (% of all publ.)	Books ^a (N)	Books ^a (% of all publ.)
2013	59,418	3780	584	15.4	3196	84.6
2017	69,997	3699	1076	29.1	2623	70.9

a Including monographs, chapters, and edited volumes.

Productivity trends

To identify potential changes in the scholarly activity of art faculties over time, we analysed differences between two evaluation exercises (2013 and 2017) in terms of artistic and publication output. First, output rates (output production per artist-academic in an evaluation period) were calculated for each output type: artistic output, publications (all), articles, scholarly books (including edited volumes and chapters) using the formula: (1) $$x_j=\frac{\sum \frac{o_{ij}}{N_{ij}}}{n}$$(1) where x_j is the output rate for the evaluation period j, o_ij is the number of outputs of a faculty i in the evaluation period j, N_ij is the yearly average of research staff employed at faculty i in the evaluation period j and n is the sample size (n = 93). Second, we calculated the output rate change between two evaluation periods for each output type by subtracting the 2013 output rate from the 2017 output rate. We conducted a series of one-sample t tests to evaluate whether the output rate change is significantly different from zero, indicating that there was a significant difference between the two evaluation periods.

Table 4 shows the results of the analysis. The difference between two evaluation periods in terms of artistic output is not significant but the production of different publication types changed significantly over time: the average number of articles increased (0.13) and the average number of books and chapters decreased (–0.34). This result suggests that the Polish PRFS has affected publication practices of artists-academics. Since publication of research results in journal articles is not common in artistic disciplines (Bennett, Wright, and Blom 2009; Wilson 2016), the increase in the number of articles, combined with the decrease in the number of monographs, is likely to be due to the influence of the evaluation system.

Table 4. Productivity trends: the results of one-sample t tests.

		Publications
	Artistic outputs	All	Articles	Books
Output rate in 2013 [Mean (SD)]	20.31 (15.64)	1.27 (1.51)	0.16 (0.28)	1.11 (1.43)
Output rate in 2017 [Mean (SD)]	20.17 (14.73)	1.06 (0.91)	0.29 (0.35)	0.77 (0.70)
Output rate change (Mean)	–0.15	–0.21	0.13	–0.34
t (df)	–0.087 (92)	–1.644 (92)	3.645 (92)	–2.723 (92)
p value	0.93	0.05	0.00	0.01

Table 5. Article publication factors: regression analysis.

	Estimate	Std. error	t value	Pr(>|t|)
Constant	0.010998	0.186836	0.059	0.953202
Scientific categories:
A+	1.111790	0.248322	4.477	2.36 e–05***
A	0.318459	0.146805	2.169	0.032888*
B	0.116828	0.131425	0.889	0.376575
Institution type, University	0.522543	0.132087	3.956	0.000159***
Discipline:
MUSIC	0.117993	0.128100	0.921	0.359635
THEATRE & FILM	0.240120	0.148363	1.618	0.109311
FINE ARTS	0.102830	0.129976	0.791	0.431088
Faculty size	–0.001927	0.001778	–1.084	0.281675

Signif. codes:

‘***’ 0.001,

‘**’0.01,

‘*’0.05.

Residual standard error: 0.2939 on 84 degrees of freedom

Multiple R-squared: 0.3471

Adjusted R-squared: 0.2849

F statistic: 5.582 on 8 and 84 DF

p value: 1.135 e–05

We found no significant change in the production of artistic outputs, and therefore no evidence of effects of evaluation on artistic production. This might be due to artists-academics submitting much more artistic output (on average, 20 artistic outputs per evaluation period per person) than required by the Polish evaluation system (no more than 3N outputs can enter the evaluation process).

Factors of publication trends

To explore the increased production of journal articles in artistic disciplines and investigate specific factors associated with this trend, regression analysis was conducted (Table 5). The dependent variable – journal article output – is the average number of journal articles per faculty member submitted for the evaluation in 2017 (published in the period 2013–2016). Independent variables are detailed in the following subsections.

Scientific categories

As scientific categories determine the amount of art faculties’ funding and contribute to their reputation, it is plausible that high categories increase faculty’s publishing results in the four years following the evaluation exercise. To explore the impact of scientific categories, we analyse how scientific categories awarded in 2013 affected the publication of journal articles by art faculties in the period 2013–2016. To identify the relative importance of different categories, the categories variable was treated as a qualitative variable with r = 4 levels and modelled by using r − 1 = 3 indicator variables: A+, A and B (Aczel and Sounderpandian 2008).

Institution type

Examples of other countries suggest that academic practices in artistic disciplines may depend on whether art colleges are independent or university-based entities (Wilson 2011; Harrison and Draper 2014; Vanlee and Ysebaert 2019; Georgii-Hemming, Johansson, and Moberg 2020). To explore this issue in the Polish context, the relationship between institution type and publication output was analysed. Institution type is coded as a dummy variable with a university faculty coded ‘1’ and an arts academy’s faculty coded ‘0’.

Discipline

Earlier studies have demonstrated differences in publication patterns between academic disciplines (e.g. see Kulczycki et al. 2018). Here, we investigate the relationship between different artistic disciplines and scholarly publishing. The discipline variable was treated as a qualitative variable with r = 4 levels and modelled by using r – 1 = 3 indicator variables: MUSIC, THEATRE & FILM and FINE ARTS.

Faculty size

Faculty size is represented as N₂₀₁₇ – the arithmetic mean of faculty members in the 2013–2016 evaluation period.

The regression model explains 29% of the variance in the journal article output, which suggests a moderate strength of the model fit to the data. The results of multiple linear regression show that the coefficients associated with ‘A+’ and ‘A’ categories are statistically significant. The estimated coefficients indicate that scientific categories A + and A have a positive effect on article publication: the A + category increases the faculty’s publication output (the number of articles per artists-academic) by an average of 1.11 and A category increases the faculty’s article output by an average of 0.32. The model coefficients associated with the ‘B’ category are not statistically significant.

The results of the analysis demonstrate that the publication of articles is influenced by the institution type. Being a university department increases article output by an average of 0.52. To further explore this result and compare the productivity of university faculties and academies of art, we analyse publication trends in both groups of higher education institutions. Table 6 shows output rates and output rate changes for academies of art and university faculties. Output rate change was calculated in both groups for three output types (publications, articles, books) using formula (1) and subtracting the 2013 output mean from the 2017 output mean. Welch’s t test for unequal sample distribution variance was used to calculate whether output production changes at art academies and universities were equal.

Table 6. Publication trends: Welch’s test for unequal variances.

	Academies of art			University faculties			t (df)	p value
	2013	2017	Change	2013	2017	Change
Publications (All)	1.24	0.96	–0.28	1.48	1.73	0.25	–2.445(33.485)	0.02
Articles	0.15	0.24	0.09	0.20	0.60	0.39	–3.134(14.533)	0.01
Books	1.09	0.72	–0.37	1.28	1.13	–0.14	–1.104(36.585)	0.28

The results show that there was a significant difference between art academies and universities as far as the production of articles is concerned [t(15) = –3.134, p = 0.01), but not in the case of books, edited volumes and chapters. Although both art academies and universities submitted more articles in 2013–2016 than in 2009–2012, the increase was significantly higher at university-based art departments. These results suggest that university art departments might be more influenced by the conventions of science-based research and susceptible to the impact of the evaluation system than academies of art. The finding also indicates that the influence of research evaluation on artistic disciplines depends on the institutional context and higher art education structures.

Finally, the coefficients associated with artistic disciplines and faculty size are not statistically significant. The results do not confirm the effects of these variables on the publication of journal articles.

Discussion and conclusions

Poland has a hybrid university research evaluation system, combining the conservative separation between art and research with a progressive PRFS design which includes assessment of artistic outputs. In the Polish system, artistic production is evaluated as ‘art’ rather than ‘research’, which makes the Polish PRFS considerably different from the PRFSs introduced in some Anglo-Saxon countries (e.g. Australia, the UK). By offering an analysis of the Polish system, this study complements the existing literature on research evaluation in artistic disciplines, which focuses almost entirely on artistic research and overlooks systems that separate research and art.

This article demonstrates that the design of the Polish system of research evaluation reflects the position of the arts in the Polish higher education system. In Poland, artistic production and research have been considered as equivalent, but at the same time distinct and incomparable forms of academic practice. The separation between art and research has for long been embedded in different parts of this country’s higher education, such as academic promotions and titles, structural independence of academies of art, etc. The higher art education based on the distinction between research and art can be regarded from two points of view. On the one hand, by acknowledging the disciplinary idiosyncrasy of artistic disciplines, it protects their practice orientation and enhances the rules of professional practice deeply rooted in particular art worlds. On the other hand, as pointed out by Biggs and Karlsson (2010), the establishment of special regulations and evaluation criteria for the arts may undermine their academic standing and suggest that they have academically irrelevant and arguably lower standards than other academic fields.

In this article, we also examined data from the Polish CRIS on outputs submitted for research evaluation by Polish art faculties to investigate scholarly production in artistic disciplines. The findings from this study are important for those who analyse and design research evaluation systems for several reasons. Most importantly, our analysis showed that national research evaluation systems may affect publishing practices in artistic disciplines. Our results demonstrate a change in scholarly practices of Polish artists-academics due to the influence of the evaluation system and suggest that the Polish PRFS fosters ‘scientification’ of publication practices in the arts, by which we mean a tendency to adopt publication practices typical for science-oriented academic disciplines. Our results also showed a positive effect of high-scientific categories on journal article output, which may be associated with increased funding and enhanced research opportunities linked with high categories. The findings also suggest that the PRFS may increase the gap between art faculties in research productivity since lower-graded faculties are less likely to publish in journals and increase their results in the future than faculties awarded with high categories.

The study was also useful in identifying other factors affecting scholarly publishing in artistic disciplines. We found that art faculties located in universities put more emphasis on journal article publishing than faculties in academies of art. This finding suggests that the institutional context influences research practices in artistic disciplines and that university structures enhance scientification of publishing practices. This result corresponds with earlier studies claiming that the merger of art schools into universities has led to the increased significance of ‘traditional’ research outputs in artistic disciplines (Wilson 2011; Harrison and Draper 2014; Georgii-Hemming, Johansson, and Moberg 2020). Our study corroborates these findings but is also the first one to present quantitative evidence on this issue.

The results of our analysis suggest that artistic disciplines are comparable as far as the publication of scholarly articles is concerned. Similarly, the results did not confirm the impact of faculty size on the publication output. Most importantly, we found no evidence of effects of the PRFS on artistic outputs. Perhaps an analysis of the level of particular output types could help identify changing trends and patterns. However, it is also possible that the PRFS does not significantly influence artistic practices, which might be for two reasons. First, since the evaluation rules – including the limit of outputs that can be submitted by a faculty – are influenced by scientific production norms and styles, which are very different from production norms in the arts, the Polish PRFS does not motivate artists-academics to produce more artistic outputs. Second, artists may find it difficult to play the ‘evaluation game’ and adjust their creative activity to the evaluation criteria (produce outputs assigned with more points, e.g. a composition for a large ensemble instead of a composition for a chamber orchestra). Producing a different type of artistic output usually implies producing a different piece of art or even becoming a different type of artist; in contrast, choosing a different publishing channel rarely involves doing entirely different research. At the same time, systems such as PRFS may affect artistic production indirectly and in ways not directly translatable to quantitative data. For example, artists-academics may experience tensions between spending time on artistic work and producing written outputs contributing to faculty’s research funding (Wilson 2016), which may affect their activity qualitatively, rather than quantitatively.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Kamila Lewandowska is an Assistant Professor at the Aleksander Zelwerowicz National Academy of Dramatic Art in Warsaw. Her work focuses on sociological valuation and evaluation of art in different policy contexts. She leads a research project “Between art and research: Evaluation of creative arts in performance-based research funding systems” funded by the National Science Centre, Poland.

Emanuel Kulczycki is an Associate Professor at the Adam Mickiewicz University in Poznań, Poland where he leads the Scholarly Communication Research Group. He is also a policy advisor for the Ministry of Science and Higher Education in Poland. Since 2018, the chair of ENRESSH (European Network for Research Evaluation in the Social Sciences and the Humanities). He got his PhD (2011) and habilitation (2016) in philosophy.

Additional information

Funding

This work was supported by the National Science Centre, Poland [grant number: 2019/35/D/HS5/00009].

Funding

This work was supported by the National Science Centre, Poland [grant number: 2019/35/D/HS5/00009].