Comparison of the Assessment Systems of Public Libraries in the United States and China

ABSTRACT

This work aims to compare the evaluation and rating systems of public libraries from the United States and China with a view to understand the similarities and differences and to learn from both systems. First, the main evaluation and rating systems of public libraries from the US are discussed, including Hennen’s American Public Library Ratings and Library Journal Index. Second, we discuss the Sixth National Evaluation and Rating System (SNERS) of public libraries above the county level in China. Finally, we compare and analyse the similarities and differences between the evaluation systems from the US and China. Between the two evaluation systems, there is one similarity (i.e. people-orientedness) and many differences in terms of goals, properties, subjects, processes, object classification, cycle, data sources, indicators, weights, scoring and result treatment. The public libraries of the US (a developed country) pay more attention to library services, whereas the public libraries of China (a developing country) pay more attention to a library’s basic construction and infrastructure. By comparing and analysing the evaluation and rating systems of public libraries from the US and China, this paper helps gain some insights into these different systems. We believe this will be helpful for librarians and library evaluators interested in assessing the service quality of public libraries in any country.

KEYWORDS:

• Comparison
• assessment system
• public library
• United States
• China

Introduction

Evaluation is an important part of any library’s operation as it is an effective means of library management and is crucial to its development; many scholars from all over the world have studied and proposed various evaluation systems. In China, the national evaluation and rating of public libraries above the county level has been conducted six times. Meanwhile, US scholars have proposed public library rating systems such as Hennen’s American Public Library Ratings (HAPLR) and the Library Journal (LJ) Index. However, with the era of Big Data and the rapid development of information technologies, there is an urgent need to re-examine the existing evaluation systems of public libraries so that the evaluation and rating results can be considered scientific and reasonable. As there are many kinds of evaluation and rating systems of public libraries in the world, they possess their respective merits and faults, which require further study. This paper will conduct a comparison between the evaluation and rating systems of public libraries from the US and China. The findings can help promote the improvement and revision of the evaluation systems of public libraries in general, irrespective of the country they are in.

Background

Because the service evaluation of public libraries is rooted in its own development history, let us look back at the development history of public library in the United States and China.

In the United States, since 1850, various local governments have gradually established their own public libraries (Joeckel, 1935). After the Second World War, to expand library services further within the country, the federal government began to improve legal norms and established many public libraries (Chatters, 1944). Since the 1890s, many state governments have also begun to participate in the establishment of libraries in order to expand services to their states (Kawasaki, 2011).

Although the American public library appeared very early, it was not until 1933 that the American Library Association (ALA) first published the ‘Public Library Standard’ (ALA, 1933). In 1943, 1956 and 1966, revisions of ‘Public Library Standard’ were made approximately every 10 years. These standards serve as a basis for self-evaluation and comparison with other libraries, and have had a tremendous impact on the development of public libraries. However, the American Library Association suspended the development of national standards in 1966 and began to explore new directions in the 1970s. After public libraries were established in almost all communities, in 1979, the ALA prepared a manual for the public library to make service plans. At the same time, ALA’s work focus was shifted to require each library to provide data to evaluate the service of the library objectively. This action demonstrates a change in direction from focusing on input to focusing on output, and from unified standards to providing the best services based on the circumstances of the respective community. After years of development, as of 2009, the United States has 9,225 public library administrative agencies, and the number of central and branch libraries is 16,698, according to the Institute of Museum and Library Services or IMLS (IMLS, 2011). With a population of approximately 310 million in the United States, it means that there is a library for every 20,000 people on average, and these libraries are located in all corners of the country. Therefore, it can be said that the American public library has developed to an advanced stage, in the number of public libraries that serve demand, and their evaluation standard is more focused on service outputs (Kawasaki & Zhu, 2015).

Chinese public libraries have also become commonplace for more than 110 years, since 1904. However, it was not until 90 years later, in 1994, that China conducted its first evaluations of public libraries above the county level. After 1994, China conducted five evaluations of public libraries above the county level: in 1998, 2003, 2009, 2013 and in 2017. From 1994 onward, there has been a rapid development of China’s public library industry as well as a gradual improvement of public welfare, universal equality, professionalism and the recognition of the social value of public libraries in China. The evaluation and grading of public libraries above the county level directly guides and promotes the development of the public library industry in China. By 2016, China had 3,153 public libraries above the county level. However, compared with the US, since China’s population is larger (1.39 billion) and the number of public libraries is smaller (CLA, 2017), Chinese public libraries are still in the intermediate stage in their development. China still needs to increase investment and build public libraries. Therefore, the evaluation standard of the public libraries in China is more focused on input.

This article compares the two kinds of currently popular public-library evaluation systems from these two countries. The significance of this article is to help librarians compare and develop evaluation systems to suit their own contexts.

Literature Review

Many documents have been published on the evaluation and rating systems of public libraries, including the HAPLR, the LJ Index, the state public library evaluation standards in the US, and the Sixth National Evaluation and Rating System (SNERS) of public libraries above the county level in China.

The HAPLR was first presented by Thomas J. Hennen Jr. in 1999 through a calculation and analysis of the data extracted from 15 indexes of inputs and outputs; its aim was to complement the process of comparing and rating on different libraries’ services. Since its publication in the American Libraries in 2000, it has had a major impact in this area. Many scholars responded on the evaluation’s aims, the basis of separated groups, the indicator selection, the calculation methods and the results of the application of the HAPLR. For example, Lance and Cox thought that the HAPLR lacked a clear evaluation goal and that half of the indicators were redundant or irrelevant (Lance & Cox, 2000). Scheppke believed that the basis of Hennen’s indicators was that they were easier to obtain (Scheppke, 1999). Crowley considered that a public library’s duty was not to comply with the HAPLR’s goal, but to serve as a community education centre, adding that the HAPLR was clearly shorthanded on the assessment of this institution (Crowley, 2006). Nelson believed that the HAPLR expanded the competition among libraries, thus causing damage to some libraries and the possible restriction of funds given to lower-ranking libraries (Nelson, 2007). In 2007, Lyons argued that the validity of the HAPLR as an assessment tool was not yet confirmed; moreover, the 15 indicators of HAPLR were unilateral and the HAPLR scores did not directly reflect the actual performance of a library (Lyons, 2007). Faced with these reviews, the HAPLR has yet to be updated since 2010 (Hennen, 2010a), even though Hennen stated that his rating system would be fully adjusted in the next edition (Hennen, 2010b; Hennen & Hennen, 2017a).

The LJ Index was jointly launched by Lance and Lyons in June 2008 because of the many criticisms heaped upon the HAPLR. Since then, the authors provided various explanations and descriptions of the LJ Index. For example, regarding the reasons for discarding the input indicators, Lance and Lyons explained that, on the one hand, the data on a library’s funds and collections do not accurately reflect their actual investment. On the other hand, there is a need to avoid the possibility of a library’s funds being cut off (Lance & Lyons, 2008). In the calculation process, Lance and Lyons adopted the standard deviation calculation method, which are relatively mature methods in the field of statistics (LJ, 2017a). In terms of abandoning the HAPLR’s weighting, Lance and Lyons explained that the use of weighting in a multi-index system has become increasingly questionable in the field of sociology and statistics. In addition, anyone can obtain the data on the LJ Index ratings of all libraries through the Library Journal website, for the purpose of self-evaluation or self-analysis (Lance & Lyons, 2017). One library’s managers said that it had adopted the results of the LJ Index rating in their strategic planning (Lance & Lyons, 2009). As for the LJ Index only having output indicators, Lyons (2008) pointed out that a library’s good manifestation on the output indicators can be considered a prerequisite for its achieving high quality, efficiency and excellent performance. Although the data from the Institute of the Museum and Library Services (IMLS) for the LJ Index rating were initially limited, with the expanding influence of the star library rating, the integrity of data from the IMLS already showed significant improvement (Lyons & Lance, 2012).

In recent years, some new digital and network service indicators of the IMLS have gradually been included in the LJ Index. For example, in 2016, the E-circulation per capita was included as an indicator in the LJ Index (IMLS, 2016; Lyons & Lance, 2016a). Meanwhile, Lance and Lyons adopted the Wireless Sessions from the IMLS’s indicator into the LJ Index (IMLS, 2014; Lyons & Lance, 2016b; Lyons & Lance, 2017).

Many state public library associations in the US have established their own public library evaluation standards since the 1960s (Lu, 2002). In 2003, Pennell reported that 40 states implemented their respective public library standards (Pennell, 2017). Of the 40 state public library standards, only a small part included tiered standards. Generally, these standards have many indicators. For example, the Wisconsin Public Library Standard has 144 indicators (management: 51, staff: 16, collections: 23, services: 16, facilities and access: 38) (Evers, 2010), and Iowa’s standard has 79 indicators (Wegner, 2010). Meanwhile, in 2012, a second edition of the National Standards and Guidelines for Australian Public Libraries was published, accompanied by a benchmarking calculator to enable libraries to assess their performance against the guidelines and standards (ALIA, 2012).

Due to the simple operation and intuitive result of the HAPLR and LJ Index, they have gained traction among Chinese researchers. For example, in 2010, Huan and Song first introduced both evaluation systems to Chinese librarians (Huang & Song, 2010), for third-party assessment of a library’s service. Some scholars also attempted to apply the HAPLR and LJ Index in assessing the Chinese libraries’ evaluation practices, including Wu and He (2012), Lu (2014) and Qiu (2016), respectively, introduced the evaluation objector, content, method and result application of the LJ Index. Wu, Sun and Li (2013) applied the HAPLR to rate the public libraries of 31 provinces in China in 2013. Li (2014) and Han (2014) also conducted similar studies in 2014.

In China, some articles about the evaluation of public libraries above the county level have been published. For example, Ke and Hu (2017) examined the newly added indicators of the SNERS of public libraries in China. Wu, Yuan and Hao (2014) analysed the results of the Fifth Assessment of public libraries in China.

To the best of our knowledge, no study has compared and studied the evaluation systems of public libraries from the US and China. Such a study can be helpful in modifying the old assessment systems of public libraries, enhancing the quality of service, and in promoting the development of public libraries in both countries.

Rating Systems for Public Libraries in the US

In 1999, Thomas J. Hennen Jr. created the HAPLR (Hennen, 2000). This index divides those libraries that take part in the assessment into 10 groups according to the library’s service reach within their population (i.e. less than 1 K, 1 K–2.5 K, 2.5 K–5 K, 5 K–10 K, 10 K–25 K, 25 K–50 K, 50 K–100 K, 100 K–250 K, 250 K–500 K and over 500 K). Its assessment data come from the IMLS, and the cycle is one year. IMLS is responsible for the development of relevant standards and the collection of library statistics. After each rating is completed, the top 10 libraries in every group in the HAPLR will be published on the American Libraries and more details are published in the HAPLR website.

The last edition (11th edition) of the HAPLR was published in 2010, and the indicators include six input indicators and nine output ones, as seen as Table 1. However, the HAPLR ratings were discontinued after the 11th edition. In 2017, the HAPLR was published once again, but its name was changed into Hennen’s American Public Library Research, in which the word ‘research’ replaced the original ‘rating’ (Hennen & Hennen, 2017b). Hennen named the new edition HAPLR 2.0 (hence, the old edition was called HAPLR 1.0). The new HAPLR 2.0 provides customised reports for libraries and groups of libraries, and emphasises graphic visualisations of library data, which allows libraries to compare their performance with their peers. Thus, the current paper considers the content of HAPLR 1.0 as the study object.

Table 1. Indicators of HAPLR (11th edition).

Input indicator	Weight	Output indicator	Weight
Expenditure per capita	3	Cost per circulation	3
Percent budget to materials	2	Visits per capita	3
Materials expend per capita	2	Collection turnover	2
FTE staff per 1000 population	2	Circulation per FTE staff hour	2
Periodicals per 1000 residents	1	Circulation per capita	2
Volumes per capita	1	Reference per capita	2
		Circulation per hour	2
		Visits per hour	1
		Circulation per visit	1

The calculation of the HAPLR index is as described below.

It is supposed that there were 72 libraries in the population category of over 500,000. For a given sample, let us assume that the expenditure per capita rank is 22. Since the weight is 3, the expenditure per capita score is then (72–22) * 3 = 150. The scores of the other 14 indicators are then calculated. However, this does not apply to the cost per circulation indicator, which should be calculated by multiplying the rank with its weight. Finally, the total score of the given sample library, S, is the sum of the scores of the 15 indicators. Since the total weight of the 15 indicators is 29, the index number of HAPLR of the library, H, is the total score is divided by 29, then divided by the number of libraries in the population group, and finally multiplied by 1000 (Hennen, 2010a), that is, H = (S/(29*72))*1000.

In 2008, Lance and Lyons published a paper entitled ‘The New LJ Index,’ through which they promoted a new public library rating system. Since then, the LJ Index (or Star Library Ratings and the LJ Index of Public Library Service) and the HAPLR became two popular rating systems for public libraries done once a year (Lance & Lyons, 2008). The libraries participating in the LJ Index rating must meet four conditions: (1) meet IMLS’ definition of public libraries, (2) its service area must have at least a population of 1,000 people, (3) total operating expenditure must be at least $10,000, (4) and the libraries must have submitted all data required for the LJ Index rating to IMLS (Lance, 2017b). All participating libraries are classified into nine categories according to the total operating expenses of the public library: 10 K–50 K, 50 K–100 K, 100 K–200 K, 200 K–400 K, 400 K to 1 million, 1 million to 5 million, 5 million to 10 million, 10 million to 30 million and more than 30 million. This data also come from the IMLS.

The LJ Index system uses only four output indicators for ratings, namely, library visits per capita, circulation per capita, program attendance per capita and public computer Internet use. The relevance and rationality of the four indicators has also been proven from the statistical point of view (LJ, 2017b).

The LJ Index score of every library is calculated, and the top 30 in every expense group, which include 10 five-star libraries, 10 four-star libraries and 10 three-star libraries, are rated based on the scores. Under the category of more than 30 million, only the top 15, which includes 5 five-star libraries, 5 four-star libraries, and 5 three-star libraries, are assessed. Finally, the results of the LJ Index rating are published in the ‘Library Journal’ magazine.

LJ Index of Public Library Service scores are calculated as described in the steps below. Libraries must meet four inclusion criteria to receive an LJ Index score: (1) Meet the IMLS definition of a public library, (2) Have a service area with at least 1000 population, (3) Have total operating expenditures of at least $10,000, and (4) Report the service output statistics for library visits, circulation, program attendance and public internet computer use. The score calculation algorithm of LJ Index is as below (LJ, 2017a):

•

Example:

$\frac{205,150\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{a}\mathrm{n}\mathrm{n}\mathrm{u}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{y}}{17,990\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{c}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{p}\mathrm{o}\mathrm{p}\mathrm{u}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}=11.4\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{a}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{a}$

• Step 2. For each of the nine expenditure groups, mean (average) per capita rates for the four indicators are calculated. Next, for each of the expenditure groups, the standard deviations of the four per capita indicators are calculated. A standard deviation is a statistical measurement of variation within a group of data – in other words, how widespread the data are. Example results for the $200 K-$399 K peer (expenditure) group: mean visits per capita is 6.2, standard deviation of visits per capita is 5.4.

• Step 1. For each library, per capita rates for the four statistical indicators described above are calculated. A per capita rate is defined as follows:

$\frac{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{r}}{\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{c}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{p}\mathrm{o}\mathrm{p}\mathrm{u}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}=\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{a}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}$

• Step 3. A standard score (a statistical measurement known also as a ‘Z-score’) is calculated for each of a given library’s four per capita rates. Calculations take this form:

$\frac{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{a}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{f}\mathrm{o}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{l}\mathrm{i}\mathrm{b}\mathrm{r}\mathrm{a}\mathrm{r}\mathrm{y}-\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{a}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{l}\mathrm{i}\mathrm{b}\mathrm{r}\mathrm{a}\mathrm{r}{\mathrm{y}}^{\prime }\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{e}\mathrm{r}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{p}}{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{a}\mathrm{r}\mathrm{d}\mathrm{d}\mathrm{e}\mathrm{v}\mathrm{i}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{o}\mathrm{f}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{a}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{l}\mathrm{i}\mathrm{b}\mathrm{r}\mathrm{a}\mathrm{r}{\mathrm{y}}^{\prime }\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{e}\mathrm{r}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{p}}$

Example for the $200 K-$399 K peer group: 11.4 visits per capita for this library, 6.2 mean visits per capita for peer group, 5.4 standard deviation of visits per capita for peer group,

$\frac{11.4-6.2}{5.4}=0.96\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{a}\mathrm{r}\mathrm{d}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{l}\mathrm{i}\mathrm{b}\mathrm{r}\mathrm{a}\mathrm{r}{\mathrm{y}}^{\prime }\mathrm{s}\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{a}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c}$

Note: Since the library’s visits per capita rate in this example is higher than the mean visits per capita for the peer group, the library’s standard score is positive. Had the library’s visits per capita rate been less than the peer group mean of 6.2, then the library’s standard score for visits per capita would be negative (less than zero).

• Step 4. The four standard scores from step 3 – one for each per capita rate – are summed.

0.96 (standard score for visits per capita)

+1.57 (standard score for circulation per capita)

+1.10 (standard score for total program attendance per capita)

±1.51 (standard score for public Internet terminal uses per capita)

5.14 (Library’s preliminary LJ Index score)

• Step 5. As noted in step 3, some libraries’ standard scores can be negative (less than zero), as can the preliminary LJ Index score. Among the entire 7,115 libraries rated, preliminary scores from step 4 included libraries scoring as low as −5.5. Therefore, a correction factor of six was added to all preliminary scores so that no LJ Index scores are negative.

• Step 6. The final LJ Index score is calculated by multiplying the results from step 5 by 100 and then rounding to whole numbers.

5.14 + 6.0 = 11.14 (Preliminary score plus correction factor)

11.14 x 100 = 1114 (Final LJ Index score).

The National Public Library Evaluation System in China

The evaluation and rating of national public libraries above the county level in China began in 1994; since then, it has been conducted six times. The evaluation object is the provincial (deputy provincial), city and county-level public libraries. The SNERS of public libraries above the county level was conducted from April 2017 to April 2018. The assessment data came from 2013 to 2016, in which the average of the four-year data was taken as the final evaluation basis. The evaluation and rating process was carried out by the Ministry of Culture of the People’s Republic of China (MCPRC) and the provincial (city) cultural offices (bureaus).

An information platform was first built for this evaluation (Ke & Gong, 2016). Each participating library prepared its own data and uploaded them to the information platform. Then, an expert group appointed by the MCPRC evaluated the data online, after which they evaluated and verified the data on site. In this process, the participating libraries can also give explanation and feedback. The information platform effectively compensated for the heavier task of the previous evaluation, which used to result in experts not carefully reviewing the data. The platform also enhanced the digitisation level of the evaluation and facilitated mutual learning between these libraries.

The criteria for the SNERS tiers first begin by dividing the libraries into three groups by administrative level, after which they are subdivided into the Eastern, Central and Western areas. The ultimate result of the competition is to identify Tier 1, Tier 2 and Tier 3 libraries in each group. After many discussions and revisions by the expert group, the SNERS resulted in a more streamlined process than the previous evaluations.

The SNERS of public libraries above the county level in China includes a prerequisite and an evaluation criterion. The prerequisite has three kinds, such as the prerequisite of Tier 1 libraries, prerequisite of Tier 2 libraries, and so on, changing with different administrative levels (provincial, city and county) and areas (Eastern, Central and Western). This paper takes a prerequisite of provincial public library of Tier 1 as an example to illustrate this issue, shown in Table 2. The content of the prerequisite consists of three parts: service benefit, business construction, and safeguard conditions. It also has eight indicators, which can change with different areas.

Table 2. Prerequisite of the provincial public library of Tier 1.

Part	Indicator	Eastern	Central	Western
Service benefit	Annual literature loan (10,000 copies)	110	90	70
	Annual amount of reading promotion activities(times)	25	22	19
	Reader satisfaction rate (%)	85	85	85
Business construction	Service system planning and co-construction sharing in the region (score)	20	20	20
	Business statistics analysis (score)	20	20	20
Safeguard condition	Annual financial allocation amount (million Yuan)	60	50	40
	General literature collection amount (10,000 copies)	450	350	250
	Construction area (10,000 M^2)	5.5	4.5	3.5

Moreover, the evaluation criteria of the public libraries in China has three different forms according to different administrative levels. The evaluation criteria of provincial public libraries are taken as an example, shown in Table 3 (Office of MCPRC, 2017). It has 3 parts, 30 primary indicators and 114 secondary indicators. For the sake of brevity, all the secondary indicators are omitted. The total basic score in the three parts of the provincial library evaluation criteria is 1000, and the additional score is 500. The basic score and additional scores of the evaluation criteria of the public library in three forms are all different.

Table 3. Evaluation criteria of the provincial public library.

Part	Primary indicator	Basic score	Additional score
Service benefit	Basic service	50	35
	Minors and others special group service	20	0
	Reading promotion and society education	65	25
	Reference service	35	30
	Network resource service	25	20
	New media service	20	10
	Service management and Innovation	25	20
	Reader evaluation	60	10
Business construction	Collection structure and development policy	30	10
	Cataloging and collection organisation management	50	35
	Digital resource construction	20	5
	Local literature work	35	0
	Construction of public library service system in the region	40	5
	Library industry collaboration and social cooperation	25	10
	Key cultural project	40	0
	Grassroots counseling and society work	35	10
	Administration and human resource management	30	0
	Finance, assets, and archives management	15	0
	Safety and environmental management	15	0
	Business management	30	5
	Business research	20	50
	Organisational culture and award	10	20
	Socialisation and management innovation	5	50
Safeguard condition	Policy and legal safeguard	30	15
	Constitution and planning	10	5
	Funding safeguard	70	40
	Document resource safeguard	55	45
	Library building facilities safeguard	35	20
	Information infrastructure safeguard	70	5
	Personnel safeguard	30	20
Total		1000	500

In the SNERS, if a library is evaluated as a specific tier library, it must meet that tier’s prerequisites. In addition, all indicators of this evaluation system are integrated with the library’s goals. Meanwhile, local government can also determine their responsibilities through these indicators.

After the SNERS, the MCPRC presented the following results: of the 2,994 public libraries above the county level that participated, 2,521 libraries were rated, comprising about 84.2%. Of these, there were 953, 501 and 1,067 Tier 1, Tier 2 and Tier 3 libraries, respectively (MCPRC, 2018.).

Comparative Evaluation and Discussion

This section compares and discusses the three evaluation systems. After the three evaluation systems were carefully studied, some similarities and differences were found and were analysed and compared point by point. Finally, these similarities and differences are summarised in a list.

The Similarities between the Evaluation Systems

The attribute of being ‘people-oriented’ is highlighted during the evaluations carried out in both countries. Many indicators in the HAPLR, LJ Index and Chinese public library evaluation system are all focused on providing good services for the users. To achieve a star rating or a tier library status, librarians have devised various methods through which patrons can better use the library resources.

The authors of this paper visited websites of some five-star public libraries that were rated based on the LJ Index, which included Avalon Public Library (2018) and Red Hook public library (2018). These libraries offer specific services for children and adults, such as access to the website for users of different levels, a summer reading bibliography, and a host of regular activities. Clearly, these libraries supply different services according to the users’ varying cultural types and levels of demand, indicating that these libraries are truly people-oriented.

Meanwhile, the service targets of public libraries in China comprise various demographic groups such as minors, adults, the elderly and people with special needs. In fact, the participating libraries in China have done much work to provide non-discriminatory services. Famous examples include the Hangzhou Library being open to the homeless, and the Nanjing Library setting up a special reading room for blind and deaf users, indicating that they are people-oriented.

Differences between These Evaluation Systems

As the US is a developed country, and China is a developing country, it causes many differences in the rating systems of public libraries from the two areas. These are described in detail in the following sections.

Goals and Properties

The goals of the evaluation systems from the US and China are different. After every evaluation, the HAPLR gives detailed data and information about the evaluation, such as the total score and total ranking of the participating libraries, and ranks the top 10 libraries in every category. The LJ Index rates the five-star, four-star and the three-star libraries on the nine funding groups, respectively. The five-star libraries have advantages over all other libraries. Both the HAPLR and the LJ Index aim at guiding public libraries to improve their own social influence and service benefits through the star library rating.

During the SNERS of public libraries above the county level in China, the MCPRC proposed a slogan: promoting the construction, management and use of the library through evaluation. It aims to comprehensively reflect and measure the financial input, management and output benefits of the public libraries, so that these libraries achieve progress and continue to develop amidst a better environment (Li & Shen, 2014).

In terms of the systems’ properties, both the HAPLR and LJ Indexes are simple rating systems, whereas the SNERS of public libraries in China entails a complex evaluation and rating process.

Assessors and Process

The assessors of the public library evaluation systems from the US and China are different. The assessors of the two rating systems from the US (HAPLR and LJ Index) are third-party institutions or persons. Usually, the third-party evaluation organisation demonstrates the objectiveness and fairness of the evaluation results.

The HAPLR establishes a comprehensive rating method for public libraries and publishes the evaluation results on American Libraries to improve the public libraries’ service quality. The assessment not only gives the final ratings, but also provides various types of analysis reports by comparing a library with other libraries, to find deficiencies and improve their investment and service quality. Based on the data from the IMLS, the LJ Index follows the evaluation criteria that have matured for many years, and calculates the final score according to its own calculation method and process. The final rating result is published in the Library Journal.

In comparison, the assessor of the SNERS in China is the MCPRC. It established the evaluation system, built an information network platform, and conducted the evaluation. In the beginning of the evaluation, every participating library supplies its own data or materials. Next, online evaluations are done by expert groups appointed by the MCPRC. The expert groups also conduct on-site evaluations and data verification. The building of the platform improved the efficiency of the preparation works and the digitisation of the assessment data. It is conducive to information communication and evaluating the merits of these libraries. The evaluation process of the SNERS in China is far more complex than the processes of the HAPLR and LJ Index.

Classification of Quality Indicators

The classification methods of the public library evaluations from the US and China are different. The HAPLR uses the population of the service area to classify the libraries, whereas the LJ Index classifies based on operating expense. The classification based on population served can better embody the fairness of social resource allocation and the principle of equalisation of information services, whereas the division by expense can reflect the service quality of public libraries. According to different administrative levels and areas, the public libraries in China can be classified into about nine groups.

Evaluation Cycle and Data Source

The star rating of the HAPLR and LJ Index is conducted once a year, and the evaluation is based on the annual data from the IMLS. The annual evaluation is beneficial for each participating library to ensure healthy competition and enhance the activity of the libraries. The data from a third-party institution ensure the impartiality and objectivity of the evaluation results. In addition, this evaluation also plays a role in maintaining the integrity and accuracy of the IMLS data.

The evaluation of public libraries in China, on the other hand, is carried out once every four years, and its evaluation data, which are verified by an expert group, come from every participating library. Clearly, the cycle of four years is too long. Although there is sufficient time to prepare for the next assessment, it is impossible to assess any oncoming crisis in the participating library and track the libraries in time. In addition, as China’s assessment cycle features both a trial evaluation and a formal evaluation, it consumes significant labour and material resources during the process.

Indicators

The evaluation systems from the US emphasise the output indicator or service benefit. The HAPLR has nine output indicators and six input ones. It can be said that HAPLR puts more attention on outputs or service benefits. Meanwhile, the LJ Index has four output indicators and no input indicator. Clearly, the LJ Index emphasises the library’s service benefit. Thus, it can be said that the evaluation systems from the US pays more attention to the service benefits offered by the libraries.

In comparison, the SNERS of public libraries in China emphasises input indicators or a library’s basic infrastructure. The indicator system is large and covers a wide range. The evaluation system of the public library in China has 30 primary indicators, which include 22 input indicators (business and) and eight output indicators (service benefit), indicating that the Chinese system pays more attention to a library’s basic. As China is a developing country, its libraries require additional infrastructure projects. It seems that the libraries in developed countries focus more on being service-oriented, but those in developing countries pay more attention to their basic infrastructure. However, too many indicators will increase the workload and complicate the assessment.

Weight and Scoring

Every indicator in HAPLR is assigned a weighting, but the indicators of the LJ Index have no assigned weighting. In China’s evaluation system, the indicators have no weighting either. Clearly, there are some differences in the setting of indicator weights among the three evaluation systems.

Further, in the HAPLR, a special calculation method, multiplication and division are used in its scoring, whereas in the scoring algorithm of the LJ Index, a statistical calculation algorithm is used based on four statistical indicators (Lance, 2017a).

The basic scores and additional scores corresponding to each indicator in the SNERS system are very clear. Once these scores are given by the expert group, the final scores of the library are the direct summary of all scores. Thus, the calculation methods of the HAPLR and LJ Index are more complicated than the Chinese evaluation system. In addition, the calculation method of the LJ Index is the most complicated, for it uses a statistical algorithm.

Results Dissemination

The results of the HAPLR are mainly published in American Libraries. The top 10 libraries in the same category can have the official logo of the HAPLR containing the words ‘A Top Ten American Library’ on its library website as a sign of honour (Hennen, 2010c). Meanwhile, each participating library can obtain a complete analysis report of the assessment, and a comparison report with the same category at a certain fee. Most public libraries consider this kind of analysis report as having a significant role in improving their service capabilities.

The results and other featured services of the LJ Index are published in the Library Journal. Of several thousands of participating public libraries in the US, only about 255 libraries can be rated a Star Library, indicating that there are more participating institutions than the star-rated libraries. The results can motivate non-rated public libraries to improve their management. Moreover, the LJ Index even provides a detailed report on the results and process of each new assessment and publishes the result on the magazine.

In contrast, the evaluation result of public libraries in China is published on the MCPRC website. The result only includes the names and numbers of the rated libraries, but the public cannot find any other details about the assessment. In the SNERS, the MCPRC gave a result as follows: Tier 1 libraries: 953, Tier 2 libraries: 501 and Tier 3 libraries: 1,067. Moreover, because the publishing style is relatively simple, the degree of exposure and publicity garnered by the results is lower. However, in order to allow a large number of grassroots public libraries to be built, MCPRC sends a letter, which includes detailed scoring results and correction requirements, to each participating library. Third parties or the public do not get to know the content or details of this letter.

A summary of the above discussions is presented in Table 4.

Table 4. Characteristics of public library evaluation systems from the US and China.

	HAPLR	LJ index	SNERS
Focus	People-oriented	People-oriented	People-oriented
Goal	Social influence, service benefit	Social influence, service benefit	Construction, service benefit
Property	Rating	Rating	Evaluation and rating
Process	Simple	Simple	Complex
Object sort	Service population	Operation expense	Administrate level, area
Cycle	1 year	1 year	4 years
Data sources	IMLS	IMLS	Library
Indicator amount	15	4	30 primary
Indicator property	6 inputs, 9 outputs	4 outputs	22 inputs, 8 outputs
Weight	Has	No	No
Scoring	Multiplication and division	Standard deviation	Direct addition
Result treatment	Publish detailed results in American Libraries	Publish detailed results in Library Journal	Publish tier results on MCPRC webpage
Current condition	Discontinued	Using	Using

Insights

From the above discussion, we can obtain some insights. For example, in China, the construction of different public libraries at the same administrative level and area are different, and even the gaps are very large. Therefore, they should be further divided under the same administrative level and area, and funding or population can be included as indicators as well. This way, the public libraries in the same group with the same basic conditions can be evaluated and rated as fairly as possible.

The coverage of the rating indicators of US public libraries is not wide enough, and factors such as reference services, cooperation among libraries, and so on are not considered. The rating system of public libraries from the US is more comprehensive.

Overall, public libraries in developing countries pay more attention to basic infrastructure, whereas those in developed countries pay more attention to the provision of better library services.

Conclusion

The evaluation systems of public libraries from the US and China are different, but they are adapted to the respective conditions and development of their own libraries. Thus, each system has its unique features. In the future, by analysing other evaluation systems and comparing them with the systems from the US and China, we can gain further insights on how to improve the library services.

The authors of this article suggest that librarians from developed countries may wish to learn or refer to the rating systems of the HAPLR and LJ indices. The indicators may be set up as 5 input indicators and 10 outputs. In addition, interested librarians from other developing countries can learn the SNERS of public libraries in China. However, its indicators are numerous and the rating process can be overly complicated. Thus, librarians may first reduce the indicators, or only save about 10 input indicators and 5 outputs before adopting the SNERS.

Public libraries have an important social responsibility and mission to eliminate the information gap, promote cultural education and ensure social progress (Wang, 2010). Therefore, an evaluation system of public libraries should guide libraries to focus on basic construction and service quality and also enhance the initiative and enthusiasm of the librarians to improve customer service.

The social influence of a public library should also be expanded, and the transparency of the evaluation should be increased. At the same time, feedback from the public should also be accepted to help public libraries improve their services and social recognition.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This research is supported by Social Science Funds of Jiangsu Province (CN) (17SYC-117, 18SWC-30); Fundamental Research Funds for the Central Universities (CN) (SKGL2017014), Educational Information Funds of Jiangsu Province (CN), 20180033, Funds of China Agricultural Society, PCE1812.

Notes on contributors

Haiying Shao

Haiying Shao is an associate professor, and a biology librarian in the Nanjing Agricultural University Library, China. She earned her master’s degree from Nanjing Agricultural University. She has published about 10 papers.

Qun He

Qun He is an assistant professor, and a chemistry librarian in the Nanjing Agricultural University Library, China. She earned her master’s degree from Nanjing Agricultural University. She has published about 10 papers.

Guiting Cha

Guiting Cha is a professor, and an economics librarian in the Nanjing Agricultural University Library, China. He earned his Ph. D. from Nanjing Agricultural University. He has published about 30 papers.

Qingkui Xi

Qingkui Xi is an associate professor, and a physics and information science librarian in the Nanjing Agricultural University Library, China. Qingkui earned his Ph. D. from Nanjing University of Science and Technology and his master’s degree from Nanjing Agricultural University, China. He has so far published about 30 papers and three patents.