Institutional Repository Interaction With Research Users: A Review of Current Practice

Abstract

The article reviews research that has examined scholarly users and institutional repository interaction within the wider scholarly communications environment. The focus is on research users as repository content creators and as eventual content users. The text explores how institutional motivations for implementing repositories match against user needs, and how consultation with users might be conducted. Some examples of innovative tailored services resulting from user needs analysis are described. The benefits of early consultation are highlighted, as well as the importance of tailoring advocacy to the needs of specific scholarly subject contexts. Understanding and engaging users mean that the benefits of repositories are more likely to be more fully realized. The article then sets out some of the current and future challenges for repository development. This includes briefly looking at opportunities for institutional and subject repositories to work together in complementary ways and consideration of research data requirements. Finally, the key area of integration is considered, first, in terms of embedding repositories in research practice, so that they become part of the researcher's daily work environment; and second, repository integration with other institutional information systems is explored to enable the sharing of repository content across other services.

Keywords:

• institutional repositories
• researchers
• scholarly communication
• user requirements
• workflows

Introduction

It is very important that institutional repository projects consult with all potential users and stakeholders prior to implementation. Perhaps above all, it is essential that repositories engage with what Aschenbrenner et al. considered to be their most important users, the researchers who are expected to deposit content in them and be major users of repository aggregation services. The consequences of any lack of consultation can be low levels of repository use and content deposition, resulting in potential problems for repository managers further down the line. Thus, while the benefits of repositories may seem persuasive to the institutions developing them, a lack of engagement with researchers may make them appear less compelling and useful to authors and other owners of content.

Repositories have many different potential users (both human and machine) with overlapping (and perhaps contradictory) requirements. However, this article has a focus on scholarly research users: first, in their role as authors and content creators for institutional repositories; and second, at the other end of the process, as users of that content (via searching and resource discovery). Institutional repositories are the primary focus, although there is brief discussion of interaction with subject repositories. In considering eventual content use, the focus is on usefulness for scholarly research, rather than usability.

The text is primarily a literature review of research user involvement in the development of institutional repositories. It explores current practice as described by practitioners and academics, and investigates the extent to which researcher requirements are sought and subsequently analyzed to influence implementation. Evidence gathered by UKOLN and partners in the JISC-funded Repositories Support Project case studies is included to illustrate best practice. The review also reveals gaps where little information is available such as how repository content is used by researchers. However, in the future, content could also be used by other services if repositories become integrated into the wider information context, as evidenced by several pioneering initiatives.

Because the repository's primary mission is to disseminate the university's or research institution's primary output, some repositories have seemed to forget that researchers—rather than institutions—are the most important users of repositories (Aschenbrenner et al.)

Methodology—Literature Review

Literature searches were carried out over an extended period from mid-2009 until early 2010. Abstract and indexing services were used such as Library Information Science and Technology Abstracts, as well as publisher search services including Emerald. Google Scholar was also searched. Electronic journals which were known to regularly contain relevant articles were searched directly—primarily D-Lib Magazine and Ariadne. Recognizing that much valuable material was likely to be contained in blogs, project reports, and other web resources, a number of standard Google searches were carried out; this was also useful because some of the most relevant peer-reviewed articles reported on older research. Useful context was often provided by discussions on the JISC-REPOSITORIES list. Initial search terms used included “repositories,” “researcher,” “user,” and “needs.”

Researcher Needs and Repositories

The first section explores how differing motivations may have negatively affected repository implementations and outlines efforts to improve consultation processes, resulting in tailored approaches.

Institutional Motivations and Researcher Needs

The initial focus on repositories as agents of reform for the scholarly communication systems and as a means of improving institutional visibility (Crow) has meant that the impetus for developing them has mainly come from institution-wide services (like libraries) rather than from direct lobbying by research staff or academics. The practical motivations for setting up an institutional repository are many and varied, but frequently include:

• Providing a showcase for scholarly output from the institution (e.g., facilitating increased visibility; generating indicators of academic quality);

• Improved dissemination of research outputs;

• The management of research (and research information);

• The long-term preservation of resources; and

• Breaking down access barriers to content (i.e., reforming the scholarly communication system).

It would be interesting to consider how these institutional objectives might match against researchers’ interests. Increased personal visibility and dissemination are generally considered to be the most important motivators for researchers to deposit in repositories (Fry et al. 45; Daoutis 3; Pickton 60). However, Aschenbrenner et al. assert that dissemination ranks relatively low as a trigger for use, largely because the existing journal system is seen as the most reliable route to career advancement. Instead, they see preservation as a much higher priority, especially for valuable and unique data, and, most important of all, mechanisms for collaboration across institutions and across disciplines.

While research staff have an obvious interest in making research outputs freely available to the widest audience, there has long been a perception that their inertia might be the greatest obstacle to developing new paradigms of scholarly communication, like repositories. In 2002, Crow commented that academic authors “publish for professional recognition and career advancement, as well as to contribute to scholarship in their discipline” (21). He added that it would be important to demonstrate to academics that repository plans and policies would be responsive to their specific needs and perceptions.

Despite this, the response of research staff to the existence of repositories has been largely disappointing. For example, with regard to deposit, it has been estimated that less-than 12% of published articles are openly available in repositories or on personal web pages, although it should be noted that there is considerable variation depending on subject discipline (Björk, Roos, and Lauri 9). Slightly more positively, Gargouri et al. (3) suggest that “self-archiving” has a spontaneous self-selective baseline of 15%. The slow growth of content in repositories has negative consequences for the development and use of content, both by researchers and by the third-party services that harvest metadata. A recent report for the PEER (Publishing and the Ecology of European Research) project concluded that, “despite the infrastructure for institutional repositories being in place, the acquisition of content has been slow and consequently current uptake by readers has also been slow” (Fry et al. 12).

One response to the apparent lack of researcher interest in repositories has been to advocate the adoption of self-archiving mandates by institutions and funding bodies. It has been observed that these mandates—where they exist—generate repository deposit rates well in excess of the 15% baseline (Gargouri et al. 3), but they are beyond the immediate scope of this paper. Instead, this paper will review selected attempts to develop repository advocacy with specific reference to researcher needs. This has become an increasingly important aspect of repository development, perhaps exemplified by Heery's recommendation in the Digital Repositories Roadmap Review (2009) that the JISC (Joint Information Systems Committee) should “analyse current communication behaviour of researchers and teachers, and involve them in development of future scholarly communication services” (4). This has also become a priority for a number of individual repositories. With reference to the Oxford University Research Archive (ORA), Rumsey has commented that a “great deal of effort is … being expended to investigate user needs and obtain feedback on how development should be prioritised in order to fulfil those needs” (3).

Exploring User Requirements

An informative study at the University of Rochester in 2005 explored the apparent misalignment between the benefits and services of an IR with the actual needs and desires of research staff (Foster and Gibbons). The authors conducted a work-practice study where researchers were observed carrying out the usual tasks associated with their work, combined with a series of interviews. The researchers concluded that their key finding seemed obvious in retrospect “what faculty members and university researchers want is to do their research, read and write about it, share it with others, and keep up in their fields.” The study revealed that research staff did not perceive the relevance of the IR as it was described and promoted by institutional staff running the repository. One conclusion from this might be that repository advocacy needs to be tailored to scholarly contexts using language that is meaningful to individual or group cultures.

Foster and Gibbons identified a further reason related to language which helps to explain lack of research user engagement: “The term ‘institutional repository’ implies that the system is designed to support and achieve the needs and goals of the institution, not necessarily those of the individual.” Researchers are perhaps not, therefore, made to feel that the repository is the optimum place to showcase their own individual work. Repository managers therefore need to identify ways of highlighting the benefits to all individual stakeholders. Gandel, Katz, and Metros take this a stage further and suggest that the emphasis should be on “personal digital repositories” rather than on institutional repositories.

The Rochester study was particularly useful in identifying many key and ongoing issues for institutional repositories. Some of these are explored further in the following sections.

Consulting on Researcher Needs

In common with other system implementations, user needs tend not always to be solicited at the beginning of a repository set-up project. The University of Rochester study was carried out after the repository had been implemented (when the “if you build it, they will come” approach was seen to have failed). It is notoriously difficult to involve end-users in development projects. A “chicken-and-egg” situation applies—how do research users know what they need when they do not know what services might be available? This was experienced by the Oxford University Research Archive:

With hindsight it might have been better to precede the two year implementation period with a longer period of fact finding and planning. Attempting to plan and implement concurrently is not easy when dealing with such a large and complex institution… Asking users what they require can be difficult if they do not actually know what they want or what is possible (Rumsey 5).

Sometimes the specification of user-requirements becomes subordinate to the related activity of stakeholder analysis. While, for example, the technical framework of the JISC Digital Repositories infoKit recommends identifying stakeholders when specifying requirements, it does not specifically suggest consulting researchers to gather information about their needs at this stage. The management framework does, however, suggest mapping out stakeholder “needs and aspirations.”

It is perhaps worth briefly considering usability investigation, which tends to follow a similar pattern to requirements identification. A recent UX2.0 project report comments that “usability studies and digital library development are not often intertwined due to the existing cultural model in system development. Usability issues are likely to be addressed post-hoc or as a priori assumptions” (Paterson and Low 2). However, in many cases users could be involved in both usability and requirements analysis at a much earlier stage. A priori assumptions are also very common in requirements documentation. In both of these work situations, earlier consultation may result in earlier (and increased) adoption by research users.

As one of the pioneers of developing open access repositories for academic research, the University of Southampton did spend some considerable time gathering information about researcher needs prior to implementation (Hey). This involved the examination of existing methods of managing research output within different communities across the University. Significantly, the aim was for a service that would add-value and save time rather than entail extra work for researchers (e.g., the ability to input data once only and to use this for multiple outputs). As a result a “user-centred route map” was drawn up: “The route we are taking has been driven by our users and even if it is a more circular route than we might have first supposed it will be the more sure for taking it” (Hey). Again, project funding (from JISC in this case) enabled this kind of research to be carried out.

More recently, the University of Surrey has also reviewed publication and dissemination practices across individual schools, departments and research groups, as a first step towards identifying cultural differences: “Our findings very much supported a pattern observed in similar reviews, namely that research outputs and the way they are produced and disseminated vary widely across disciplines” (Daoutis 54) An author survey fulfilled three functions: as an advocacy tool, a way to identify key issues to be addressed, and as a recruitment tool for early adopters.

There are indications that learning from earlier projects and initiatives has taken place (Maness et al.) and that some implementation projects have begun to place users at the beginning of the process. One example is the JISC-funded Building the Research Information Infrastructure (BRII) project at the University of Oxford: ‘Understanding our users is a crucial first step to achieving acceptance and adoption’ (Loureiro-Koechlin Institutional Repositories). BRII aims at developing an infrastructure that will collect and re-use data about research from a range of services across the university.

Therefore, it can be seen that repository project managers will need to consider the exigencies and constraints of their own individual implementation, in order to decide the optimum time to consult scholarly (and other) users. Largely user-driven projects are likely to take longer to set up initially, although their longer term success is probably more assured. Bolting-on user consultation in order to re-orientate the project after it has failed is clearly sub-optimal.

Responding to Researcher Needs

Once specific user requirements have been identified, it will be necessary to identify ways of implementing these within the institutional context. A good example of this would be the “Researcher Pages” developed at the University of Rochester for their repository. Anthropological surveys of academic staff and PhD students at the university (Foster and Gibbons; Randall et al.) showed that they were primarily interested in knowing how their work was being used and in having an online space where they could showcase work and undertake collaboration. Researcher Pages enabled researchers to generate online CVs that were fully integrated with the repository (as well as repositories elsewhere) and able to provide statistics about downloads and other usage. Further changes included the involvement of subject librarians, who were able to use their subject understanding to recruit content using a personalized approach, and working with an “early-adopter group” to network to colleagues.

Having consulted users at an early stage, the University of Southampton also adopted a tailored individual approach to advocacy using early-adopters. Instead of traditional publicity using posters and flyers, “a personal approach was used, where a relationship was built with the people who were interested in the repository. Then it was often the case that these people sold it to other people” (Zuccala 29). The success of this method was revealed in a user survey, where 43% of e-Prints Southampton users indicated that they had learned about the existence of the repository from a colleague or friend.

In conclusion, it is worth highlighting some of the critical factors which have emerged in section one, which may enable a more effective response to user needs:

• early consultation with researchers,

• demonstration of responsiveness to identified needs,

• tailoring to scholarly context, and

• consideration of the importance of preservation as a motivator for deposit.

Challenges

The second section turns to the challenges that institutional repositories face as they respond to user expectations for resource discovery, seek to interoperate with complementary external services, and also position themselves within the wider institutional environment.

Institutional Versus Subject Repositories

It has often been argued that since many researchers are focused on very specific academic interests, and work within (potentially small) international research communities with shared interests, it is counter-intuitive to expect them to be interested in depositing their scholarly output in an institutional repository which has no specific subject context. However, it is increasingly being recognized that the locus of actual deposit may be irrelevant; repository content can easily be accessed using generic search tools like Google or specialist repository portals like OAIster or DRIVER. Additionally, flexible repository architectures enable multiple deposits to be made, that is, enabling content (or metadata) to be deposited in either or both institutional and subject repositories. With sufficient high-quality metadata, resources are likely to be discovered regardless of location.

There is scope, however, for subject and institutional services to work together in complementary ways that are able to build on their respective strengths. So, for example, subject-based repository search services could be built based on metadata harvested from multiple institutional repositories using OAI-PMH. Rowland et al. (2004) recommended a model in which full-text articles remain in distributed archives—whether institutional, subject-based or national—but that subject-based services could be developed from the metadata harvested from these archives. Economists Online is a good example of this kind of service, harvesting its content (including datasets and enriched metadata) from 22 institutional repositories across Europe.

Subject-based repositories may have benefits for certain classes of material, for example, research data. At the Subject Repositories Conference held at the British Library in January 2010, Clifford Lynch suggested that while institutional repositories may be financially more sustainable than subject repositories, subject repositories could be better positioned to manage research data within disciplinary structures rather than institutional ones (Puplett). A possible alternative might be to involve researchers’ departments in a more direct way. For example, the JISC Keeping Research Data Safe report concluded in 2008 that federated approaches involving academic departments may be more sustainable than institutional repositories for the management of research data (Beagrie 69–70). The reasons for this are mainly cultural. Citing a University of Southampton case study, the report comments that the academic department is “an academic's natural affiliation and an environment they understand and can often have an influence on” (Beagrie 70). Furthermore, research data on a whole-institution level is likely to be extremely diverse, and its specific metadata requirements are unlikely to be handled very well by typical repository software or OAI-PMH.

The Use of Repositories

It is, perhaps, unsurprising that there is little detailed information available about the use and usefulness of repositories generally, and for resource discovery in particular. Since the population of repositories with content has taken much longer than expected, there is often limited content available for research access (full text especially), depending on the subject area. In user surveys it has sometimes not been possible to gather views because of lack of relevant content (Pickton).

Despite the Cornell University study discovery that most research staff (9 out of 11) believed repositories to be stand-alone services which had to be searched separately (Davis), it is unlikely that much direct searching of repositories (either local or remote) would be carried out by research users. Searching is more likely to be conducted via a search engine or portal. Repository managers have little detailed information about who is accessing their content. While usage statistics are available, these will usually include search engine spiders as well as human users. The web server log will capture the IP address of each user visiting the repository (and therefore location information), but cannot report on content access. JavaScript analytical tools can offer complex information about the browsing behavior of the user (Digital Repository infoKit). These tools may provide valuable usability as well as management information, but cannot provide any qualitative information on how useful a user found their visit, (i.e., was a known item located [although in some situations this may be able to be surmised from the search terms]), or a new resource discovered, or was an expected item missing? They will also be unaware of other services that may have been accessed in the same search (with or without success).

Usefulness and usability are necessarily two complementary concepts. For example, users who think an application is useful to their job will not use it if they find it is hard to use (Loureiro-Koechlin Uncovering User Perceptions). Also, as Carter and Bélanger indicate, “perceived ease of use is predicted to influence perceived usefulness, because the easier a system is to use, the more useful it can be” (8).

User expectations of usefulness of discovered content may be high, in comparison to their reluctance to deposit. Interviews with research students at Loughborough University established that, as readers, they wanted to find many more types of material in the repository than, as authors, they were willing to deposit. (Pickton 53–55). This particularly applied to research data: 61.8% said they would not deposit the dataset from their thesis, while 65% said they would like to access datasets themselves! Analogous findings were reported at Cranfield University: despite limited use of the web to disseminate their own work (43% claimed to have deposited work in the Cranfield repository), all the academic authors interviewed said that they use the web to search for the work of others in their field (Watson 226). Also, the Economists Online project found that academics were far more reluctant to share their data freely than other types of research output (Blake 209).

The Integration of Repositories with Research Practice

One way in which repositories might be able to engage usefully with researchers would be to integrate deposit and other repository interactions into research practice and workflows. Simple deposit tools like the Atom-based SWORD (Simple Web-service Offering Repository Deposit) protocol already exist, supporting the bulk transfer of content into repositories. Specific applications, like the OfficeSWORD tool developed by Microsoft Research, facilitate the deposit of content from things like standard office applications.

In general, however, researcher interaction with repositories needs to be far more than just about deposit. Repositories are fundamentally a means of sharing and managing content and its associated metadata. Where suitable tools exist, repositories are able to become one part of a wider infrastructure enabling the ongoing management of research outputs that is essential to the research process. In order for this to succeed from the researcher perspective, repositories need to become what Aschenbrenner et al. describe as a “natural part of the user's daily work environment.” The kinds of additional services that repositories might offer might vary according to institutional preference or user requirements analysis, but might include things like the long-term stewardship of content or facilitating the sharing and reuse of primary research data. Peter Murray-Rust's general principles for data repositories are that they must be “intimately embedded” in the current practice of scientists (ideally invisible), and that they “must directly support the scientific effort and be seen as doing so rather than being confused with metrics, business processes, etc.” (Murray-Rust). The same approach would seem to be useful for repositories more generally.

Integration with Other Institutional Information Systems

In order to make repositories even more useful to researchers, it will be necessary to explore how they might need to interact with the many other information systems that exist to manage research activities within the institution or outside. Typically, this may need to include the information held in institutional research information management systems. The need for repository interaction with other systems has been emphasized by Cardiff University's JISC-funded I-WIRE Project, which has produced a comprehensive list of user requirements (I-WIRE Requirements). These include the easy reuse of bibliographic data by researchers and the ongoing collection of usage or impact metrics.

An ambitious project that is exploring the information integration challenge is VIVO at Cornell University, which has been attempting to gather together all “publicly available information on the people, departments, graduate fields, facilities, and other resources that collectively make up the research and scholarship environment in all disciplines at Cornell” (VIVO). The VIVO technology is currently being extended to explore the connection of biomedical researchers across several US institutions. In the UK, a project with similar aims is the University of Oxford's BRII. This is exploring the use of semantic web technologies to share information about the institution's research activities, and has developed a Web-interface known as the “Oxford Blue Pages” that captures information about research activity from a wide range of sources (Loureiro-Koechlin Uncovering User Perceptions).

A growing number of other projects are exploring how best to enhance repositories by integrating them into a much wider context of diverse information systems. The key point that underlies this is that once content has been deposited in repositories (with sufficient metadata), it can in turn be used to generate data for a range of other operations, including researcher-focused things like the Researcher Pages developed at the University of Rochester as well as feeding into institutional reporting systems that can be used to support the production of grant proposals or institutional submissions for research assessment.

Conclusions

Since some of the earlier behavioral studies were carried out, understanding of scholarly communications practices has improved and there is evidence that research user requirements are feeding in to innovative repository developments. Even if institutions do not have the resources to carry out lengthy behavioral analysis of their own research users, a lot can be learned from the findings of existing studies.

Despite this, the continuing low rates of deposit experienced by many institutional repositories mean that there is still much that can be done. Some institutions still need to engage more with their research users and adapt advocacy methods to suit the needs of their own staff, using language that is meaningful in different cultural contexts. In discussions with researchers it is also important to be capable of demonstrating that repository plans and policies will be responsive to their specific needs and perceptions. The importance of preservation as a motivating factor for research deposit should not be under-estimated.

Subject repositories, rather than being seen as rivals to institutional repositories, should be seen as complementary: there is scope for both services to work together. There is also scope for integration with institutional research management systems. However, repository interactions need to be integrated into research practice and workflows (SWORD is a useful deposit tool).

At the Repository Fringe 2008 at the University of Edinburgh, David de Roure proposed: “Don't think roll-out of services, think roll-in of researchers.” Engagement with and understanding of researchers will mean that the benefits of repositories (for all stakeholders) are likely to be more fully realized, and at an earlier stage.

However, in the longer term it is likely that progress will depend on institutions integrating institutional repositories into their wider strategies for digital collections and research information management. As repository content grows, other services will be increasingly keen to access that data.