Access to Supplemental Journal Article Materials

ABSTRACT

The use of supplemental journal article materials is increasing in all disciplines. These materials may be datasets, source code, tables/figures, multimedia, or other materials that previously went unpublished, were attached as appendices, or were included within the body of the work. Current emphasis on critical appraisal and reproducibility demands that researchers have access to the complete life cycle to fully evaluate research. As more libraries become dependent on secondary aggregators and interlibrary loan, we questioned if access to these materials is equitable and sustainable. While NISO RP-15-2013 Recommended Practices for Online Supplemental Journal Article Materials was published in 2013, it is unclear if these recommended practices fully meet the needs of users; if aggregators and publishers are following these standards; and if library processes and procedures are facilitating access to supplemental journal article materials. While studies have surveyed authors, reviewers, and readers, or examined journal supplemental materials practices, no studies have surveyed library staff and librarians about their experience with access to supplemental materials and requesting and receiving supplemental materials through interlibrary loan. This presentation reported on a study surveying library employees from academic, hospital, public and special library settings in the United States about their experiences identifying, finding, and retrieving supplemental journal article materials; and proposes ways that libraries, publishers and aggregators can enable access to the complete published life cycle.

KEYWORDS:

• Supplemental journal article materials
• supplementary materials
• reproducibility
• publishing trends
• associated data
• interlibrary loan

Introduction

Supplemental journal article materials (SJAM) support efforts to reproduce research, to make data accessible and transparent for further research, and to trace the impact of research. Librarians and library staff may confuse journal supplements with SJAM. They differ widely in that journal supplements contain original articles or conference abstracts not found elsewhere, while SJAM, supplementary materials, supplemental information (SI) or additional files, may consist of tables and figures, checklists, flow diagrams, interview transcripts, data, software codes, visualizations, methods, media files, reagents, dynamic models, and other materials found outside the article.¹ Supplemental journal article materials are a new and growing phenomenon brought on first by the onset of electronic publishing and article length restrictions, and more recently by the demand for critical appraisal, reproducibility/transparency of research, and funder open data policies.²

Recognizing growing problems with SJAM, the National Information Standards Organization (NISO) and the National Federation of Advanced Information Services (NFAIS) created RP-15-2013 Recommended Practices for Online Supplemental Journal Article Materials in 2013.³ This document distinguishes two functional categories of SJAM, integral content essential for understanding of the work and additional content that expands reader understanding but is not essential. Recommendations include 1) publishers should provide written guidelines for acceptance or outsource archiving of SJAM, 2) SJAM should be reviewed at the same level as the article, 3) SJAM should have a persistent identifier, 4) SJAM should have standardized naming conventions, metadata including file types, indexing in databases, and navigation in the table of contents and article, and 5) bibliographic references within SJAM should be treated differently based on designation of the SJAM as integral or additional. Some feel that current recommended practices do not go far enough in defining standard terminology and how SJAM should be organized, formatted and indexed.⁴

Despite these recommended practices, this study stems from requests that librarians and library staff continue to receive from research faculty and clinicians, or observe in library listservs, looking for SJAM. These requests come to a variety of liaison, reference, systems, access services, and interlibrary loan librarians and library staff, and often go undocumented and uncollected. As well, we are anecdotally aware that libraries do not understand the full extent of the problem as many researchers bypass libraries altogether and turn to colleagues when unable to access SJAM. In many respects libraries and research communities are having separate parallel conversations about supplemental materials, each centered around their own siloed practices.

Journals currently lack consistent policies on SJAM.⁵ Some journals may request that SJAM be submitted as one file while other journals require that they be submitted separately and tied to elements of the article, and yet others do not allow submission of any SJAM.⁶ As the reproducibility movement expands and awareness of FAIR (findability, accessibility, interoperability, reusability) principles grows, individual journals and professional organizations have published best practices for SJAM. Alter and Gonzalez recommend that data be deposited in domain-specific repositories, with a separate and unique identity, and associated with publications through citation.⁷ However, this best practice does not take into account the variety of integral materials submitted as SJAM, nor does it account for reader frustrations with link rot.⁸ Surveys of authors, readers, and reviewers show a preference for publication of SJAM on the journal website.⁹ Since journals have not traditionally viewed SJAM as stand-alone items, continuation of this practice comes with recommendations for structure, standards, internal navigation and micro-digital object identifiers for specific sections of a paper.¹⁰

Historical dependence on the Portable Document Format (PDF) is shifting as researchers recognize value in reusable, interactive and machine-readable file types.¹¹ Despite Kenyon’s finding in 2016 that disciplines continue to share data primarily as Word files or PDF documents on journal websites, a growing number of disciplines now depend on high-throughput methods where access to SJAM in formats that facilitate data mining tools is imperative to use.¹² This more recent need for machine-readable data may be one reason why some authors suggest that data be deposited in domain-specific repositories, instead of journal websites or institutional repositories, because curation, metadata application services, and supported file types are robust and consistent with disciplinary expectations.¹³

Library systems like interlibrary loan (ILL) are also structured around the PDF. While the ALA Interlibrary Loan Code for the United States has undergone slight revisions over time, it does not address SJAM.¹⁴ It is merely a voluntary code of behavior for lending and requesting libraries and it is not enforced by any oversight body. Similarly, International Federation of Library Associations and Institutions (IFLA) Guidelines for Best Practice in Interlibrary Loan and Document Delivery and International Resource Sharing and Document Delivery: Principles and Guidelines for Procedure, 2009 revision make no mention of SJAM.¹⁵

Many historical complaints about SJAM can be traced to non-indexing.¹⁶ This may stem from some of the more heavily used scholarly search engines, PubMed and Google Scholar not indicating the presence of SJAM. While ours is not meant to be a study of indexing and abstracting services, a cursory look reveals that many library databases also do not index SJAM. An exception may be Web of Science where limited associated data are displayed through the connection with Data Citation Index. On the other hand, some Open Access repositories are rapidly adopting consistent user interfaces for identification of SJAM. PubMed Central (PMC) has a requirement that all SJAM be supplied with articles submitted to PMC, even when they are available in a public repository, and designated with unique identifiers and associated Extensible Markup Language (XML).¹⁷ The arXiv pre-print servers also identify the presence of supplementary material and data/code.

While previous authors have surveyed authors and reviewers or evaluated SJAM practices and adherence to standards in disciplinary publications, these studies rarely trickle down to the level of library employees working to identify and access SJAM on a daily basis.¹⁸ No studies have interviewed librarians and library staff about their collective experience with SJAM. Greater understanding of access to SJAM is needed as more libraries obtain journal articles from secondary aggregators and through ILL. This study aims to understand librarians and library staff members experience with SJAM, and provide recommendations for identifying, finding and retrieving SJAM.

Methods

The survey was formed and validated by two librarians and a library access services manager. We chose a mixed method survey design using Qualtrics. Quantitative questions were chosen to get an overview of librarian and library staff demographics, library types, what subject areas were requesting these materials and where the SJAM’s become bottlenecked. We provided several open-ended questions to understand difficulties of acquiring supplemental materials, knowledge of NISO guidelines around SJAM and frequency of library workers’ interaction with SJAM based on ILL requests.

Qualitative and quantitative data were collected, analyzed separately, and integrated to inform the final discussion. Survey distribution was aimed to reach a wide variety of library workers, including library staff and librarians that may work daily with ILL or encounter SJAM peripherally through requests from faculty. We also sought to find a variety of types of libraries. After Institutional Review Board (IRB) exemption was received, the survey was distributed to listservs for academic, special and medical librarians and library staff who fulfill ILL and have access services roles. The survey was sent twice over a two-week period in July 2019.

Quantitative demographic data were compiled, and descriptive statistical comparisons were run in Excel Pivot tables. For qualitative analysis, open-ended responses were coded and analyzed by librarians based on grounded theory. Grounded theory uses constant comparative analysis to identify common themes and categories emerging from rich descriptive data.¹⁹ For inter-rater reliability, responses were coded separately by the authors and exchanged for verification.

Results

Of the 21,451 members of listservs to which the survey was distributed, 211 opened the survey. This is a 1% response rate, although many individuals subscribe to multiple listservs. A total of 197 answered the question about U.S. residency and 191 indicated they live in the United States as required by our IRB (Table 1).

Table 1. Participant demographics

Completion of Survey
Total opened survey	211
Total answered US question	197
Total in US	191
Total moved past US question	180
Total moved past library size question	120
Library type
Hospital	56	29%
Academic	107	56%
Public	0	0%
Other (govt, publisher, institute, organization)	17	9%
No response	11	6%
Library Size (cleaned to remove all explanatory text, plus marks, units (like FTE and students), or obvious unit’s errors such as number of librarians/number of campuses)
1–1000	25	13%
1001–5,000	58	30%
5,000–10,000	28	15%
10,000–20,000	22	12%
20,000+	23	12%
No response or inaccurate interpretation	35	18%
How often asked for SJAM
Never	5	3%
1–5 x/year	39	20%
6–10 x/year	25	13%
More than 10x/year	37	19%
Other (Always, Don’t know)	13	7%
No response	72	38%
Institution Provides SJAM when filling ILL Requests
Yes, always look and provide if accessible	44	23%
No, only provide if requested	56	29%
Other (unsure, if aware, if easy, Docline only)	16	8%
No response	75	39%

Average time to completion was 21 min. Respondents work in academic (56%), hospital (29%) and other special libraries (9% – government, publisher, institute, and organization). None work in public libraries and 6% did not respond. While hospital librarians made up a smaller percent of survey respondents, they were more likely to complete the entire survey. (Figure 1) A total of 120 answered at least one question specifically about SJAM. The average library size within limitations of the survey was 11,457 potential users, with a range of 50 to 200,000.

Figure 1. Completion of survey by library type N = 180

Quantitative

While access to SJAM is not only relevant to those librarians and library staff members working with ILL, we know that questions about SJAM frequently come up in the context of ILL and thus, several survey questions asked specifically about ILL. In comparing how much responders work with ILL (Q3 N = 179) by library type (Figure 2) it was apparent that ILL usually makes up at least part of hospital library work, while academic library responders were more likely to respond that it was either never a part of their work or most of their job duty.

Figure 2. Work with ILL by library type N = 179

When asked about frequency of ILL requests for SJAM (Q4 N = 119), 85% of respondents affirmed that patrons request supplemental materials, with 31% indicating more than ten requests a year. Requests for supplemental materials come to all respondents regardless of their level of involvement with interlibrary loan, though those who described their job as mostly with ILL receive the most requests for SJAM. (Figure 3)

Figure 3. Supplemental requests by amount of work with ILL N = 119

When asked at what point patrons are aware of SJAM in an ILL request (Q9 N = 116), of those that replied 69% felt that requesters were not aware of SJAM until after they received the article, 9% felt that they were aware prior to making an initial ILL request, and 22% said they do not know or a combination of before and after the request. (Figure 4)

Figure 4. Point at which patron is aware of SJAM N = 116

Forty-seven percent of respondents say patrons assume that SJAM will be included without asking for it (Q10 N = 116), 22% do not believe that patrons assume it will be provided without asking, and 32% said other. Of those who replied other, most don’t know (N = 19) and others replied sometimes (N = 12) or that they always provide it (N = 1).

When asked if their institution provides SJAM even when not requested when filling ILL requests (Q11 N = 116 Table 1), respondents fall into four categories. Nearly half (48%) only provide SJAM if it is asked for, 38% always look for and provide SJAM if accessible, and 8% replied other which included responses like if aware, if easy and that they always provide only with DOCLINE requests. Recall though that at this point in the survey over a third of the initial respondents did not answer the questions about SJAM. This results in a fourth category and we might assume that those who did not respond to this question do not know what their institution does in regard to SJAM based on qualitative responses.

More work with ILL (Q3) was not associated with any greater tendency to look for and provide SJAM (Q11 N = 116). (Figure 5)

Figure 5. Work with ILL vs. provide SJAM when not requested N = 116

Most telling are correlations between Q10 (Do patrons assume that SJAM will be included without asking for it?) and Q11 (Do you look for and provide SJAM even when not asked for?). While over 40% replied that patrons assume SJAM will be provided without asking, 47% (N = 22) of those who responded that patrons assume SJAM will be provided are not actually providing SJAM (Figure 6 N = 116) unless it is requested. We purposely surveyed both librarians and library staff to include perspective of both those who may be involved in setting institutional policy and those who work within policy set by others. Thus, there are apparent gaps and inconsistencies between what we feel patrons assume we do, what library policies state that we do, and what we actually do, and librarians and library staff are aware that we are not always meeting patron expectation.

Figure 6. Patrons assume SJAM included without request vs. provide SJAM without asking N = 116

When asked if patrons put in a second request because they did not receive SJAM (Q12 N = 114), a majority of those who responded say they rarely submit a second ILL request and acknowledge that institutions usually handle the need to track down SJAM through e-mail, which goes unrecorded in ILL statistics. Some responders noted that they have identified those ILL providers that never supply SJAM and removed them from their routing lists.

Qualitative themes (Table 2) were gleaned from open-ended responses to Q7, 8, 9, 12, and 13.

Table 2. Themes from responses to open-ended questions

Theme	Sub-theme	Representative quotes
Why SJAM are a problem?	Behind publisher paywall	We have struggled with getting supplementary materials for [publisher] articles requested through ILL. The supplementary materials are often accessible through a link that as a non-subscriber, we (our patron) cannot access.
	Separate from article	On the whole, I think Supplementary material is difficult to find. Publishers should make SJAMs more readily available with articles in the first place.
	Multiple files	I would like it if articles with multiple supplements would put them all in one file. If an article has 8 separate supplements in 8 separate files, I won’t retrieve and send for ILL unless requested. If all 8 supps are in one file, I will.
	Multiple file types	Often if there is supplemental material it is only visible thru the HTML view, and not the PDF. This presents a problem as the lender attaches PDF files not HTML, so the reader discovers the references to supplemental material after receiving the main article in PDF. Wish there was an easier way to send materials other than combining PDFs before sending or emailing. Also interested in the accessibility of these materials when sent via PDF, etc. Also we are seeing more audio and video supplements. We are not able to provide these for ILL’s, and it is difficult to even save and send to our own patrons.
	Not indexed	We also struggle with doi not reflecting a difference between the original article and SJAM a consistent format for listing available SJAM with the article would improve everyone’s awareness Sometimes SJAMs may not be indexed – Medline is one source that only infrequently includes SJAMs.
	ILL systems do not support SJAM	Part of the problem with obtaining SJAMs is that ILLiad is not really equipped to handle them … which is something that needs to be addressed soon by the vendor, in my opinion. We use Odyssey and deliver articles, often from other libraries, so are unaware of SJAMs unless the customer asks for them.
	Not available from secondary aggregators	Packages we buy through [secondary aggregators] never identify or deliver supps. Occasionally, I have not been able to find an identified item on the publisher site, but if I work with the publisher, they will eventually get the content up. Most journals we buy separate subscriptions to (not part of a package) make supplementary material easily identifiable and accessible. None of the supplementary content is available in [secondary aggregator]
Lack of understanding of SJAM	By patrons	some don’t know the SJAMs exist, and it’s staff who notice and ask for them on behalf of the patron. I as the librarian catch it and attach the supplements. Most of our patrons do not know about supplementary materials when requesting articles. But are a few heavy users that always ask for supplementary materials.
	By libraries	I hadn’t heard of them. Don’t e-articles include all related things like charts, tables, etc.? when we ask other libraries for the SJAM, they provide another copy of the article and not the SJAMs. It’s very frustrating. Also many lenders fail to read notes regarding request to include all available supplements.
	By publishers	I find that patrons have difficulty finding SJAM even for articles that we do have, particularly since seems as if every publisher has a different layout for identifying SJAMs It seems that SJAMs are an afterthought for journal editors and publishers. When you edit a manuscript, you forget that the author(s) put the survey in an appendix or offered additional data – so it dangles.
Expectations for SJAM	Patrons expect SJAM	Some can be quite indignant that the SJAM weren’t supplied along with the article They would not mention it to me until they get the article without the supplement. Some do indicate on the initial request that that they want the supplement, but others just seem to expect it.
	Requesting Libraries expect that lending libraries are looking for SJAM	Also many lenders fail to read notes regarding request to include all available supplements. I attach several (1–25) SJAMs to Lend almost every day, but rarely ever receive them for my Borrow I.L.L.s even though I specifically ask for them. SJAMs should really be considered just one more part of the article, NOT something special that we need to practically beg to be included. WHY can’t they just make a download that INCLUDES all the supplemental material (in addition to the regular article download). That way, other libraries could just click that and send it, instead of my arguing with them about sending it. I actually don’t have that problem as much anymore because I deleted the problem lenders from our ILL routing table. I don’t want the stress!
Why libraries are not routinely supplying SJAM?	Time	Time consuming and cumbersome; that’s why we have stopped supplying unless specifically asked for. Publishers really need to make the content more obvious. We can’t skim every article to see if they mention supplemental material.
	Consider it something special	depends on how serious they are about the article
	Not in the habit	getting veteran ILL technicians to get in the habit of including suppl mat is another hurdle
	Difficult	It’s a pain in the neck to download all of those supplementary files in various formats. Often there are quite a number of files!

Limitations of the study

In Q7 we asked specifically about experience with compliance of specific publishers and aggregators to SJAM standards. There were certain aggregators and publishers mentioned by multiple responders as not complying. This potentially impacts both local patrons and ILL. While open-ended replies to this question suggest that participants anecdotally experience more problems with access through secondary aggregators, we did not ask this question in such a way that it allows direct comparison with access to SJAM in general.

Questions about SJAM are presented to librarians and library staff in a variety of positions, not only to those who work with ILL. As liaison librarians, we know that patrons contact different people at different times when the library is not meeting their needs, or they may not contact the library at all. We purposely sought to include perspectives of both those library employees involved in setting institutional policy and those who work within policy set by others. As part of our purpose in the survey was to identify who receives questions about supplementary material, we did not identify a target population and pilot our survey in advance. However, this exploratory approach did allow us to see that librarians and library staff working in hospital settings when taken as a whole are more aware of patrons’ needs for SJAM and issues with access than academic librarians and library staff as a whole. We also found that more library employees feel that patrons assume that SJAM will be included in an ILL request without it being requested, than are actually supplying SJAM without it being requested.

However, we acknowledge that piloting the survey would have helped us to word questions in a way that made them more consistently understood by a broader audience. For instance, we left the question about library size open-ended. In hindsight, this should have been a multiple-choice question with stated standard units of potential number of institutional library users, as it was interpreted in various ways by responders. Also, we asked several questions that were not fully dichotomous yes/no questions, instead adding an other category. These provided qualitative data (such as only providing SJAM for DOCLINE requests); however, they hindered ability to perform statistical analysis.

Discussion

Requests for access to SJAM are often presented to librarians, and particularly to library staff members, in the form of ILL requests. Our study’s qualitative data confirmed the findings in Price et al. that librarians and library staff prefer that SJAM come in a standardized single zipped file for downloading or as a persistent link to a PDF file.²⁰ While this fits well with existing library systems, it does not consider the breadth of interactive data and visualization necessary for current scholarly activity and learning.

Based on our results we recommend the following:

• Publishers, aggregators, databases, repositories and scholarly search engines should:

  • Comply with all NISO recommended practices for SJAM, including fixing broken links.

  • Provide SJAM in one file or as a permanent link with a digital object identifier (DOI).

  • Index SJAM consistently in a way that facilitates discovery of SJAM without reading the article.

  • Assure that the industry develops a collective understanding of data as a stand-alone item, increasingly not intended to be linked with one study.

  • Use consistent/standard terminology, organization, file formats, and systems of reporting SJAM access problems.

• Library ILL system vendors should:

  • Have a way to communicate to the ILL patron that the library checked for supplemental materials, and a way to designate if it was unavailable.

  • Create ILL systems that are not PDF-dependent and can accommodate the variety of file types made available by publishers (HTML, etc.).

• Librarians and library staff should:

  • Understand how important SJAM is to critical appraisal of research.

  • Adopt a standard practice of supplying SJAM even when not requested.

  • Train academic library staff to identify the presence of SJAM using database indexing.

  • Communicate effectively with patrons about library policies on providing SJAM and instruct patrons on how to best access SJAM.

  • Understand that data are increasingly separate for a reason. Data are no longer associated with just one article. It is an entity in and of itself.

• Patrons and library liaisons who teach patrons should:

  • Stay current on new journal and database indexing and interface features used to identify the presence of SJAM.

  • Learn to always look for and request SJAM through ILL.

• Standards should:

  • Provide guidance for consistent access to expanding file formats and linked data.

  • Discontinue the distinction between integral and additional content as that can only be determined by the reader.

  • Provide short plain language summaries and infographic visualization of standards for training of library technical staff.

Future studies are needed on how publishers, aggregators, catalogers, databases and library systems can better accommodate a growing amount of linked data and materials. Librarians and library staff members should be trained to ensure that patrons have access to the full research life cycle for reproducibility and critical appraisal purposes. As well, outcomes-based behavior studies are needed to know if patrons and library staff members filling ILL requests are actively looking for, finding, and providing SJAM through library provided systems and subscriptions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Suzanne Fricke

Suzanne Fricke is Animal Health Sciences Librarian, Washington State University Libraries, Pullman, WA.

Electra Enslow

Electra Enslow is Assistant Director of Clinical Research and Data Services, University of Washington Health Sciences Library, Seattle, WA.

Susan Shipman

Sue Shipman is Access Services Manager, Washington State University Libraries, Pullman, WA.

Notes

1 Ángel Borrego and Francesc Garcia, “Provision of Supplementary Materials in Library and Information Science Scholarly Journals,” Aslib Proceedings 65, no. 5 (2013): 503-14, doi: 10.1108/AP-10-2012-0083; John P. Kastelic and Fulvio Gandolfi, “On-Line Publication of Supplementary Material,” Theriogenology 72, no. 1 (2009): 1, doi: 10.1016/j.theriogenology.2009.04.001; Lior Pachter, “Stories from the Supplement” (keynote address, Genome Informatics 2013, Cold Spring Harbor Laboratory, NY, November 2, 2013), YouTube video, 46:39, uploaded November 4, 2013, accessed April 23, 2020, https://youtu.be/5NiFibnbE8o; Jeremy Kenyon, Nancy Sprague, and Edward Flathers, “The Journal Article as a Means to Share Data: A Content Analysis of Supplementary Materials from Two Disciplines,” Journal of Librarianship & Scholarly Communication 4 (2016): 1-17, doi: 10.7710/2162-3309.2112; Todd Carpenter, “Outside the Core: Working Towards an Industry Recommended Practice for Supplemental Journal Materials,” Serials 23, no. 2 (2010): 155-58, doi: 10.1629/23155; Thomas Schaffer and Kathy M. Jackson, “The Use of Online Supplementary Material in High-Impact Scientific Journals,” Science & Technology Libraries 25, no. 1-2 (2004): 73-85, doi: 10.1300/J122v25n01_06.

2 Ibid.; Carpenter, “Outside the Core.”; Jeremy Kenyon and Nancy R. Sprague, “Trends in the Use of Supplementary Materials in Environmental Science Journals,” Issues in Science & Technology Librarianship 75 (2014): 7, doi: 10.5062/F40Z717Z; National Information Standards Organization and National Federation of Advanced Information Services, NISO RP-15-2013. Recommended Practices for Online Supplemental Journal Article Materials (NISO, January 2013), https://www.niso.org/publications/niso-rp-15-2013-recommended-practices-online-supplemental-journal-article-materials (accessed April 23, 2020).

3 Ibid.

4 Barbara Cheifet, “Making Supplemental Information More Accessible,” Genome Biology 19, no. 1 (2018): 225, doi: 10.1186/s13059-018-1609-8.

5 Mihai Pop and Steven L. Salzberg, “Use and Mis-Use of Supplementary Material in Science Publications,” BMC Bioinformatics 16 (2015): 237, doi: 10.1186/s12859-015-0668-z.

6 American Association for the Advancement of Science, “Science: Information for Authors,” Science, https://www.sciencemag.org/authors/science-information-authors (accessed April 25, 2020); American Society for Microbiology, “mBio Instructions to Authors,” 2019, https://mbio.asm.org/sites/default/files/additional-assets/mbio-ITA.pdf (accessed April 25, 2020); Cell Press, “Supplemental Information Guidelines,” https://www.cell.com/supplemental-information (accessed April 25, 2020); Nature, “On Supplementary Information: Nature Immunology is Implementing New Editorial Guidelines for the Supplementary Material of Research Articles,” Nature Immunology 13, no. 6 (2012): 519, doi: 10.1038/ni.2325; John Maunsell, “Announcement Regarding Supplemental Material,” The Journal of Neuroscience 30, no. 32 (2010): 10599-600, https://www.jneurosci.org/content/30/32/10599 (accessed April 25, 2020).

7 Dov Greenbaum et al., “Structuring Supplemental Materials in Support of Reproducibility,” Genome Biology 18, no. 1 (2017): 64, doi: 10.1186/s13059-017-1205-3; Maunsell, “Announcement Regarding Supplemental Material”; George Alter and Richard Gonzalez, “Responsible Practices for Data Sharing,” The American Psychologist 73, no. 2 (2018): 146-156, doi: 10.1037/amp0000258.

8 Evangelos Evangelou, Thomas A Trikalinos, and John PA Ioannidis, “Unavailability of Online Supplementary Scientific Information from Articles Published in Major Journals,” The FASEB Journal 19, no. 14 (2005): 1943-44, doi: 10.1096/fj.05-4784lsf; Nicholas R. Anderson, Peter Tarczy-Hornoch, and Roger E. Bumgarner, “On the Persistence of Supplementary Resources in Biomedical Publications,” BMC Bioinformatics 7 (2006): 260, doi: 10.1186/1471-2105-7-260.

9 Amy Price et al., “Role of Supplementary Material in Biomedical Journal Articles: Surveys of Authors, Reviewers and Readers,” BMJ Open 8, no. 9 (2018): e021753, doi: 10.1136/bmjopen-2018-021753.

10 Greenbaum et al., “Structuring Supplemental Materials in Support of Reproducibility.”

11 Pop and Salzberg, “Use and Mis-Use of Supplementary Material in Science Publications.”

12 Kenyon, Sprague, and Flathers, “The Journal Article as a Means to Share Data.”; Antonio Jimeno Yepes and Karin Verspoor, “Literature Mining of Genetic Variants for Curation: Quantifying the Importance of Supplementary Material,” Database 2014, (2014), doi: 10.1093/database/bau003.

13 Alter and Gonzalez, “Responsible Practices for Data Sharing.”

14 Codes, Guidelines, and Technical Standards Committee, Sharing and Transforming Access to Resources Section (STARS), “Interlibrary Loan Code for the United States,” American Library Association, Reference and User Services Association, January 11, 2016, http://www.ala.org/rusa/guidelines/interlibrary (accessed April 25, 2020).

15 International Federation of Library Associations and Institutions, Guidelines for Best Practice in Interlibrary Loan and Document Delivery, https://www.ifla.org/publications/guidelines-for-best-practice-in-interlibrary-loan-and-document-delivery (accessed April 23, 2020); International Federation of Library Associations and Institutions, International Resource Sharing and Document Delivery: Principles and Guidelines for Procedure, 2009, https://www.ifla.org/publications/international-resource-sharing-and-document-delivery-principles-and-guidelines-for-proc (accessed April 23, 2020).

16 Frank Seeber, “Citations in Supplementary Information are Invisible,” Nature 451, no. 7181 (2008): 887, doi: 10.1038/451887d; Pop and Salzberg, “Use and Mis-Use of Supplementary Material in Science Publications.”

17 National Library of Medicine, “Associated Data in PMC,” NLM Technical Bulletin, no. 425 (November-December 2018): b4, https://www.nlm.nih.gov/pubs/techbull/nd18/brief/nd18_associated_data_pmc.html (accessed April 23, 2020); Price et al., “Role of Supplementary Material in Biomedical Journal Articles.” Evangelou, Trikalinos, and Ioannidis, “Unavailability of Online Supplementary Scientific Information.”

18 Schaffer and Jackson, “The Use of Online Supplementary Material in High-Impact Scientific Journals.”; Kenyon and Sprague, “Trends in the Use of Supplementary Materials in Environmental Science Journals.”; David L. Schriger et al., “Use of the Internet by Print Medical Journals in 2003 to 2009: A Longitudinal Observational Study,” Annals of Emergency Medicine 57, no. 2 (2011): 153-60.e3, doi: 10.1016/j.annemergmed.2010.10.008; Dave Shutler and Ashley Murray, “Trends, Costs, Benefits, Challenges, and Prognoses for Supplementary Materials,” The Auk 133, no. 4 (2016): 733-37, doi: 10.1642/AUK-16-92.1; Antonio Vidal-Infer et al., “The Availability of Raw Data in Substance Abuse Scientific Journals,” Journal of Substance Use 24, no. 1 (2019): 36-40, doi: 10.1080/14659891.2018.1489905; Sarah C. Williams, “Practices, Policies, and Persistence: A Study of Supplementary Materials in Crop Science Journals,” Journal of Agricultural & Food Information 17, no. 1 (2016): 11-22, doi: 10.1080/10496505.2015.1120213.

19 Antony Bryant and Kathy Charmaz, eds., The SAGE Handbook of Current Developments in Grounded Theory (London: SAGE Publications, 2019).

20 Price et al., “Role of Supplementary Material in Biomedical Journal Articles.”