ABSTRACT
Scientific integrity is at the forefront of the scientific research enterprise. This paper provides an overview of key existing efforts on scientific integrity by federal agencies, foundations, nonprofit organizations, professional societies, and academia from 1989 to April 2016. It serves as a resource for the scientific community on scientific integrity work and helps to identify areas in which more action is needed. Overall, there is tremendous activity in this area and there are clear linkages among the efforts of the five sectors. All the same, scientific integrity needs to remain visible in the scientific community and evolve along with new research paradigms. High priority in instilling these values falls upon all stakeholders.
GLOSSARY OF ACRONYMS | ||
AAAS | = | American Association for the Advancement of Science |
AAMC | = | Association of American Medical Colleges |
ACS | = | American Chemical Society |
ASA | = | American Sociological Association |
BITSS | = | Berkeley Initiative for Transparency in the Social Sciences |
CASBS | = | Center for Advanced Study in the Behavioral Sciences |
CCRE | = | Center for Clinical Research Ethics |
CDC | = | Centers for Disease Control and Prevention |
CFR | = | Code of Federal Regulations |
COPE | = | Committee on Publication Ethics |
COS | = | Center for Open Science |
CSE | = | Council of Science Editors |
DR | = | Departmental Regulation |
EPA | = | US Environmental Protection Agency |
FASEB | = | Federation of American Societies for Experimental Biology |
FDA | = | US Food and Drug Administration |
FR | = | Federal Register |
HHS | = | US Department of Health and Human Services |
HMD | = | Health and Medicine Division of NAS (formerly known as IOM) |
ICs | = | NIH Institutes and Centers |
ICTS | = | Institute of Clinical and Translational Sciences |
IOM | = | Institute of Medicine (former name of HMD) |
LJAF | = | Laura and John Arnold Foundation |
NAS | = | National Academy of Sciences |
NIH | = | National Institutes of Health |
NSF | = | National Science Foundation |
OER | = | NIH Office of Extramural Research |
ORI | = | Office of Research Integrity |
OSF | = | Center for Open Science's Open Science Framework |
OSI | = | Office of Scientific Integrity |
OSTP | = | Office of Science and Technology Policy |
PEER | = | Public Employees for Environmental Responsibility |
PHS | = | Public Health Service |
RCR | = | Responsible Conduct of Research |
USAID | = | US Agency for International Development |
USDA | = | US Department of Agriculture |
Introduction
Scientific integrity is at the forefront of the scientific research enterprise, more so than ever in recent years owing to the apparent increase in misconduct and misuse of research data. This paper provides a resource for the scientific community on the extensive work being done on scientific integrity and helps to identify areas in which more attention is needed. This work provides an overview of key existing efforts by federal agencies, foundations, nonprofit organizations, professional societies, and academia on their scientific integrity efforts from 1989 to April 2016, as described on their websites. A description of the linkages among the efforts is provided. A literature search in PubMed was not conducted for this paper. It is designed as a living document that will be updated online to include existing efforts that have not been captured and new work in the future.
Key work on scientific integrity
Federal agencies
Key work by federal agencies is included in .
In 2009, President Barack Obama issued a Presidential Memorandum on Scientific Integrity for the heads of executive branch departments and agencies to ensure “the highest level of integrity in all aspects of… scientific and technological processes” (White House Office of the Press Secretary, Citation2009). The memorandum articulated on “six principles central to the preservation and promotion of scientific integrity” (Holdren, Citation2010).
Office of Science and Technology Policy (OSTP)
OSTP issued an implementation guide for federal agencies to respond to the directive in 2010. The implementation guide provides guidance on four areas: foundations of scientific integrity in government, public communications, use of federal advisory committees, and professional development of government scientists and engineers (Holdren, Citation2010).
Individual federal agencies then began work to develop their own set of scientific integrity policies. In 2013, Public Employees for Environmental Responsibility (PEER) issued a scientific integrity report card for individual federal agencies. The report card provided a snapshot of progress that had been made across federal agencies in drafting and adopting scientific integrity policies, and PEER describes the report's aims as follows (PEER, Citation2013a):
This report card is meant to provide a basis for comparing the agencies' progress in shielding science from improper political manipulation through their scientific integrity policies… Policies were scored on three main areas of interest to PEER's work: (a) scientific misconduct regards whether the agency has prohibited political manipulation of science, has in place a defined process for investigating allegations of such misconduct, as well as protections for complainants and sanctions for those who have engaged in misconduct; (b) public communications of science regards generally how free agency scientists are to speak to peers and the public regarding their work; and (c) transparency of policy decision-making regards to what extent the public, external scientists, or legal challenger to a policy might become aware of any political manipulation of science.
Fifteen federal agencies had scientific integrity policies in place, and their report cards were compiled in a comparison chart (PEER, Citation2013b).
Even before the 2009 Presidential Memorandum, OSTP had been working on issues of scientific integrity. In 2000, the OSTP National Science and Technology Council published the federal policy on research misconduct in the Federal Register (FR), 65 FR 76260, as the final, government-wide policy that addressed research misconduct. The policy's purpose was to establish (1) uniformity among the federal agencies' definitions of research misconduct and (2) consistency in federal agencies' processes for responding to allegations of research misconduct. The OSTP policy covers both intramural research as well as extramural research (Executive Office of the President, Citation2000). Federal agencies have codified these definitions in their agency's regulations, the Code of Federal Regulations (CFR).
National Science Foundation (NSF)
NSF is an independent federal agency with an annual budget of $7.3 billion (for fiscal year 2015). NSF funds approximately 24% of all federally supported basic research conducted by America's colleges and universities (NSF, Citation2016).
In 2002, in response to OSTP 65 FR 76260, NSF published its regulation on Research Misconduct (45 CFR 689). NSF defines research misconduct as “…fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results. Fabrication means making up results and recording or reporting them. Falsification means manipulating research materials, equipment, or processes or changing or omitting data or results such that the research is not accurately represented in the research record. Plagiarism means the appropriation of another person's ideas, processes, results, or words without giving appropriate credit” (US Government Publishing Office, Citation2012).
In 2009, NSF published a paper on “Responsible Conduct of Research” in response to the 2007 America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Act. The act requires “each institution that applies for financial assistance from NSF for science and engineering research or education describe in its grant proposal a plan to provide appropriate training and oversight in the responsible and ethical conduct of research to undergraduate students, graduate students, and postdoctoral researchers participating in the proposed research project” (NSF, Citation2009b). Since 2010, NSF has required that an institution's authorized organizational representative must certify that the institution has a plan to implement the training at the time of proposal submission (NSF, Citation2009b).
In 2009, NSF and the National Academy of Sciences (NAS) jointly funded the third edition of On Being a Scientist: A Guide to Responsible Conduct in Research (for more detail on the guide, see NAS, under Nonprofit Organizations) (NAS, Citation2009a).
In accordance with the 2009 Presidential Memorandum on Scientific Integrity, NSF released a draft Scientific Integrity Policy in 2011 for public comments, which states that “NSF's policy applies to civil service employees; visiting scientists, engineers, and educators; those working at NSF under the Intergovernmental Personnel Act; and political appointees” (NSF, 2011).
NSF has also supported research grants on scientific integrity, especially in the 1990s, when a large number of grants were funded in this area. More recently, NSF funded grants including “The Nature of Ethical Decision-Making in Research” at Western Michigan University (NSF, Citation2009a) and “Gaming Against Plagiarism” funded at the University of Florida (Digital Worlds Institute, Citation2016). “Gaming Against Plagiarism” includes an educational video game for students that provides an understanding and appreciation of the issues involved in ethical conduct related to intellectual property. In 2015, NSF began funding a new project at Purdue University titled “Collaborative Research: Foundations of Social and Ethical Responsibility Among Undergraduate Engineering Students: Comparing Across Time, Institutions, and Interventions” (NSF, Citation2014).
US Department of Health and Human Services (HHS)
HHS is a cabinet-level department of the US federal government and aims to protect the health of Americans (HHS, Citation2016a). HHS has 11 operating divisions, including eight Public Health Service (PHS) agencies and three human services agencies (HHS, Citation2016b). PHS is composed of the following offices and agencies: Office of Public Health and Science, National Institutes of Health (NIH), Centers for Disease Control and Prevention (CDC), US Food and Drug Administration (FDA), Substance Abuse and Mental Health Services Administration, Health Resources and Services Administration, Agency for Healthcare Research and Quality, Agency for Toxic Substances and Disease Registry, Indian Health Service, and Office of Regional Health Administrators. HHS issued Policies and Principles for Assuring Scientific Integrity in 2011 in response to the 2009 Presidential Memorandum (HHS, Citation2016c). Within HHS, several of the agencies also developed their own policies on scientific integrity, and efforts by the Office of Research Integrity (ORI), NIH, FDA, and CDC are described below.
Office of Research Integrity (ORI)
ORI oversees and directs PHS research integrity activities on behalf of the HHS Secretary, with the exception of the regulatory research integrity activities of FDA. ORI was established as an independent entity within HHS as part of the NIH Revitalization Act of 1993.
In its work on research integrity, ORI is focused on the research community and supports activities within professional societies. The NIH Revitalization Act of 1993 also “mandated that a Commission on Research Integrity be created to review the system for protecting against research misconduct. The Commission delivered its report, Integrity and Misconduct in Research, to the Secretary of HHS in November 1995. The Commission… made 33 recommendations including the development of a regulation on the protection of whistleblowers in research misconduct cases and the extension of the misconduct in science assurance to required institution to establish educational programs on the responsible conduct of research (RCR)” (ORI, Citation2016a).The Commission on Research Integrity recommended that “Professional societies [should] adopt a code of ethics in research… [and] should consider initiating activities that will further promote the ethical conduct of research” (AAAS, Citation2000).
In 2000, ORI held the first of several biennial Research Conferences on Research Integrity (ORI, Citation2016b) “to expand the knowledge base and develop a research community focused on the responsible conduct of research, research integrity, and research misconduct” (ORI, Citation2016a). In 2001, ORI began the Research on Research Integrity Program (HHS ORI, Citation2016f) and the Rapid Response for Technical Assistance Program “to provide technical assistance to any institution that is responding to an allegation of research misconduct” and “to facilitate high-quality and well-documented investigations and help resolve research misconduct cases promptly” (HHS ORI, Citation2016d).
In 2002, ORI developed the RCR Resource Development Program (HHS ORI, Citation2016e) and the RCR Program for Academic Societies (HHS ORI, Citation2016c) based on an ORI-commissioned report from the Health and Medicine Division (HMD) of NAS (formerly known as the Institute of Medicine [IOM]) (see NAS, under Nonprofit Organizations). According to ORI, “The former program was designed to facilitate the development of materials for teaching the responsible conduct of research by the research community for use in the research community. The latter program, a collaboration with the Association of American Medical Colleges (AAMC), supported activities within academic societies designed to promote the responsible conduct of research among their members. The first RCR Expo was held in 2003 to call attention to the new RCR materials” (ORI, Citation2016a). ORI also published the ORI Introduction to the Responsible Conduct of Research, last updated in 2007 (Steneck, Citation2007), and created the RCR Program for Graduate Schools in collaboration with the Council of Graduate Schools to institutionalize RCR education in graduate training (Council of Graduate Schools, Citation2016).
A new regulation, PHS Policies on Research Misconduct (42 CFR Part 93), became effective in 2005 (HHS, Citation2005). Based on this regulation, ORI developed a training program for institutional Research Integrity Officers in 2005, which led to the production of a 2006 orientation video and 2007 boot camps. Also in 2007, ORI began the RCR Program for Postdocs and developed a laboratory management training program in collaboration with the Laboratory Management Institute at the University of California–Davis (ORI, Citation2016a).
In 2011, ORI developed an interactive video on research misconduct titled “The Lab: Avoiding Research Misconduct,” which includes role playing by participants as they make decisions about research integrity that can lead to consequences for the character or others in the lab. The RCR simulation includes topics such as avoiding research misconduct, mentorship responsibilities, data handling, responsible authorship, and questionable research practices. “The Lab” is available in English, Spanish, and Chinese (HHS ORI, Citation2016b).
ORI investigates research misconduct and confirms multiple cases a year. ORI posts case studies of confirmed research misconduct in which administrative actions were enforced, and only case studies that currently have an administrative action imposed are posted. ORI does not post the names of individuals whose administrative action periods have expired (HHS ORI, Citation2016a).
National Institutes of Health (NIH)
NIH is the nation's medical research agency and is the largest single funder of biomedical research (NIH, Citation2016c). In November 2012, NIH developed the NIH Policies and Procedures for Promoting Scientific Integrity (NIH Office of the Director, Citation2012). The agency describes its efforts in this area as follows (NIH Office of the Director, Citation2012):
The NIH Office of Extramural Research (OER) develops and oversees the implementation of policy for the NIH peer review process. This two-tiered system involves initial peer review for scientific and technical merit and subsequent review by advisory councils or boards… in the 27 institutes and centers of NIH (ICs) that are considering applications for funding… Scientific integrity is the cornerstone of the NIH peer review process and is exemplified in its core values: (1) expert assessment; (2) transparency; (3) impartiality; (4) fairness; (5) confidentiality; (6) integrity; and (7) efficiency… To help ensure scientific integrity in the initial peer review process, OER has developed a policy (NIH, Citation2011) for managing conflict of interest, the appearance of conflict of interest, prejudice, bias, or predisposition.
In December 2013, NIH launched an online forum called PubMed Commons to enable authors to share opinions and information about scientific publications in PubMed (http://www.ncbi.nlm.nih.gov/pubmedcommons/). To be eligible to submit comments on PubMed Commons, an individual must have at least one publication in PubMed. According to PubMed Commons (Citation2015), “About half the comments are on clinical or health-related publications. Members have been using PubMed Commons to: update and expand the public record, for instance by pointing to new data, relevant publications, or alternative interpretations; note corrections and retractions to publications; post discussion and critique, either directly or via links to blog posts and other platforms; provide links to datasets, code, or publicly accessible versions of publications; call attention to issues affecting reproducibility, such as cell line misidentification.”
In 2014, NIH held a joint workshop on reproducibility and rigor of research findings with Nature Publishing Group and Science, the journal of the American Association for the Advancement of Science (AAAS). Journal editors representing over 30 basic/preclinical science journals, representatives from funding agencies, and scientific leaders attended. The goal was to identify “the common opportunities in the scientific publishing arena to enhance rigor and further support research that is reproducible, robust, and transparent” (NIH, Citation2016b). Attendees reached consensus on a set of principles to facilitate these goals, including the following (NIH, Citation2016b):
(1) rigorous statistical analysis; (2) transparency in reporting; (3) data and material sharing; (4) consideration of refutations; (5) consider establishing best practice guidelines for: image-based data (image screening for manipulation, i.e., Western blots), description of biological material with enough information to uniquely identify the reagents (for example, unique accession number in repository), in particular for: antibodies: also report source, characteristics, dilutions and how they were validated; cell lines: also report source, authentication and mycoplasma contamination status; and animals: also report source, species, strain, sex, age, husbandry, inbred and strain characteristics of transgenic animals.
Since then, several journals have endorsed these principles.
NIH (Citation2015b) states that it is “…fully committed to maintaining public trust in the NIH research enterprise. Attempts to influence the outcome of the peer review process through inappropriate or unethical means result in needless expenditure of government funds and resources, and erode public trust in science.” In 2015, NIH launched a new website to help guide reviewers through the peer review process (https://grants.nih.gov/grants/policy/review.htm). The site provides step-by-step instructions, policy notices, guidelines, videos, and critique templates (NIH OER, 2016a). The NIH also released its Statement on Integrity in Peer Review in 2015 (NIH, Citation2015b), which included, “the responsibilities of all participants, including individuals named on applications and officials of applicant organizations, in maintaining the integrity of the NIH peer review process and potential consequences for any unlawful or unethical attempt to influence the outcome of NIH peer review” (NIH, Citation2016a).
NIH also published new guidelines on Enhancing Reproducibility Through Rigor and Transparency in Grant Applications with the goal “to enhance reproducibility of research findings” (NIH, Citation2015a). The mandatory guidelines went into effect for new grant applications starting in January 2016. The revised application instructions “focus on four areas deemed important for enhancing rigor and transparency: the scientific premise of the proposed research, rigorous experimental design for robust and unbiased results, consideration of relevant biological variables, and authentication of key biological and/or chemical resources” (NIH OER, 2016b). NIH developed a training module for intramural postdoctoral fellows on these guidelines, which emphasized good experimental design (NIH OER, 2016b).
US Food and Drug Administration (FDA)
FDA is responsible for protecting the public health by assuring the safety, efficacy, and security of human and veterinary drugs, biological products, medical devices, the nation's food supply, cosmetics, and products that emit radiation (FDA, Citation2016b). In 2009, FDA established the Office of Scientific Integrity (OSI). OSI works to “ensure that FDA's policies and procedures are current and applied across the Agency, and resolve scientific disputes that may arise internally or externally, among other functions” (FDA, Citation2016a). In response to the 2009 Presidential Memorandum, FDA issued an agency-wide policy in 2012 that presents 11 key principles of scientific integrity (FDA, Citation2012).
Centers for Disease Control and Prevention (CDC)
CDC is the nation's health protection agency and works to protect Americans from health, safety, and security threats, both foreign and domestic (CDC, Citation2016a). The CDC Office of the Associate Director for Science “provides service and support to the CDC scientists” and is “home to the Office of Science Quality, which is responsible for advancing the quality of CDC's science and championing the translation of research through the development of science policies and best practices (e.g., authorship, scientific clearance, peer review, and extramural research policies); and the Office of Scientific Integrity, which ensures that CDC science and research activities comply with various federal laws, regulations, and policies…,” among other functions (CDC, Citation2016b). The Office of the Associate Director for Science released the latest version of CDC's Guidance on Scientific Integrity in April 2016 (CDC Office of the Associate Director for Science, Citation2016).
US Environmental Protection Agency (EPA)
The mission of EPA is to protect human health and the environment (EPA, Citation2016a). EPA believes that “scientific integrity results from adherence to professional values and practices, when conducting and applying the results of science and scholarship. It ensures: objectivity; clarity; reproducibility; and utility. Scientific integrity is important because it provides insulation from: bias; fabrication; falsification; plagiarism; outside interference; censorship; and inadequate procedural and information security” (EPA, Citation2016b).
EPA's Principles of Scientific Integrity were published in 1999 and laid out basic rules for ethical behavior by all agency employees in conducting scientific research, interpreting and presenting results, and using scientific information and data (EPA, Citation2016c).
In response to the 2009 Presidential Memorandum, EPA released a draft policy on scientific integrity for public comment. EPA (Citation2014) stated that
All of the public comments were considered and, in combination with discussions with other Federal agencies, contributed to an improved final policy which was released in February 2012. The EPA Scientific Integrity Policy builds upon EPA's significant earlier scientific integrity efforts, focusing on the: (1) promotion of a culture of scientific integrity throughout the EPA; (2) release of scientific information to the public; (3) consistent use of peer review and federal advisory committees; and (4) professional development of government scientists.
EPA's scientific integrity official is the agency's focal point on scientific integrity and serves as the agency's expert on such matters. EPA's Scientific Integrity Committee implements the policy, and this committee consists of deputy scientific integrity officials representing each of the agency's program offices and regions (EPA, Citation2016b).
In 2014, EPA's recognition of the need for a greater statistical presence in science led to the creation of a Community of Practice for Statistics. It has the goal of strengthening statistics, considering study design, and ensuring that these same factors are evaluated during the review and approval of study protocols. Three working groups developed process and guidance documents to add value during the design phase for three types of research approaches: experimental, observational, and modeling (George et al., Citation2015).
In 1998, EPA developed the Peer Review Handbook to “provide guidance to EPA staff and managers who are planning and conducting peer reviews. It is intended to improve the internal management of EPA peer review by providing recommended procedures and approaches for EPA staff and managers” (EPA Science and Technology Policy Council, Citation2015a). The handbook was revised in 2000 (second edition), 2006 (third edition), and 2009 (with an addendum on the “Appearance of a Lack of Impartiality in External Peer Review”). The fourth edition in 2015 incorporated agency organizational changes, the 2009 addendum, and additional processes put in place after 2006 (EPA Science and Technology Policy Council, Citation2015b).
US Department of Agriculture (USDA)
USDA provides leadership on food, agriculture, natural resources, rural development, and nutrition. In 2013, USDA issued a Departmental Regulation (DR) on Scientific Integrity in response to the 2010 OSTP implementation guide. The DR provides guidance to all employees and contractors on “the proper use of scientific findings and the principles of conducting scientific activities” (USDA Office of the Chief Scientist, Citation2013). USDA then released its Scientific Integrity Policy Handbook in 2013 (USDA, Citation2013). USDA has agency scientific integrity officers in place at each of its agencies to ensure that policies are aligned with the USDA policy (USDA Office of the Chief Scientist, Citation2016) and issues a Scientific Integrity Allegations Summary Report on an annual basis (USDA, Citation2015).
US Agency for International Development (USAID)
The USAID is the lead US government agency that works to end extreme global poverty and enable resilient, democratic societies to realize their potential (USAID, Citation2016). In March 2012, USAID released its Scientific Integrity Policy (USAID, Citation2012), which states the following:
This document articulates the principles the Agency will follow to ensure the integrity of its scientific and scholarly activities, including how they are supported and carried out, and research findings are used and disseminated. These principles pertain to five specific aspects of the Agency's activities: (1) protecting the scientific process from misconduct and from inappropriate influence; (2) promoting access to scientific and technical information; (3) maintaining a highly skilled technical and scientific staff; (4) using federal advisory committees ethically and transparently; and (5) ensuring quality, methodological rigor, and ethical standards in all USAID-funded research activities. Future development of specific, implementation-focused policies and updates to existing policies in each of these areas will use the framework provided by this scientific integrity policy.
USAID released its Scientific Research Policy in December 2014, which consists of “operational policies of particular relevance to the design and management of research activities,” including quality standards for research plans and reports and open data. The policy also covers scientific peer review and publication (USAID, Citation2014).
Foundations
presents details of scientific integrity by foundations.
Laura and John Arnold Foundation (LJAF)
Research integrity is one main area of focus for the LJAF. According to the foundation, “LJAF's Research Integrity initiative aims to improve the reliability and validity of scientific evidence across fields that inform governmental policy, philanthropic endeavors, and individual decision making” (LJAF, Citation2016b). LJAF supports organizations that are committed to improving the openness, transparency, and quality of research. Between 2011 and 2015, the foundation provided over $85 million in grants for research integrity projects. Between 2013 and 2018, the foundation is funding eight grants to the Center for Open Science (COS), totaling over $10 million (see COS, under Nonprofit Organizations). LJAF provided a $100,000 grant, between 2014 and 2016, to AAAS (see AAAS, under Professional Societies) to foster open, reliable, and rigorous scientific research by sponsoring three workshops on publication standards (LJAF, Citation2016a). LJAF has awarded another $300,000 grant to the Center for Scientific Integrity to continue work from 2015 to 2017 on the Center's Retraction Watch database and support other projects (see Center for Scientific Integrity, under Nonprofit Organizations) (Retraction Watch, Citation2015).
John Templeton Foundation
The John Templeton Foundation is a philanthropic organization whose vision includes a commitment to rigorous scientific research and related scholarship. The foundation provided over $2 million to COS for activities in 2014–2016 to increase scientific openness and integrity and to expand the features and connectivity of the Open Science Framework (OSF) (see COS, under Nonprofit Organizations) (Templeton Foundation, Citation2016).
In 2015, the foundation provided an $80,000 grant to the Berkeley Initiative for Transparency in the Social Sciences (BITSS) to establish the Leamer–Rosenthal Prizes for Open Social Science (see BITTS, under Academia) (PRWeb, Citation2015).
John D. and Catherine T. MacArthur Foundation
The John D. and Catherine T. MacArthur Foundation is one of the nation's largest independent foundations (MacArthur Foundation, Citation2016). In 2014, the MacArthur Foundation awarded the Center for Scientific Integrity a $400,000 grant over 2 years to help the Center operate and expand Retraction Watch. The goal of the grant is to create a comprehensive and freely available database of retractions, something that did not previously exist. This was a gap that deprived scholarly publishing of a critical mechanism for self-correction (see the Center for Scientific Integrity, under Nonprofit Organizations) (Retraction Watch, Citation2014).
Alfred P. Sloan Foundation
The Alfred P. Sloan Foundation is a private philanthropic organization that provides grants on various subjects, including digital information technology (Sloan Foundation, Citation2016). The program, launched in 2011, has the following aims (Sloan Foundation, Citation2013):
[T]o leverage developments in digital information technology to empower scientists, and enable new forms of data intensive research. The program works directly with researchers, technologists, engineers, academic publishers, and university administrators to speed the development and adoption of tools, standards, norms, and practices that will enable researchers to better communicate with one another and more effectively work with large datasets. The program also seeks to support and encourage the effective use of new datasets, data repositories, data dissemination, and computational techniques across the Sloan Foundations' other grant making programs.
In 2014, the Sloan Foundation awarded $168,600 grant to COS to connect the OSF with tools created by other open-source and open-science service providers. The funds support data management planning, preregistration of research designs, data archiving, data analysis, and journal management (see COS, under Nonprofit Organizations) (COS, Citation2016c).
Nonprofit organizations
Key work by nonprofit organizations is included in .
National Academy of Sciences (NAS)
NAS is a private, nonprofit organization of the country's leading researchers. Through its National Academies of Sciences, Engineering, and Medicine, the NAS provides objective, science-based advice on critical issues affecting the nation (NAS, Citation2016c). The NAS IOM issued a report in 1989, which recommended that “scientific organizations representing the research community should develop educational and training activities and materials to improve the integrity of research [and that] scientific journals should develop policies to promote responsible authorship practices, including procedures for responding to allegations or indications of misconduct in published research or reports submitted for publication” (AAAS, Citation2000).
The NAS Committee on Science, Engineering, and Public Policy convened a panel on scientific responsibility and the conduct of research and published Responsible Science: Ensuring the Integrity of the Research Process in 1992 (NAS, Citation1992). Also in 1992, AAAS issued a report titled Good Science and Responsible Scientists, which examined misconduct in science and the response of the scientific community. The report noted that “scientific societies serve as custodians of their disciplines' distinct knowledge, traditions, and professional norms. The… standards of proper research practices adopted by a scientific society embody the collective conscience of the discipline and are an expression of its ethical responsibilities” (AAAS, Citation2000). Among the report's findings was that scientific societies “play a major role in influencing the moral tone and ethical climate in which research is conducted” (AAAS, Citation2000).
In 2002, HHS ORI commissioned NAS to form a Committee on Assessing Integrity in Research Environments, which published its report Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct. The report focused on “the research environment and attempts to define and describe those elements that allow and encourage unique individuals, regardless of their role in the research organization or their backgrounds on entry, to act with integrity” (NAS, Citation2002).
Several overarching conclusions emerged, including: (1) attention to issues of integrity in scientific research is very important to the public, scientists, the institutions in which they work, and the scientific enterprise itself; (2) no established measures for assessing integrity in the research environment exist; (3) promulgation of and adherence to policies and procedures are necessary, but they are not sufficient means to ensure the responsible conduct of research; (4) there is a lack of evidence to definitively support any one way to approach the problem of promoting and evaluating research integrity; (5) education in the responsible conduct of research is critical, but if it is not done appropriately and in a creative way, it is likely to be of only modest help and may be ineffective; and (6) institutional self-assessment is one promising approach to assessing and continually improving integrity in research.
As a result of the report, HHS ORI established two key programs in 2002, the RCR Resource Development Program and the RCR Program for Academic Societies (NAS, Citation2002) (see HHS ORI, under Federal Agencies).
Recognizing that the foundation of the scientific research enterprise is built on trust, both NAS and NSF funded a study that was the basis for the third edition of On Being a Scientist: A Guide to Responsible Conduct in Research in 2009, which was originally published in 1989 (NAS, Citation2009a). The guide (NAS, Citation2009a) aims to ensure that
…the scientific community devotes itself to exemplifying and transmitting the values associated with ethical scientific conduct… On Being a Scientist is designed to supplement the informal lessons in ethics provided by research supervisors and mentors… and …describes the ethical foundations of scientific practices and some of the personal and professional issues that researchers encounter in their work… The book is aimed primarily at graduate students and beginning researchers, but its lessons apply to all scientists at all stages of their scientific careers. A continuing feature of this Guide is the inclusion of a number of hypothetical scenarios offering guidance in thinking about and discussing these scenarios.
Also in 2009, NAS published the report Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age, which included three principles: (1) researchers are responsible for ensuring the integrity of their data, (2) data from published papers should be publicly accessible, and (3) data should be properly archived. The report offered 11 recommendations urging scientists, institutions, journals, and other players to develop standards and provide proper training (Kaiser, Citation2009; NAS, Citation2009b).
The 2014 HMD (formerly known as IOM) workshop on “Conflict of Interest and Medical Innovation: Ensuring Integrity While Facilitating Innovation in Medical Research” addressed the scientific integrity issues that arise with collaborative efforts among different stakeholders. According to Beachy et al. (Citation2014), “While the potential benefits of collaboration are significant, the fact that the relationships among development partners are often financial means that it is vital to ensure trust by identifying, disclosing, and managing any potential sources of conflict that could create bias in the research being performed together.”
The National Research Council Roundtable on Public Interfaces of the Life Sciences hosted a workshop in 2015 titled “Does the Public Trust Science? Trust and Confidence at the Intersections of the Life Sciences and Society.” The event aimed to explore “the nuanced nature of trust in science,” including “what the elements of trust are, and how trust is built, maintained, or lost” (NAS, Citation2015a). The workshop summary was published in 2015 (NAS, Citation2015b).
NAS also developed the Arthur M. Sackler Colloquia, which addresses scientific topics of broad and current interest that cut across the boundaries of traditional disciplines (NAS, Citation2016a). The colloquia will present a conference on “Reproducibility of Research: Issues and Proposed Remedies” in Washington, DC, in 2017 (NAS, Citation2016b).
Center for Open Science (COS)
COS is a nonprofit technology startup founded in 2013 with the mission to “…increase openness, integrity, and reproducibility of scientific research” (COS, Citation2016a). COS pursues this mission by building communities around open science practices, supporting metascience research, and developing and maintaining free, open-source software tools. COS's efforts aim to inform best practices and serve as platforms to demonstrate reproducible research methods (COS, Citation2016a).
Although it is a relatively new organization, COS has become a major player in advancing scientific integrity and has secured funding from three major foundations (see LJAF, Templeton Foundation, and Sloan Foundation, under Foundations).
COS has a 2-year $2.1 million grant from the Templeton Foundation, which began in 2014, to build the infrastructure for the OSF. This is a web application that connects and supports research workflow, enabling scientists to increase the efficiency and effectiveness of their research (OSF, Citation2016). The organization describes the OSF as “a scholarly commons to connect the entire research cycle. Researchers can use the OSF to collaborate, document, archive, share, and register research projects, materials, and data. The grant allows COS to build communities around open science values. Scientific integrity can be improved with strategies that make the fundamental but abstract accuracy motive—getting it right—competitive with the more tangible and concrete incentive—getting it published.” Three major activities of the OSF are as follows: “(1) building the OSF to provide features that provide value to the scientist's existing workflow and enables or automates good practices, (2) building community, training and outreach to facilitate use of the OSF, and (3) connecting a variety of tools (e.g., data repositories, data visualization tools, analytic tools) through the OSF to support the entire research lifecycle and facilitate documentation and archival of research materials and data” (Templeton Foundation, Citation2016).
COS held a workshop in 2014 that was “aimed at creating standards to promote transparent and reproducible research in the social-behavioral sciences” (Christensen, Citation2016). Representatives came together from across the scientific disciplines, including funders, publishers, editors, and researchers from over 40 leading institutions (Christensen, Citation2016).
COS has received seven grants from LJAF (Citation2016a), with the following goals: (1) to encourage preregistration of scientific studies by offering monetary awards to selected researchers who publish studies that have been preregistered on the OSF, (2) to support the creation of a new database of clinical trials, and (3) to set up four COS communities (Publishing Initiatives, Metascience, Infrastructure, and Interest Groups).
Projects within the Publishing Initiatives community include the following (COS, Citation2016b):
(1) COS Communities Badges to Acknowledge Open Practices: Badges incentivize open research by rewarding authors for three distinct practices: open data, open materials, and preregistration. As a visual reward, badges appear directly on publications… (2) Registered Reports offer journals an alternative structure to the current publishing format to promote transparency and reproducibility in scientific research. In this model, peer review occurs twice. Each study procedure and analysis plan are evaluated prior to data collection for in-principle acceptance; if accepted, the final manuscript is essentially guaranteed publication regardless of the reported outcome, with a second peer review to ensure the accepted methodology was conducted. This supports the publication of negative or neutral results instead of only positive results into the scientific literature.
Projects within the Metascience community include the following (COS, Citation2016b):
(1) Reproducibility Project–Psychology: The project is a collaborative community effort to replicate published psychology experiments from three journals. Replication teams follow a standard protocol to maximize consistency and quality across replications, and the accumulated … workflow are to be open for critical review on the OSF. (2) Reproducibility Project–Cancer Biology: The project is an initiative to conduct direct replications of 50 high-impact cancer biology studies. The project anticipates learning more about predictors of reproducibility…. (3) Collaborative Replications and Education Project: The project facilitates student research training through conducting replications. The community-led team composed a list of studies that could be replicated as part of research methods courses… Replication teams are encouraged to submit their results to an information commons for aggregation for potential publication…(4) Crowdsourcing a Dataset: …a method of data analysis in which multiple independent analysts investigate the same research question on the same data set in whatever manner they consider to be best. This first crowdsourcing project establishes a protocol for independent simultaneous analysis of a single dataset by multiple teams, and resolution of the variation in analytic strategies and effect estimates among them.
Projects within the Infrastructure community include open-source developers and the OSF.
Projects within the Interest Groups community include the following (COS, Citation2016b):
(1) Ambassadors: COS trains ambassadors to act as the local authority on COS, the OSF, and open science practices in their community…. (2) Open Science Collaboration is a network of researchers, professionals, citizen scientists, and others with an interest in open science, metascience, and good scientific practices. The goal is to promote open collaboration of scientific ideas(COS, 2016b).
Center for Scientific Integrity
The Center for Scientific Integrity's mission is “to promote transparency and integrity in science and scientific publishing, and to disseminate best practices and increase efficiency in science” (Retraction Watch, Citation2016). A major product of the Center is a database of retractions generated by the work of Retraction Watch (http://retractionwatch.com/), the first publicly available database to compile retracted scientific publications (Retraction Watch, Citation2016). Retraction Watch is funded mainly by a $400,000 grant from LJAF (see LJAF, under Foundations) (Retraction Watch, Citation2014).
Professional societies
Key work by professional societies is included in .
Committee on Publication Ethics (COPE)
COPE was initiated by medical journal editors in the United Kingdom in 1997 but has expanded to over 10,000 editors of academic journals and persons interested in publication ethics worldwide. COPE “…provides advice to editors and publishers on all aspects of publication ethics and, in particular, how to handle cases of research and publication misconduct. It also provides a forum for its members to discuss individual cases. COPE does not investigate individual cases but encourages editors to ensure that cases are investigated by the appropriate authorities (usually a research institution or employer)” (COPE, Citation2016a). COPE maintains a database of all cases that have been discussed at the COPE Forum since 1997 as well as podcasts of the discussion, including the outcome of the cases and the advice given. The database now contains over 500 cases (COPE, Citation2016b).
COPE has produced an eLearning course that provides the tools and knowledge to enhance one's skills in publication ethics. The course has 11 modules, including 10 that are currently available to members: selective reporting; reviewer misconduct; redundant publication; plagiarism; introduction to publication ethics; falsification; fabrication; corrections, retractions and expressions of concern; conflict of interest; and authorship (COPE, Citation2016d).
COPE also funds research on publication ethics (COPE, Citation2016e), organizes annual seminars globally (COPE, Citation2016f), and has created an audit tool for members to measure compliance with its Code of Conduct and Best Practices Guidelines for Journal Editors, which all members are expected to follow (COPE, Citation2016c).
Council of Science Editors (CSE)
CSE is an international membership organization for editorial professionals publishing in the sciences (CSE, 2016). CSE works with other professional organizations to shape the scientific journal environment so that the integrity of publications is upheld.
Since 2006, CSE's Editorial Policy Committee has maintained a white paper titled “Promoting Integrity in Scientific Journal Publications”(CSE, Citation2012b). The most recent update of the white paper in 2012 includes “information on citation manipulation, publication planning by study sponsors, and ethical conduct of sponsors; reorganization of the section on reporting suspect manuscripts; updated information on international models for responding to research misconduct; and more recent examples of corrections, retractions, and expressions of concern” (CSE, Citation2012a). The white paper is a living document that is continuously updated through feedback received from members.
American Association for the Advancement of Science (AAAS)
AAAS is an international nonprofit organization dedicated to advancing science for the benefit of all people. It is the world's largest general scientific society and publishes the journal, Science (AAAS, Citation2016a).
One of the highest priorities for AAAS in the late 1990s and early 2000s focused on the role of scientific societies in promoting research integrity. In 2000, AAAS published a report summarizing proceedings from an April 2000 conference on “The Role and Activities of Scientific Societies in Promoting Research Integrity,” which was cosponsored by AAAS and ORI. The report reviewed some of the recent history of the perceived roles and activities of scientific societies in promoting ethical conduct, discusses codes of ethics and support activities, and concluded with some findings and recommendations for research and action related to the societies' roles in promoting research integrity (AAAS, Citation2000).
At the 31st Annual AAAS Forum on Science and Technology Policy in 2006, a panel titled “Protecting the Integrity of Science” explored the subtle dilemmas of science ethics and integrity, with presentations on “Attacks on Peer Review,” “Political Uses of Science,” and “Protecting the Integrity of Science: Scientific Misconduct” (AAAS, Citation2006).
AAAS was awarded a $100,000 grant from LJAF (which runs from 2014 to 2016) to “foster open, reliable, and rigorous scientific research by sponsoring [three] workshops on publication standards” (LJAF, Citation2016a) (see NIH, under Federal Agencies; and LJAF, under Foundations).
AAAS has developed a video series to “help improve the ability of scientists, post-doctoral fellows, undergraduate and graduate students, administrators, and technicians to develop informed and well-reasoned responses to ethical issues that arise in scientific research” (AAAS, Citation2016b). The series includes (AAAS, Citation2016b):
…five ‘trigger’ videos, short dramatizations aimed at provoking discussion on a series of ethical issues confronting various participants in the research process, and a Discussion and Resource Guide. Among the topics addressed in the videos are: role and responsibilities of mentors and lab chiefs; determination of authorship; allocation of credit; impact of legal rules on conduct of science; data retention, selection, sharing, and reporting; pressures in the research environment; sloppiness in research; scientific misconduct and institutional responses; whistle blowing; peer review; animals in research; intellectual property; commingling of private and public funds for research; privileged information; and responsibilities of collaborators.
Federation of American Societies for Experimental Biology (FASEB)
FASEB is the nation's largest coalition of biomedical researchers, representing 30 scientific societies and over 125,000 researchers from around the world (FASEB, Citation2016b). FASEB believes that maintaining public trust in medical research and preventing the introduction of bias is absolutely critical, and the organization supports efforts to preserve the integrity of science (FASEB, Citation2016c).
In January 2016, FASEB published a set of recommendations on enhancing research reproducibility, which resulted from FASEB's 2015 Science Policy Symposium and subsequent roundtable discussions. Participants agreed that “three general factors impede the ability to reproduce experimental results: lack of uniform definitions to describe the problem, insufficient reporting of key experimental details, and gaps in scientific training” (FASEB, Citation2016a). The recommendations included definitions for terms used in describing research, improvements in reporting of research materials and methods, and focus on “two key tools critical to basic research: mouse models and antibodies,” and advocacy for “robust training of researchers in rigorous experimental design” (FASEB, Citation2016a).
American Sociological Association (ASA)
ASA, founded in 1905, is a nonprofit membership association dedicated to advancing sociology as a scientific discipline and profession serving the public good. In 2004, ASA issued a statement on maintaining the integrity of U.S. Presidential Appointments of scientists, strongly urging the President of the United States to “consider scientific expertise as the primary basis for soliciting and nominating or appointing advisors to scientific, technological, and health-related posts or governmental advisory committees” (ASA, Citation2004). ASA believes that scientific expertise should play the dominant role (implicitly or explicitly) in the President's decision about whom to select for these positions. This will ensure that “scientific expertise is the primary consideration in such appointments, as a means to help protect the influence of America's critical scientific, technological, and health science enterprises” (ASA, Citation2004).
In 2006, ASA issued a statement on academic independence and scientific integrity. The statement affirmed ASA's “ongoing support for the protection of academic independence and the integrity of scientific research through the open movement of faculty and students between universities irrespective of nationality or political views. Similarly, ASA also strongly endorses the principle of scientific worthiness as the primary basis for assessing articles for scholarly publication, service on editorial boards of scholarly journals, and participation in scholarly conferences, not excluding persons on the basis of nationality or political views” (ASA, Citation2006).
American Chemical Society (ACS)
ACS is a scientific society that supports scientific inquiry in the field of chemistry (ACS, Citation2016a). ACS published a Position Statement on Scientific Integrity in Public Policy for 2014–2017. Within this statement, ACS “encourages scientific integrity policies that help the federal government obtain and integrate scientific assessments into policy development and implementation” (ACS, Citation2016c). The statement includes sections on recommendations for government, scientific processes and procedures, data quality use and review, and scientific access and advice (ACS, Citation2016c).
ACS's 2015 webinar titled “Catching Errors: Peer Review and Retractions in Publishing” included speakers from Retraction Watch, COPE, and the Center for Clinical Research Ethics (CCRE). The speakers discussed the efforts that are being made to combat issues in peer review as well as what could be changed to improve the review process (ACS, Citation2016b).
Academia
Key work by academia is included in .
Institute of Clinical And Translational Sciences (ICTS)
ICTS at Washington University in St. Louis was established in 2007 through funding from the NIH Clinical and Translational Science Award program, Washington University, and BJC HealthCare (ICTS, Citation2016e). According to its website, “ICTS and the Office of the Vice Chancellor for Research at Washington University jointly sponsor a course on RCR. Some of the topics covered in the course include: Conflicts of Interest; Research Misconduct; Mentor/Mentee Responsibilities and Relationships; Collaborative Research; Responsible Authorship and Publication; The Scientist as a Responsible Member of Society; and Data Management, Ownership, and Sharing. The course is designed to satisfy both the NIH and the NSF RCR training requirements” (ICTS, Citation2016f).
Center for Clinical Research Ethics (CCRE)
CCRE is a partnership between ICTS at Washington University in St. Louis and the Center for Health Care Ethics at Saint Louis University. CCRE “exists to provide education in the areas of clinical research ethics and RCR, to conduct research on ethics and integrity in research, and to provide consultations to investigators on ethical issues in clinical and translational research or the design of studies on ethical issues” (ICTS, Citation2016a).
CCRE sponsored a Professional Integrity program, which began in 2012. This is the first remediation education program that provides “intensive professional development education for investigators who have engaged in wrongdoing or unprofessional behavior” (ICTS, Citation2016b).
CCRE has also developed a Library of Research Ethics Case Studies (ICTS, Citation2016d). In addition, CCRE faculty have ongoing research and scholarly projects in a variety of areas, including the understanding and preventing wrongdoing in research and environmental factors predictive of misbehavior. CCRE faculty researchers have received grants from HHS ORI and NIH, among other agencies (ICTS, Citation2016c).
Center for Advanced Study in the Behavioral Sciences (CASBS)
CASBS at Stanford University was founded in 1954 (CASBS, Citation2016a). Since 2013, CASBS has hosted a group of scholars called the Group on Best Practices in Science. The group is a collaboration coordinated by Stanford University, Rutgers University, University of California–Davis, and many other institutions. The collaboration “represents one attempt to increase the validity and credibility of scientific research. It (1) documents research on how scientific practices may become compromised and (2) provides a platform to spur discussion about such issues” (CASBS, Citation2016b). The group is currently engaged in the following efforts (CASBS, Citation2016c):
(1) writing a memorandum for the White House about how scientific practice can become compromised and how the federal government can help improve scientific practices; (2) writing a grant proposal for the NSF; (3) developing archival studies assessing: the evidentiary value and prevalence of p-hacking, the association of sample size with journal impact factor, changes in sample size over time, the rate at which hypotheses are clearly stated and operationalized, and how well suggested reforms have succeeded at increasing the credibility and validity of research findings; and (4) developing questionnaire studies on: exploratory interviews to design questionnaires assessing scientists' beliefs about the prevalence of integrity-impairing practices in their discipline, etc.
CASBS hosted a “Best Practices in Science” conference in June 2015. The conference had two primary purposes: (1) to share ideas among those actively conducting research on scientific integrity with one another and the wider scholarly community and (2) to set the stage for the participants to generate articles on scientific integrity to be published in a special issue of a journal or edited book (CASBS, Citation2015).
Berkeley Initiative for Transparency in the Social Sciences (BITSS)
BITSS is “an international network of researchers and institutions committed to improving the standards of openness and integrity in economics, political science, psychology, and related disciplines” (BITSS, Citation2016c). BITSS further describes its efforts as follows (BITSS, Citation2016c):
BITSS is a program of the Center for Effective Global Action at the University of California, Berkeley. Central to BITSS efforts is the identification of useful tools and strategies for increasing transparency and reproducibility in research, including the use of study registries, pre-analysis plans, version control, data sharing platforms, disclosure standards, and replications. BITSS seeks to achieve its objectives by identifying practical resources and tools for increasing openness and reproducibility, in close partnership with data scientists and empirical researchers; and promoting the dissemination and adoption of effective approaches through training courses, grant competitions, and coordination with funders, government agencies, professional societies, and academic journals.
Furthermore, BITSS (Citation2016a) states that “Today, researchers are not explicitly rewarded for disclosing their data collection and analysis methods, registering detailed pre-analysis plans, or making data and other research materials available to the public… In order to promote… transparent research, and to offer recognition and visibility to scholars practicing open social science, BITSS has launched two prizes named for pioneers who laid the foundation for transparency…,” and these efforts are supported by the Templeton Foundation (see Templeton Foundation, under Foundations). Beginning in 2015, the Leamer–Rosenthal Prizes for Open Social Science offered prizes for emerging researchers and leaders in education. Prizes for Emerging Researchers focus on “early-career researchers—junior faculty, postdoctoral researchers or graduate students—who adopt transparent research practices or pioneer new methods to increase the rigor of research” (BITSS, Citation2015). The Leaders in Education Prize “awards the work of professors who incorporate instruction in transparent practices in social science research into their curricula” (BITSS, Citation2015). The total available prize money is $60,000 for emerging researchers (in $10,000–$15,000 increments) and $20,000 for leaders in education (in $10,000 increments) (BITSS, Citation2016b).
Linkages of activities across entities working on scientific integrity
illustrates where the five different sectors are focusing on scientific integrity issues and helps to identify areas in which more attention is needed.
Federal agencies have focused on developing their individual scientific integrity policies in response to the 2009 Presidential Memorandum with guidance from the OSTP implementation guide. Within their policies and work, the agencies have concentrated on different aspects of scientific integrity, depending on their unique mission. Several agencies, including the USDA, CDC, EPA, and FDA, have created an office of scientific integrity that oversees the implementation of their scientific integrity policies. Some have created officials within these departments, including deputy scientific integrity officials at EPA and scientific integrity officers at USDA. NSF has formal CFR regulations on what defines research misconduct, whereas HHS ORI publishes cases of research misconduct and USDA issues an annual allegations summary report. NIH has focused on creating policies for integrity in the peer review process and establishing guidelines on enhancing reproducibility through rigor and transparency in grant applications. Peer review is also a focus for EPA, which developed the Peer Review Handbook. In addition, EPA has a program of work to address the need for a greater statistical presence in scientific research. HHS ORI and NSF have developed training for scientists on responsible conduct in research, such as HHS ORI's interactive video on research misconduct.
The development of training modules and educational tools is also an important aspect of addressing scientific integrity for other organizations and institutions. NAS has published reports on integrity in scientific research and developed a guide on responsible conduct in research to train scientists. COPE has an eLearning course that provides extensive training for journal editors and publishers on publication ethics. AAAS has a video series for scientists and students on ethical issues in scientific research. ICTS teaches a course on the responsible conduct of research, and CCRE produces a course that is designed to be a remediation course for those who have engaged in misconduct. Similar to HHS ORI and USDA, CCRE also has its Library of Research Ethics Case Studies that can be used as an educational tool.
Three successful efforts to create online forums for discussion of scientific integrity include NIH PubMed Commons, the Center for Scientific Integrity's Retraction Watch, and the OSF created by COS. PubMed Commons and Retraction Watch are public forums for discussion on the integrity of publications and the research they present, whereas the OSF is a scholarly commons to connect the entire research cycle and provides the tools to share protocols, raw data, and analysis. This commitment to open science is possible in part through the generosity of foundations cited in this paper, who are funding grants to nonprofit organizations and academia to build the necessary infrastructure and communities needed to drive open science. These funds also support the development of tools needed to effectively work within large data sets.
Like NIH, other groups have focused on issues of reproducibility. COS held a workshop on this topic that included 40 leading institutions. Recently, FASEB published a set of recommendations on enhancing research reproducibility. BITSS focuses on the identification of tools and strategies for increasing transparency and reproducibility. NAS will examine the issues and remedies for reproducibility of research at a future colloquium that is already scheduled.
Beyond NIH and EPA, peer review and publication ethics have been addressed by professional societies including COPE, CSE, and AAAS. ACS held a webinar for members that addressed issues of peer review and retractions.
In academia, CASBS hosts the Group on Best Practices in Science for researchers to convene and attempt to increase the validity and credibility of scientific integrity. This includes documenting research on how scientific practices may become compromised, which will be outlined in a memorandum to the White House. BITTS has created two monetary prizes that are awarded to researchers and educators who exemplify aspects of scientific integrity.
Overall, there is tremendous activity in the area of scientific integrity and there are clear linkages among the efforts of the federal agencies, foundations, nonprofit organizations, professional societies, and academia. Yet there continues to be a need to address the issue of scientific integrity despite the work that has already been done. The better the pressures that lead to misconduct are understood, the more effective efforts will be to instill scientific integrity in researchers. Scientific integrity needs to remain visible in the scientific community and evolve along with new research paradigms. All stakeholders must place high priority in instilling these values.
A comprehensive analysis of the details of the individual federal agency scientific integrity policies as well as the work done by foundations, nonprofit organizations, professional societies, and academia was not conducted for this publication. A second manuscript will be forthcoming with the goal to synthesize the policies of these federal agencies and organizations into a set of principles or best practices for scientific integrity.
Acknowledgments
This paper is intended to be a living document and updates will be posted on the ILSI North America website (http://ilsina.org; E-mail: [email protected]). Please help keep this document current by pointing out areas that need to be expanded or updated or additional organizations that should be included. Please send comments or suggestions to [email protected] with the subject line “Scientific Integrity.”
ILSI North America is a public, nonprofit foundation that provides a forum to advance understanding of scientific issues related to the nutritional quality and safety of the food supply by sponsoring research programs, educational seminars and workshops, and publications. ILSI North America receives support primarily from its industry membership.
References
- AAAS. (2000). The Role and Activities of Scientific Societies in Promoting Research Integrity: A Report of a Conference, April 10–11, 2000. Available from: http://www.aaas.org/sites/default/files/content_files/The%20Role%20and%20Activities%20of%20Scientific%20Societies%20in%20Promoting%20Research%20Integrity.pdf [Accessed August 3, 2016].
- AAAS. (2006). 2006 Forum, 20–21 April 2006, Washington, DC. Available from: http://www.aaas.org/news/2006-forum [Accessed August 3, 2016].
- AAAS. (2016a). About AAAS.Available from: http://www.aaas.org/about-aaas[Accessed August 3, 2016].
- AAAS. (2016b). Integrity in Scientific Research Video Series. Available from: http://www.aaas.org/page/integrity-scientific-research-video-series[Accessed August 3, 2016].
- ACS. (2016a). About ACS. Available from: http://www.acs.org/content/acs/en/about.html[Accessed August 3, 2016].
- ACS. (2016b). ACS Webinars: Catching Errors: Peer Review and Retractions in Publishing. Available from: http://www.acs.org/content/acs/en/acs-webinars/popular-chemistry/catching-errors.html[Accessed August 3, 2016].
- ACS. (2016c). Scientific Integrity in Public Policy. Available from: http://www.acs.org/content/acs/en/policy/publicpolicies/promote/scientificintegrity.html[Accessed August 3, 2016].
- ASA. (2004). Statement of the American Sociological Association on Maintaining the Integrity of U.S. Presidential Appointments of Scientists. Available from: http://www.asanet.org/about-asa/how-asa-operates/council-statements/integrity-presidential-appointment-scientists[Accessed August 3, 2016].
- ASA. (2006). Statement by the Council of the American Sociological Association on Academic Independence and Scientific Integrity. Available from: http://www.asanet.org/about-asa/how-asa-operates/council-statements/academic-independence-and-scientific-integrity[Accessed August 3, 2016].
- Beachy, S. H., Berger, A. C., and Olson. S.(Rapporteurs). (2014). Roundtable on Translating Genomic-Based Research for Health, Board on Health Sciences Policy, Institute of Medicine. Conflict of Interest and Medical Innovation: Ensuring Integrity While Facilitating Innovation in Medical Research: Workshop Summary. Washington, DC, National Academies Press. Available from: http://www.nap.edu/catalog/18723/conflict-of-interest-and-medical-innovation-ensuring-integrity-while-facilitating.
- BITSS. (2015). Winners of Inaugural Social Science Transparency Prizes Announced (Press Release, December 10, 2015). Available from: http://www.bitss.org/lr-prizes/leamer-rosenthal-prizes-press-release/[Accessed August 3, 2016].
- BITSS. (2016a). The Leamer–Rosenthal Prizes for Open Social Science. Available from: https://bitssblog.wordpress.com/prizes/[Accessed August 3, 2016].
- BITSS. (2016b). Leamer–Rosenthal Prizes Nomination Process. Available from: http://www.bitss.org/lr-prizes/leamer-rosenthal-prizes-nomination-process/[Accessed August 3, 2016].
- BITSS. (2016c). Mission. Available from: https://bitssblog.wordpress.com/about/mission/[Accessed August 3, 2016].
- CASBS. (2015). CASB Group on Best Practices in Science: 2015 Conference. Available from: http://bps.stanford.edu/?page_id=4587[Accessed August 3, 2016].
- CASBS. (2016a). About CASBS. Available from: https://casbs.stanford.edu/about-casbs[Accessed August 3, 2016].
- CASBS. (2016b). CASB Group on Best Practices in Science. Available from: http://bps.stanford.edu/[Accessed August 3, 2016].
- CASBS. (2016c). CASB Group on Best Practices in Science: Group Activities. Available from: https://bps.stanford.edu/?page_id=1977[Accessed August 3, 2016].
- CDC Office of the Associate Director for Science. (2016). CDC Guidance on Scientific Integrity, Version 2.1. Available from: https://www.cdc.gov/od/science/docs/cdcsiguide_042516.pdf[Accessed August 3, 2016].
- CDC. (2016a). About CDC 24-7: Mission, Role and Pledge. Available from: http://www.cdc.gov/about/organization/mission.htm[Accessed August 3, 2016].
- CDC. (2016b). Advancing Excellence & Integrity of CDC Science: Overview. Available from: http://www.cdc.gov/od/science/aboutus/index.htm[Accessed August 3, 2016].
- Christensen, G. (2016). Creating Standards for Reproducible Research: Overview of COS Meeting. Available from: https://bitssblog.wordpress.com/2014/11/06/creating-standards-for-reproducible-research-overview-of-cos-meeting/[Accessed August 3, 2016].
- Collins, F. S., and Tabak, L. A. (2014). Policy: NIH plans to enhance reproducibility (comment). Nature.505:612–613.
- COPE. (2016a). About COPE. Available from: http://publicationethics.org/about[Accessed August 3, 2016].
- COPE. (2016b). Cases Search. Available from: http://publicationethics.org/cases[Accessed August 3, 2016].
- COPE. (2016c). Code of Conduct and Best Practice Guidelines for Journal Editors.Available from: http://publicationethics.org/files/Code_of_conduct_for_journal_editors_Mar11.pdf[Accessed August 3, 2016].
- COPE. (2016d). E-Learning.Available from: http://publicationethics.org/resources/e-learning[Accessed August 3, 2016].
- COPE. (2016e). Research.Available from: http://publicationethics.org/resources/research[Accessed August 3, 2016].
- COPE. (2016f). Seminars.Available from: http://publicationethics.org/resources/seminars[Accessed August 3, 2016].
- COS. (2016a). Center for Open Science website.Available from: https://cos.io/[Accessed August 3, 2016].
- COS. (2016b). COS Communities.Available from: https://cos.io/communities/[Accessed August 3, 2016].
- COS. (2016c). Sponsors of COS.Available from: https://cos.io/about_sponsors/[Accessed August 3, 2016].
- Council of Graduate Schools. (2016). Scholarly Integrity and Responsible Conduct of Research (RCR).Available from: http://cgsnet.org/scholarly-integrity-and-responsible-conduct-research-rcr?tabid=123[Accessed August 3, 2016].
- CSE. (2012a). 1.0 Introduction.Available from: http://www.councilscienceeditors.org/resource-library/editorial-policies/white-paper-on-publication-ethics/1-0-introduction/[Accessed August 3, 2016].
- CSE. (2012b). White Paper on Publication Ethics: CSE's White Paper on Promoting Integrity in Scientific Journal Publications, 2012 Update.Available from: http://www.councilscienceeditors.org/resource-library/editorial-policies/white-paper-on-publication-ethics/[Accessed August 3, 2016].
- CSE. (2016). About CSE.Available from: http://www.councilscienceeditors.org/about/about-cse/[Accessed August 3, 2016].
- Digital Worlds Institute. (2016). The GAP Project: Gaming Against Plagiarism.Available from: http://digitalworlds.ufl.edu/projects/gap/[Accessed August 3, 2016].
- EPA Science and Technology Policy Council. (2015a). Peer Review Handbook.4th ed. Washington, DC,US Environmental Protection Agency Science and Technology Policy Council. Available from: http://www2.epa.gov/sites/production/files/2015-09/documents/final_epa_peer_review_handbook-_4th_ed_091415_dummy_link.pdf.
- EPA Science and Technology Policy Council. (2015b). Peer Review Handbook: Executive Summary. 4th ed. Washington, DC,US Environmental Protection Agency Science and Technology Policy Council. Available from: https://www.epa.gov/sites/production/files/2015-10/documents/october_19-20_2015_hsrb_meeting_-_final_agenda.pdf .
- EPA. (2014). US Environmental Protection Agency Scientific Integrity Policy.Available from: http://www2.epa.gov/sites/production/files/2014-02/documents/scientific_integrity_policy_2012.pdf[Accessed August 3, 2016].
- EPA. (2016a). About EPA: Our Mission and What We Do.Available from: http://www2.epa.gov/aboutepa/our-mission-and-what-we-do[Accessed August 3, 2016].
- EPA. (2016b). Programs of the Office of the Science Advisor (OSA): Basic Information About Scientific Integrity.Available from: http://www2.epa.gov/osa/basic-information-about-scientific-integrity[Accessed August 3, 2016].
- EPA. (2016c). Programs of the Office of the Science Advisor (OSA): EPA's Principles of Scientific Integrity Fact Sheet.Available from: http://www2.epa.gov/osa/epas-principles-scientific-integrity-fact-sheet[Accessed August 3, 2016].
- Executive Office of the President. (2000). Federal Policy on Research Misconduct; Preamble for Research Misconduct Policy: A Notice by the Science and Technology Policy Office on 12/06/2000.Available from: https://www.federalregister.gov/articles/2000/12/06/00-30852/executive-office-of-the-president-federal-policy-on-research-misconduct-preamble-for-research[Accessed August 3, 2016].
- FASEB. (2016a). Enhancing Research Reproducibility: Recommendations from the Federation of American Societies for Experimental Biology.Available from: https://www.faseb.org/Portals/2/PDFs/opa/2016/FASEB_Enhancing%20Research%20Reproducibility.pdf[Accessed August 3, 2016].
- FASEB. (2016b). FASEB's Mission.Available from: https://www.faseb.org/About-FASEB/Our-Mission.aspx[Accessed August 3, 2016].
- FASEB. (2016c). Research Integrity.Available from: https://www.faseb.org/Science-Policy-and-Advocacy/Science-Policy-and-Research-Issues/Research-Integrity.aspx[Accessed August 3, 2016].
- FDA. (2012). SMG 9001.1: FDA Staff Manual Guides, Volume IV, Agency Program Directives, General or Multidiscipline Scientific Integrity, Scientific Integrity at FDA.Available from: http://www.fda.gov/AboutFDA/ReportsManualsForms/StaffManualGuides/ucm289975.htm[Accessed August 3, 2016].
- FDA. (2016a). About FDA: Office of Scientific Integrity.Available from: http://www.fda.gov/AboutFDA/CentersOffices/OC/OfficeofScientificandMedicalPrograms/ucm197861.htm[Accessed August 3, 2016].
- FDA. (2016b). What We Do.Available from: http://www.fda.gov/AboutFDA/WhatWeDo/[Accessed August 3, 2016].
- George, B. J., Sobus, J. R., Phelps, L. P., Rashleigh, B., Simmons, J. E., and Hines, R. N. (2015). Raising the bar for reproducible science at the U.S. environmental protection agency office of research and development. Toxicol Sci. 145:16–22.
- HHS ORI. (2016a). Case Summaries.Available from: https://ori.hhs.gov/case_summary[Accessed August 3, 2016].
- HHS ORI. (2016b). The Lab: Interactive Movie on Research Misconduct.Available from: http://ori.hhs.gov/thelab[Accessed August 3, 2016].
- HHS ORI. (2016c). Program for Academic Societies.Available from: https://ori.hhs.gov/program-academic-societies[Accessed August 3, 2016].
- HHS ORI. (2016d). Rapid Response for Technical Assistance.Available from: http://ori.hhs.gov/rapid-response-technical-assistance[Accessed August 3, 2016].
- HHS ORI. (2016e). RCR Resource Development Program.Available from: https://ori.hhs.gov/rcr-resource-development-program[Accessed August 3, 2016].
- HHS ORI. (2016f). RRI Program.Available from: http://ori.hhs.gov/rri-program[Accessed August 3, 2016].
- HHS. (2005). Federal Register: 42 CFR Parts 50 and 93 Public Health Service Policies on Research Misconduct; Final Rule.Available from: https://ori.hhs.gov/sites/default/files/42_cfr_parts_50_and_93_2005.pdf[Accessed August 3, 2016].
- HHS. (2016a). About HHS.Available from: http://www.hhs.gov/about/index.html[Accessed August 3, 2016].
- HHS. (2016b). HHS Agencies & Offices.Available from: http://www.hhs.gov/about/agencies/hhs-agencies-and-offices/index.html[Accessed August 3, 2016].
- HHS. (2016c). United States Department of Health and Human Services Policies and Principles for Assuring Scientific Integrity.Available from: http://www.hhs.gov/sites/default/files/open/pres-actions/scientifc-integrity-principles-12-19-11.pdf[Accessed August 3, 2016].
- Holdren, J. P. (2010). Memorandum for the Heads of Executive Departments and Agencies; Subject: Scientific Integrity.Available from: https://www.whitehouse.gov/sites/default/files/microsites/ostp/scientific-integrity-memo-12172010.pdf[Accessed August 3, 2016].
- ICTS. (2016a). Center for Clinical Research Ethics.Available from: http://icts.wustl.edu/icts-researchers/icts-cores/find-services/by-core-name/center-for-clinical-and-research-ethics[Accessed August 3, 2016].
- ICTS. (2016b). Center for Clinical Research Ethics: Education.Available from: http://icts.wustl.edu/icts-researchers/education[Accessed August 3, 2016].
- ICTS. (2016c). Center for Clinical Research Ethics: Funding Programs.Available from: http://icts.wustl.edu/icts-researchers/funding/funding-opportunities[Accessed August 3, 2016].
- ICTS. (2016d). Center for Clinical Research Ethics: Research Ethics Case Studies.Available from: http://ethicsresearchcore.org/education/case-studies/[Accessed August 3, 2016].
- ICTS. (2016e). Director's Message.Available from: http://icts.wustl.edu/icts-researchers/about/directors-message[Accessed August 3, 2016].
- ICTS. (2016f). Responsible Conduct of Research (RCR) Course.Available from: http://ethicsresearchcore.org/education/rcr/[Accessed August 3, 2016].
- Kaiser, J. (2009). Data integrity report sends journals back to the drawing board. Science.325:381.
- LJAF. (2016a). Grants.Available from: http://www.arnoldfoundation.org/grants/[Accessed August 3, 2016].
- LJAF. (2016b). Research Integrity.Available from: http://www.arnoldfoundation.org/initiative/research-integrity/[Accessed August 3, 2016].
- MacArthur Foundation. (2016). About Us.Available from: https://www.macfound.org/about/[Accessed August 3, 2016].
- NAS. (1992). Responsible Science: Ensuring the Integrity of the Research Process. Vol. 1.Washington, DC, National Academies Press. Available from: http://www.nap.edu/openbook.php?record_id=1864&page=R1.
- NAS. (2002). Report: Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct.Available from: http://iom.nationalacademies.org/Reports/2002/Integrity-in-Scientific-Research-Creating-an-Environment-That-Promotes-Responsible-Conduct.aspx[Accessed August 3, 2016].
- NAS. (2009a). On Being a Scientist: A Guide to Responsible Conduct in Research. 3rd ed. Washington, DC, National Academies Press. Available from: http://www.nap.edu/catalog/12192/on-being-a-scientist-a-guide-to-responsible-conduct-in.
- NAS. (2009b). Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age. Washington, DC, National Academies Press. Available from: http://www.nap.edu/catalog/12615/ensuring-the-integrity-accessibility-and-stewardship-of-research-data-in-the-digital-age.
- NAS. (2015a). Does the Public Trust Science? Trust and Confidence at the Intersections of the Life Sciences and Society (May 5–6, 2015).Washington, DC, National Academy of Sciences. Available from: http://nas-sites.org/publicinterfaces/roundtable/events/trust/.
- NAS. (2015b). Trust and Confidence at the Interfaces of the Life Sciences and Society: Does the Public Trust Science? A Workshop Summary. Washington, DC, National Academies Press. Available from: http://www.nap.edu/catalog/21798/trust-and-confidence-at-the-interfaces-of-the-life-sciences-and-society.
- NAS. (2016a). Arthur M. Sackler Colloquia: About the Sackler Colloquia.Available from: http://www.nasonline.org/programs/sackler-colloquia/about-the-sackler-colloquia.html[Accessed August 3, 2016].
- NAS. (2016b). Arthur M. Sackler Colloquia: Upcoming Colloquia.Available from: http://www.nasonline.org/programs/sackler-colloquia/upcoming-colloquia/[Accessed August 3, 2016].
- NAS. (2016c). National Academy of Sciences website.Available from: http://www.nasonline.org[Accessed August 3, 2016].
- NIH OER. (2016a). Guidance for Reviewers.Available from: https://grants.nih.gov/grants/policy/review.htm[Accessed August 3, 2016].
- NIH OER. (2016b). Rigor and Reproducibility.Available from: http://grants.nih.gov/reproducibility/index.htm[Accessed August 3, 2016].
- NIH Office of the Director. (2012). NIH Policies and Procedures for Promoting Scientific Integrity.Available from: https://ombudsman.nih.gov/ScientificIntegritynov2012.pdf[Accessed August 3, 2016].
- NIH. (2011). Notice Number NOT-OD-11-120: Revised Policy: Managing Conflict of Interest in the Initial Peer Review of NIH Grant and Cooperative Agreement Applications.Available from: http://grants.nih.gov/grants/guide/notice-files/NOT-OD-11-120.html[Accessed August 3, 2016].
- NIH. (2015a). Notice Number NOT-OD-15-103: Enhancing Reproducibility Through Rigor and Transparency.Available from: http://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-103.html[Accessed August 3, 2016].
- NIH. (2015b). Notice Number NOT-OD-15-106: Applicant Responsibilities in Maintaining the Integrity of NIH Peer Review.Available from: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-106.html[Accessed August 3, 2016].
- NIH. (2016a). Peer Review Policies and Practices.Available from: http://grants.nih.gov/grants/peer/peer.htm[Accessed August 3, 2016].
- NIH. (2016b). Rigor and Reproducibility: Principles and Guidelines for Reporting Preclinical Research.Available from: http://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-research[Accessed August 3, 2016].
- NIH. (2016c). Who We Are.Available from: http://www.nih.gov/about-nih/who-we-are[Accessed August 3, 2016].
- NSF. (2009a). Award Abstract 0924751: The Nature of Ethical Decision-Making in Research (R. Wayne Fuqua, Principal Investigator; David Hartmann, Co-Principal Investigator; and Thomas Van Valey, Co-Principal Investigator).Available from: http://www.nsf.gov/awardsearch/showAward?AWD_ID=0924751&HistoricalAwards=false[Accessed August 3, 2016].
- NSF. (2009b). Responsible conduct of research. Fed Regist.75:42126–42128.
- NSF. (2011). NSF Scientific Integrity Policy.Available from: http://www.nsf.gov/bfa/dias/policy/si/sipolicy.pdf[Accessed August 3, 2016].
- NSF. (2014). Award Abstract 1449479: Collaborative Research: Foundations of Social and Ethical Responsibility Among Undergraduate Engineering Students: Comparing Across Time, Institutions, and Interventions (Brent Jesiek, Principal Investigator; Carla Zoltowski, Co-Principal Investigator).Available from: http://www.nsf.gov/awardsearch/showAward?AWD_ID=1449479&HistoricalAwards=false[Accessed August 3, 2016].
- NSF. (2016). About the National Science Foundation.Available from: http://www.nsf.gov/about/[Accessed August 3, 2016].
- ORI. (2016a). Historical Background.Available from: http://ori.hhs.gov/historical-background[Accessed August 3, 2016].
- ORI. (2016b). RRI Conferences.Available from: https://ori.hhs.gov/rri_conference[Accessed August 3, 2016].
- OSF. (2016). Open Science Framework website.Available from: http://openscienceframework.org/[Accessed August 3, 2016].
- PEER. (2013a). About the Scientific Integrity Report Card and Disclaimer.Available from: http://www.peer.org/assets/docs/About_SI_Report_Card_and_Disclaimer.pdf[Accessed August 3, 2016].
- PEER. (2013b). Scientific Integrity Report Card Comparison Charts.Available from: http://www.peer.org/assets/docs/SI_Report_Card_Comparison_Chart%20-%20Sorted%20by%20Score.pdf[Accessed August 3, 2016].
- PRWeb. (2015). New Prizes Will Recognize and Reward Transparency in Social Science Research (Press Release May 18, 2015).Available from: http://www.prweb.com/releases/2015/05/prweb12727015.htm[Accessed August 3, 2016].
- PubMed Commons. (2015). Commenting on PubMed: A Successful Pilot.Available from: https://pubmedcommonsblog.ncbi.nlm.nih.gov/2015/12/17/commenting-on-pubmed-a-successful-pilot/[Accessed August 3, 2016].
- Retraction Watch. (2014). Retraction Watch is Growing, Thanks to a $400,000 Grant from the MacArthur Foundation.Available from: http://retractionwatch.com/2014/12/15/retraction-watch-growing-thanks-400000-grant-macarthur-foundation/[Accessed August 3, 2016].
- Retraction Watch. (2015). New $300,000 Grant Marks the Fifth Anniversary of Retraction Watch.Available from: http://retractionwatch.com/2015/08/03/new-300000-grant-marks-the-fifth-anniversary-of-retraction-watch/[Accessed August 3, 2016].
- Retraction Watch. (2016). The Center for Scientific Integrity.Available from: http://retractionwatch.com/the-center-for-scientific-integrity/[Accessed August 3, 2016].
- Sloan Foundation. (2013). 2013 Annual Report.Available from: http://www.sloan.org/fileadmin/media/files/annual_reports/2013-Annual-Report.pdf[Accessed August 3, 2016].
- Sloan Foundation. (2016). Major Program Areas.Available from: http://www.sloan.org/major-program-areas/[Accessed August 3, 2016].
- Steneck, N. (2007). Introduction to the Responsible Conduct of Research. Washington, DC,US Department of Health and Human Services Office of Research Integrity. Available from: https://ori.hhs.gov/ori-intro.
- Templeton Foundation. (2016). Increasing Scientific Openness and Integrity.Available from: https://www.templeton.org/what-we-fund/grants/increasing-scientific-openness-and-integrity[Accessed August 3, 2016].
- US Government Publishing Office. (2012). Publication No. 45 CFR 689: Research Misconduct.Available from: http://www.gpo.gov/fdsys/granule/CFR-2012-title45-vol3/CFR-2012-title45-vol3-part689/content-detail.html[Accessed August 3, 2016].
- USAID. (2012). USAID Scientific Integrity Policy.Available from: https://www.usaid.gov/sites/default/files/documents/15396/integrity.pdf[Accessed August 3, 2016].
- USAID. (2014). USAID Scientific Research Policy.Available from: https://www.usaid.gov/sites/default/files/documents/15396/USAID%20Scientific%20Research%20Policy%2012-3-14.pdf[Accessed August 3, 2016].
- USAID. (2016). Who We Are.Available from: https://www.usaid.gov/who-we-are[Accessed August 3, 2016].
- USDA Office of the Chief Scientist. (2013). Departmental Regulation DR Number 1074-001: Scientific Integrity.Available from: http://www.ocio.usda.gov/sites/default/files/docs/2012/DR%201074-001_0.pdf[Accessed August 3, 2016].
- USDA Office of the Chief Scientist. (2016). USDA Agency Scientific Integrity Officers.Available from: http://www.usda.gov/wps/portal/usda/usdahome?contentidonly=true&contentid=ocs-agency-scientific-integrity-officers.xml[Accessed August 3, 2016].
- USDA. (2013). USDA Scientific Integrity Policy Handbook (Guidance for Implementation of DR 1074-001).Available from: http://www.usda.gov/documents/usda-scientific-integrity-policy-handbook.pdf[Accessed August 3, 2016].
- USDA. (2015). Annual USDA Scientific Integrity Allegations Summary Report May 2014–April 2015.Available from: http://www.usda.gov/documents/usda-updated-scientific-integrity-summary-report.pdf[Accessed August 3, 2016].
- White House Office of the Press Secretary. (2009). Memorandum for the Heads of Executive Departments and Agencies 3-9-09; Subject: Scientific Integrity (Press Release).Available from: https://www.whitehouse.gov/the_press_office/Memorandum-for-the-Heads-of-Executive-Departments-and-Agencies-3-9-09/[Accessed August 3, 2016].
- Wikipedia. (2016). United States Department of Health and Human Services. Available from: https://en.wikipedia.org/wiki/United_States_Department_of_Health_and_Human_Services[Accessed August 3, 2016].