Audit of Research Data

Every few years, a spate of reports is in the media on scandals in research on issues of irreproducibility of data, plagiarism, fabrication, and falsification. It is not at all surprising since the United States’ budget for research and development is about $450 billion, split between the government and private sectors. The research enterprise has five million employees. This enormous enterprise reflects the mores of the society as a whole. Reports of misconduct have come for priests, FBI agents, police officers, bankers, accountants, and political figures. Researchers have human failings, too.

The current concern (The Washington Post 2015) is from the report in Science magazine (2015) by a group of researchers at the University of Virginia. They selected 100 published articles on behavioral studies. With the help of the original authors, they attempted to reproduce the results. They found that only 39% of the studies were reproducible with the same degree of strength as claimed by the original authors. These results are disturbing. However, research data from behavioral studies are more prone to variations because human behavior is much more complex than physics, chemistry, and chemicals that interact with biology (drugs). Furthermore, it is cost prohibitive to reproduce all of the research data. After all, there are over 200,000 journal articles per year in the United States. Nevertheless, all those involved in the project should be commended as well as Science magazine for publishing the study.

In 1981, then Congressman Al Gore (Price 2013) held oversight hearings of the House Science and Technology subcommittee on research fraud. A few years later, Rep. Ted Weiss held hearings of the Government Operation subcommittee on the same topic. It was those hearings that sparked my interest in publishing with a colleague, in 1987, an article in Nature magazine entitled “Ensuring Scientific Integrity,” suggesting the use of data audit. In the article, I stated “The procedures for data auditing will have to be non-obtrusive so that scientists can carry out their work without fear of unnecessary interruptions” (Shamoo and Annau, 1987). My colleagues and I held three international conferences in the late 1980s and early 1990s on data audit. As a result of these activities, there were nearly 25 articles and a book detailing on how to conduct a successful audit of research data with an audit of an actual published research article.

Since the late 1980s, fraud in research confirmed by the federal government is well below one percent. However, estimates of fraud in research range from a few percentage points to 30%. If one takes even 5% of research as fraudulent, the cost to the treasury alone is in the tens of billions of dollars. The cost to the public of fraud in research is enormous when data are used in the production of drugs to alleviate diseases, national security projects, prescribing behavioral changes, and the loss in public trust. Recent data indicate that misconduct is on the rise.

Internal Revenue Service (IRS) conducts random and “for cause” audits. IRS estimates (Find Law 2015) that tax crime convictions are less than .0022% of taxpayers annually. However, IRS estimates that failure to comply with tax code is 17%. Can one imagine what the rates of tax fraud would be without the risk of an audit?

The current mode of mitigating fraud is by hoping for greater compliance with ethical norms, by education, and by encouraging more whistleblowers to come forward. None of these methods has had much of an impact on reducing misconduct in research. Two years ago, and on the 20-year anniversary of founding the federal Office of Research Integrity (ORI), ORI asked me to speak on the subject of Data Audit as a Way to Prevent/Contain Misconduct. Auditing (Shamoo and Annau 1987) is the process of objectively validating the degree of correspondence of the published research data with source data. Audit should be conducted by an external independent organization.

Cost estimates for an audit is less than 1% of the cost of producing the data. Estimates indicate that, for each dollar spent on audit, there will be a saving equivalent to $25–60 in 20 years.

Many organizations such as the industry have adopted at least an internal data audit under different names such as “data validation.” It is time for academic institutions to adopt this certain way to reduce misconduct and restore the admiration and trust of the public to this noble profession—research.

Adil E. Shamoo, Ph.D.

Editor-in-Chief