Editorial: What Makes for a Great Applications and Case Studies Paper?

One of the most common reasons a submission to JASA Applications and Case Studies is rejected is that it is deemed inappropriate for this section of the journal. If we look for guidance in the journal’s instructions for authors, the opening sentence of the section on Applications and Case Studies states, “The Applications and Case Studies section publishes original articles that cogently demonstrate statistical usage in applications from any research area.” In contrast, the instructions for Theory and Methods papers say, “The research reported should be motivated by a scientific or practical problem and, ideally, illustrated by application of the proposed methodology to that problem. Illustration of techniques with real data is especially welcomed and strongly encouraged.” Many potential authors may find the distinctions between these two statements difficult to discern, which perhaps partly explains the high frequency of submissions rejected for being inappropriate. This editorial is an attempt to clarify what I think these distinctions are. In particular, in what ways is a paper with new methodology motivated by a “scientific or practical problem” and illustrated with “real data” not necessarily appropriate for Applications and Case Studies?

First, let me be clear that the views expressed here are my own and are not part of official journal policy. Every editor is the final arbiter of what papers should be published in the journal and I think it is appropriate and even desirable that different editors use somewhat different criteria in making these decisions. Nevertheless, it is my hope that authors, referees, associate editors, and future editors will find it helpful for me to spell out in greater detail than is appropriate for a journal’s website some of the things I look for when evaluating a submission.

There is not and should not be a clear and wide dividing line between Applications and Case Studies papers and Theory and Methods papers. Nevertheless, the use of the word “illustration” in the instructions for Theory and Methods papers points at a key distinction. An illustrative example possesses a feature that a proposed methodology is meant to address. The resulting data analysis may be rather brief, focusing on how the methodology can handle this feature better than previously proposed methods. The example thus serves in a supporting role, with the novel methodology and possibly accompanying theory being the main research contributions.

In contrast, the specific application plays a much more prominent role in an Applications and Case Studies paper. A typical Applications and Case Studies paper begins with a description of the applied problem, generally one of current scientific or policy interest, which then leads into a discussion of the proposed methodology. Note that while most Applications and Case Studies papers include novel methodology, that is not a requirement for publication. For example, a paper that adapts existing methodology to a new field of application may be publishable if this adaptation leads to substantial scientific insights in the application beyond what could be learned from previously used methods. Indeed, a good Applications and Case Studies paper might provide thoughtful analyses of real data to demonstrate the strengths and/or weaknesses of various statistical methods presently used to address an important substantive problem.

All data analyses appearing in Applications and Case Studies should correspond to good statistical and scientific practice. Statisticians and other researchers using statistical methods should be able to look to papers in Applications and Case Studies as examples of statistical practice they can emulate. Here are some of the features of an application or case study that should generally appear in any Applications and Case Studies paper:

• Describe the substantive problem your work addresses and why the dataset you are using is appropriate for addressing this problem. In particular, datasets should generally be of the size and scope that might be used in a paper for a relevant subject matter journal.

• Describe the provenance of the data, including, how they were collected and, if relevant, how they were selected from some larger data source. Discuss how these factors might affect the validity of any statistical or substantive conclusions drawn from the analysis. This description should generally include a discussion of the measurement process and any errors it might introduce. Of course, give units for all measured quantities and model parameters as needed.

• Provide adequate subject matter background so that a reader from outside that field can make some sense of how this background information affects model choices and substantive results. In particular, any substantive conclusions should be credible to a subject matter specialist.

• Recognizing that scientific breakthroughs are unlikely to be published in a statistics journal, the results of the data analysis should be of some subject matter and/or policy relevance.

• Use appropriate diagnostics to assess the appropriateness of key modeling assumptions.

• Compare results to those obtained using plausible alternative methodologies.

• Honestly discuss weaknesses in the proposed analysis and their possible impact on statistical and substantive conclusions. In particular, within a single paper, it will often not be possible to address all the statistical challenges that arise when analyzing a complex dataset, in which case, authors should clearly note these issues and explain why they are not addressed in the present work. This kind of discussion is crucial to provide guidance for others who might want to build on your work.

These requirements imply that even when the proposed methodological advance is motivated by a specific application, the paper may still be inappropriate for Applications and Case Studies. For example, datasets that are not of current substantive interest or that have already been analyzed many times in the statistical literature would generally not provide a suitable basis for an Applications and Case Studies paper unless the proposed methods led to substantial new scientific findings. Even when the data motivating the proposed statistical methods are of considerable current scientific interest, if the manuscript does not, for example, address most of the bullet points above, it will likely be inadequate for Applications and Case Studies. In particular, it will often be essential to address issues in the modeling and analysis that are of little or no relevance to the methodological thrust of the paper, but are crucial to drawing appropriate substantive conclusions from the data.

Some submissions to Applications and Case Studies include multiple applications, often from unrelated fields. For the purposes of demonstrating the broad applicability of a new methodology, providing diverse applications could be a positive feature and, in principle, an Applications and Case Studies paper could include multiple applications. However, especially when the datasets are from unrelated fields, given the page limit on submissions, it is difficult to provide adequate in-depth analyses of multiple datasets in a paper that has a substantial component of new methodology. A paper with a shorter methodological development will be more likely able to accommodate more than one dataset, especially if they are from related areas.

In addition to descriptions of methodologies and specific applications, Applications and Case Studies papers often contain simulations and/or theoretical results as a way to demonstrate the properties of proposed or existing methodologies. Simulation studies are commonly used to compare different methodologies and should be designed with care to allow for sufficient power to detect practically relevant differences between the statistical properties of these methods. Simulations should be chosen to reflect conditions that are likely to exist for relevant applications, especially if these conditions differ from stated conditions that motivate a methodology or underlie a theoretical result in the paper. In particular, simulation settings that are wholly or in part derived from observations or a complex numerical model, such as a climate model, may be more compelling than simulations from a purely hypothetical statistical model. While it is tempting to use simulations to show off how well one’s proposed methods work, simulation studies should elucidate both the strengths and the limitations of a methodology. Personally, I tend to view with suspicion any simulation study showing a proposed method uniformly dominating all other methods considered.

A substantial fraction of papers published by Applications and Case Studies include theoretical results, often about asymptotic properties of proposed methodologies. Proofs are almost always consigned to supplementary material as are, in many cases, needed technical conditions. Details of proofs in Applications and Case Studies submissions commonly receive little or no scrutiny by reviewers. Thus, if you think your theoretical results or your methods of proof are of broad interest, you should consider placing this material in a separate paper. There is a bit of a conundrum here if the theoretical results play an important role in justifying the methodology. Perhaps a solution in some cases would be to include just narrow statements of theoretical results needed to support the methodology in the Applications and Case Studies submission and to write a separate and concurrent paper that focuses on the theory.

Since 2016, all papers published in Applications and Case Studies undergo a reproducibility review. In 2021, this process was extended to all publications in JASA. These reviews focus on the important but fairly narrow issue of how the numerical results of a data analysis or simulation were computed. However, credibility of scientific conclusions depends on much more than numerical reproducibility. In addition to issues related to data provenance noted earlier, it may be important to describe preliminary analyses that led to inclusion or exclusion of certain covariates or specific functional forms of relationships between variables, which can effect the interpretation of findings of statistical significance. I would further encourage authors to address broader issues relating to the robustness of substantive conclusions, which often come under the heading of replicability (National Academies of Sciences, Engineering, and Medicine 2019).

Finally, a few specific tips for those of you planning to submit papers to Applications and Case Studies:

• Make sure your application appears prominently in your abstract, introduction and, in most cases, the paper’s title.

• Never use words like “illustration” or “example” when referring to your application. Such words give the impression that the application is not the focus of the paper, which is a good way to induce referees and Associate Editors to conclude that your submission is not suitable for Applications and Case Studies.

• If the parts of your paper that specifically address your application are, in total, much less than, say, 25% of the paper, then perhaps you have not written an Applications and Case Studies paper.

• If you downloaded your data from some data repository and have not made a serious effort to learn about the data source and the underlying substantive issues, you most likely have not written an Applications and Case Studies paper.

And, most importantly, have a really good idea.

Michael L. Stein
[email protected]