A Letter from the Editors: Pearls and Pitfalls for Writing a Methods Section

Abstract

The Methods section is the core of any research manuscript, yet writing this section may feel like a daunting task. In this letter, two of our methods and statistics editors provide some guidance on common pitfalls to avoid and pearls for writing the Methods section. From study design to analytic approach, this letter gives a high-level look at keys to success.

In his book How to Write a Lot: A Practical Guide to Productive Academic Writing, Paul Silvia says that for some, “it’s easier to embalm the dead than to write an article about it” (1). Perhaps this rings even more true when it comes to writing the Methods section of a research manuscript. But before you rush off to enroll in mortuary science, have a read through this friendly guide as we seek to demystify the Methods section. We provide some tips to try, and point out grave mistakes to avoid (pun intended - who said statisticians can’t have fun?).

Leverage Reporting Guidelines

As the core of the paper, the Methods section should clearly explain what you did and how you did it. With this section, the reader should have enough information to replicate your study, verify your results, or maybe even identify details that were not considered or were more relevant than expected. This does not mean that a Methods section should be a long, boring list of details. Your task is to concisely lead the reader through the pertinent components that help tell your story. But, don’t give away the punchline – make sure to keep your results, including how many people were included in your study, out of the Methods section.

When it comes to figuring out what details should be included in the Methods section, there are several helpful frameworks for reporting research that both readers and researchers can, and should, leverage. To foster high-quality reporting, the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network (www.equator-network.org) maintains a collection of reporting guidelines with accompanying checklists. The appropriate guideline depends on the study design: for example, CONSORT for randomized trials, STROBE for observational studies, or PRISMA for systematic reviews and meta-analyses. There are also frameworks for quality improvement research (SQUIRE) and qualitative research (COREQ). Referring to and using these checklists when writing a manuscript can help avoid missing key details, and enhance the flow of the methods section. These reporting frameworks are also useful when you are designing and conducting your study.

Use Subsections

To make the Methods section easy to follow and to avoid leaving out key details, subsections are an incredibly useful tool. Methods sections typically follow a common order of presentation, and examples of common subheadings include: Study Design and Setting, Data Collection/Data Source, Selection of Participants, Intervention, Measures/Outcomes, and Statistical Analysis. The best subheadings depend on your study (e.g., survey work should include a description of the survey instrument development and pilot testing).

Obtain and Report Institutional Review Board (Ethics) Determination

An institutional review board or IRB (sometimes called an ethics committee) is a group, with both scientific and nonscientific members charged with ensuring that research studies involving human subjects meet regulations and ethical standards. Like warning labels, IRBs exist for unfortunate reasons. Before IRB review was part of research, unethical experiments (see Tuskegee for a famous example) were carried out, often on marginalized and vulnerable populations (2). The IRB’s task is to ensure that the study question you are proposing to answer is worth putting human subjects at some degree of risk (even if that risk is just accidental disclosure of private information), and that your proposed methods are a reasonable approach to answer that study question while minimizing the risks to the subjects. The study design information that the IRB will request is essentially your Methods section, so write it once, and use it for both purposes.

If you are debating whether you need review and determination from an IRB, beware of trying to squeeze research under the guise of quality improvement (QI). A single-center study does not necessarily mean QI. The goal of research is to test new knowledge. Meanwhile, the goal of QI is to make sustained improvements toward a given metric. When in doubt, you should always consult the IRB. Third party determination of ethics is a key to ensuring history does not repeat itself. The IRB will review your study protocol, intervention, data collection procedures, informed consent forms, surveys, etc. Depending on the level of interaction with human subjects, the study may undergo a full review (for studies with more than minimal risk) or expedited review (for no more than minimal risk) before approval. The IRB may also determine that the project does not meet the official federal definition of human subjects research, or that it is exempt from review according to federal regulations. Even if you are fairly sure that your project will not meet the definition for human subjects research, it is important to let the IRB make this decision (third party determination). IRBs are often found within academic institutions but can exist at hospitals and as private entities. Every research paper should include a statement of IRB determination, typically at the end of the first paragraph of the Study Design & Setting subsection of the Methods.

Reporting Statistical Analyses

Statistically speaking, we know that most people didn’t love stats classes. But reporting statistical analyses does not have to be intimidating, and there is actually little to no magic involved (but don’t tell the other statisticians we told you this).

The Analysis Funnel

The flow of the analysis and thus the analysis subsection of the Methods follows a funnel approach starting with descriptive statistics, then comparative statistics, and any regression analysis. Subgroup and sensitivity analyses may also be presented next. While it may be tempting to jump right into a fancy-sounding regression model, following this flow is critical to telling a story with your data.

Selecting a Basic Analytic Approach

Knowing which statistical test or approach to use may sound like a daunting task, but making the right choice really starts with asking yourself two simple questions:

1. What am I trying to accomplish? (Goal)

   You may be interested in simply describing a single group, or perhaps you want to compare two distinct groups. But what if you want to compare before and after measurements on the same people? Or compare three or more groups? Don’t worry, there’s a statistical test for that, but before you know which one, you’ll need to answer the second question.

2. What kind of outcome data do I have?

   Determine whether your outcome is time to an event, binary (yes/no), categorical, or continuous. If your outcome variable is continuous, it is critical to know whether or not you have normally distributed data. Keep in mind that like non-normal people, non-normal data are more common and often more interesting. While there are several statistical tests you can use to determine normality, often it is easier to simply take a look at a histogram. If after visual inspection, the data do not look like a bell curve, you are likely dealing with non-normal data. Another tip: if the data are normally distributed, the mean and median will be similar. Ultimately, if you are unsure, it is better to err on the side of following reporting for non-normal data, as these methods are appropriate for both non-normal and normal data.

   Once you have answered these two questions, find the intersection of your answers in Table 1 to determine an appropriate statistical approach. It’s as easy* as that! (*Note: This approach covers many of the more commonly used statistical analysis techniques, but the list is not all-inclusive. Make sure to consult your statistician early and often).

Table 1. Selecting an analytic approach.

Step 1: What is your GOAL?	Step 2: What type of OUTCOME data do you have?
	Continuous, normally distributed	Continuous, non-normally distributed	Binary	Time to event
Describe one group	Mean, SD	Median, IQR	Proportion, n	Kaplan-Meier Survival Curve
Compare two unpaired groups	Two-sample t-test	Wilcoxon Rank Sum test	Chi-square test	Log-rank test
Compare two paired* groups	One-sample t-test on differences	Wilcoxon Signed Rank test	McNemar’s test	Conditional Proportional Hazards Regression
Compare three or more unmatched groups	One-way ANOVA	Kruskal-Wallis test	Chi-square test	Cox Proportional Hazards Regression
Predict value or quantify association between variables	Simple or Multiple Linear Regression	Quantile or Ordinal Regression	Simple or Multiple Logistic Regression	Cox Proportional Hazards Regression

* Paired measurements refer to two measurements taken from the same individual, object, or related units (e.g., pretest and post-test scores on the same individuals or 1:1 matched samples).

Building Statistical Models

When it comes to statistical modeling, the approach to variable selection should be driven by the goals of the analysis: are you trying to predict an outcome, describe an association, or estimate a causal effect? With predictive models, variables are selected for the best statistical performance of the model and may not always be easily interpretable (e.g., what in the world does a cubic term mean anyway?). This approach often includes decision-making based on statistical significance of variables. Conversely, when describing an association or estimating a causal effect, the approach should instead be guided by theory, prior literature, and substantive reasoning.

A useful tool for selecting confounding variables is the causal diagram. Most commonly used when estimating a causal effect, these diagrams can also help us design a study and select covariates in a logical and meaningful way. Directed acyclic graphs (DAGs) are one type of causal diagram. Briefly, a DAG can be used to help us explain how we think variables are related to each other, which allows the research team to make informed decisions about data that should be collected and the potential pros and cons of including variables as covariates in a regression model (3, 4).

Beware of “Naked” p-Values

Ah, yes – the p-value. For many, the notorious p-value embodies the very essence of statistical fear. If our software tells us that the p-value is less than 0.05, we rejoice in our scientific discovery, but if it spits out something even the slightest bit greater than 0.05 we are instantly awash with despair. However, this need not be the case. While the p-value can serve as a useful statistical measure, too often this value is misinterpreted, over trusted, and misused (5).

Incorrect interpretations of p-values abound. For example, a p-value is NOT the probability that the null hypothesis is true, and a p-value is NOT the probability that your findings are due to random chance alone. In reality, the p-value is a statistical summary of the compatibility between the observed data and what would be expected if the null hypothesis was true, given that all the assumptions used to compute the p-value were correct (6). Smaller p-values do mean that the observed data would be more unusual if the null hypothesis were true, only if (this is a BIG if) all of the assumptions of the given statistical model were met. However, p-values can be affected by sample size and do not tell us whether the hypothesis being tested is true or not. Thus, the commonly adopted 0.05 threshold is sadly not a magical arbiter of discovery. In fact, use of this threshold to divide statistical significance often incorrectly leads to the interpretation of “no difference” instead of “no significant difference” and weakens our capacity to interpret the data.

As such, emphasis in statistical reporting should be on estimates (and how these estimates were reached–i.e., the methods) (7). The p-value alone tells the reader little to nothing about the actual effect observed. For this reason, avoid reporting p-values without their point estimates. For example, instead of saying “the rate was higher in women than in men (p < 0.01)”, say “the estimated rate among women was 56% compared to 23% in men (p < 0.01)”. For more complete reporting, consider providing a measure of variability around the point estimate as well, such as a 95% confidence interval for a risk difference, or interquartile range for a median, or standard deviation for a mean.

The Big Picture

The Methods section is truly the core of your paper. Missing or confusing details in this section often result in early rejection of a paper or multiple rounds of substantial revisions. The Methods section should clearly connect with the objective and information laid out in the Background section. You should present your Results section following the same flow you created in the Methods section. To set yourself up for success, we strongly recommend you start writing the Background and Methods sections, and even the Results section, before you ever start collecting the data. This concept, known as the “zeroth draft”, has been around for nearly four decades and remains one of the greatest tools for strong study design and writing (8). Taking the time to write your Methods section and create table shells for how your results will be presented prior to undertaking the work will ensure that you know what data need to be collected to answer your study question, and what statistical tests will be needed to analyze those data. Having a zeroth draft will also be useful in your IRB/ethics application, saving duplicate writing. And lastly, be sure to consult your friendly neighborhood methodologist and statistician early and often to avoid issues down the line.

Editor Pearls

• Refer to the appropriate reporting guidelines based on your study design as a framework for the content to include (and to assist with study design).

• Write a “zeroth” draft of your Methods and Results sections prior to collecting and analyzing data.

• Use subheadings to guide the reader through the Methods section and provide key details. Examples include: Study Design and Setting, Selection of Participants, Intervention, Measures/Outcomes, and Statistical Analysis.

• Include a statement of IRB determination at the end of the Study Design & Setting subsection.

• To choose the right statistical test, know your goal and know your outcome data. Remember that non-normal data are more common and choose appropriate tests and summary measures.

• Avoid presenting naked p-values and instead be sure to report point estimates with measures of variability such as 95% confidence intervals.

• When building statistical models, select covariates based on the goal of your analysis.

• Consult a trained methodologist and statistician early and often!

Disclosure Statement

The authors have no conflicts of interest to report.