Impact of Length Difference in Risk and Comparison Windows: A Cautionary Note

Abstract

Postmarketing observational safety studies seek to identify potential rare events that may be associated with a licensed product but could not be detected in clinical studies. In vaccine postmarketing safety studies, it is often useful to compare incidence rate in a risk period immediately following vaccination versus a longer self-comparison period for many (usually several hundreds or even thousands) medical events. It is recognized that when the risk and comparison event rates are the same, some of the tests for a difference between the two periods will reach significance by chance alone, and in this case, one would expect the differences reaching significance to be equally likely to be favorable or unfavorable. However, when the risk and comparison periods differ in length and events are rare, all or most of the significant findings could favor the group with longer follow-up even after accounting for the differential length of follow-up. An investigation of this phenomenon confirms that unequal comparison periods can affect the direction of chance findings and shows that the magnitude depends on the expected number of events in the risk period.

Key Words:

• Conditional test
• Observational study
• Rare event
• Self-control

Acknowledgments

We are indebted to the Zostavax^TM study team at the Kaiser Permanente Vaccine Study Center, Oakland, CA, USA, especially Bruce Fireman (who first explored this phenomenon) and Dr. Roger Baxter (principal investigator). We would also like to acknowledge the study independent Safety Review Committee (Dr. Neal Halsey, Dr. Samy Suissa, and Dr. Stefan Gravenstein) and Dr. Patricia Saddier at Merck Research Laboratories for requesting further exploration of this phenomenon, especially Dr. Halsey who first noticed it when reviewing the study interim analysis results. We are also grateful to the referees for their constructive comments.