One Step Forward, One Step Back: Changes in News Coverage of Medical Interventions

ABSTRACT

During 2005–2013, the award-winning website HealthNewsReview.org offered reviews of major media outlets’ news stories related to health interventions, including tests, treatments, dietary changes, and prescription drugs. The reviews offered a measure by which the public and journalists themselves could assess the completeness and usefulness of health coverage across 10 criteria for quality reporting. This study produced an analysis of those reviews from 2005 to 2013, indicating significant changes in key areas. Analysis of 1,889 health news story reviews published by HealthNewsReview.org (HNR) between 2005 and 2013 showed that, on average, the stories reviewed during 2005–2010 successfully met just less than half of the criteria, but by 2010–2013, that average had improved to almost 70%. There were significant improvements over time in news organizations’ success in meeting six of HNR’s 10 criteria for a successful health news story related to drugs, devices, surgery and other medical procedures, and diet; however, when data for television stories were excluded, only the improvement in avoiding disease-mongering remained significant. In addition, there was a statistically significant decline in the percentage of stories rated satisfactory on establishing the true novelty of the intervention discussed in the story. There was no improvement in quantification of possible harms from medical interventions. Changes over time in meeting the criteria were related to outlet type and story topic.

Notes

¹ This criterion is meant to assess whether the news story points out the limitations of study evidence, encourages caution about interpretation of data, cautions readers about how the data might apply to their own health outcomes, notes the extent to which the reported findings have been peer reviewed, and presents anecdotes as illustrations of the possible benefit or harm of an intervention, rather than as evidence (“Review criteria,” n.d.).

² “This criterion assesses whether a story exaggerates or over-sells a condition” (“Review criteria,” n.d., 5) by presenting risk factors as diseases, overplaying the severity of a disease, medicalizing normal states such as menopause, baldness, etc., or “exaggerating the prevalence of a disorder” (“Review criteria,” n.d., 5).

³ HNR published no story reviews between April 2013 and January 2015, due to a lack of funding. A new grant enabled story reviews to resume in January 2015.

⁴ We realize it is customary to double-code at least 10% of the sample to establish intercoder reliability. Our initial plan had been to code only the 2012 and early 2013 reviews, which would have meant the 36 double-coded stories would have constituted 17% of the sample. However, because the intercoder reliability scores are so high, primarily due to the objective nature of the elements being coded, we believe the double-coded sample is sufficient.

⁵ The variable with the lowest Scott’s pi coefficient—whether or not the story was about surgical or other medical procedures—had only one disagreement; however, there was little variation in responses to this item, resulting in the lower coefficient.

⁶ We analyzed the 2010–2013 data separately from the 2005–2009 data because 2009 was the last year during which HNR routinely reviewed television health stories.

⁷ According to HNR’s website, this decision was made because the television stories did not seem to be improving and the HealthNewsReview staff decided “to apply our time, resources and energy to new and different media with whom we might have more impact” (“How we rate stories,” n.d.).

⁸ A criterion would be judged “not applicable” if it were not relevant to the intervention discussed in the story. For instance, for a story about the health benefits of sleeping late on weekends, the criteria of costs, harms, and availability would not be relevant; sleep is free and available to everyone, for the most part, and normal sleep does not have negative side effects.

⁹ The table excludes the 10th criterion, which rates whether the story appears to rely solely on a news release, because reviewers deemed this criterion applicable for only 5 stories during the earlier period and 225 stories during the later period. Reviewers rate stories “unsatisfactory” on this criterion only when the story includes “the exact same wording from a news release” and does not include any independent sources not quoted in the news release. Thus, this criterion was rated “not applicable” for the great majority of reviews during both time periods.

¹⁰ By comparison, during the earlier period, an increase of 1.7 percentage points would require satisfactory ratings on an additional four stories.

¹¹ Because one of HNR’s goals was to help journalists improve reporting on health topics, these guidelines were made available to journalists (e.g., through AHCJ training sessions), which provided HNR an opportunity to receive feedback from health journalists on the standards it used to assess their work (Schwitzer, personal communication, June 6, 2014).