Using the probability-probability plot and index to augment interpretation of treatment effect for patient-reported outcome measures

Abstract

The interpretation of treatment effect can pose challenges, especially for patient-reported outcomes. As subjective assessments, patient-reported outcomes frequently lack a historical record to support what their scores mean, making their interpretation of treatment differences challenging. We show how the probability-probability (p-p) plot a graph of the test-treatment distribution percentiles versus the control-treatment distribution percentiles, can complement and supplement interpretation of treatment effect. From this plot, we introduce the p-p index as a new measure of treatment effect, illustrating the method with two examples. The p-p index represents, across all percentiles, the average difference in percentile rank for any pair of subjects on two different treatments with the same outcome score. This measure, which complements other metrics of treatment effect, captures full information by integrating across all percentiles and thus accurately summarizes and augments the interpretation of treatment effect.

Keywords::

• effect size
• interpretation
• patient-reported outcomes
• probability-probability plot
• probability-probability index
• treatment effect

Acknowledgements

The authors thank the editor and the anonymous reviewers for their useful critique, and Kelly H Zou from Pfizer Inc. for her thoughtful and constructive comments.

Disclaimer

This work is strictly educational and methodological. It expresses the views of the authors, not their employer (Pfizer Inc.) or any other institution.

Financial & competing interests disclosure

Both authors are employees and shareholders of Pfizer Inc. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. This article is intended solely as an instructional, methodological and educational work. It expresses the views of the authors, not Pfizer Inc.

No writing assistance was utilized in the production of this manuscript.

Key issues

• • Interpretation of treatment effect can pose challenges for patient-reported outcomes (PROs) because of their subjective nature and lack of historical precedent.

• • The probability-probability (p-p) index is introduced as a way to augment the interpretation of treatment effect on a PRO measure.

• • The p-p index is based on the time-honored p-p plot, a curve of the cumulative percentage of individuals with outcome scores in the control-treated group on the horizontal axis and the cumulative percentage of individuals with outcome scores in the test-treated group on the vertical axis, which has received little or no attention for the interpretation of PRO measures.

• • The p-p index can be viewed as a generalization of two measures, the U₃ measure and the binomial effect size display, that concentrate on a particular part of the joint treatment-control outcome distribution.

• • The p-p index can be interpreted as giving, across all percentiles and the entire span of data, the average difference in percentile rank for any pair of test-treatment and control-treated subjects with the same outcome score.

• • It can range from -0.5 (control treatment completely dominates) to 0.5 (test treatment completely dominates).

• • Values of the p-p index (in absolute value) from 0 to 0.17 (lower tertile) can be interpreted as a ‘small’ difference between the test-treated group and the control-treated group; from 0.18 to 0.33 (middle tertile) can be interpreted as a ‘moderate’ difference; and from 0.34 to 0.50 (upper tertile) can be interpreted as a ‘large’ difference.

• • For example, a p-p index of 0.295 would indicate that, relative to a subject on control treatment, a subject on treatment with the same score ranked 29.5 percentile points higher (better) on average; the value of 0.295 would represent a moderate difference between groups.

• • Although not directly addressing the clinical relevance per se, the p-p index is an addition to distribution-based methods that offer valuable insights about the magnitude of an effect based solely on the distribution of the data.

• • Like the standardized effect size and other distribution-based methods, the p-p index also allows for a standardization of different scales with different ranges and ways of scoring.

• • Its appeal and use will depend on knowledge of its existence, its perception as adding useful and incremental interpretative value complementary to or beyond existing metrics, and its ease or convenience of computation.