Paired or Pooled Analyses in Continuing Medical Education, Which One is Better?

ABSTRACT

In data analyses, pairing participant responses is often thought to yield the purest results. However, ensuring all participants answer all questions can be challenging. Concerns exist that pooling all responses together may diminish the robustness of a statistical analysis, but the practical insights may still exist. Data from a live, in-person, continuing education series for health professionals was analysed. For each topic, identical questions were asked prior to the educational content (pre), immediately following the content (post), and on a rolling 4 to 6 week follow-up survey (follow-up). For each educational topic, responses were matched by participant for a paired analysis and aggregated for a pooled analysis. A paired analysis was done for matched responses on pre vs post and pre vs follow-up questions. A pooled analysis was done for the aggregate responses on pre vs post and pre vs follow-up questions. Responses from 55 questions were included in the analysis. In both the paired and pooled pre vs post analyses, all questions yielded a statistically significant improvement in correct responses. In the paired pre vs follow-up analysis, 59% (n = 33) of questions demonstrated a statistically significant improvement in correct responses, compared to 62% (n = 35) in the pooled pre vs follow-up analysis. Paired and pooled data yielded similar results at the immediate post-content and follow-up time periods.

KEYWORDS:

• Education
• outcomes assessment
• health care professionals
• primary care

Introduction

Traditionally, in educational research, assessment data is best represented when analysed as paired data with the largest attainable sample size. This allows for clarity in individual learner improvement as a result of educational material [1]. However, there may be situations when this strategy should be challenged. In certain situations, pooled data may allow for an earlier and still insightful analysis. For example, in studies with small sample sizes, using pooled data to analyse intricate results may reveal early insights that paired data cannot [2]. Likewise, participant heterogeneity and homogeneity may influence the need for paired or pooled analyses [3]. To date, there is limited research on what type of analysis is ideal in continuing medical education (CME). Available research has found no significant difference between paired and pooled data when comparing pre and follow-up responses [1].

Materials and Methods

Practicing Clinicians Exchange (PCE) is a medical education company providing free continuing education (CE) to advanced practice providers (APP). Over 95% of PCE learners are nurse practitioners or physician assistants/associates, with the majority practicing in a community setting. Annually, PCE offers a live primary care series, covering various therapeutic topics in multiple cities across the USA.

Data from the 2019 PCE live CE series for primary care APP was analysed. Eight educational sessions, with varying topics, were held across six different dates. Participants could engage in as many or as few topics as desired. For each topic, identical questions were asked prior to the educational content (pre), immediately following the content (post), and on a rolling 4 to 6 week follow-up survey, with a maximum of two weekly reminder emails (follow-up). Each session asked nine questions, split between knowledge, competence, and performance, yielding a total of 72 questions asked across all eight sessions [4]. Questions were analysed following each meeting, using a point biserial analysis. Poor performing questions were modified for future sessions but excluded from the current study. For each educational topic, responses were matched by participants and aggregated for a pooled analysis. A t-test was used for the paired analysis with matched responses on pre vs post and pre vs follow-up questions. A pooled analysis was done for the aggregate responses on pre vs post and pre vs follow-up questions, using a chi-squared. Paired and pooled groups were divided into tertiles, grouping the question items into three separate groups based on the distribution of responses for analyses. This allowed for determining if a threshold existed for the number of responses, in gleaning any statistical significance in correct item responses (Table 1).

Table 1. Number of responses to 55 questions, broken into tertiles for analyses.

Group	Lower Tertile	Middle Tertile	Upper Tertile
Paired
Pre vs Post	<544 (n = 19)	544–757 (n = 18)	>757 (n = 18)
Pre vs Follow-Up	<97 (n = 16)	97–148 (n = 27)	>148 (n = 12)
Pooled
Pre vs Post	<1739 (n = 18)	1739–1900 (n = 18)	>1900 (n = 19)
Pre vs Follow-Up	<1077 (n = 17)	1077–1166 (n = 20)	>1166 (n = 18)

Results

Responses from 55 (76%) questions were included in the analysis. The pre vs post content questions yielded 323–956 matched response pairs, and the pre vs follow-up questions yielded 41–236 matched response pairs. The pre vs post content pooled analysis yielded 1260–2285 responses, and the pre vs follow-up pooled analysis yielded 777–1431 responses.

In both the paired and pooled pre vs post analysis, all individual questions yielded a statistically significant improvement in correct responses. In the paired pre vs follow-up analysis, 59% (n = 33) of individual questions demonstrated a statistically significant improvement in correct responses, compared to 62% (n = 35) in the pooled pre vs follow-up analysis. Since all questions in the pre vs post analysis yielded statistically significant improvement, tertile analysis was only conducted on the pre vs follow-up groups. In both the paired and pooled pre vs follow-up analyses, no difference in the number of statistically significant questions was found between tertiles (Figure 1).

Figure 1. A comparison of the percentage of questions yielding statistically significant improvement between paired and pooled analyses in pre vs follow-up item responses. *No statistically significant differences found between groups.

Discussion

Statistical significance and use of effect size are often sought metrics with questionable utility following a single event. This study revealed the potential utility of a pooled analysis, rather than a paired analysis. Matching pre and post responses using a paired analysis did not yield improvements in knowledge, competence, or performance as statistically significant improvement in correct responses was achieved in all questions in both the paired and pooled analyses. In the pre vs follow-up analysis, the pooled analysis demonstrated numerically more questions achieving statistical significance than that of the paired analysis. However, the practical relevance of this small difference is questionable. There was no threshold found when comparing tertiles and the improvement in correct responses at the paired and pooled pre vs follow-up analysis, suggesting sample size did not influence our findings.

A similar study conducted by Heintz and Fagerlie had comparable findings. The authors analysed paired and pooled outcomes data from 5 different CME topics (lung cancer, gastric cancer, non-Hodgkin lymphoma, B-cell non-Hodgkin lymphoma, and pancreatic cancer). For each topic, a pre and post assessment was administered. The outcomes were assessed using both paired and pooled data. The authors found that there was no significant difference between paired and pooled data when looking at pre and post responses [1]. In contrast, the current study found a slight increase in the number of correct responses in the pooled analysis when compared to the paired analysis. These studies together suggest that there may be value in utilising pooled data.

While the utility of a paired analysis is limited by a small sample size, a pooled analysis can increase the sample size to reveal statistical significance. Laur et al sought to evaluate the knowledge, attitudes, and practices (KAP) of hospital staff pre and post implementation of nutrition care activities. The KAP questionnaire involved 4- and 5-point Likert scale items. Example knowledge and attitudes (KA) questions included positively and negatively worded statements for assessing nutrition-related content, while questions assessing practices included statements for determining how often participants engage in nutrition-related practices. There were similar trends in the pooled (pre, n = 189; post, n = 147) and paired (n = 57) analyses, but only the pooled analysis showed statistically significant improvement in KAP. This yielded valuable insights in the use of pooled data for analysis of intricate results in studies with small sample sizes [2].

Importantly, a pooled analysis may not be appropriate for heterogeneous data. A previous study analysed pre vs post questionnaire scores of pharmacy students at different stages in their academic careers at three different universities in the USA, UK, and Australia. The questionnaire, composed of 5-point Likert scale items, true/false, and multiple-choice questions, evaluated the perceived confidence and knowledge of students who participated in a workshop on communicating effectively with adolescents in a pharmacy setting. The authors report a significantly lower pre-knowledge score in the pooled group compared to the paired group, potentially due to diversity in educational and experiential backgrounds [3].

The importance of CME assessment quality has previously been discussed [5–8]. Use of an assessment tool that is both valid and reliable can help gauge how effective the program is at helping a healthcare professional move across the learning continuum. It is also pertinent to consider how the assessment should be tailored to its setting, learner, and purpose. Careful use of assessment formats, such as multiple-choice questions (MCQs) or Objective Structured Clinical examinations (OSCEs), are designed to evaluate a specific subset of knowledge and skills [5]. Each assessment strategy also has a most effective way of writing it into the assessment tool. As an example, previous researchers have explored the best strategies to write quality MCQs [6]. Regardless of the assessment used, recognising learners require repeat exposure to material to shift changes in practice is essential [7,8]. Setting realistic expectations for practice change following a single educational touchpoint is crucial in developing content that re-engages learners over time.

Limitations/Future Areas of Study

While this study included a robust sample size and an analysis to examine the influence of sample sizes, the range of responses for individual questions remained wide and sample bias cannot be overlooked. Another area of CME with a paucity of information is how an assessment question is constructed. Poorly constructed questions directly impact the findings of analyses. While the current study excluded questions necessitating modification, potential inclusion bias cannot be ignored. A detailed item analysis to compare components of a CME assessment could evaluate how well questions are written and whether the question is specific to the relevant knowledge or skills being taught.

Conclusion

This preliminary research questions the need for paired data in CME data and sparks opportunities for more conversations and data insights in this space. Further research in the CME space should explore whether other variables or nuances within the data can yield meaningful insights to move healthcare professionals within the learning continuum.

Acknowledgments

Robert C. Gresham III, PharmD Candidate, UNC Eschelman School of Pharmacy, Chapel Hill, NC

Disclosure statement

No financial or non-financial competing interests to report by the authors.