Critical analysis of a computer-assisted tutorial on ECG interpretation and its ability to determine competency

Figures & data

Table

Practice points
•A computer-based tutorial can improve skills for the identification of abnormal features in ECGs.
•The tutorial and posttest generated information about strengths and weaknesses in baseline knowledge in residents at various stages of training.
•The strategy for critical evaluation of the ECG program could be applied to any computer-based program, regardless of the topic.

Figures 1a–d. Evaluation of questions in the tutorial and posttest: • Each data point represents the data on the difficulty and discriminative value of questions. Difficulty was judged by how many participants answered questions in each module correctly. From 50 to 85% was considered an ideal range and indicated that the questions were moderately difficult. Discriminative value was determined by the ratio of the number of participants who ranked in the upper quintile in all the questions in the tutorial who answered the questions correctly vs. the number in the lower quintile. The upper and lower quantiles were defined by grades in the tutorial. A question was considered discriminative if the ratio was greater than 1.25 and very discriminative if over 1.5. The ideal range of difficulty of discriminative questions is shown in the shaded areas. • Figure 1a: Data from the evaluation of questions of feature identification in the 12 Modules in the Tutorial. Each Module contains 3 to 5 ECGs relevant to the topic of the Module. Each data point represents the mean of difficulty and discriminative values for all the ECGs in a Module. • Figure 1b: Data on the evaluation of questions on diagnosis in the 12 Modules in the Tutorial. • Figure 1c: Data points represent data on the evaluation of the difficulty and discriminative value of questions on feature identification each of the 20 ECGs in the Posttest. • Figure 1d: Data points represent data on the evaluation of the difficulty and discriminative value of questions on the identification of abnormal features in each of the 20 ECGs in the posttest. The ECG that is circled was difficult and non-discriminative, and therefore was not graded. Feature identification in the EKG designated by the arrow was difficult but very discriminative and was graded.

Table 1a.  Baseline grades in tutorial (n = 50)

	PGY1 (n = 18)	PGY2 (n = 10)	PGY3 (n = 12)	*Experienced physicians (n = 10)	ANOVA (CF = Critical F)
Features	52.6 ± 22.7	56.8% ± 12.8	55.0% ± 21.9	68.8% ± 15.8	F = 1.584032
					CF.05 = 2.8.06843
Diagnosis	70.3 ± 12.3	78.9% ± 11.5	85.2% ± 15.1	96.2% ± 13.1	F = 9.174631
					CF.05 = 2.806843

*Experienced physicians: Cardiology fellows, cardiologists and hospital-based internists.

Table 1b.  Mean ± SD for grades in the posttest (n = 46)

	PGY1 (n = 18)	PGY2 (n = 10)	PGY3 (n = 12)	Cardiology Fellows (n = 6)	ANOVA (CF = Critical F)
Features	64.1 ± 10.7	69.8% ± 11.5	69.2% ± 15.8	79.0% ± 13.0	F = 2.125157
					CF.05 = 2.827051
Diagnosis	69.9 ± 10.3	77.3% ± 11.8b	79.3% ± 12.5	90.5% ± 7.15	F = 5.747487
					CF.05 = 2.827051

Table 1a: Grades in diagnosis in the tutorial • Difference in grades in diagnosis between for all residents and experienced physicians was significant, 75.3 ± 15.5% and 96.2 ± 13.1%, respectively (p < 0.001 per t-test). Differences between PGY1 or PGY2 and experienced physicians were significant. • Differences between PGY1 and PGY3 were significant. (ANOVA and Scheffe post-hoc analysis)Table 1b: Grades in features in the tutorial • The difference in grades in feature identification between all residents and cardiology fellows was significant, 54.4 ± 19.7% and 68 ± 15.8%, respectively (p < 0.03 per t-test).Table 1c: Grades in diagnosis in the posttest • Differences between PGY1 and Experienced Physicians and between other groups were not significant. (ANOVA) • Diagnosis: Difference in grades in diagnosis between for all Residents and Experienced Physicians was significant, 74.6 ± 11.7% and 90.5 ± 7.2%, respectively (p < 0.005 per t-test). • Differences between PGY1 and Experience Physicians were significant (ANOVA and Scheffe post-hoc analysis).Table 1d: Grades in features in the posttest • Difference in grades in feature identification between all residents and cardiology fellows was significant, 67.1 ± 12.4% and 79.0 ± 13.0%, respectively (p < 0.05 per t-test). • There was no significant difference between the groups (ANOVA).

Figure 2. Improvement of Grades for Feature Identification for participants at different stages of training: As described in the results there was a significant improvement grades in the Posttest from that in the Tutorial for all residents. This figure shows the improvement in participants at different stages of training. Only data from cardiology fellows and residents are shown since cardiologists and hospital based physicians did not take the posttest.

Figures 3a–e. The correlation of grades in feature identification and diagnosis: (a) For all participants; (b) For PGY1; (c) For PGY2; (d) For PGY3; (e) For cardiology fellows (cardiologists and hospital-based physicians did not take the posttest).

Table 2a.  Participants with “Outstanding” & “Poor” grades for diagnosis in the posttest

Mean grade = 76.5%	PGY 1	PGY 2	PGY 3	Cardiology fellows	ALL
“Outstanding Performance”	1/18	2/10	4/13	4/6	11
>89.3 (Mean + one SD)
“Poor Performance”	7/18	3/10	2/13	0	12
<63.6% (Mean–one SD)

Table 2b.  Participants with “Outstanding” & “Poor” grades for feature identification in the posttest

Mean grade = 69.9%	PGY 1	PGY 2	PGY 3	Cardiology Fellows	ALL
Outstanding Performance”	0/18	1/10	4/13	2/6	7
> 83.1 (Mean + one SD)
“Poor Performance”	3/18	2/10	2/13	1/6	8
< 56.8 (Mean - one SD)

Tables 2a & 2b. The mean grade for all participants for diagnosis was 76.5% and for feature identification was 69.9%. Mean grades ± One SD were used a criteria for outstanding and poor grades. The table shows the number of participants that were in the “Outstanding” and “Poor” groups per the total number of PGY1, PGY2, PGY3 and cardiology fellows who took the posttest. Six out of 11 in the outstanding category for diagnosis also were in the outstanding in feature identification. This is an impressive correlation since only residents were outstanding in feature identification Seven out of 12 in the poor performance category for diagnosis were also in the poor category in feature identification.