Assessing methodological quality of randomized and quasi-experimental trials: A summary of stuttering treatment research

Figures & data

Table I. Databases and search engine supplier used to electronically search for stuttering treatment studies.

Database	Supplier
PsychINFO	EBSCO Host
Education Resource Information Center (ERIC)	EBSCO Host
MEDLINE	PubMed
Cumulative Index to Nursing and Allied Health Literature (CINAHL)	EBSCO Host
Cochrane Central Register of Controlled Trials	OVID
Communication & Mass Media	EBSCO Host
Canadian Education Index	CSA
FRANCIS	CSA
British Education Index

Figure 1.  Decision tree for study inclusion criteria.

Table II. Reporting quality sub-scale scores.

Quality indicator: Does study provide clear description of …	Percentage (n) of studies meeting quality indicator
1. Hypothesis/aims/objectives	91% (21)
2. Main study outcomes	100% (23)
3. Participant characteristics	100% (23)
4. Intervention and comparison  conditions	100% (23)
5. Distribution of confounders in  each participant group	78% (18)
6. Main findings	83% (19)
7. Estimates of random variability  in data for main outcomes	65% (15)
8. Adverse events	87% (20)
9. Characteristics of patients  lost to follow-up	74% (17)
10. Actual probability values  for the main outcomes	35% (8)

Table III. External validity quality sub-scale scores.

Quality indicator	Percentage (n) of studies meeting quality indicator
1. Participants recruited were  representative of population	17% (4)
2. Participants who agreed to  participate were representative  of population	26% (6)
3. Study context (e.g., staff, facilities)  representative of population	65% (15)

Table IV. Internal validity-bias quality sub-scale scores.

Quality indicator	Percentage (n) of studies meeting quality indicator
1. Blinding of participants to  intervention received	4% (1)
2. Blinding of assessors measuring  main study outcomes	48% (11)
3. Unplanned analyses identified  (no data dredging)	96% (22)
4. Analyses adjusts for different  lengths of follow-ups	87% (20)
5. Statistical tests appropriate  for main outcomes	83% (19)
6. Reliable compliance with  intervention	44% (10)
7. Scores from main outcome  measures were reliable and valid	91% (21)

Table V. Internal validity-confounding quality sub-scale scores.

Quality indicator	Percentage (n) of studies meeting quality indicator
1. Participants in different interventions  recruited from same population	44% (10)
2. Participants in different interventions  recruited during same time period	17% (4)
3. Participants randomly assigned  to treatment conditions	65% (15)
4. Random assignment concealed  from participants and staff	17% (4)
5. Statistical adjustments for confounding  (i.e., intent to treat)	61% (14)
6. Loss of participants (i.e., attrition)  addressed in analyses	65% (15)

Table VI. Study characteristics (frequency and percentage).

Study characteristic		Frequency (%)*
Research design	Randomized controlled trial (RCT)	16 (70%)
	Quasi-experimental design (QED)	7 (30%)
Treatment focus	Treatment vs control	9 (39%)
	Two non-traditional treatment comparisons	6 (26%)
	Non-traditional treatment vs traditional treatment	5 (22%)
	Treatment vs modification of that same treatment	3 (13%)
Age classification	Children	13 (56%)
	Adult	10 (44%)
Year of publication	1969–1979	5 (22%)
	1980–1989	8 (35%)
	1990–1999	3 (13%)
	2000–2008	7 (30%)
Journal	Journal of Speech Language Hearing Research (JSLHR)	7 (30%)
	Journal of Fluency Disorders (JFD)	5 (22%)
	Journal of Behavior Therapy and Experimental Psychiatry (JBTEP)	3 (13%)
	International Journal of Speech-Language Pathology (IJSLP)	3 (13%)
	American Journal of Speech-Language Pathology (AJSLP)	1 (4%)
	British Medical Journal (BMJ)	1 (4%)
	Behavior Therapy (BT)	1 (4%)
	Journal of Communication Disorders (JCD)	1 (4%)
	Perceptual and Motor Skills (P&MS)	1 (4%)

*May not round to 100% due to rounding error.

Table VII. Mean (SD) quality ratings by research design and K-S test results.

Quality category (maximum score)	Research design		Results
	RCT (n = 15)	QED (n = 7)	K-S Z	p	r
Reporting (10)	8.69 (1.30)	6.86 (1.35)	1.16	.13	.24
External validity (3)	1.13 (.89)	1.00 (.58)	.65	.79	.14
Internal validity-bias (7)	4.88 (.89)	3.71 (1.11)	.95	.33	.20
Internal validity-confounding (6)	3.44 (1.31)	1.00 (1.29)	1.48	.03	.31
Quality index (26)	18.13 (2.73)	12.57 (2.57)	1.75	.004	.37

Table VIII. Mean (SD) quality ratings by age group and K-S test results.

	Age group		Analysis summary
Quality category (maximum score)	Child (n = 13)	Adult (n = 10)	K-S Z	p	r
Reporting (10)	8.38 (1.61)	7.80 (1.48)	.64	.81	.13
External validity (3)	1.23 (.73)	.90 (.88)	.59	.88	.12
Internal validity-bias (7)	4.46 (.88)	4.60 (1.35)	.29	1.00	.06
Internal validity-confounding (6)	2.15 (1.57)	3.40 (1.71)	.81	.54	.17
Quality index (26)	16.23 (3.68)	16.70 (3.92)	.48	.98	.10

Table IX. Mean (SD) quality ratings by year of publication and Kruskal Wallis test results.

	Year of publication				Results
Quality category (maximum score)	1969–1979 (n = 5)	1980–1989 (n = 8)	1990–1999 (n = 3)	2000–2008 (n = 7)	χ2	p	η2
Reporting (10)	7.40 (1.67)	7.50 (1.61)	8.00 (1.00)	9.43 (.79)	7.64	.05	.35
External validity (3)	.40 (.55)	1.13 (.83)	1.33 (.58)	1.43 (.79)	5.33	.15	.24
Internal validity-bias (7)	4.40 (.55)	4.38 (1.60)	4.00 (1.00)	5.00 (.58)	3.39	.34	.15
Internal validity-confounding (6)	3.20 (1.10)	2.88 (2.17)	.67 (1.15)	3.00 (1.29)	4.94	.18	.22
Quality index (26)	15.40 (1.67)	15.89 (5.11)	14.00 (3.00)	18.86 (1.95)	5.71	.13	.26

Table X. Mean (SD) quality ratings by journal.

Quality category (maximum score)	Journal
	IJLCD & BJDC** (n = 3)	JBT&EP** (n = 3)	JFD (n = 5)	JSLHR & JSHR** (n = 7)
Reporting (10)	7.67 (2.08)	8.00 (1.00)	7.60 (2.07)	8.71 (1.11)
External validity (3)	.67 (.58)	.67 (1.15)	1.40 (.55)	1.00 (.82)
Internal validity-bias (7)	4.00 (1.73)	4.00 (0.00)	4.60 (1.14)	4.29 (.76)
Internal validity-confounding (6)	2.67 (2.52)	4.00 (1.00)	1.60 (1.14)	2.14 (1.77)
Quality index (26)	15.00 (5.58)	16.67 (2.08)	15.20 (3.96)	16.14 (3.44)

*Standard deviations are not reported for journals in which only one study is published.

**These journals were merged to form the first listed.