A reliability generalization meta-analysis of coefficient alpha and test–retest coefficient for the aging males’ symptoms (AMS) scale

Abstract

Purpose: The aging males’ symptoms (AMS) scale is an instrument used to determine the health-related quality of life in adult and elderly men. The purpose of this study was to synthesize internal consistency (Cronbach’s alpha) and test–retest reliability for the AMS scale and its three subscales.

Methods: Of the 123 studies reviewed, 12 provided alpha coefficients which were then used in the meta-analyses of internal consistency. Seven of the 12 included studies provided test–retest coefficients, and these were used in the meta-analyses of test–retest reliability.

Results: The AMS scale had excellent internal consistency [α = 0.89 (95% CI 0.88–0.90)]; the mean alpha estimates across the AMS subscales ranged from 0.79 to 0.82. The AMS scale also had good test–retest reliability [r = 0.85 (95% CI 0.82–0.88]; the test–retest reliability coefficients of the AMS subscales ranged from 0.76 to 0.83. There was significant heterogeneity among the included studies.

Conclusions: The AMS scale and the three subscales had fairly good internal consistency and test–retest reliability. Future psychometric studies of the AMS scale should report important characteristics of the participants, details of item scores, and test–retest reliability.

• Meta-analysis
• internal consistency
• test–retest reliability
• aging males’ symptoms
• health-related quality of life

Introduction

The aging males’ symptoms (AMS) scale is an instrument which has been used for over 15 years to determine the level of health-related quality of life in adult and elderly men [1]. The scale was designed as a self-administered scale to: (a) assess symptoms of aging (independent from those which are disease-related) between groups of males under different conditions, (b) evaluate the severity of symptoms over time and (c) measure changes pre- and post-androgen therapy [1].

The AMS scale has been well-accepted internationally and has been translated into more than 30 languages [2–11]. As the AMS scale is widely used worldwide, one potential problem with the AMS scale is that the symptoms of aging could be affected by factors such as culture, race, religion and economic status. If the reliability statistics of the AMS scale vary internationally, then the utility of the AMS scale would be significantly limited with regard to pooling results of clinical studies among countries. Therefore, it is important to compare the reliability of the AMS scale among countries.

Daig et al. [11] demonstrated that the AMS had good reliability among different countries; however, in their report, the reliability statistics of the AMS scale were based largely on samples from the Western world, and there were only two small samples from Asia, each of which had fewer than 30 participants. Such results might not be applicable in non-Western countries. Recently, several psychometric studies of the AMS scale in non-Western countries have been published [2–8,12]. Therefore, it would be relevant to update the review of the reliability statistics of the AMS scale.

The objective of this systematic review and meta-analysis was to specifically evaluate the available evidence for the reliability of the AMS scale.

Methods

Instrument

The AMS scale is a 17-item polychotomous Likert-type scale. Respondents rate each item on a five-point response scale (1–5: none = 1; mild = 2; moderate = 3; severe = 4; and extremely severe = 5), so that the possible scores range from 17 to 85. Based on the total score, the severity of the symptoms is then classified as none/little (17–26), mild (27–36), moderate (37–49) or severe (50–85) [1]. The AMS scale can be divided into three subscales: (1) psychological subscale (AMS-PSY) which contains depressed mood, burned out, increased irritability, anxiety and nervousness; (2) somatovegetative subscale (AMS-SOM) which contains impaired well-being, increased joint complaints, increased sweating, need for more sleep, increased sleep disturbance, muscular weakness and physical exhaustion; and (3) sexual subscale (AMS-SEX) which contains past peak, decreased beard growth, impaired sexual potency, fewer morning erections and disturbed libido.

Literature search

We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) Statement [13]. We searched two electronic databases (PubMed and Google Scholar), using the search terms “Aging Males' symptoms scale” OR “Ageing Males' symptoms scale” AND “reliability”, in the title/abstract field without language restrictions from 1 January 1993 to 31 July 2016 to identify publications about the AMS scale. In addition, we screened references in the included studies and reviews, and hand searched The Aging Male archives. Journal articles, dissertations, theses and electronically available unpublished manuscripts were included to insure that all the available published and unpublished studies could be located in an attempt to control for publication bias.

Study selection

Two reviewers independently assessed publications for inclusion in the review. Discrepancies were resolved through discussion by the review team.

Psychometric variables analyzed and statistical methods used

All the statistical analyses were conducted using R version 3.3.1 (R Foundation for Statistical Computing, Vienna, Austria). Meta-analyses were conducted using the metafor package [14]. The p values were two-tailed, and the significance level was set at 0.05.

Three primary variables were of interest in this psychometric meta-analysis: internal consistency, test–retest reliability and descriptive statistics (i.e. means and standard deviations) of age and scale scores. If means and standard deviations were not reported in the included studies, these statistics were then derived using median and range with the formulas provided by Hozo et al. [15].

Cronbach’s alpha (α) was used for the internal consistency analyses. The raw α values were transformed using Bonett’s method [16,17] for analyses, and were back-transformed to α for presentation. Pearson’s product moment correlation coefficient r was used as the effect size for test–retest reliability and convergent validity analyses. Pearson’s rs were transformed into z values using the Fisher's method [18] for analyses, and were back-transformed to r. Assessments of homogeneity of α and r were conducted using random-effects (RE) models with restricted maximum-likelihood estimators, which assume the selected studies were sampled from a larger set of studies, thus allowing for greater external generalizability than fixed effects models. The 95% confidence interval (95% CI) was chosen to facilitate a simple assessment of whether an effect size was greater than zero.

To estimate the heterogeneity in effect sizes, the degree of inconsistency (I²) was calculated and interpreted as follows: 0% meant no inconsistency, 25% low inconsistency, 75% high inconsistency and 100% total inconsistency [19]. I² values greater than 75% implied significant heterogeneity, indicating that meta-regression analyses should be conducted for mediator and moderator variables.

Five methods were used to detect publication bias in the present study: funnel plot analysis, trim and fill procedure [20], Orwin’s fail-safe N [21], Egger regression [22] and the Begg rank test [23]. If there were possible publication bias, the method of Henmi and Copas [24] was used to estimate the average true effect and corresponding CI in a RE model.

Results

Literature search

We identified 10 publications in Pubmed, 121 in Google Scholar, and 22 in The Aging Male. A total of 12 studies were eligible for further analyses (Figure 1). These studies were conducted in 16 countries between 1996 and 2016. The number of participants per study varied considerably (range 21–4633, median = 210).

Figure 1. Flowchart of the search strategy and selection process for identifying papers for meta-analysis.

Internal consistency

A total of 12 studies [2–9,11,12,25,26] with a combined sample size of 12  747 participants reported coefficient α results for the AMS scale (Table 1). When all the studies were weighted and then averaged, α = 0.89 (95% CI 0.88–0.90) (Figure 2). Except for one study [25] that reported α results for only the AMS scale, the remaining 11 articles reported α results for all the AMS subscales, with a combined sample size of 12 135. The generalized α results for the three AMS subscales were as follows: α = 0.82 (95% CI 0.78–0.84) for the AMS-PSY (Figure 3); α = 0.81 (95% CI 0.79–0.83) for the AMS-SOM (Figure 4); and α = 0.79 (95% CI 0.77–0.82) for the AMS-SEX (Figure 5). Therefore, the pooled mean αs, and their upper and lower limits of CIs fell between the recommended cut-offs of 0.70–0.95 [27]. With regard to the meta-analyses of α results, all the I² values for the AMS scale and its subscales were above 75%, indicating significant heterogeneity. We conducted a meta-regression analysis with α coefficients as dependent variables, and found that the standard deviation (SD) and coefficient of variation (CV) of the AMS total scores were statistically significant positive moderators of α for the AMS scale (Table 2). There was a trend in the association between the SD of the AMS total score and the α coefficients of the AMS scale (r = 0.55, t = 2.18, df = 11, p = 0.052). The CV of the AMS total score was correlated with the α coefficients of the AMS scale (r = 0.67, t = 2.96, df = 11, p = 0.01).

Figure 2. Forest plot of the alpha coefficients of the AMS scale using a RE model.

Figure 3. Forest plot of the alpha coefficients of the AMS-PSY using a RE model.

Figure 4. Forest plot of the alpha coefficients of the AMS-SOM using a RE model.

Figure 5. Forest plot of the alpha coefficients of the AMS-SEX using a RE model.

Table 1. Internal consistency of the AMS scale and its three subscales – alpha coefficients.

						Alpha coefficients
Study	Country	Publication year	Sample size	Age (years)	Setting	AMS	AMS- PSY	AMS- SOM	AMS- SEX
Daig et al. (2003) [11]	German	1996	116	range: 40–70	Clinical	0.88	0.82	0.84	0.80
	German	1997	958	range: 40–70	Community	0.86	0.76	0.79	0.72
	German	2003	4633	range: 40–70	Community	0.90	0.83	0.82	0.81
	United Kingdom	2000	96	range: 40–70	Community	0.90	0.84	0.85	0.80
	Rest of Europe*	2003	118	range: 40–70	Community	0.88	0.76	0.77	0.80
	South Korea	2002	25	range: 40–70	Community	0.94	0.89	0.87	0.76
	Leungwattanakij et al. (2002) [9]	Thailand	2002	20	range: 40–70	Community	0.79	0.58	0.70	0.62
Chen et al. (2008) [8]	Taiwan	2007	176	mean (SD): 54.3 (10.7); median: 53; range: 40–80	Clinic	0.90	0.86†	0.79†	0.81†
Lee et al. (2016) [12]	Taiwan	2016	1787	mean (SD): 43.8 (11.5); median: 43; range: 17–86	Clinic	0.92†	0.89†	0.86†	0.78†
Akinyemi et al. (2008) [7]	Nigeria	2008	456	range: 60–90†	Community	0.92	0.80	0.78	0.81
Myon et al. (2005) [26]	France	2005	903	range: 15–60†	Community	0.89	0.75	0.81	0.76
Kong et al. (2014) [3]	China	2014	244	mean (SD): 52 (7.3); range: 40–79	Community	0.91	0.88	0.83	0.85
	China	2014	27	mean (SD): 25.1 (4.3); range: 22–36	Community	0.86	0.62	0.75	0.78
	Kobayashi et al. (2008) [6]	Japan	2008	39	median: 21; range: 18–29	Community	0.88	0.81	0.71	0.73
Japan		2008	125	median: 53; range: 40–77	Community	0.92	0.89	0.83	0.83
Japan		2008	93	median: 53; range: 34–84	Clinic	0.82	0.79	0.71	0.77
Ren et al. (2013) [5]		China	2013	973	range: 40–70†	Community	0.90	0.87	0.82	0.88
Yuen et al. (2016) [2]	Hong Kong	2016	825	mean (SD): 58.97 (11.09); range: 40–96	Community	0.88	0.70	0.85	0.80
Ardebili et al. (2014) [4]	Iran	2014	521	mean (SD): 50.8 (5.75); range: 40–65	Clinic	0.88	0.78	0.78	0.73
Mas et al. (2008) [25]	Spain	2008	612	range: 40–79	Community	0.917	NA	NA	NA

AMS-PSY: psychological subscale; AMS-SOM: somatovegetative subscale; AMS-SEX: sexual subscale; NA: not available; SD: standard deviation.

* France, Spain, Portugal, Italy and Sweden.

† The alpha coefficients were calculated from the raw data.

Table 2. Meta-regression analyses of the internal consistency of the AMS scale.

Independent variable	Unstandardized regression coefficients (ß)	SE	p
Publication year	0.0013	0.0142	0.93
Study setting (community versus clinical)	0.0438	0.1480	0.77
Country (Western versus Non-western)	−0.0187	0.1593	0.91
Mean of age	0.0060	0.0054	0.27
SD of age	0.0065	0.0253	0.80
Mean of AMS total score	−0.0027	0.0092	0.77
SD of AMS total score	0.0539	0.0230	0.02
CV of AMS total score	2.0646	1.0006	0.04

CV: coefficient of variation; SD: standard deviation; SE: standard error.

Bold values indicate p < 0.05.

Publication bias was evaluated as the possible bias for internal consistency of the AMS scale. All of the Egger’s regression and Begg rank tests and the trim and fill procedure suggested no publication bias (p > 0.05), and the fail safe N = 69. Visual inspection of plots suggested no obvious causes for concern.

Test–retest reliability

A total of seven studies [2–4,6,8,9,11] with a sample size of 594 reported test–retest r coefficients for the AMS scale (Table 3). Such results were weighted and then combined to yield a test–retest reliability coefficient (r_tt) of 0.85 (95% CI 0.82–0.88) for the AMS scale (Figure 6); r_tt of 0.76 (95% CI 0.72–0.80) for the AMS-PSY (Figure 7); r_tt of 0.83 (95% CI 0.77–0.87) for the AMS-SOM (Figure 8); and r_tt of 0.83 (95% CI 0.77–0.88) for the AMS-SEX (Figure 9). The interval between the measurements ranged from 1 week to 3 weeks. The duration between two measurements was not associated with the test–retest reliability coefficients for the AMS scale and its three subscales (p > 0.05). With regard to the meta-analyses of test–retest reliability coefficients, all the I² values for the AMS scale and its subscales were below 75%, indicating non-significant heterogeneity.

Figure 6. Forest plot of the test–retest coefficients (r) of the AMS scale using a RE model.

Figure 7. Forest plot of the test–retest coefficients (r) of the AMS-PSY using a RE model.

Figure 8. Forest plot of the test–retest coefficients (r) of the AMS-SOM using a RE model.

Figure 9. Forest plot of the test–retest coefficients (r) of the AMS-SEX using a RE model.

Table 3. Test–retest reliability of the AMS scale and its three subscales – Pearson’s correlation coefficients.

							Correlation coefficients (r)
Study	Country	Publication year	Sample size	Age (years)	Setting	Duration (weeks)	AMS	AMS- PSY	AMS- SOM	AMS- SEX
Daig et al. (2003) [11]	German		1996	102	range: 40–70		Community	2	0.86	0.81	0.85	0.80
	United Kingdom	2000	96	range: 40–70	Community	2	0.91	0.81	0.91	0.81
	France	2002	21	range: 40–70	Community	2	0.85	0.82	0.71	0.88
	Spain	2002	33	range: 40–70	Community	2	0.80	0.50	0.89	0.74
	Portugal	2002	20	range: 40–70	Community	2	0.74	0.67	0.45	0.67
	Sweden	2002	24	range: 40–70	Community	2	0.88	0.69	0.94	0.84
	Italy	2002	20	range: 40–70	Community	2	0.79	0.54	0.76	0.98
	South Korea	2002	25	range: 40–70	Community	2	0.90	0.85	0.87	0.77
	Leungwattanakij et al.  (2002) [9]	Thailand	2002	20	range: 40–70	Community	2	0.87	0.78	0.81	0.68
Chen et al. (2008) [8]	Taiwan	2007	45	mean (SD): 42.2 (5.5)	Community	3	0.72	0.73	0.65	0.82
Kong et al. (2014) [3]	China	2014	27	mean (SD): 25.1 (4.3); range: 22–36	Community	4	0.91	0.75	0.90	0.94
	China	2014	46	range: 40–79	Community	4	0.83	0.74	0.82	0.79
	Kobayashi et al. (2008) [6]	Japan	2008	20	NA	Clinic	2	0.88	0.90	0.86	0.73
Yuen et al. (2016) [2]	Hong Kong	2016	24	NA	Community	2	0.79	0.69	0.66	0.87
Ardebili et al. (2014) [4]	Iran	2014	71	range: 40–65	Clinic	1	0.87	0.79	0.85	0.79

AMS-PSY: psychological subscale; AMS-SOM: somatovegetative subscale; AMS-SEX: sexual subscale; NA: not available; SD: standard deviation.

Publication bias was evaluated as the possible basis for the test–retest reliability of AMS scale. All of the Egger’s regression and Begg rank tests suggested no publication bias (p > 0.05), and the fail safe N = 7. The trim and fill procedure suggested four missing studies on the right side of the funnel plot (p = 0.03). Based on the method of Henmi and Copas [24], the estimate of r_tt was 0.86 (95% CI 0.83–0.89) for the AMS scale.

Discussion

This meta-analytic review examined two types of reliability for the AMS scale. With regard to internal consistency, our meta-analysis showed that the generalized αs of the AMS scale and its subscales were good to excellent, although such estimates might have been inflated by significant heterogeneity. The results of a meta-regression indicated that the SD and CV of the AMS total scores were the most important predictors of internal consistency of the AMS. The relationships among SD, CV and internal consistency are consistent with the classical test theory, so that with increased observed score variance, the reliability also increases, provided that the error variance remains constant. Our α estimates should also be robust because the fail-safe number was 69, immensely larger than the number of included studies.

Second, test–retest reliability – a consistency of measurement across time – is an important aspect of an instrument’s reliability. If an assessment instrument has low test–retest reliability, it is very difficult to distinguish between changes in scores that reflect real changes in the severity of illness from variations in scores due to the instrument's low test–retest reliability. According to the criteria of Gliner et al. [28], the test–retest reliability is high if it is above 0.80. Our meta-analysis demonstrated that the AMS scale, AMS-SOM and AMS-SEX had good test–retest reliability, whereas the AMS-PSY has slightly lower test–retest reliability (0.76). With regard to potential publication bias, the trim and fill procedure suggested some missing studies on the right side of the funnel plot of the AMS scale, and the fail-safe number was 7, indicating that the estimates would be liable to new studies. The test–retest reliability of the AMS scale might be under-estimated in RE models so it would be better estimated as 0.86 (95% CI 0.83–0.89) using the method of Henmi and Copas [24].

There were some difficulties in performing this meta-analysis. First, important data were often missing, i.e. many studies did not report the characteristics of the participants and other relevant information necessary for moderator analyses or transformations, e.g. the mean and standard deviation of age of participants, means and standard deviations of the AMS score and subscores, and the clinical status of participants. Second, some reliability statistics, such as item-total correlation coefficients, were generally not reported in the included studies.

Conclusions

Our meta-analyses demonstrated that the AMS scale and three subscales had fairly good internal consistency and test–retest reliability internationally. Future psychometric studies of the AMS scale should include important characteristics of the participants, details of item scores and test–retest reliability.

Ethical approval

This article contains two studies [8,12] with human participants performed by the authors. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent was obtained from all individual participants included in the two included studies that were performed by the authors.

Declaration of interest

All authors declare no conflicts of interest. The authors are responsible for the content and writing of this paper.

This research received no specific grant from any funding agency in the public, commercial, or non-for-profit sectors.

Acknowledgements

We would like to thank Ms. Ya-Shan Chen for assistance with the managerial work of the present study.