Item-level psychometrics of a brief self-reported memory problem screening measure in breast cancer survivors

Abstract

Background: We previously developed and psychometrically validated a self-reported memory problem (SRMP) measure using principal component analysis. In the present study, we applied item response theory (IRT) analysis to further examined the construct validity and determine item-level psychometric properties for the SRMP.

Methods: The sample included 530 female breast cancer survivors (61% non-Hispanic White, mean age = 57 years) who were recruited from community-based organizations and large health care systems. We examined the construct validity, item-level psychometric properties, and differential item functioning (DIF) of the SRMP using confirmatory factor analysis (CFA), IRT and logistic regression analysis models.

Results: The CFA confirmed a one-factor structure for the SRMP (comparative fit index [CFI] = 0.996, root-mean-square-error-approximation [RMSEA] = 0.059). As expected, SRMP scores correlated significantly with pain, mood, and fatigue; but not spiritual health locus of control (SPR). DIF analysis showed no measurement differences based on race/ethnicity and age groups.

Conclusion: The CFA and DIF analysis supported the construct validity of the SRMP and its use in an ethnically diverse breast cancer population. These findings provide further evidence of the generalizability for the SRMP, and support its utilization as a psychometrically valid and reliable screening measure of cancer and treatment-related memory difficulties.

Introduction

Cancer and treatment-related neurocognitive dysfunction (CRND) is a debilitating condition that can impact multiple dimensions of psychosocial functioning, activities of daily living, and overall quality of life (QoL). CRND occurs in about 75% of survivors of non-central nervous system cancers (non-CNS) [1–7] and involves mild to moderate difficulties in attention, memory, processing speed and executive functioning [4,8–10]. Difficulties remembering are the most commonly reported cancer-related neurocognitive sequelae [11].

Memory performance is central to successful engagement in normal mental operations and activities of daily living. Studies have shown strong associations between subjective complaints of cognitive difficulties and deficits in daily functioning and QoL [12–14]. The magnitude of CRND is also noteworthy. Individuals in the United States with a history of non-CNS cancer and cancer treatments are 40% more likely to experience memory problems compared with age-matched healthy controls without a history of cancer or cancer treatments [10]. Finding ways to systematically screen cognitive impairments can have a positive health impact for cancer patients/survivors.

CRND is usually determined based on self-reports of cognitive difficulties or objective neuropsychological tests [9–11,15,16]. However, these methods are debatable and commonly yield inconsistent outcomes [9,17,18]. Self-reports are subjective and available objective neuropsychological tests may lack ecological validity for cancer populations [9,15,19]. Consequently, there are no ‘gold standard’ measures for CRND, which underscores the need to develop ecologically validated screening measures [11,15,19,20].

We previously validated the self-reported memory problem (SRMP) measure in a sample of adult-onset non-CNS survivors using classical test theory (CTT) methods that yielded evidence of unidimensionality and high internal consistency [19]. The CTT analysis focused on the internal structure of SRMP but did not establish criterion-related validity by examining the relationship of SMRP with other constructs. Additionally, the CTT analysis did not (a) distinguish between examinee’s and test items’ characteristics, (b) establish measurement invariance based on sociodemographics and (c) treated reliability or measurement precision as a constant (i.e., CTT considered reliability a property of test scores that applies equally to every participant, whereas reliability can vary depending on the latent trait level in IRT analysis). In the present study, we extend the CTT validation of the SRMP using IRT analysis and differential item functioning (DIF) based on age and race/ethnicity. Item parameter estimates (e.g., item difficulty, and discrimination and slope parameters based on item characteristic curve) obtained from this study will inform future applications of the SRMP.

Methods

Study population

The sample included 530 female breast cancer survivors, 25–82 years old, identified via community-based organizations (Bridge Breast Network, Sisters Network Inc., and Army of Women) and hospital systems in the Dallas/Fort Worth Metroplex (JPS Center for Cancer Care and Texas Health Resources). Participants identified via hospital systems were ascertained from hospital tumor registries. Most participants resided in the Dallas/Fort Worth Metroplex area, except those recruited from the Army of Women who were located throughout the United States. Participants completed a web-based survey (mostly non-Hispanic whites) from September to October 2015 or a paper survey (mostly African Americans) via postal mail between June 2014 and April 2015. Data were inputted into IBM-SPSS statistical analysis software package. Institutional Review Boards at the JPS Center for Cancer Care, Texas Health Resources, and academic institutions including the University of Alabama, Tuscaloosa, Alabama approved this study.

Measures

We analyzed data from a subset of items that were assessed as part of a screening survey for a larger randomized controlled trial (RCT). The larger RCT included questions about participants’ sociodemographics, medical history, lifestyle variables (i.e., diet, exercise, weight), and health-related QoL. This analysis is based on the QoL data.

SRMP. Participants completed this brief screening measure of global memory problems experienced over the past 2 weeks, and rated SRMP items [(1) my head feels heavy, (2) my brain feels muddled or scrambled, (3) it is difficult to think, (4) I cannot concentrate, and (5) it’s easy for me to forget things] on a Likert-type scale ranging from ‘1 = never’ to ‘5 = Several times a day’. Our previous psychometric validation of the SRMP using CTT in 781 cancer patients revealed adequate internal consistency and test–retest reliability, data factorability based on Kaiser–Meyer–Olkin test (KMO = 0.87), Bartlett’s sphericity test (p < .001), Kaiser’s simplest criterion test of eigenvalue ([λ] > 1), item-correlation coefficients ≥ 0.30 [19]. Additionally, the contents of attention and memory often overlap. Therefore, SRMP items are expected to probe cognitive domains (e.g., aspects of attention) that are linked with memory. Previous studies showed significant relationships between attention and working memory [21–23].

Fatigue interference. Participants completed an abbreviated version of the Fatigue Symptom Inventory (FSI) [24,25] that assessed how much fatigue interfered with general activity, walking ability, normal work, relationships with other people, and mood in the past 7 days. Items were rated on a Likert-type response scale ranging from ‘1 = never’ to ‘5 = all the time’. Internal consistency reliability (ICR) for this sample is appropriate (α = 0.93).

Pain interference. Patients completed a 7-item pain interference measure that assessed how much pain interfered with general activity, mood, walking ability, normal work, relations with other people, sleep, and enjoyment of life in the past 7 days. Items were rated on a Likert-type scale ranging from ‘1 = never’ to ‘5 = all the time’. The validity and reliability of the measure was previously established [26]. ICR for this sample is suitable (α = 0.95).

Mood. We assessed mood using the 4-item Patient Health Questionnaire (PHQ-4) – a brief screening measure that quantifies symptoms of anxiety and depression over the past 2 weeks. Items were rated on a Likert-type scale ranging from ‘0 = Not at all’ to ‘3 = Nearly every day’ [27]. The PHQ-4 is commonly used in clinical settings and has high sensitivity and specificity for diagnosing psychiatric outcomes. PHQ-4 scores ≥3 are indicative of anxiety and depression [27]. ICR for this sample was adequate (α = 0.85).

Spiritual health locus of control (SPR). The SPR was developed for African Americans and includes subscales designed to capture four aspects of spiritual beliefs: (1) spiritual life/faith subscale that pertains to being healthy because of living a ‘good’ spiritual life (i.e., Through my faith in God, I can stay healthy), (2) active subscale that relates to the belief that one works with God to stay in good health (i.e., God gives me the strength to take care of myself), (3) passive subscale that relates to the belief that God has complete control over one’s health (i.e., There is no point in taking care of myself when it’s all up to God anyway), and (4) God’s grace subscale that refers to being healthy through prayer and God’s grace (i.e., God works through doctors to heal us). SPR items were rated on a Likert-type scale ranging from ‘1 = strongly disagree’ to ‘5 = strongly agree’. The SPR has appropriate construct and predictive validity, as well as ICR with Cronbach’s alphas ranging from 0.51 (Passive Spirituality subscale) to 0.81 (Spiritual Life/Faith subscale) [28,29].

Socio-demographics and medical data. The socio-demographics and medical data were self-reported by participants and included age (current, and at diagnosis), race/ethnicity, treatment, and disease stage at diagnosis. We categorized age as: Group 1 < 50 years, Group 2 (50–59 years), and Group 3 (60 years or older), and derived years since diagnosis from age at diagnosis and current age.

Statistical analysis

We conducted an exploratory factor analysis (EFA) that revealed only one-of-five eigenvalues (λ) > 1 (i.e., 3.469, 0.743, 0.365, 0.284 and 0.138), and applied Kaiser’s rule and scree plot inspection that confirmed the unidimensionality of the SRMP. We completed a confirmatory factor analysis (CFA) to evaluate the internal structure of the SRMP, DIF to assess measurement invariance/equivalence based on age and race/ethnicity, evaluation of criterion-related and discriminant validity using correlations among similar and different psychological constructs, and item response theory (IRT) analysis to examine item-level psychometrics. We used Mplus/IBM-SPSS for statistical analysis.

Confirmatory factor analysis

We conducted the CFA using Unweighted-Least-Squares-with-Mean-and-Variance adjustment that produced similar parameter estimates as Unweighted-Least-Squares and provided standard errors for the parameter estimates corrected by robust procedures. We treated item scores as ordinal. We estimated thresholds and polychoric correlations using a two-stage maximum likelihood estimation and obtained parameter estimates by minimizing the least squares fit function. We used recommended criteria (CFI ≥ 0.95 and RMSEA ≤ 0.06) to confirm the measurement model fit to the data [30,31].

Differential item functioning analysis

We calculated DIF to evaluate the SRMP performance based on age and race/ethnicity [32]. A series of logistic regression models were computed to determine the presence or absence of DIF based on age groups and racial/ethnic categories: $$X_j={\beta }_{0j}+{\beta }_{1j}F+{\beta }_{2j}G+{\beta }_{3j}FG+e_j,$$ where $X_j$ is the response to item j, $F$ is the matching variable (typically the total score on the scale), and G is group membership. G is 0 for the reference group (e.g., the majority group) and 1 for the focal group (e.g., the minority group). A ${\beta }_{2j}$ that is significantly smaller than ‘0’ indicates that individuals at the ability level (i.e., the same level of F) have a lower chance of scoring high on item j if they belong to the minority group, suggesting a DIF effect against the minority group. In contrast, a ${\beta }_{2j}$ that is significantly larger than ‘0’ indicates a DIF effect favoring the minority group.

We fitted five ordinal logistic regression models (one per item) because $X_j\mathrm{ }$ is ordinal instead of continuous. Either the total score or the latent trait estimate $\theta$ could be used as matching variable. The groups were race/ethnicity (White vs. Others) and age (<50 years, 50–59 years, and 60+ years). Main parameters of interest were ${\beta }_{2j}$ (main effect) and ${\beta }_{3j}\mathrm{ }$(interaction effect). A ${\beta }_{2j}\mathrm{ }$ significantly different from ‘0’ suggests that even when conditioning on the level of memory problems, scores on item j differ by race/ethnicity or age groups. This would indicate the uniform DIF effect of item j. A significant ${\beta }_{3j}\mathrm{ }$ would suggest non-uniform DIF. Whether DIF favors one group or another would depend on the level of the matching variable.

Criterion-related validity

To establish criterion-related validity, we calculated Pearson correlation coefficients between SRMP, and measures of pain, mood, fatigue, and SPR. Significant correlations were expected between SRMP and pain, mood and fatigue (i.e., concurrent validity). However, the correlation between SRMP and SPR was expected to be low (i.e., discriminant validity). We selected these variables because of previously reported associations between pain, fatigue, psychological distress and cognitive functioning [33,34]. Studies showed that pain can impact neurocognitive processes [34]. Studies also reported that spirituality is linked to a positive life theme that may influence how one thinks [35]. Although a direct link between spirituality and cognitive operations (e.g., memory) is not established, researchers have reported that spirituality may lead to cognitive biases [35]. While spirituality may influence beliefs, it is not expected to directly affect memory processing. We expected age to influence SRMP scores. Thus, we completed an analysis of variance (ANOVA) with post-hoc comparisons to assess SRMP score differences among different age groups (<50 years, 50–59 years, and 60 years or above).

Item response theory analysis

We established the SRMP unidimensionality using CFA and fitted a unidimensional IRT model to the data. Prior to performing the latent trait analysis, we assessed the data to ensure all response categories contained sufficient sample sizes for stable estimates. This led us to collapse response categories 4 and 5 for item 1 since only two participants endorsed category 5. The unidimensional normal-ogive graded response model (NO-GRM) [36] was fitted to the resulting data (i.e., collapsed categories 4 and 5 for item 1). This analysis was completed in Mplus using delta parameterization [37] and assuming the latent factor had a variance of 1. Additionally, the unidimensional NO-GRM can be mathematically equivalent to the one-factor CFA under certain regularity conditions [38,39]. We present general model fit results under the CFA framework since that approach is often the most familiar to the general audience. However, item information is rarely discussed under the CFA framework. Therefore, we discussed item characteristics (e.g., item parameters and item information) using the IRT framework.

GRM is one of the most widely used IRT models to fit Likert-scale data. Suppose that the item responses for item j are scored 1, 2, 3, …, K_j. The cumulative response function of the unidimensional NO-GRM is defined as: $$P_{\mathit{ijk}}^{\mathrm{*}}\left(\theta \right)\mathrm{ }=P\left(X_{ij}\ge k|{\theta }_i,{\alpha }_j,d_{jk}\right)=\phi \left({\alpha }_j{\theta }_i-d_{jk}\right),$$ for j = 1, 2, …, J, J being the total number of items of the scale (in this case 5), ${\theta }_i$ is the latent trait (memory problem) of individual i, and $\phi$ is the normal CDF. The cumulative response function $P_{\mathit{ijk}}^{\mathrm{*}}\left(\theta \right)\mathrm{ }$ captures the probability of individual i with latent trait ${\theta }_i$ endorsing response category k or above on item j. When none of the items need to be reverse-coded, an individual with higher latent trait is more likely to endorse high response categories. In the case of SRMP, individuals with more severe memory problems will be more likely to report more frequent occurrences of memory difficulties. The category response function is defined as the difference between two adjacent cumulative response functions: $$P_{\mathit{ijk}}\left(\theta \right)=P_{\mathit{ijk}}^{\mathrm{*}}\left(\theta \right)-P_{ij\left(k+1\right)}^{\mathrm{*}}\left(\theta \right).$$

Here, $P_{\mathit{ijk}}\left(\theta \right)$ is the probability of patient i endorsing exactly response category k on item j. When k = 1, $P_{\mathit{ijk}}^{\mathrm{*}}\left(\theta \right)=1;\mathrm{ }$ when k = K_j, $P_{ij\left(k+1\right)}^{\mathrm{*}}\left(\theta \right)=0.$ We obtained parameter estimates and standard errors for ${\alpha }_j,$ $d_{jk},$ and ${\theta }_i.$ We plotted category response curves for each SRMP item.

Results

Study participants

The sample consisted mostly of non-Hispanic White (61%) and African American (33%) breast cancer survivors, mean age 57 years (range: 25–82 years), and about 8 years since initial diagnosis. Participants were diagnosed with localized tumors (60%), m that spread to nearby tissues (30%), and distant metastases (3%). Treatment modalities included surgery (94%), supplemental chemotherapy (62%), radiotherapy (60%), and hormonal therapy (50%).

Confirmatory factor analysis and differential item functioning

The CFA fit the data adequately (${\chi }^2$ = 13.5, df = 5, p-value = .02, CFI = 0.996, RMSEA = 0.059). The CFI and RMSEA indicate a close fit of the one-dimensional model. DIF analysis showed the SRMP performed similarly among different race/ethnic and age groups. A logistic regression model was completed for each item, and we tested ${\beta }_2$ and ${\beta }_3$ for each model. Each item was evaluated for DIF separately. We applied a Bonferroni correction (p = .01) to correct for multiple tests [40]. None of the ${\beta }_{2j}$ or ${\beta }_{3j}$ were significantly different from ‘0’ (p > .01). The smallest p-value obtained was p = .044 for item 3. The findings indicate invariance across groups and will enable interpretation of SRMP differences via group means comparison.

Discriminant validity

SRMP was significantly correlated with pain (r = 0.44, p < .01), fatigue interference (r = 0.58, p < .01) and mood (r = 0.54, p < .01), but not with SPR (r = −0.05, p = .40). The ANOVA showed significant differences in SRMP scores based on age (F = 10.51, p < .001). Post-hoc tests revealed significant differences between age Groups 1 (<50 years) and 2 (51–59 years), and Groups 1 and 3 (60 years or older). There was no significant difference between Groups 2 and 3. Overall, Group 1 had the highest mean SMRP score. Mean SRMP score difference was 1.55 (p = .01) between Groups 1 and 2, and 2.41 (p < .001) between Groups 1 and 3, suggesting that younger survivors reported more memory problems.

Item response theory analysis

Distribution of responses. Participants responded primarily at the lower end of the scale (i.e., ‘never’ and ‘about once a week’) for items 1 (91%), 2 (76%) and 3 (75%). Few participants responded at the higher end of the scale (i.e., ‘nearly every day’ and ‘several times a day’). Items 4 and 5 were distributed more toward the middle of the scale (i.e., ‘about once a week’ and ‘two-to-three time a week’). Response categories for ‘often’ and ‘all the time’ were collapsed for item 1 due to insufficient responses for ‘all the time’. The proportions for the remaining items and response categories were retained. The distribution of item responses is summarized in Table 1. These responses are positively skewed for all SRMP items, which explain the statistically significant ${\chi }^2$ test in the CFA.

Table 1. Distribution of item responses.

Item	Category 1	Category 2	Category 3	Category 4	Category 5
1	0.782	0.129	0.057	0.032	–
2	0.529	0.234	0.131	0.073	0.032
3	0.535	0.218	0.145	0.077	0.024
4	0.428	0.267	0.206	0.071	0.028
5	0.248	0.289	0.222	0.145	0.095

Parameter estimation. We obtained the factor loadings, threshold estimates and corresponding SEs using Delta Parameterization in Mplus (see Table 2). The factor loadings range from 0.60 (‘item 1’) to 0.986 (‘item 3’) and were consistent with the CFA results confirming the unidimensionality of the SRMP. The loading of ‘item 3’ seems high. However, estimation with weighted least squares (WLS) corroborated the results from ULSMV, suggesting their robustness. Threshold parameters increased across the response scale and parameters’ SEs were small (mostly <0.10), indicating that the parameter estimates were precise. These parameter estimates can be translated to ${\alpha }_j$ and $d_{jk}$’s in the NO-GRM metric (see Table 3). Given the invariance of the IRT model parameters, these parameter estimates can be applied in future analysis estimating latent SRMP scores using maximum likelihood estimation.

Table 2. Parameter estimates and SEs for fitting the item factor analytic model.

Item	λ	t1	t2	t3	t4
1	0.600 (0.046)	0.778 (0.063)	1.348 (0.079)	1.848 (0.110)	–
2	0.918 (0.013)	0.073 (0.056)	0.718 (0.062)	1.253 (0.076)	1.848 (0.110)
3	0.986 (0.009)	0.089 (0.056)	0.686 (0.061)	1.276 (0.077)	1.973 (0.121)
4	0.894 (0.014)	−0.181 (0.057)	0.510 (0.059)	1.287 (0.077)	1.907 (0.115)
5	0.806 (0.020)	−0.679 (0.061)	0.094 (0.056)	0.705 (0.062)	1.311 (0.078)

Table 3. Item parameter estimates for the normal-ogive graded response model.

Item	${\alpha }_j$	d1	d2	d3	d4
1	0.750	1.297	2.247	3.080
2	2.315	0.080	0.782	1.365	2.013
3	5.913	0.090	0.696	1.294	2.001
4	1.995	−0.202	0.570	1.430	2.133
5	1.362	−0.842	0.117	0.875	1.627

Item category characteristic curves (ICCCs) and item information functions (IIFs). ICCCs were plotted in blue for each item (see Figure 1) and showed the probability of endorsing each response category for each item at different latent trait levels. For example, respondents low on θ (i.e., low memory problems) would be more likely to endorse response category 1 (i.e., ‘never’) for item 4. In fact, when θ < −1 the chance of responding ‘never’ on item 4 is nearly 100%. Respondents with θ slightly above ‘0’ would have a much higher chance to endorse response category 2. Respondents with θ close to ‘1’ would have the highest chance to endorse category 3 of item 4 and so forth. The chance of endorsing higher response categories increases with higher θ. The patterns of the ICCCs met our expectations (see Figure 1).

Figure 1. Item category characteristic curves (blue) and item information functions (pink) for the five items of the self-reported memory problems (SRMP) measure.

The IIF describes information that a specific item provides in differentiating individuals who are either high or low on θ. Item information curves were plotted and are represented in pink (see Figure 1). An item with higher information at a certain θ level allows us to better differentiate people who are higher or lower than that level. Both items 2 and 4 have high information for θ between 0 and 2.5, suggesting that these items are suitable for people with CRND. However, items 2 and 4 provide little information for people with θ = −1. Item 5 provides substantial information for people with θ = −1 (i.e., relatively fewer memory problems).

Discussion

We applied unidimensional IRT analysis to assess item-level psychometrics for the SRMP. The results confirmed previous CTT analysis [19] and revealed that the SRMP has appropriate psychometric properties, measurement invariance and construct validity. Younger breast cancer survivors scored higher on the SRMP, which could be due to younger individuals having higher baseline cognitive functioning, being more attuned to subtle changes in memory performance and experiencing higher level of distress. Additionally, younger survivors maybe engaging in more cognitively demanding activities that can exacerbate their cognitive problems. Interventions to mitigate cancer-related memory problems should consider the effects of age on cognition and rehabilitation and recovery of cognitive abilities.

The findings reveal specific characterization of the psychometric properties of each item through item parameter estimates, category characteristic curves and item information curves. Oftentimes, composite scores can show suitable psychometric properties (e.g., reliability) while it is unclear whether individual items function appropriately or need to be modified based on a scale-level analysis. IRT provides item-level information that allows examination of each item, complements scale-level analysis (i.e., CTT), and strengthens the psychometric evidence of an SRMP. These findings will inform future SRMP applications and strategies to enhance screening/monitoring of memory performance for cancer survivors.

Despite known deleterious impact of cancer and its treatments on cognition, CRND is not commonly assessed in oncology care. Self-report measures (e.g., SRMP) may be influenced by subjectivity biases. However, individuals have the benefit of self-knowledge and may report impairments that objective tests are not capturing. The SRMP can be used to facilitate screening/monitoring of CRND and inform referral decisions for comprehensive objective testing. Albeit not representative of objective cognitive dysfunction, self-reports of CRND are important and should be integrated into the treatment process. In fact, studies have reported strong associations between subjective complaints of cognitive difficulties and decreased participation in activities of daily living and QoL [12,13]. Future research should integrate self-reports with objective measures (e.g., blood biomarkers, neuropsychological tests, and neuroimaging) to facilitate the systematic evaluation of CRND. This approach will bridge gaps between self-reports and neuropsychological tests for CRND [15].

Data in the present analysis are cross-sectional and include self-reports. We also used two modes of survey administration. Nevertheless, this study has several strengths. The results demonstrated the construct and discriminant validity of the SRMP in a diverse sample. We used brief measures with established reliability and validity to quantify psychosocial outcomes that reduced participant burden, and we applied robust statistical methods to determine item-level characteristics. Overall, findings of this IRT analysis support the use of the SRMP as a brief screening measure for CRND.

Conclusions

Establishing the psychometric properties of the SRMP addresses a significant research gap. This IRT analysis complements previous CTT validation and supports the use of the SRMP as a reliable and valid screening measure of cancer-related memory sequelae that could inform decisions for follow-up objective testing to determine the presence/severity of CRND. Future studies should examine the relationships between the SRMP and other measures of cognitive performance, establish the ecological validation of objective neuropsychological tests, and facilitate the comparison and integration of self-reports and objective measures for CRND.

Disclosure statement

No potential conflict of interest was reported by the authors.