Psychometric validation and gender invariance analysis of a revised version of the attitudes towards research scale (EACIN-23) in a Chilean university student sample

Abstract

Scientific research is vital for student’s education, fostering critical thinking, problem-solving skills, and deepening subject knowledge. To assess students’ attitudes towards research, the attitude towards research scale was developed (EACIN). This study addresses three gaps regarding this instrument: inconsistent latent structure, lack of comparison between available versions, and absence of gender invariance analysis given previously found gender differences. We analyzed data from 358 Chilean undergraduates, assessing the different version’s dimensionality, reliability. Results revealed inadequate fit indices for all EACIN available versions. Consequently, we proposed a new solution through exploratory structural equation modeling, EACIN-23 (or the Attitude toward Research Scale, ATRS-23), with a three-factor latent structure (replicating a previously found latent structure by the scale authors) and acceptable to good goodness-of-fit. The ATRS-23 demonstrated adequate to optimal factor loadings and good reliability. Moreover, it achieved scalar invariance across genders and demonstrated excellent criterion validity. This study provides a reliable and valid psychometric version of the EACIN, capable of accurately measuring and predicting attitudes towards research, including discriminating between students with and without intentions for postgraduate studies and research careers. Given that the ATRS-23 is only available in Spanish, we provide its English version. These findings have significant implications for undergraduate education and scientific research training.

Keywords:

• EACIN-23
• ATRS-23
• attitude toward research
• psychometric properties
• validity
• invariance

PUBLIC INTEREST STATEMENT

In education, students’ outlook on research plays a big role in their experiences and career choices. Therefore, it’s important to have a reliable tool to measure these attitudes. The ATRS-23 scale, a modified version of the original, is a reliable and valid instrument that works equally well for men and women. Its previous versions did not match up well with the data. However, the ATRS-23 abides by high standards of scientific rigor, supporting well its theoretical model. The present study provides evidence its statistical properties and capacity to predict student’s career choices. Therefore, the ATRS-23 is a promising tool for researchers and educators to better assess and understand these attitudes not only in students but also in educators and administrative staff.

1. Introduction

Scientific research is a crucial component of undergraduate education, as it enables students to develop their analytical and critical thinking skills, problem-solving abilities, and gain a deeper understanding of their fields of study (Petrella & Jung, 2008). Some of the benefits of undergraduate research activities (e.g., development of thinking and communication skills; Bradforth et al., 2015; Lopatto, 2003) extend beyond the individual, as they also contributes to the advancement of knowledge in various academic fields (Kardash, 2000). Likewise, students who engage in research activities are more likely to pursue graduate studies and careers in research-related fields (Lopatto, 2006). Thus, understanding what students already know and any potential preconceived ideas they may have about scientific research is a crucial element of their education, with far-reaching implications.

One effective approach to gaining insight in this matter is by exploring their attitudes. These refer to the general evaluations or judgments that individuals hold about specific objects, people, or events (Wood, 2000). Attitudes play a significant role in shaping students’ behaviors, including their course selection, academic performance, and educational outcomes. A positive attitude towards learning has been linked to higher academic achievement, greater persistence, and motivation to learn (Chen et al., 2018; Gardner & MacIntyre, 1993), as well as to correlate with higher grades and better performance (Mao et al., 2021). Similarly, attitudes have shown to influence student’s performance (Else-Quest et al., 2013), as well as career choices (Curran & Rosen, 2006; Hofstein et al., 2010; Wiebe et al., 2018), potentially having long lasting repercussions from as early as primary school (Turner & Ireson, 2010).

Amongst the multitude of variables that modulate student’s attitudes towards scientific research and science, school and university students often report gender differences (Osborne et al., 2003; Sofiani et al., 2017), varying by country (Emily Oon & Subramaniam, 2023), field of scientific interest (Khan et al., 2022), etc. The magnitude of gender differences towards science have been so that Jones et al. (2000) predicted that the gender landscape of STEM fields in the USA will remain the same despite 20 years of efforts to change that, a prediction that has largely been corroborated (National Science Foundation, 2014). Given that student’s (Cheng & Lo, 2022) and also teacher’s attitudes (Kollmayer et al., 2020) have shown to contribute to this reality, it becomes crucial to be able to accurately measure and compare attitudes within individuals and across parties.

As a way to objectively evaluate this phenomenon, Aldana et al. (2016) developed the attitude towards research scale (EACIN). Its initial latent model is based on McGuire three-factor model of attitudes (McGuire, 1969): affective (i.e., what the subject feels or the emotions that research produces in the subject), cognitive (i.e., what the subject knows or thinks they know about research), and behavioral (i.e., what the subject does or is willing to do with respect to research). Originally composed of 34 items measured through a five-points Likert scale, the scaled has been revised into two new versions, one with 32 (Quezada-Berumen et al., 2019) with a three-factorial latent structure (i.e., affective-behavior, cognitive, and behavioral-affective factors), and another with 28 (EACIN-R; Aldana et al., 2020) with a different three-factorial latent structure (i.e., disinterest in research, vocation for research, and research valuation). Studies evaluating the EACIN/EACIN-R psychometric properties have shown mix results in terms of their goodness of fit and explanatory power, yet only the EACIN-R latent structure has been replicated with good fit indicators, and by different groups of researchers (Aldana et al., 2020; Hidalgo et al., 2023). The reason as to why the latent structure varies across studies could be methodological (e.g., differences in the study samples, design, techniques, etc.), theoretical (e.g., different emphasis of the underlying theory), or likely both (e.g., different samples leading to different factor loadings, and ultimately leading to a different interpretation of the latent construct).

There is, to our knowledge, five other alternatives to this scale that measure the same or a similar construct. The attitude towards research (ATR; Papanastasiou, 2005) is perhaps the better known of all. It is a self-report tool measuring students’ attitudes towards scientific research as a field, regardless of their orientation (i.e., quantitative, qualitative, or mixed methods). Moreover, Castro Molinares (2017) developed a scale to measure attitude towards formative research in university students. The scale is composed by 26 items measured though a Likert scale, with a five-factor structure explaining 56% of the variance, and with good overall reliability (Cronbach’s alpha = .827). Likewise, Ozturk (2011) developed a similar scale to measure attitudes towards education research, with an eight-factor structure measured through 29-Likert items. Similarly, Muthuswamy et al. (2017) proposed also a scale measuring attitudes towards research, with a six-factor structure, measured through 28-Likert items. Finally, Muñoz et al. (2020) developed a scale to measure attitude towards research in university students. The scale is composed by 17 items measured through a Likert scale, with a three-factor structure, and alpha coefficients ranging between .700–.901. Neither of these scales, except the ATR, have been replicated by other psychometric and independent studies. While there are also other scales that also assess student’s perceptions of scientific research (e.g., Gregory, 2010; Kirk, 1981), they are a bit more specific to other contexts and phenomena.

1.1. Statement of the problem

The study of student’s attitudes towards scientific research provides valuable insight of its impact in student’s skill development, performance, career choices, etc. While there are alternatives as to how to measure this phenomenon, these are unable to compare student’s attitude with other relevant roles such as teachers and academic managers. However, the EACIN-R stands out due to its versatility by being able to compare across these educational roles. Yet, the inconsistency of its theoretical structure and capability to meaningfully compare between men and women remain unexplored. Namely, 1) the internal latent structure of the EACIN/EACIN-R is not consistent throughout studies and lacks an explanation as to what its dimensions represent; 2) there has not been a comparison of the original and revised versions to evaluate which one offers a better fit to the data; and 3) neither of the previous studies provides an invariance analysis of the instrument for any variable. Therefore, the following study evaluated the internal structure of the scale by comparing its different versions, so as to ascertain which factorial structure and item structure version provided a better fit to the data, as well as to offer a gender invariance analysis.

2. Methods

2.1. Design

The study used a cross-sectional design, using both the EACIN and EACIN-R scales by including all 34 original items, yet analysing all versions separately. We chose not to include #35 from Quezada et al. study (Quezada-Berumen et al., 2019), yet still assessed its 31-items version so as to evaluate how the original EACIN items would adjust without item #35. We assessed the scales psychometric properties to provide a definitive version of it. This study was pre-registered before the data was analyzed in the open science framework (https://osf.io/xtdyn/), which did not include modifications made after the peer-review process.

2.2. Participants

The sample was originally comprised of 408 Chilean undergraduate students of Psychology from different Universities from Iquique and Arica, from all year classes. The study used a non-probabilistic convenience sample of ≥18 years old participants. After removing participants who did not provide consent to participate, and repeated responses, the final database was comprised of 358 people. Participant’s descriptive statistics are presented in Table 1. Individuals whose gender responses were “non-binary” or “others” were not included in the invariance and validity analyses given their small sample size.

Table 1. Sample descriptive statistics

Variables (n = 358)	n	%
Gender
Man	103	28.8
Woman	241	67.3
Non-binary	11	3.1
Other	3	.8
Cohort
1st year	95	26.5
2nd year	62	17.3
3rd year	64	17.9
4th year	44	12.3
5th year	93	26
Have you performed as a Teaching Assistant?
Total
Yes	61	17.0
No	297	83.0
Men
Yes	22	21.4
No	81	78.6
Women
Yes	36	14.9
No	205	85.1
Do you intend to pursue postgraduate studies (Master/PhD)?
Total
Yes	268	74.9
I am not sure	84	23.5
No	6	1.7
Men
Yes	73	70.9
I am not sure	29	28.2
No	1	1
Women
Yes	186	77.2
I am not sure	50	20.7
No	5	2.1
Do you plan to dedicate yourself to do research for living?
Total
Yes	63	17.6
I am not sure	183	51.1
No	112	31.3
Men
Yes	26	25.2
I am not sure	49	47.2
No	28	27.2
Women
Yes	35	14.5
I am not sure	126	52.3
No	80	33.2

2.3. Procedure

The project was conducted online between June 2022 and March 2023. Our study was evaluated and authorized by the University of Tarapacá’s Ethic Committee (N°35/2022). Participants who responded consented to participate completed an anonymous self-report survey on an online Google form platform that lasted approximately 15–20 min.

2.4. Instruments

The project took into account the following demographic variables characteristics related to career choices: gender, have they performed as a teaching assistant, intention to go into graduate school, intention to work as a researcher.

The Attitudes towards Research (EACIN) is a scale developed by Aldana de Becerra and collaborators (Aldana de Becerra & Joya Ramírez, 2011; Aldana et al., 2016, 2020). Initially, the original scale was based on a multidimensional model and is composed of 34 items answered Likert style, distributed in three dimensions: affective, cognitive, and behavioral. Furthermore, Quezada et al. proposed a similar structure blending some aspects of the original dimensions into each other (Quezada-Berumen et al., 2019). Subsequently, Hidalgo et al. elaborated the revised version, EACIN-R, with the following subscales: (a) disinterest in research; (b) vocation for research, and (c) valuing research (Hidalgo et al., 2023). Since we aimed to determine the latent factorial structure of the EACIN, we tested all three versions of it to evaluate which latent factorial structure would provide a better fit for the data.

2.5. Statistical analyses

The statistical plan as per pre-registered was composed of the following steps: data imputation, descriptive analysis, test of dimensionality, test of reliability, measurement invariance, and criterion validity test.

We planned to analyse missing values in the database to identify the type of missing data pattern. However, contrary to what was expected in the pre-registration, we had no missing data, hence imputation was unnecessary. Thus, we first used demographic statistics of central tendency and dispersion to describe the sample, items, and dimensions for all EACIN scale versions (i.e., M, SD, n, %, Skewness and Kurtosis, see Tables 1 and 2).

Table 2. Descriptive statistics for all EACIN items

Items	Range	M	SD	Skewness	SE	Kurtosis	SE
Item 1		4.32	1.06	−1.77	.13	2.56	.26
Item 2		2.97	1.11	−.06	.13	−.59	.26
Item 3		3.42	1.02	−.42	.13	−.19	.26
Item 4		4.06	1.00	−1.00	.13	.62	.26
Item 5		4.51	.80	−2.04	.13	4.79	.26
Item 6		3.37	1.10	−.47	.13	−.40	.26
Item 7		4.06	1.08	−1.16	.13	.83	.26
Item 8		3.94	.96	−.88	.13	.61	.26
Item 9		3.11	1.05	−.02	.13	−.44	.26
Item 10		3.35	.95	−.22	.13	−.18	.26
Item 11		3.60	1.03	−.49	.13	−.27	.26
Item 12		4.44	.84	−2.00	.13	4.79	.26
Item 13		3.54	1.06	−.32	.13	−.52	.26
Item 14		3.53	1.04	−.20	.13	−.74	.26
Item 15		4.27	.89	−1.52	.13	2.63	.26
Item 16		3.18	1.08	−.16	.13	−.52	.26
Item 17		3.21	1.12	−.22	.13	−.61	.26
Item 18		3.25	1.01	−.24	.13	−.33	.26
Item 19		3.79	.98	−.50	.13	−.20	.26
Item 20		4.17	.85	−1.13	.13	1.51	.26
Item 21		3.37	1.03	−.38	.13	−.38	.26
Item 22		4.27	.83	−1.49	.13	3.05	.26
Item 23		3.67	1.16	−.61	.13	−.44	.26
Item 24		2.67	1.09	.26	.13	−.47	.26
Item 25		3.19	1.04	−.33	.13	−.14	.26
Item 26		4.33	.88	−1.84	.13	4.20	.26
Item 27		3.46	1.09	−.44	.13	−.33	.26
Item 28		3.42	1.14	−.27	.13	−.58	.26
Item 29		4.49	.85	−2.20	.13	5.49	.26
Item 30		3.39	1.03	−.12	.13	−.40	.26
Item 31		4.30	.86	−1.54	.13	2.89	.26
Item 32		3.23	1.14	−.13	.13	−.52	.26
Item 33		4.08	.94	−.98	.13	.81	.26
Item 34		3.67	1.03	−.45	.13	−.27	.26
34-items version	0–136	109.77	13.028	−2.45	.129	10.384	.257
Affective		26.97	3.94	−1.04	.13	3.04	.26
Cognitive		45.91	6.52	−2.21	.13	8.09	.26
Behavioral		36.90	4.89	−1.27	.13	5.02	.26
28-items version	0–112	88.79	10.818	−2.181	.129	8.638	.257
Vocation		20.16	5.38	.15	.13	−.26	.26
Valuation		39.69	8.11	−.45	.13	.59	.26
Disinterest		28.94	4.52	−1.98	.13	6.28	.26
23-items version	0–92	38.41	5.81	−2.355	.129	8.176	.257
Vocation		18.32	3.40	−.16	.13	−.19	.26
Valuation		28.97	6.50	−.35	.13	.29	.26
Disinterest		85.70	12.24	−.72	.13	1.14	.26

M = Mean; SD = Standard deviation; SE = Standard error.

In order to confirm the internal structural validity of the model, due to the ordinal nature of the EACIN items, and the lack of multivariate normality in the data detected through the Mardia’s test (Mardia, 1970) (bSkew = 211.425; bKurt = 1395.329; ps < .001), the Weighted Least Squares Means- and Variance-Adjusted (WLSMV) was used as the estimation method for model and invariance testing (Barrett et al., 2008; Browne & Cudeck, 1992; Finney & DiStefano, 2013; Kline, 2015).

The tested models were evaluated through commonly used goodness of fit indices (Marsh & Hau, 2005; Schermelleh-Engel et al., 2003): Comparative Fit Index (CFI ≥ .90 adequate; ≥ .95 good), Tucker—Lewis index (TLI ≥ .90 adequate; ≥ .95 good), and Root-Mean-Square Error of Approximation with its 90% confidence interval (RMSEA ≤ .08 adequate, ≤ .05 good) and SRMR < .08 good (Schreiber, 2017). Regarding factor loadings, we will use the criteria proposed by Jöreskog & Sörbom (Jöreskog & Sörbom, 1993), optimal (λ > .7) and adequate (.7 < λ > .3). Furthermore, the criteria of strong (r > .7) and moderate (.7 < r > 0.3) correlations were used for correlations between dimensions (Cohen, 1992). Furthermore, to assess the scale’s criterion validity, we used student’s career plans (see Table 1) as a way to evaluate the scale’s capability to discriminate high and low research attitudes, which have been previously found to correlate with career choices related to science (Balta et al., 2023; Sellami et al., 2023; Wiebe et al., 2018). Finally, we performed a discriminant validity assessment invariance-based SEM test using the Heterotrait-monotrait (HTMT) ratio of correlations in order to assess the distinctiveness of the factors, while also prevent multicollinearity issues (Hamid et al., 2017; Henseler et al., 2015).¹ The HTMT value is contrasted with a predefined threshold value, typically established at 0.85 (Kline, 2022). If the computed HTMT ratio falls below this threshold, it indicates that the two constructs are separate and exhibit satisfactory discriminant validity.

We reported reliability for each dimension and whole scale using Cronbach’s alpha coefficient. We also used McDonald’s hierarchical omega in order to report a more efficient reliability criteria (Goodboy & Martin, 2020; Hayes & Coutts, 2020; McNeish, 2018). Acceptable levels of reliability has been established to be ≥ .7; however, this criterion is a guide, more than an empirical one (Cho & Kim, 2015).

Finally, given the often reported gender differences/disparities in attitudes towards scientific research (e.g., Osborne et al., 2003), and therefore to assess whether the EACIN-R can be interpreted in the same way across gender, we performed tests of measurement invariance (Vandenberg & Lance, 2000). We tested three levels of invariance: configural (i.e., same structure), metric (i.e., same factor loadings), scalar (i.e., same item intercepts). Finally, to establish the evidence of criterion validity with other variables, we performed one-way ANOVA analyses and independent sample Student t-test’s using career choices as variables of interest. All results were conducted through Mplus 7.0 and SPSS 22.

3. Results

3.1. Study 1 - EACIN previous versions

3.1.1. Test of dimensionality

Standardized factor loadings and reliabilities of both three-factors, 34-items and 28-items, models CFA can be seen in Table 3. The standardized factor loadings for the 34-items version ranged .101–.841 (ps < .001, except for item 28, p = .07), whereas the inter-factor correlations were positive and moderate (r = .556–.606; p < .01). Meanwhile, standardized factor loadings for the 28-items version ranged .381–.804 (ps < .001), whereas the inter-factor correlation were positive and negative, while also weak-to-moderate (r = .201–.691; p < .001). Moreover, both models yielded overall and dimensional acceptable-to-good levels of reliability.

Table 3. Previous EACIN version’s factor loadings and reliability

34-items version (ω = .902; α = .905)				28-items version (ω = .900; α = .901)
	Estimator†	SE	Est./SE		Estimator†	SE	Est./SE
Factor 1 - Affective (ω = .825; α = .821)				Factor 1 - Disinterest (ω = .765; α = .767)
Item 2	.556**	.040	14.004	Item 1	.381**	.054	6.998
Item 3	.532**	.041	12.915	Item 4	.549**	.045	12.143
Item 6	.667**	.030	21.890	Item 5	.640**	.049	13.187
Item 11	.780**	.026	29.867	Item 9	.577**	.042	13.702
Item 14	.299**	.047	6.414	Item 14	.427**	.049	8.679
Item 17	.769**	.027	28.983	Item 19	.773**	.031	25.075
Item 19	.605**	.035	17.133	Item 27	.837**	.031	27.330
Item 25	.727**	.027	27.158	Item 30	.587**	.042	14.123
Item 27	.634**	.035	18.265	Item 34	.497**	.047	10.499
Factor 2 - Cognitive (ω = .824; α = .808)				Factor 2 - Vocation (ω = .874; α = .872)
Item 1	.286**	.054	5.293	Item 2	.556**	.039	14.241
Item 7	.627**	.043	14.501	Item 3	.524**	.041	12.677
Item 12	.709**	.041	17.29	Item 6	.663**	.03	21.800
Item 15	.65**	.036	17.841	Item 8	.704**	.03	23.199
Item 20	.703**	.032	21.757	Item 10	.548**	.037	14.935
Item 22	.816**	.028	29.488	Item 11	.772**	.025	31.058
Item 26	.841**	.027	31.638	Item 13	.574**	.037	15.563
Item 28	.101*	.056	1.815	Item 16	.652**	.032	20.382
Item 29	.756**	.040	19.031	Item 17	.763**	.027	28.650
Item 31	.739**	.033	22.679	Item 18	.648**	.031	20.839
Item 32	.403**	.048	8.423	Item 24	.559**	.037	15.220
Item 33	.66**	.035	18.703	Item 25	.716**	.026	27.157
Factor 3 - Behavioral (ω = .807; α = .808)				Factor 3 - Valuation (ω = .834; α = .832)
Item 4	.420**	.046	9.206	Item 7	.600**	.044	13.500
Item 5	.511**	.049	10.444	Item 12	.668**	.043	15.681
Item 8	.724**	.031	23.470	Item 20	.669**	.033	20.099
Item 9	.392**	.044	8.890	Item 21	.754**	.031	24.091
Item 10	.557**	.037	15.042	Item 22	.758**	.031	24.117
Item 13	.604**	.036	16.713	Item 26	.804**	.029	28.053
Item 16	.659**	.032	20.557	Item 31	.671**	.037	18.104
Item 18	.662**	.031	21.329	Interfactor Correlations
Item 21	.696**	.030	23.279	Disinterest	–
Item 23	.475**	.042	11.316	Vocation	−.448*	–
Item 24	.556**	.038	14.666	Valuation	−.201*	.691*	–
Item 30	.426**	.041	10.313	† = two-tailed; * = < .01; ** = < .001.
Item 34	.365**	.047	7.761
Interfactor Correlations
Affective	–
Cognitive	.556*	–
Behavioral	.606*	.563*	–

All versions yielded unacceptable fit indices to the data (34-items: WLSMV χ² (524) = 2,107.935, p < .001, CFI = .798, TLI = 0.784, RMSEA = .092, SRMR = .091 [90% CI = .088–.096]; 31-items: WLSMV χ² (431) = 1226.803, p < .001, CFI = .895, TLI = .887, RMSEA = .072, SRMR = .067 [90% CI = .067–.077]; 28-items: WLSMV χ² (374) = 1183.135, p < .001, CFI = .877, TLI = .865, RMSEA = .082, SRMR = .07 [90% CI = .077–.087]; see Table 4).

Table 4. Fit indices for all attitude towards research scale versions (EACIN, EACIN-R, and EACIN-23)

Models	FP	χ2	df	RMSEA	90% CI	CFI	TLI	SRMR
CFA—EACIN 34-items	173	2107.935*	524	.092	.088–.096	.798	.784	.091
CFA—EACIN-R 31-items	158	1226.803*	431	.072	.067–.077	.895	.887	.067
CFA—EACIN-R 28-items	143	1183.135*	347	.082	.077–.087	.877	.865	.070
eSEM − 23-items	158	433.952*	187	.061	.053–.068	.955	.939	.038
CFA—EACIN-23	118	510.395*	227	.060	.053–.067	.944	.938	.052

FP = Free parameters; χ2 = Chi square; df = Degrees of freedom; RMSEA = Root Mean Square Error of Approximation; CI = 90% Confidence interval; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; SRMR = Standardized Root Mean Square Residual; eSEM = exploratory Structural Equation Modeling; * = p < .001.

In light of the current findings, and while it was not part of the pre-registration, we explored a new factorial solution through structural equation modeling using the same estimator and analyses to evaluate its goodness of fit.

3.2. Study 2 - EACIN-23: The Attitude Towards Research Scale (ATRS-23)

We first evaluated the face validity of all 34 items, ultimately keeping all of them. Subsequently, we conducted exploratory structural equation modeling (eSEM) analyses on different iterations of the scale until we found a suitable version of it, by eliminating items step-by-step following three established criteria: a) redundancy of items (Abad et al., 2011); b) high cross-loadings ≥ .3 (Muthén & Asparouhov, 2012; Xiao et al., 2019); and c) low factor loadings (λ < .3). Based on these criteria, the scale was iteratively reduced until a solution of 23 items was reached (see Supplementary Table S1). Furthermore, given the lack of good fit to the data of previous models, we conducted measurement invariance only on the EACIN-23 (henceforth the Attitude Towards Research Scale [ATRS-23]) so as to ascertain potential structural differences in this version of the scale for gender (van de Schoot et al., 2012; Vandenberg & Lance, 2000). Finally, we conducted parametric comparisons evaluating the ATRS-23 association with theoretically relevant correlates to examine its criterion validity. We used independent sample t-test or one-way ANOVA’s (using the Bonferroni correction) to determine if the scale was able to differentiate amongst different alternatives to the sociodemographic questions related to research attitude in our study.

3.2.1. Test of dimensionality

The new revised version was ultimately composed of 23 items, and yielded a three-factor structure, keeping the same interpretation as the EACIN-R (Aldana et al., 2020; Hidalgo et al., 2023). This version yielded adequate to good goodness-of-fit (WLSMV χ² (227) = 510.395, p < .001; CFI = .944, TLI = .938, RMSEA = .06; SRMR = .052 [90% CI = .053–.067]). Standardized factor loadings for this version ranged between adequate to optimal (λ = .409–.912), whereas the inter-factor correlation were positive, ranging from weak to moderate (r = −.17–.58). Reliability for the total scale and dimensions ranged from acceptable to good (see Table 5).

Table 5. ATRS-23 factor’s loadings and reliability

ATRS-23	Descriptive statistics			Factor loadings
Factor 1†	M (SD)	Skewness	Kurtosis	Factor 1	Factor 2	Factor 3
Item 2	2.97 (1.11)	−.06	−.58	.451
Item 3	3.42 (1.02)	−.41	−.19	.497
Item 6	3.37 (1.10)	−.46	−.39	.641
Item 11	3.60 (1.03)	−.48	−.26	.557
Item 16	3.18 (1.08)	−.15	−.52	.606
Item 17	3.21 (1.12)	−.22	−.61	.912
Item 21	3.37(1.03)	−.37	−.38	.704
Item 24	2.67 (1.09)	.26	−.46	.566
Item 25	3.19 (1.04)	−.33	−.13	.523
Factor 2	Valuation
Item 7	4.06 (1.08)	−1.16	.83		.409
Item 12	4.44 (.84)	−2.00	4.79		.764
Item 15	4.27 (.89)	−1.53	2.63		.727
Item 20	4.17 (.85)	−1.13	1.50		.713
Item 22	4.27 (.83)	−1.49	3.04		.793
Item 26	4.33 (.88)	−1.84	4.20		.769
Item 29	4.49 (.85)	−2.20	5.48		.732
Item 31	4.30 (.86)	−1.54	2.89		.704
Item 33	4.08 (.94)	−0.98	.81		.672
Factor 3	Disinterest
Item 4	4.06 (1.00)	−1.00	.62			.572
Item 14	3.53 (1.04)	−.20	−.74			.437
Item 23	3.67 (1.15)	−.61	.44			.473
Item 30	3.39 (1.03)	−.12	.40			.693
Item 34	3.67 (1.03)	−.44	−.27			.537
		Correlations			ω	α
Vocation		–			.832	.825
Valuation		.58*	–		.886	.891
Disinterest		.25*	−.17*	–	.648	.654

† = Vocation; M = Mean; SD = Standard deviation; * = p < .001.

Table 6 shows the Heterotrait-Monotrait (HTMT) criterion assessing the discriminant validity of the ATRS-23 factors amongst themselves. Results show that all factors exhibit a good level of discriminant validity given that all scores fell below the established criterion (i.e., < .85).

Table 6. ATRS-23 factor’s discriminant validity through the HTMT criterion

	Vocation	Valuation	Desinterest
Vocation	-
Valuation	0.660	-
Disinterest	0.454	0.132	-

3.3. Measurement invariance

Fit indices for the three-factor model are shown in Table 7 for both groups examined. All factor loadings were statistically significant and similar between the groups (ps < .001). The configural model showed acceptable fit indices (CFI = .934; TLI = .927; RMSEA = .067 [90% CI = .059–.074]), whereas the lack of substantive differences to the fit detriment after imposing constraints (ΔCFIs = −.002; ΔTLIs = .006; ΔRMSEA = −.003; see Table 4) indicate the ATRS-23 reached scalar level of invariance, suggesting that all parameters have an equivalent same latent structure, factor loadings, and intercepts, regardless if participants were men or women.

Table 7. Measurement invariance by gender for the ATRS-23

Models	Fit indices							Model comparison
	FP	χ2	df	RMSEA	90% CI	CFI	TLI	Pair	Δdf	ΔRMSEA	ΔCFI	ΔTLI
CFA – Men	123	368.984*	249	0.068	0.053–0.083	0.944	0.949
CFA – Women	123	522.612*	249	0.068	0.059–0.076	0.913	0.904
Model 1 – Configural	236	800.888*	454	0.067	0.059–0.074	0.934	0.927
Model 2 – Metric	216	837.576*	474	0.067	0.059–0.074	0.931	0.927	M2 – M1	20	0	−0.003	0
Model 3 – Scalar	150	916.428*	506	0.064	0.057–0.071	0.929	0.933	M3 – M2	32	−0.003	−0.002	0.006

FP = Free Parameters; χ2 = Chi-square; df = Degrees of freedom; RMSEA = Root Mean Square Error of Approximation; CI = 90% Confidence interval; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; SRMR = Standardized Root Mean Square Residual; * = p < .001.

3.4. Validity test

The analyses demonstrated that students who have or currently perform as a teaching assistant scored significantly higher in “Vocation” and the Total score (ps < .004). Furthermore, students who intend to pursue postgraduate studies scored significantly higher in “Vocation” and the Total score, and significantly lower than those who do not plan to do so in “Disinterest” (ps < .001). Finally, students who plan to dedicate to do research for living scored significantly higher in all dimensions and the Total score than those who do not (ps < .001), whereas those who were “not sure” also scored significantly higher in “Vocation” and the Total score than those who intend not to do so (ps < .001; see Table 8).

Table 8. Criterion validity analysis of the ATRS-23

	Yes	Not sure	No	F(t)	df1(df2)	p	$\mathit{\eta }\frac{2}{p}$(d)
Have you performed as a Teaching Assistant?
Valuation	38.33(5.98)	-	38.41(5.60)	0.096	342	0.924	(.162)
Vocation	32.52(6.40)	-	28.35(6.21)	4.637	342	<.001	(.669)
Disinterest	10.93(3.51)	-	11.84(3.31)	−1.805	342	0.061	(.277)
Total scale	89.91(12.55)	-	84.92(11.82)	2.903	342	0.004	(.418)
Do you intend to pursue postgraduate studies?
Valuation	38.71(5.68)	37.49(5.56)	36.67(5.16)	1.683	2(341)	0.187	0.100
Vocation	30.32(6.20)*	25.59(5.25)	19.83(6.18)	25.953	2(341)	<.001	0.132
Disinterest	11.24(3.29)‡	13.00(3.11)	13.50(4.76)	9.668	2(341)	<.001	0.054
Total scale	87.79(11.85)*	80.09(10.23)	73.00(14.24)	17.201	2(341)	<.001	0.092
Do you plan to dedicate yourself to do research for living?
Valuation	39.98(5.12)†	38.57(5.27)	37.21(6.30)	4.956	2(341)	0.008	0.028
Vocation	34.64(5.38)*	29.19(5.39)•	25.67(6.28)	48.688	2(341)	<.001	0.222
Disinterest	10.51(3.47)*	11.78(3.12)	12.19(3.55)	5.133	2(341)	<.001	0.029
Total scale	94.11(10.06)*	85.98(10.95)•	80.69(12.24)	27.923	2(341)	<.001	0.141

*: “Yes” significantly differs from “Not sure” and “No”; †: “Yes” significantly differs from “No”; ‡: “Yes” significantly differs from “Not sure”; •: “Not sure” significantly differs from “No”; df1: n-1 degrees of freedom; df2: k-1 degrees of freedom.

4. Discussion

Attitudes towards research is a pivotal component of higher education, common to all scientific programs, able to modulate, and to some extend determine, students’ experiences, performance, and career decisions. Consequently, having a reliable and valid psychometric instrument capable of accurately measuring and predicting such attitudes becomes immensely valuable. Thus, our findings suggest that the EACIN-23/ATRS-23 scale is a reliable, valid, and gender-invariant version of the original scale, capable of distinguishing between students with and without intentions to seek-out postgraduate studies and to dedicate themselves to research for living.

After thorough examination of the psychometric properties of the ATRS-23 in a sample of university students, our study demonstrated that: 1) previous versions of the EACIN scale did not provide a good fit for the data, rendering them problematic; 2) the new version of the scale, ATRS-23, supported the three-factor structure of the EACIN-R (Aldana et al., 2020); 3) the new version possessed acceptable-to-good fit indices, acceptable-to-good reliability indices for its dimensions and whole scale, and good discriminant validity amongst its latent structure sub-dimensions; 4) the ATRS-23 is a gender-invariant tool capable of comparing results between men and women, and finally 5) the ATRS-23 provides a valid distinction between students with different attitudes towards research career choices.

Previous studies assessing the EACIN psychometric properties have shown mixed results regarding its factorial structure and goodness of fit, with three different factorial solutions offered. The first factorial solution (Aldana et al., 2016) proposed three dimensions resembling a basic theoretical attitude structure (McGuire, 1969). However, the authors did not perform any statistical analysis beyond calculating Cronbach’s alpha. The following study did perform a psychometric validation using an exploratory factor analysis (EFA), partially corroborating this three-factorial solution, with a good fit for the data (Quezada-Berumen et al., 2019). The authors proposed the following interpretation of their dimensions: affective-behavioral, behavioral-affective, and cognitive. These combine items from the three original factorial structure proposed. Finally, they deleted two items based on low communality, factor loading, inter-correlation, and reliability, while also added one. Meanwhile, the newer three-factor solution, EACIN-R (Aldana et al., 2020) conducted an EFA, using Horn’s parallel analysis (Garrido et al., 2013; Horn, 1965). Horn’s parallel analysis is a reliable technique to identify the number of underlying factors, yet it has been noted that the analysis performs best when factors are either minimally or uncorrelated (Crawford et al., 2010; Debelak et al., 2016), which has not been the case in any previous psychometric validation of the EACIN. Still, their factorial solution has been the only one replicated (Hidalgo et al., 2023; Maguiña Custodio, 2021), including the present study. They removed six items based solely on low reliability (i.e., < .30). However, neither of these studies provided an interpretation of their factorial latent structure, or compared the factorial solutions to evaluate which one provided a better fit to the data, or why. Thus, since the sample characteristics and statistical techniques used are the two main differences across validations, while using with robust psychometric techniques within the same sample, the present study demonstrated that all previous versions of the scale did not provide a good and consistent fit to the data, whereas our new solution, ATRS-23, did.

Our study provides further evidence to the three-factor solution proposed by Hidalgo et al (Hidalgo et al., 2023). as a better representation of the latent construct measured by the EACIN-R. After an initial assessment, we took all items representing these dimensions and asked Chat GPT to return the “common characteristic underlying all items”, one dimensions at the time. The software explained that the “Vocation” dimension items represent “an inclination or passion for research and scientific exploration”. Meanwhile, the “Valuation” dimension items represent “the recognition and belief in the importance and value of research”. Finally, the “Disinterest” dimension items represent “the lack of engagement or disconnection with current or ongoing activities and information”. We believe that its analysis is indeed a valid rendering of the latent factorial structure, which also replicates with the previous one offered by the scale authors (Hernández et al., 2017). Furthermore, the HTMT criterion provides further proof of the suitability of the factorial structure proposed in the ATRS-23 by virtue of the discriminant validity scores amongst its latent factors.

The ATRS-23 is a new, parsimonious proposal to the previous EACIN 34-, 32-, and 28-items versions, while replicating the scale author’s most recent latent factorial structure. Whereas the overall reliability of the ATRS-23 was not as good as in its previous versions, it is worth noticing that this version has 11 items less than the original, particularly leaving only five items in the “disinterest for research” dimension. Whereas it may be tempting to offer a low number of items as an explanation for the lower reliability, and to some extend there is true to that given due to more opportunities for inter-item correlations (Cortina, 1993), it has been shown that Conbach’s α can inflate reliability estimation (Kopalle & Lehmann, 1997) or even become a misleading estimator of it (Raykov, 2007, 2008), especially when its assumption are not taken into account (Cho & Kim, 2015). However, following McDonald’s ω, we do bring attention to this dimension reliability—potentially in need of a more in-depth theoretically-driven analysis to improve its reliability, not only necessarily because the reliability falling below .7, but mostly since this new version is yet to be replicated. This is because despite that .7 or .8 are often cited as the acceptable levels of reliability (Nunnally & Bernstein, 1994), these are not empirically driven, but the author’s intuition, and that McDonald’s ω is not the only alternative to Conbach’s α (Cho & Kim, 2015). Still, the ATRS-23 yielded the best fit for the data, good overall reliability, and excellent criterion validity able to distinguish across different levels of interest regarding student’s intentions of specialization and pursuing a career in research. Furthermore, the ATRS-23 is the only peer-review article demonstrating this scale is invariant when it comes to comparing men and women. There is, though, only one record—a non-published undergraduate thesis—that showed the EACIN-R has a similar goodness of fit to the ATRS-23, and invariant regarding gender (Maguiña Custodio, 2021).

The results of the present study also demonstrated that the latent factorial structure of the ATRS-23 is able to distinguish between students intentions related to research. On the one hand, the dimensions “Research Vocation” and “Research Disinterest” are able to distinguish significantly amongst students who have performed as teaching assistants or not, as well as amongst those who are sure they intend to pursue postgraduate studies, and plan to dedicate themselves to research for living than those who are not sure or will not. Nevertheless, “Research Vocation” yielded a much stronger effect size than any other ATRS-23 dimension, including the total score. On the other hand, the dimension “Research Valuation” was only able to distinguish between those who plan to dedicate themselves to research for living from those who will not, yet still with a small effect size than other comparisons. This does not mean this dimension has no discriminant value. In fact, Bolin et al. (2012) demonstrated the perception of research utility/value was significantly related to a decreased research anxiety, much like Gredig and Bartelsen-Raemy (2018) demonstrated that interest in research courses was predicted by research anxiety, the perceived value, the attributed usefulness, and the perceived unbiased nature of research. This could explain, in part, why this dimension was not able to distinguish between those who answered “yes” or “not sure” to the sociodemographic questions, assuming the latter may experience some degree of stressed or anxiety related to answering the question.

Amongst the other psychometric alternatives to measure attitudes towards research, Papanastasiou’ scale stands out, offering a different theoretical model than the EACIN’s (Papanastasiou, 2005). The original self-report scale consisted of 32 items, answered through a seven-Likert scale, from strongly disagree to strongly agree. Its initial exploratory analysis of its structure proposed a five-factor solution: “Research usefulness” (α = .919), “Research anxiety” (α = .918), “Positive research predisposition” (α = .929), “Relevance to life” (α = .767), and “Research difficulty” (α = .711). Subsequently, the authors suggested to drop two items (i.e., #6, #11) (Papanastasiou & Schumacker, 2014). However, a more recent confirmatory factor analysis proposed a three-factor solution (Research usefulness, Research anxiety, Positive research predisposition) only using 13 items, demonstrating good indicators of fit than the original model (Papanastasiou, 2014). Other studies using the ATR scale have replicated the latter factorial solution with good reliability levels (Howard & Michael, 2019; van der Westhuizen, 2015). More than suggesting which scale is better, these authors believe they serve similar purposes, and researchers and educators are provided with two instruments that measure the same construct with two different approaches, allowing them to choose which approach is best for their purposes. For instance, Sizemore & Lewandowski compared sophomore-level student’s knowledge and attitudes towards research and statistics at the beginning and end of the semester. They found that the course did improved their knowledge, yet their attitudes towards the subject declined significantly (Sizemore & Lewandowski, 2009). Thus, by using the ATR scale researchers would be able to assess how anxiety towards research and their predisposition may modulate these findings, whereas by using the ATRS-23 researchers would be able to ascertain if these results are based on a lack of interest or vocation for scientific research, meanwhile using both scales could represent an overall criterion validity to each other.

Since the ATRS-23 is only available in Spanish, we provide researchers an English version translated following Beaton’s guidelines for cross-cultural adaptations of self-report measures (Beaton et al., 2000). This offered English version was translated from Spanish to English by two experienced translators. Any discrepancy between the two versions was addressed, to come up with one revised English version, which was later on translated by a third translator into Spanish to be compared with the original (see Supplementary Table S2). The authors hope the evidence presented and this translated version of the ATRS-23 encourages others to conduct validations in different contexts, with samples of different programs, in different educational roles, in other cultures/countries and languages.

4.1. Limitations

The current study is offers a new, simpler, version of the EACIN scale, yet not without limitations. Our main caveats has to do with the diversity of the sample. We included only undergraduate university students of one program. Future validation studies should include not only students of different programs, but also from different levels (undergraduate and postgraduate), and people in other university roles, as the EACIN is capable to do so. An ideal validation would include a sample big in every role to also provide invariance analysis amongst these groups. Moreover, the provided criterion validity was performed only through sociodemographic questions. Future studies are encouraged to utilize other validated psychometric scales of constructs of interest theoretically relevant to research attitudes, such as anxiety (Kracker, 2002; Yau & Leung, 2018), self-efficacy (Wajid & Jami, 2020; Yau & Leung, 2018), student performance (Mao et al., 2021), etc.

Acknowledgments

The authors would like to thank José Piña for his contribution in the data collection process.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The study data was made available in the following link https://osf.io/r645c/

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/2331186X.2023.2292825

Additional information

Funding

This work was supported by the Investigación para la Innovación en Educación Superior UTA grant #3771-22.

Notes

1. See the author’s website for a calculator of the HTMT criterion.