A preliminary validation of the Regional Proficiency Assessment Tool (RPAT)

ABSTRACT

Regional proficiency is a multidimensional construct created by the Department of Defense (DoD) to characterize a person’s knowledge of a region’s social, economic, political, and linguistic features. The Regional Proficiency Assessment Tool (RPAT) was designed to capture the regional proficiency of the military workforce. The RPAT collects biographical self-report data (e.g., professional travel, language test scores) to provide military personnel with scores for each dimension of regional proficiency (e.g., study of a region, utility of language skills) for each of the different global regions. Cluster scores combine to render the overall ratings. In two studies, we provide evidence for the RPAT’s construct validity. In Study 1, 44 military participants completed the RPAT and individual difference measures. Aggregated, person-level RPAT ratings and appropriate cluster scores were moderately positively associated with cultural intelligence (.35–.47), but unrelated with self-reported cognitive styles and the Big Five personality dimensions, except for negative correlations with neuroticism. In Study 2, 11 subject matter experts rated the estimated RP of individuals by several world regions via reading narrative biographical sketches. The aggregate of the expert ratings was strongly, positively associated with the RPAT ratings (M across regions = .72), and was moderately correlated with cultural intelligence (.38–.48). This finding suggests that the RPAT provides assessments in line with the best of current practice (individual expert heuristic judgments), and that cultural intelligence assessments cannot replace the RPAT. Future research can examine the validity of the RPAT across larger samples and using different types of validation.

KEYWORDS:

• Regional proficiency
• assessment
• validity

Acknowledgments

The authors are at the Center for Advanced Study of Language at the University of Maryland. This material is based upon work supported, in whole or in part, with funding from the United States Government. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the University of Maryland, College Park and/or any agency or entity of the United States Government. We wish to thank Shauna Sweet for statistical advice, as well as Dr. Jeffrey Watson and Ms. Beverly Rouse for assistance with recruitment.

Notes

1. This Instruction was updated December 2016, after the RPAT was initially deployed: The new version of the Instruction includes the six rating levels, which have been changed from the earlier version (U.S. DoD, 2016).

2. Although not recognized outside the DoD context, RP meets the definition of construct as “theoretical psychological characteristics, attributes, processes, or states that cannot be directly observed” (Furr & Bacharach, 2014, p. 5).

3. Copies of the questionnaire copyrighted to the University of Maryland are available upon request for academic purposes only. Other uses need to go through the University of Maryland Office of Technology Commercialization at [email protected] or [phone]. Commercial customers can receive a free evaluation copy.

4. The 15 global regions came out of a joint effort between the Defense Language Office, National Geospatial Agency, and Defense Language Action Panel, and they were approved by the Defense Language Steering Committee in late 2006. The regions are Caribbean, Central America, Central Asia, East Asia, Eastern Europe, Eastern Oceania, Middle East/North Africa, North America, South America, South Asia, Southeast Asia, Sub-Saharan Africa, West Africa, Western Europe, and Western Oceania.

5. For the full correlation matrices between all four CQ subscales and the RPAT, see the unpublished report by Paletz et al. (2016). The general pattern of results was the same for each of the subscales as for the combined CQ score.

6. Although the participants completed a penultimate version of the RPAT questionnaire (version 0.3), the follow-on supplemental survey contained questions that were then used in the final version. The RPAT scores and ratings reported here thus reflect the most recent version of the RPAT questionnaire (version 0.4). The production version of the RPAT is hosted by the Defense Manpower Data Center (DMDC), a system accessible only to military personnel.

7. Cohen (1988) provided the following guidelines for interpreting correlation effect sizes in psychological research: correlations in the order of .10 are small, on the order of .30 are medium, and of .50 are large.

8. On the chance that the relationship was due to neuroticism being negatively related to number of service years, and CQ and RPAT levels being positively related to service years, the correlation between neuroticism and service years was also tested. It was not statistically significant, r_s = -.15, p > .34.

9. Although it uses Pearson’s r instead of Spearman’s rho, we used the SPSS statistical software to test partial correlations.

10. The metric of reliability used was the intraclass correlation coefficient (ICC) two-way mixed model, using the consistency definition. When the underlying data are ordinal and skewed, as with RP ratings, this metric tends to underestimate the potential true reliability (Gadermann, Guhn, & Zumbo, 2012).

Additional information

Funding

This work was supported by the United States Department of Defense; H98230-07-D-0175.