Obtaining Content Weights for Test Specifications From Job Analysis Task Surveys: An Application of the Many-Facets Rasch Model

Abstract

This article discusses the use of the Many-Facets Rasch Model, via the FACETS computer program (Linacre, 2006a), to scale job/practice analysis survey data as well as to combine multiple rating scales into single composite weights representing the tasks’ relative importance. Results from the Many-Facets Rasch Model are compared with those calculated from the Rasch Rating Scale Model (RRSM) (Spray & Huang, 2000) using two examples of actual job analysis data from diverse professions. In addition, this article proposes a solution for establishing the origin of the percentage scale when transferring the task importance weights from a logit unit into percentage weights. Although the resulting test specifications from the two compared methods are not radically different, a case is made that the use of the Many-Facets Rasch Model with a zero point based on the frequency rating scale provides a more justifiable basis for combining multiple rating scales and transforming task survey data. In addition, this study found that the Many-Facets Rasch Rating Model can better accommodate missing data than the RRSM method in situations in which respondents only rate subsets of the multiple scales and not all of the scales for the tasks being surveyed.

Keywords:

• job/practice analysis
• licensure/certification examination
• test specifications

Acknowledgments

The authors acknowledge and appreciate the permission given by Pearson VUE to use their job analysis data. Any opinions expressed in the article are those of the authors and do not necessarily represent the views of Pearson VUE.

Notes

1. There were a total of 67 job tasks in the final list of this job analysis. A detailed description of this job analysis is provided in a later section of this article.

2. As a default, in the FACETS analysis, a lower value of task parameter, D, corresponds to a higher rating of the task, meaning that the task is less “difficult,” which is similar to the interpretation of the traditional item difficulty index. In the job analysis context, a higher rating of a task indicates the task's greater importance. By changing the sign of the task parameter estimates, higher task estimates correspond to higher ratings obtained for the tasks and indicate the greater importance of the tasks. Alternatively, you can specify which facets of the analysis will be scored in a positive orientation and which will be scored in a negative orientation in the FACETS control file.

3. In this study, FACETS analyses were conducted first and the RRSM analyses were performed after the quality of the job analysis data were examined using the rater and task logit measures as well as fit statistics from the FACETS analyses. Therefore, the same data set from each job analysis across the two methods was used to produce the test specifications.