Weighing the difference: the validity of multiplicative and subtractive approaches to item weights in an instrument assessing college choice decisions

Abstract

The Admitted Student Questionnaire Plus (ASQ⁺) is a standardised measure that provides an analysis of the student's college selection process. Among other things, the instrument inquires about the importance of 16 college characteristics, followed by quality ratings of specific colleges that the student considered on these same characteristics. This study investigated the utility of importance weights in the assessment of college choice, examining how much the importance rating would improve one's ability to predict the student's actual college choice over and above what is possible with just the quality ratings. Another purpose of the study was to determine if importance ratings and quality ratings were independent of each other or associated in some way. Two types of weights were studied: (1) standardised weights created by averaging the importance ratings of the entire sample; and (2) subjective weights unique to each respondent. The weights were combined with quality ratings by either: (1) multiplying the quality rating by the importance rating; or by (2) subtracting the quality rating from the importance rating (gap score). Standardised weights did not improve prediction at all, and subjective weights only improved the predictability of college choice by a very miniscule amount (about 1%). Importance and quality ratings were found to be associated, especially in the ratings of the college that the student decided to attend. Some correlations were linear in nature, but many were non-linear, such that characteristics rated high or low were perceived as more important than characteristics assigned mid-range quality ratings. It was concluded that importance weights do not enhance prediction of college choice, but they may be useful for administrators in prioritising interventions.

Keywords:

• ratings
• importance
• weights
• quality
• college choice
• gap scores
• Admitted Student Questionnaire Plus (ASQ⁺)