The illusion of validity: how effort inflates the perceived validity of interview questions

ABSTRACT

Interviewers are often confident in the validity of their interview questions. What drives this confidence and is it justified? In three studies, we found that question creators judged their own interview questions as more valid than when the same questions are judged by an evaluator. We also found that effort expenditure inflated the perceived validity of interview questions but not question quality. Question creators’ perceptions of validity were primarily driven by their self-confidence, and not the question quality. As an intervention, we nudged participants into holding more favourable attitudes towards better questions (i.e., structured questions) by allowing them to choose a subset of them from a pre-written list. Together, we found that while effort expenditure was responsible for the illusion of validity when evaluating unstructured (i.e., low-quality) questions, the same mechanism could also be used to improve interviewers’ acceptance of structured questions. Implications for structured interviews and the scientist-practitioner gap are discussed.

KEYWORDS:

• Overconfidence
• structured interviews
• judgement and decision making
• employee selection
• cognitive dissonance

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. Interview performance may be operationalized as the quality of questions, quality of the evaluation, or quality of the interview itself. Our studies will focus on the quality of the questions as the primary operationalization of interview performance.

2. We examined the available individual difference characteristics and found no difference of interview experience (p = .12), interview self-confidence (p = .20), and preference for intuition (p = .20) between the two conditions (creator vs. evaluator).

3. Given the non-normality of the dependent variable, a non-parametric test (Friedman test) was also conducted for robustness, χ² = 5.67, p = .017.

4. Society for Human Resource Management.

5. The three items are highly correlated (rs = .59, .54, .65) and a one factor solution was suggested with exploratory factor analysis. The eigenvalues for a 1-factor and 2-factor solutions were 1.178 and 0.003 respectively.

6. The main effect reported in the manuscript across both samples were significant for the lay sample: F(2, 134) = 4.22, p = .017 as well as the experienced sample F(2,195) = 5.25, p = .006.