Response Surface Analysis with Missing Data

Abstract

Response Surface Analysis (RSA) is gaining popularity in psychological research as a tool for investigating congruence hypotheses (e.g., consequences of self-other agreement, person-job fit, dyadic similarity). RSA involves the estimation of a nonlinear polynomial regression model and the interpretation of the resulting response surface. However, little is known about how best to conduct RSA when the underlying data are incomplete. In this article, we compare different methods for handling missing data in RSA. This includes different strategies for multiple imputation (MI) and maximum-likelihood (ML) estimation. Specifically, we consider the “just another variable” (JAV) approach to MI and ML, an approach that is in regular use in applications of RSA, and the more novel “substantive-model-compatible” (SMC) approach. In a simulation study, we evaluate the impact of these methods on focal outcomes of RSA, including the accuracy of parameter estimates, the shape of the response surface, and the testing of congruence hypotheses. Our findings suggest that the JAV approach can sometimes distort parameter estimates and conclusions about the shape of the response surface, whereas the SMC approach performs well overall. We illustrate applications of the methods in a worked example with real data and provide recommendations for their application in practice.

Keywords:

• Response surface analysis
• polynomial regression
• missing data
• multiple imputation
• maximum likelihood

Article information

Conflict of interest disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Acknowledgments: We thank Steffen Filz for supporting the preparation of the example analysis.

Notes

1 Further examples of applications of RSA in psychological research are summarized in Barranti et al. (2017), Humberg et al. (2019), and Humberg et al. (2020). The research questions that have been previously addressed with RSA include, for example, the consequences of person-environment or person-job fit (e.g., Kristof, 1996), the adaptiveness of social reciprocity (e.g., Nahum-Shani et al., 2011), the consequences of dyadic or person-group similarity (e.g., Kim et al., 2020), and effect-size estimation in meta-analyses (Rubin, 1992; Shadish et al., 2000).

2 Specifically, we considered the following four journals: Journal of Personality and Social Psychology, Social Psychological and Personality Science, European Journal of Personality, Journal of Management. The search included all articles published in these journals through March 2020 containing one of the terms “response surface” or “polynomial regression” anywhere in the text. Of these articles, 31 included an application of RSA. Of these 31 articles, 29 reported missing data, and 23 provided sufficient detail to identify the methods used to handle the missing data.

3 The polynomial regression model underlying RSA assumes that the errors are independently and identically normally distributed with a constant variance and a mean of zero.

4 For reasons of clarity, we focus on congruence hypotheses that posit that some outcome variable y is maximized when x₁ and x₂ are congruent. Alternatively, one could be interested in a reverse congruence hypothesis, which posits a minimal y value for congruent predictors. The test of a reverse congruence hypothesis is equivalent to the test of a ”regular” congruence hypothesis with a reverse-scored y.

5 If the data are MNAR, handling missing data is more challenging and requires the missing data mechanism to be specified directly. In such a case, researchers are often advised to conduct sensitivity analyses to evaluate how the results change under various MNAR mechanisms (Carpenter & Kenward, 2013).

6 Note that, because all variables are standardized, choosing R² is sufficient for determining both the curvature of the response surface and the residual variance. Specifically, given R², the curvature $c=\pm \sqrt{\frac{R^2}{V}}$ and the residual variance ${\sigma }^2=c^{2}V\left(\frac{1-R^2}{R^2}\right),$ where $V=4C^2{S}^2+4C^2-8C^{2}S\rho +2S^4+2+4S^2-8S^3\rho -8S\rho +8S^2{\rho }^2.$

7 An alternative to the delta method is computing confidence intervals through a percentile bootstrap, which can provide more accurate intervals in small samples (Fox, 2016). Our choice to focus on the delta method was motivated by a preliminary simulation, which suggested that the delta method provides confidence intervals with acceptable properties in the conditions considered.

8 This study used data collected by the Swiss Household Panel (SHP), which is based at the Swiss Center of Expertise in the Social Sciences FORS. The project is financed by the Swiss National Science Foundation.