Evaluating Dosage Effects in a Social-Emotional Skills Training Program for Children: An Application of Generalized Propensity Scores

ABSTRACT

In a study of the Making Choices social-emotional skills training program for children, generalized propensity scores (GPS) were used to estimate treatment effects by dosage. Dosage analyses provide information regarding the optimal amount of exposure to interventions. In addition to applying dosage analysis to an evaluation of the Making Choices program, this article reviews issues encountered during dosage analyses. It introduces GPS methods, a relatively recent development within the family of propensity score methods. Based on data from 267 3rd graders who participated in a trial of Making Choices, intervention effects varied significantly by dosage, with greater social competence demonstrated by children who had higher intervention exposure.

KEYWORDS:

• Social intervention research
• dosage analysis
• generalized propensity score
• social-emotional skills training

ACKNOWLEDGMENTS

We thank Susan T. Ennett, Maeda J. Galinsky, Joelle D. Powers, Kathleen A. Rounds, and Christopher A. Wiesen for their assistance with data analysis and their comments on drafts of this report. Special thanks also to Diane Wyant for her generous editorial help.

FUNDING

This project was supported, in part, by a grant from the U.S. Department of Education (R305L030162). In addition, the first author received funding from the Royster Society of Fellows program at the University of North Carolina at Chapel Hill.

APPENDIX

Stata Codes and List of Variables Included in Estimating Generalized Propensity Score

1. Stata Codes:

   • The dosage analysis includes five steps. The first step estimates the conditional distribution of treatment (i.e., number of minutes). The second step estimates the overall generalized propensity score (GPS). The two steps were accomplished using a user-developed Stata package gspscore.ado. The syntax is provided below:

     gpscore varlist , t(MCminute3) predict(hat_treat) sigma(sd) gpscore(pscore) index(p50) nq gps(2) t_transf(ln) detail

     The command gpscore estimates the predicted value of treatment (i.e., hat_treat),standard deviation (i.e., sd), and the overall GPS (i.e., pscore).

   • The third step estimated three sets of GPS used for identifying the common support region. The three sets of GPS were estimated at the median (i.e., 906, 1,088, and 1,234) of each of the three treatment intervals defined by the two cut points of 1,030 min and 1,105 min. The estimation of the three sets of GPS used the standard deviation (i.e., sd) and predicted value of treatment (i.e., hat_treat) estimated in previous steps.

     generate sqsd = sd × sd generate pi = 3.14159265 generate tt1 = log(906) generate gps1 = (1/sqrt(2 × pi × sqsd)) × exp((–1/(2 × sqsd)) × (tt1 – hat_treat) × (tt1 – hat_treat)) generate tt2 = log(1,088)generate gps2 = (1/sqrt(2 × pi × sqsd)) × exp((–1/(2 × sqsd)) × (tt2 – hat_treat) × (tt2 – hat_treat)) generate tt3 = log(1,234) generate gps3 = (1/sqrt(2 × pi × sqsd)) × exp((–1/(2 × sqsd)) × (tt3 – hat_treat) × (tt3 – hat_treat))

   • Step 4 estimates the conditional expectation of the outcome (i.e., social competence) using a simple regression model. min_gps is an interaction term between minutes and GPS; cccccon_2 is cognitive concentration; hattp3 is hostile attribution; s0_b4_wk is weeks devoted to tolerance and diversity activities; S0_sle_profinterst is professional interest. The common support region is applied in this analysis.

     regress DV minutes pscore min_gps /// cccccon_2 hattp3 s0_b4_wk S0_sle_profinterst ///if gps1 > = 0.0002477 & gps1 < = 3.584874 & /// gps2 > = 0.0166222 & gps2 < = 3.678357 & ///gps3 > = 0.0004511 & gps3 < = 3.676587

   • Step 5 estimates the average potential outcome for each minute level of interest. The syntax for the estimation at the lowest level is provided here.

     generate cccscom906 = 4.178038 + –0.0050401 × MCminute3 + –1.978737 × pscore /// + 0.0019741 × min_gps + –0.1348259 × cccccon_2 /// + –0.2579656 × hattp3 + –0.0370335 × s0_b4_wk /// + .3625391 × S0_sle_profinterst /// if MCminute3 = 906

2. Covariates included in the model that estimated GPS:

   • Student covariates

   • gender, race, and ethnicity (African American, White, Hispanic, Asian, Multirace), father in household, no parent in household, primary caregiver employment, cognitive concentration, social aggression, emotional regulation, academic achievement, overt aggression, school affiliation, student popularity, encoding, hostile attribution, goal formulation, and respond decision

   • Teacher covariates

   • weeks devoted to tolerance and diversity activities, perception of student support, and professional interest

   • School covariates

   • percentage of students receiving free or reduced lunch, adequate yearly progress, and income-to-poverty ratio