The Influence of High Schools on Developing Public Service Motivation

Abstract

ABSTRACT: Despite the theoretical importance of public service motivation (PSM) for the public sector, little is known about the malleability and causal determinants of PSM. Formal schooling is one possible determinant of PSM. Using longitudinal data, this study analyzes the effects of civics courses and school-based community service in high school on PSM-related values. A propensity score matching strategy that compares observationally similar individuals finds that participation in school-based service activities for credit increases students’ reported prosocial values, such as helping others in the community, one underlying component of PSM-related values. However, taking a civics course does not affect PSM or associated values. The results suggest that PSM-related values are malleable and responsive to early service experiences. The values underlying PSM may evolve over time. That is, the development of prosocial values in adolescence may lead to the future development of other values important to PSM.

Notes

Notes

1 Table A1 in the Appendix lists the items in the social values section of the ELS:2002 and maps the items to the closest item in the Perry (1996) instrument. Notably, the items cover four dimensions of the original six dimensions of PSM proposed by Perry and four of the five dimensions in the more recent instrument proposed by Kim et al. (2013). Moreover, as Wright, Christensen, and Pandey (2013) demonstrate, more global measures of PSM that use fewer items are strongly correlated with PSM measured using longer instruments.

2 Among the full set of items in the values section of the survey, five latent factors were identified. The PSM factor has a Crohnbach’s alpha of 0.72, which suggests that responses to the public service motivation related items are highly correlated. After identifying the latent factors using principal components factor analysis, I conducted confirmatory factor analysis on the PSM items to evaluate the reliability of the items in measuring the underlying PSM factor. Following Kim et al. (2013), I evaluate the reliability of each factor per the comparative fit index (CFI), root mean square error of approximation (RMSEA), and the standardized root mean residual (SRMR), which are robust to the use of large samples. The PSM factor in the ELS has a CFI of 0.97, RMSEA of 0.08, and SRMR of 0.03, all of which suggest that the model of PSM using the ELS items fit the data well.

3 Using a binary indicator for the highest level of a given dimension allows for the identification of movement in a consistent direction and eases the intuition of inference made when interpreting the estimated coefficients. Multinomial logistic regressions with controls for observables provide estimates consistent with the primary findings of the analysis presented here. However, multinomial logistic regressions, appropriate for estimating models with a discrete categorical set of outcomes, do not account for potential selection bias.

4 Carnegie units of credits are provided by the ELS. A Carnegie unit measures the equivalent of taking one secondary-level course that meets every day for a full year. For instance, a 0.5 Carnegie unit of civics course work would be equivalent to being in a civics course that met every day for half of the year.

5 See Appendix Table A2 for a list of civics courses and credited school-based service activities used in the analysis.

6 A small number of students take either treatment in their freshman or sophomore years. Including them in the analysis yields strongly similar results.

7 Note that the full sample with non-missing data on relevant variables includes approximately 10,880 students; however, I restrict the sample to schools in which there is variation in participation in civics coursework and school-based service activities. If all students or no students participate at a given school, the school is dropped from the sample. All instances in which schools are dropped is due to no participation in civics coursework or civics service within the school, rather than complete participation. In line with prior research (e.g., Kahne and Middaugh 2009), overall in the sample, only about 11% of students take a civics course their junior year, and these students tend to be from higher SES backgrounds. Similarly, about 7% of students participate in school-based service activities in their junior year and are also more likely to come from higher SES backgrounds. As a result, this study is generalizable to the extent that it reflects national gaps in access to civics coursework and credited school-based service activities; however, observed effects of civics coursework and school service are limited to those who attend a school with a civics course or school service for credit available. Consistent with NCES regulations on restricted-use data, the number of observations is rounded to the nearest ten.

8 Table A7 in the Appendix recreates the main results, excluding students who took both treatments, and demonstrates that the results are robust to their exclusion.

9 Logit estimates of (1) in Appendix Table A8 are similar to OLS estimates in Table A4 in direction and significance. The coefficients of logit regressions are not directly comparable to OLS estimates. The inclusion of school fixed-effects makes the calculation of average partial effects (APE) comparable to OLS impossible.

10 Appendix Table A4 contains LPM estimates of (1).

11 The same concern applies to logistic and probit regression estimators. While nonlinear estimators account for the binary nature of the outcome variable, they do not account for unobserved characteristics that drive selection into the treatments.

12 An alternative approach to account for endogenous selection into the treatment follows an instrumental variable (IV) approach proposed by Altonji (1995). Since student course selection might be, in part, attributable to endogenous selection into the school based on average course offerings, Altonji proposes using school-level average credits per subject in eight subjects as an instrument for participation in a given course. However, as Altonji notes, variation in curricula across schools is likely to be correlated with student and family characteristics and, as a result, the IV estimate might also be biased. While the IV approach accounts for endogenous selection into the school, it fails to account for endogenous student-level factors associated with sorting into the treatment. Appendix Table A4 implements Altonji’s IV approach using two-stage least squares (2SLS).

13 A caliper is a cut-point in the allowable distance in propensity score between a treatment and control observation for a match to be made. Observations without a match within the defined caliper are dropped from the analysis. As Dehejia and Wahba (2002) show, estimated effects can vary substantially, conditional on the caliper chosen. Since caliper selection can lead to significant reductions in the sample size, the choice of a caliper involves tradeoffs between bias and efficiency (Caliendo and Kopeinig 2008). I present estimates using a caliper of 0.1, which resulted in the lowest estimated bias. Appendix Table A6 presents the primary results using calipers of .01 and .001, respectively. As shown in the Appendix, the direction of the estimates is consistent across matching estimators; however, using more restrictive calipers, the estimates are less precise due to the substantial reduction in sample size.

14 One could also consider the possibility that schools that do not offer the treatment could serve as a good control group. That is, examining the effect of either treatment relative to a student who would be equally likely to take the treatment, but did not have the opportunity, could also be insightful. Appendix Table A9 replicates the baseline results, but matches treatment students with controls from schools that did not have civics courses or service activities observed at their school during the junior year. The results are strikingly similar to the baseline results.

15 Matching with replacement ensures that control group observations most closely resemble the treatment observations with which they are matched. While there is common support across most of the range of propensity scores, some “bins” of scores have fewer observations than others. Thus, matching without replacement would likely lead to less similar matches. Moreover, matching within schools using outcome measures collected in the same survey instrument for both treatment and control groups satisfies the requirements for consistent matching estimators identified by Heckman et al. (1998) and Smith and Todd (2005). Appendix Table A5 replicates Table 6 in the main text using a matching strategy that allows matches to occur across schools, and the results are strikingly consistent.

16 Appendix Tables A10 and A11 present the difference in average characteristics between treatment and control groups before and after matching on propensity scores using a variety of matching algorithms. Intuitively, the differences in the effectiveness of the matching algorithms represent a trade-off between the precision offered by a larger sample of students and the increased bias of using control group observations which may not be as well-matched. The Rosenbaum and Rubin (1985) measure of bias, the standardized percent of the average difference between treatment and control on observables commonly used in evaluating matching on propensity scores, is reported in the final two rows.

17 Conducting multiple hypothesis tests increases the likelihood of Type I or family-wise error. That is, the likelihood of finding false positive results increases as the number of hypothesis tests increases. These concerns are unlikely to apply to the primary results of the current study. First, the primary findings, regarding both the effect of service credits on the public interest dimension of PSM and the relationship between PSM and civic engagement, are robust to both Bonferroni and Benjamini-Hochberg adjustments of the t-test (see Schochet 2009 for a discussion of these adjustments). Second, there is strong theoretical grounding and support for both the analysis and results. Third, the results are robust to a variety of estimators and matching approaches.

18 The results presented here are consistent in both the school-service and civics coursework analytic samples.

19 The results are robust to the inclusion of all categories of responses (e.g., adding a binary for “somewhat important” responses to each item in the PSM index). However, for clarity and consistency with previous analyses, the estimates using only binaries for the highest category versus all others are taken as the preferred estimates.

Additional information

Notes on contributors

Stephen B. Holt

Stephen B. Holt ([email protected]) is an assistant professor of public management at the Rockefeller College of Public Affairs and Policy, University at Albany, SUNY. He received his Ph.D. in public administration from the School of Public Affairs, American University. His current research focuses on the dynamics of motivation and representation in the public sector workforce.