Evaluating Ethics Education Programs: A Multilevel Approach

REFERENCES

• Aguinis, H., & Kraiger, K. (2009). Benefits of training and development for individuals and teams, organizations, and society. Annual Review of Psychology, 60, 451–474.
   PubMed Web of Science ®Google Scholar
• Anderson, M. S. (2010). Creating the ethical academy: A systems approach to understanding misconduct and empowering change. New York, NY: Routledge.
   Google Scholar
• Antes, A. L., Brown, R. P., Murphy, S. T., Waples, E. P., Mumford, M. D., Connelly, S., & Devenport, L. D. (2007). Personality and ethical decision-making in research: The role of perceptions of self and others. Journal of Empirical Research on Human Research Ethics: International Journal, 2, 15–34.
   PubMed Web of Science ®Google Scholar
• Antes, A. L., Murphy, S. T., Waples, E. P., Mumford, M. D., Brown, R. P., Connelly, S., & Devenport, L. D. (2009). A meta-analysis of ethics instruction effectiveness in the sciences. Ethics & Behavior, 19, 379–402.
   PubMed Web of Science ®Google Scholar
• Antes, A. L., Thiel, C. E., Martin, L. E., Stenmark, C. K., Connelly, S., Devenport, L. D., & Mumford, M. D. (2012). Applying cases to solve ethical problems: The significance of positive and process-oriented reflection. Ethics & Behavior, 22, 113–130.
   PubMed Web of Science ®Google Scholar
• Antes, A. L., Wang, X., Mumford, M. D., Brown, R. P., Connelly, S., & Devenport, L. D. (2010). Evaluating the effects that existing instruction on responsible conduct of research has on ethical decision making. Academic Medicine, 85, 519–526.
   PubMed Web of Science ®Google Scholar
• Arthur, W., J., Bennett, W., J., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88, 234–245.
   PubMed Web of Science ®Google Scholar
• Arvey, R. D., Cole, D. A., Hazucha, J. F., & Hartanto, F. M. (1985). Statistical power of training evaluation designs. Personnel Psychology, 38, 493–507.
   Web of Science ®Google Scholar
• Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41, 63–105.
   Web of Science ®Google Scholar
• Baldwin, T. T., Ford, J. K., & Blume, B. D. (2009). Transfer of training 1988–2008: An updated review and agenda for future research. International Review of Industrial and Organizational Psychology, 24, 41–70.
   Google Scholar
• Bebeau, M. J., & Thoma, S. J. (1994). The impact of a dental ethics curriculum on moral reasoning. Journal of Dental Education, 58, 684–692.
   PubMedGoogle Scholar
• Borman, W. C., & Motowidlo, S. J. (1993). Expanding the criterion domain to include elements of contextual performance. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 71-98). San Francisco, CA: Jossey-Bass.
   Google Scholar
• Braunschweiger, P., & Goodman, K. W. (2007). The CITI program: An international online resource for education in human subjects protection and the responsible conduct of research. Academic Medicine, 82, 861–864.
   PubMed Web of Science ®Google Scholar
• Brock, M. E., Vert, A., Kligyte, V., Waples, E. P., Sevier, S. T., & Mumford, M. D. (2008). Mental models: An alternative evaluation of a sensemaking approach to ethics instruction. Science and Engineering Ethics, 14, 449–472.
   PubMed Web of Science ®Google Scholar
• Campion, M. A., & Campion, J. E. (1987). Evaluation of an interviewee skills training program in a natural field experiment. Personnel Psychology, 40, 675–691.
   Web of Science ®Google Scholar
• Clarkeburn, H. (2002). A test for ethical sensitivity in science. Journal of Moral Education, 31, 439–453.
   Web of Science ®Google Scholar
• Clarkeburn, H., Downie, J. R., & Matthew, B. O. B. (2002). Impact of an ethics programme in a life sciences curriculum. Teaching in Higher Education, 7, 65–79.
   Google Scholar
• Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues for field settings. Boston, MA: Houghton Mifflin.
   Google Scholar
• Council of Graduate Schools. (2012). Research and scholarly integrity in graduate education. Washington, DC: Author.
   Google Scholar
• Crain, A. L., Martinson, B. C., & Thrush, C. R. (2013). Relationships between the survey of organizational research climate (SORC) and self-reported research practices. Science and Engineering Ethics, 19, 835–850.
   PubMed Web of Science ®Google Scholar
• Cullen, J. G., Sawzin, S. A., Sisson, G. R., & Swanson, R. A. (1978). Cost effectiveness: A model for assessing the training investment. Training and Development Journal, 32, 24–29.
   Google Scholar
• Dalal, R. S. (2005). A meta-analysis of the relationship between organizational citizenship behavior and counterproductive work behavior. The Journal of Applied Psychology, 90, 1241–1255.
   PubMed Web of Science ®Google Scholar
• Dansereau, F., & Yammarino, F. J. (2002). Overview: The many faces of multi-level issues. Research in Multi Level Issues, 1, xiii–xix.
   Google Scholar
• Davis, M. A., Curtis, M. B., & Tschetter, J. D. (2003). Evaluating cognitive training outcomes: Validity and utility of structural knowledge assessment. Journal of Business and Psychology, 18, 191–206.
   Web of Science ®Google Scholar
• Drake, M. J., Griffin, P. M., Kirkman, R., & Swann, J. L. (2005). Engineering ethical curricula: Assessment and comparison of two approaches. Journal of Engineering Education, 94, 223–231.
   Web of Science ®Google Scholar
• DuBois, J. M., & Dueker, J. M. (2009). Teaching and assessing the responsible conduct of research: A delphi consensus panel report. Journal of Research Administration, 40, 49–70.
   Google Scholar
• DuBois, J. M., Dueker, J. M., Anderson, E. E., & Campbell, J. (2008). The development and assessment of an NIH-funded research ethics training program. Academic Medicine, 83, 596–603.
   PubMed Web of Science ®Google Scholar
• Ericsson, K. A., & Charness, N. (1994). Expert performance. American Psychologist, 49, 725–747.
   Web of Science ®Google Scholar
• Fisher, C. B., Fried, A. L., & Feldman, L. G. (2009). Graduate socialization in the responsible conduct of research: A national survey on the research ethics training experiences of psychology doctoral students. Ethics & Behavior, 19, 496–518.
   PubMed Web of Science ®Google Scholar
• Ford, J. K., Smith, E. M., Weissbein, D. A., Gully, S. M., & Salas, E. (1998). Relationships of goal orientation, metacognitive activity, and practice strategies with learning outcomes and transfer. Journal of Applied Psychology, 83, 218–233.
   Web of Science ®Google Scholar
• Ford, J. K., & Weissbein, D. A. (1997). Transfer of training: An updated review and analysis. Performance Improvement Quarterly, 10, 22–41.
   Google Scholar
• Gawthrop, J. C., & Uhlemann, M. R. (1992). Effects of the problem-solving approach in ethics training. Professional Psychology: Research and Practice, 23, 38–42.
   Web of Science ®Google Scholar
• Goldstein, I. L. (1980). Training in work organizations. Annual Review of Psychology, 31, 229–272.
   Web of Science ®Google Scholar
• Griffin, R. P. (2010). Means and ends: Effective training evaluation. Industrial and Commercial Training, 42, 220–225.
   Google Scholar
• Griffin, R. P. (2011). Workplace learning evaluation: A conceptual model and framework. Industrial and Commercial Training, 43, 172–178.
   Google Scholar
• Gruys, M. L., & Sackett, P. R. (2003). Investigating the dimensionality of counterproductive work behavior. International Journal of Selection and Assessment, 11, 30–42.
   Web of Science ®Google Scholar
• Harkrider, L., MacDougall, A. E., Bagdasarov, Z., Johnson, J. F., Thiel, C. E., Mumford, M. D., … Devenport, L. D. (2013). Structuring case-based ethics training: How comparing cases and structured prompts influence training effectiveness. Ethics & Behavior, 23, 179–198.
   Web of Science ®Google Scholar
• Harkrider, L. N., Thiel, C. E., Bagdasarov, Z., Mumford, M. D., Johnson, J. F., Connelly, S., & Devenport, L. D. (2012). Improving case-based ethics training with codes of conduct and forecasting content. Ethics & Behavior, 22, 258–280.
   Web of Science ®Google Scholar
• Hauenstein, N. M., Findlay, R. A., & McDonald, D. P. (2010). Using situational judgment tests to assess training effectiveness: Lessons learned evaluating military equal opportunity advisor trainees. Military Psychology, 22, 262–281.
   Web of Science ®Google Scholar
• Heitman, E., & Bulger, R. E. (2005). Assessing the educational literature in the responsible conduct of research for core content. Accountability in Research, 12, 207–224.
   PubMedGoogle Scholar
• Helton-Fauth, W., Gaddis, B., Scott, G., Mumford, M., Devenport, L., Connelly, S., & Brown, R. (2003). A new approach to assessing ethical conduct in scientific work. Accountability in Research: Policies and Quality Assurance, 10, 205–228.
   PubMedGoogle Scholar
• Izzo, G. (2000). Compulsory ethics education and the cognitive moral development of salespeople: A quasi-experimental assessment. Journal of Business Ethics, 28, 223–241.
   Web of Science ®Google Scholar
• James, L., James, L., & Asch, D. (1990). The measuring of organizations: The role of cognition and values. In B. Schneider (Ed.), Organizational climate and culture (pp. 40–89). San Francisco, CA: Jossey-Bass.
   Google Scholar
• Johnson, J. F., Bagdasarov, Z., Harkrider, L. N., MacDougall, A. E., Connelly, S., Devenport, L. D., & Mumford, M. D. (2013). The effects of note-taking and review on sensemaking and ethical decision-making. Ethics & Behavior, 23, 299–323.
   Web of Science ®Google Scholar
• Kirkpatrick, D. (1996). Revisiting Kirkpatrick’s four-level model. Training & Development, 50, 54–59.
   Google Scholar
• Kirkpatrick, D. L., & Kirkpatrick, J. D. (2007). Implementing the four levels: A practical guide for effective evaluation of training programs. San Francisco, CA: Berrett-Koehler.
   Google Scholar
• Kligyte, V., Marcy, R. T., Waples, E. P., Sevier, S. T., Godfrey, E. S., Mumford, M. D., & Hougen, D. F. (2008). Application of a sensemaking approach to ethics training in the physical sciences and engineering. Science and Engineering Ethics, 14, 251–278.
   PubMed Web of Science ®Google Scholar
• Kohlberg, L. (1984). Essays on moral development: Vol. 2. The psychology of moral development: The nature and validity of moral stages. San Francisco, CA: Harper & Row.
   Google Scholar
• Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology Monograph, 78, 311–328.
   Web of Science ®Google Scholar
• Loe, T. W., Ferrell, L., & Mansfield, P. (2000). A review of empirical studies assessing ethical decision making in business. Journal of Business Ethics, 25, 185–204.
   Web of Science ®Google Scholar
• MacDougall, A. E., Harkrider, L. N., Bagdasarov, Z., Johnson, J. F., Thiel, C. E., Peacock, J., … Connelly, S. (2014). Examining the effects of incremental case presentation and forecasting outcomes on case-based ethics instruction. Ethics & Behavior, 24, 126–150.
   Web of Science ®Google Scholar
• Mann, S., & Robertson, I. T. (1996). What should training evaluations evaluate? Journal of European Industrial Training, 20, 14–20.
   Google Scholar
• Mecca, J. T., Medeiros, K. E., Giorgini, V., Gibson, C., Mumford, M. D., Connelly, S., & Devenport, L. D. (2014). The influence of compensatory strategies on ethical decision making. Ethics & Behavior, 24, 73–89.
   PubMed Web of Science ®Google Scholar
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.). New York, NY: Macmillan.
   Google Scholar
• Mirabal, T. E. (1978). Forecasting future training costs. Training and Development Journal, 32, 78–87.
   Google Scholar
• Morgan, R. B., & Casper, W. J. (2000). Examining the factor structure of participant reactions to training: A multidimensional approach. Human Resource Development Quarterly, 11, 301–317.
   Google Scholar
• Mumford, M. D. (in press). Assessing the effectiveness of responsible conduct of research training: Key findings and viable procedures. In R. Nerem (Ed.), Responsible science. Washington, DC: National Academies of Science.
   Google Scholar
• Mumford, M. D., Connelly, S., Brown, R. P., Murphy, S. T., Hill, J. H., Antes, A. L., … Devenport, L. D. (2008). A sensemaking approach to ethics training for scientists: Preliminary evidence of training effectiveness. Ethics & Behavior, 18, 315–339.
   PubMed Web of Science ®Google Scholar
• Mumford, M. D., Connelly, S., Murphy, S. T., Devenport, L. D., Antes, A. L., Brown, R. P., … Waples, E. P. (2009). Field and experience influences on ethical decision-making in the sciences. Ethics & Behavior, 19, 263–289.
   PubMed Web of Science ®Google Scholar
• Mumford, M. D., Devenport, L. D., Brown, R. P., Connelly, S., Murphy, S. T., Hill, J. H., & Antes, A. L. (2006). Validation of ethical decision making measures: Evidence for a new set of measures. Ethics & Behavior, 16, 319–345.
   Web of Science ®Google Scholar
• Mumford, M. D., Marks, M. A., Connelly, M. S., Zaccaro, S. J., & Reiter-Palmon, R. (2000). Development of leadership skills: Experience and timing. Leadership Quarterly, 11, 87–114.
   Web of Science ®Google Scholar
• Mumford, M. D., Murphy, S. T., Connelly, S., Hill, J. H., Antes, A. L., Brown, R. P., & Devenport, L. D. (2007). Environmental influences on ethical decision making: Climate and environmental predictors of research integrity. Ethics & Behavior, 17, 337–366.
   Web of Science ®Google Scholar
• Mumford, M. D., Waples, E. P., Antes, A. L., Brown, R. P., Connelly, S., Murphy, S. T., & Devenport, L. D. (2010). Creativity and ethics: The relationship of creative and ethical problem-solving. Creativity Research Journal, 22, 74–89.
   PubMed Web of Science ®Google Scholar
• National Academy of Engineering. (2009). Ethics education and scientific and engineering research. Washington, DC: National Academies Press.
   Google Scholar
• National Academy of Sciences. (1992). Responsible science: Ensuring integrity of the research process. Washington, DC: National Academies Press.
   Google Scholar
• National Institute of Medicine. (2002). Integrity in scientific research. Washington, DC: National Academies Press.
   Google Scholar
• Ostroff, C. (1991). Training effectiveness measures and scoring schemes: A comparison. Personnel Psychology, 44, 353–374.
   Web of Science ®Google Scholar
• Pieters, H. M., Touw-Otten, F. W., & De Melker, R. A. (1994). Simulated patients in assessing consultation skills of trainees in general practice vocational training: A validity study. Medical Education, 28, 226–233.
   PubMed Web of Science ®Google Scholar
• Peterson, D. K. (2002). The relationship between unethical behavior and the dimensions of the ethical climate questionnaire. Journal of Business Ethics, 41, 313–326.
   Web of Science ®Google Scholar
• Podsakoff, P. M., MacKenzie, S. B., Moorman, R. H., & Fetter, R. (1990). Transformational leader behaviors and their effects on followers’ trust in leader, satisfaction, and organizational citizenship behaviors. The Leadership Quarterly, 1, 107–142.
   Google Scholar
• Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93, 223–231.
   Web of Science ®Google Scholar
• Rest, J. R. (1986). Moral development: Advances in research and theory. New York, NY: Praeger.
   Google Scholar
• Saks, A. M., & Burke, L. A. (2012). An investigation into the relationship between training evaluation and the transfer of training. International Journal of Training and Development, 16, 118–127.
   Google Scholar
• Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress. Annual Review of Psychology, 52, 471–499.
   PubMed Web of Science ®Google Scholar
• Salas, E., Milham, L. M., & Bowers, C. A. (2003). Training evaluation in the military: Misconceptions, opportunities, and challenges. Military Psychology, 15, 3–16.
   Web of Science ®Google Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
   Google Scholar
• Siegel, D. H. (1983). Can research and practice be integrated in social work education? Journal of Education for Social Work, 19, 12–19.
   Web of Science ®Google Scholar
• Sonenshein, S. (2007). The role of construction, intuition, and justification in responding to ethical issues at work: The sensemaking-intuition model. Academy of Management Journal, 32, 1022–1040.
   Web of Science ®Google Scholar
• Stenmark, C. K., Antes, A. L., Thiel, C. E., Caughron, J. J., Wang, X., & Mumford, M. D. (2011). Consequences identification in forecasting and ethical decision-making. Journal of Empirical Research on Human Research Ethics, 6, 25–32.
   PubMed Web of Science ®Google Scholar
• Thiel, C. E., Bagdasarov, Z., Harkrider, L., Johnson, J. F., & Mumford, M. D. (2012). Leader ethical decision-making in organizations: Strategies for sensemaking. Journal of Business Ethics, 107, 49–64.
   Web of Science ®Google Scholar
• Thiel, C. E., Connelly, S., & Griffith, J. A. (2011). The influence of anger on ethical decision making: Comparison of a primary and secondary appraisal. Ethics & Behavior, 21, 380–403.
   Web of Science ®Google Scholar
• Thiel, C. E., Connelly, S., Harkrider, L., Devenport, L. D., Bagdasarov, Z., Johnson, J. F., & Mumford, M. D. (2013). Case-based knowledge and ethics education: Improving learning and transfer through emotionally rich cases. Science and Engineering Ethics, 19, 265–286.
   PubMed Web of Science ®Google Scholar
• Victor, B., & Cullen, J. B. (1988). The organizational bases of ethical work climate. Administrative Science Quarterly, 33, 101–125.
   Web of Science ®Google Scholar
• Waples, E. P., Antes, A. L., Murphy, S. T., Connelly, S., & Mumford, M. D. (2009). A meta-analytic investigation of business ethics instruction. Journal of Business Ethics, 87, 133–151.
   Web of Science ®Google Scholar
• Wells, D., & Schminke, M. (2001). Ethical development and human resources training: An integrative framework. Human Resource Management Review, 11, 135–158.
   Google Scholar