Applying a participatory systems and value approach in a transdisciplinary exercise: on assessing the impact of training and education initiatives

Figures & data

Figure 1. A generic picture of a system impact model – from Akinluyi et al. (2019)’s paper on outcome identification (with permission).

Figure 2. Core Purpose Translation Tool (With permission) - Developed by Akinluyi (11) based on a cross-examination of Statements of Core Purpose form leading Non-Profits, Consumer Companies and the NHS.

Figure 3. Developing an Impact Model Systematically, using a Sequence of Questions from the Outcome Identification Loop – from Akinluyi et al. (2019)’s paper on outcome identification (with permission).

Table 1. Workshop content in overview.

Workshop/Stage	System Impact Model Stage	General description of stage
[Planning]	1 – Assembling a Team	The knowledge and understanding needed to produce a system model sit with those people who interact with it, who bring with them different perspectives and insights (As Figure 1 illustrates).
Workshop 1	2 - Establishing Core Purpose	More often than not, an improvement exercise has a defined scope or an overall objective before the system modelling stage is reached. It is important to establish a common understanding of this – especially in handling a transdisciplinary challenge.
	3 - Seeding the System Impact Model (SIM)	Statements of purpose are gathered, and (re)articulated as “outcomes”, which are framed as attributes that can be increased or decreased. A “core purpose translation” tool (Figure 2) may be applied, to support this process.
	4 - Developing the SIM	Mapping out’ the SIM means asking participants a series of questions in sequence (Akinluyi etal., 2019) to elicit more outcomes, which they gradually add to the SIM, representing (with arrows) which outcomes influence others. The SIM then grows as described in the lower portion of Figure 3.
	5 - Arranging the impact model and drawing system boundaries	Defining the “boundary” of a designed system means identifying the outcomes that directly affect and are meaningful to those individuals that we aim to serve. Akinluyi etal. (2019) set out various criteria for defining boundary outcomes.
Workshop 2
	6 - Deriving measures and indicators	Measures and indicators should map as closely as possible to the identified boundary outcomes. Where outcomes are difficult to measure directly, connected or associated outcomes may be as a proxy. Proxy measures/indicators must be kept in context, to avoid a bias towards more measure-able, less meaningful outcomes.
Follow-up workshop: on nursing retention	Further SIM development and utilisation	-

Table 2. Participant Backgrounds.

Staff Group	Participant Job Titles as disclosed
Nurses	Deputy Chief Nurse Nurse Senior Nurse- Project Lead
SC Dentistry	Consultant in special care Dentistry
Paediatrics	Consultant in Paediatric Disability
Psychiatrics/Psychology	Head of Education and Training, Psychosis Clinical Academic Group Consultant Clinical Psychologist
Haematology	Haematologist
Therapies and rehabilitation	Occupational therapist Consultant Occupational therapist Deputy Chief Therapist
Genomics	Education and Training Lead, Genomic Laboratory Hub
Acute medicine	Consultant and Honorary Senior Lecturer in Acute Medicine
Surgery	Project Lead, Surgical Simulation Team
Human resources stakeholders	Director of Equality, Diversity and Inclusion
Education training operational (within organisation)	Head of Learning and Development Associate Director Education Training and Development
Academic (Clinical Academic) training	Visiting Senior Lecturer Reader in Clinical Education
Strategic training organisation	Organisation Development Coordinator Clinical Educator for Medical Physics in London Head of CPPD and Education Funding Director of Lambeth Living Well Network Alliance Organisational Development Change Manager/Consultant Organisational Development Associate Equality and Human Rights Advisor at NHS Regional Director for Health Education England
Strategic leadership	Director of Organisational Development Director of Operations and Partnerships, GSTT Head of Research and Development

Figure 4. A System Impact Model for Education, Training and Development – with filled circles representing “boundary outcomes”, directly affecting stakeholders.

Table 3. Outcome measures/indicators suggested in the workshops.

Stakeholder	Outcome	Potential Indicators
Patient	Service capacity ⇒ [waiting times]	Number of new roles created. Procedural data; analysis of procedural volume. Actual waiting time data.
	Service proficiency ⇒ [Procedural Success, Patient Safety]	Procedural outcome data. Incident data.
	Range of Services ⇒ [Service proficiency]	Analysis of procedural data.
	Service Affirmation ⇒ [Confidence in Services]	Awards & Publications.
	Investment in personalised patient care ⇒ [Positive Patient Experience, Procedural Success]	Structured patient feedback. Reports/surveys on time invested per case (estimated).
Trainee	Attainment of knowledge	Attainment of (accredited) qualifications. Review at appraisal.
	[Attainment of knowledge & Ability to Communicate & Receptiveness of Service] ⇒ Diffusion of knowledge	Post-hoc survey on training impact, e.g., 0–5 star reviews. Reflection on, and documentation of: “How many did you tell about this… in your immediate team? Outside your immediate team, within your wider Department? Outside your Department, in the organisation?” i.e., consider both diversity and size of audience. Review at appraisal and/or 360-degree review, noting changes in individual behaviour, number of change/improvement projects initiated and any discernible impact on others.
	Perspective and confidence	Review and appraisal (as described above), evidence of applying corporate values. (Documented) reflection on attitudinal and cultural change. Psychometric tests.
	[Satisfaction of trainee] ⇒ Staff Retention	Clarification of factors in exit surveys/interviews Surveys and other possible subjective indicators of wellbeing and functioning of team.
	Personal Resilience	Feedback from supervisor(s). Psychometric tests (e.g., “I-resilience” questionnaire (Ng & Nicholas, 2015)).
	Consumption of Time/Effort	Reports on human resource allocation (i.e., time commitment). Volume of exception reports submitted where staff must step in to work outside of contracted hours.
	Competence of trainee	Incident reduction (estimates). Simulation training & review (medical staff).
	[Resilience & Reduced Isolation & Satisfaction] ⇒ Team Cohesion and Motivation	360-degree appraisal of teams, including input from teams. Consider shared goals, collaboration. Engagement surveys.
Organisation	Training Costs ⇒ [Financial Impact]	Reporting on financial outlay, commensurate with estimated pay-off; similar to NICE’s cost-effectiveness measures.

Figure 5. An Impact Model for Nursing Retention.

Akinluyi, E. A., Ison, K., & Clarkson, P. J. (2019). Mapping outcomes in quality improvement and system design activities: The outcome identification loop and system impact model. BMJ Open Quality, 8, e000439. https://doi.org/10.1136/bmjoq-2018-000439

 PubMedGoogle Scholar

Akinluyi, E. A. (2017). The modelling of value for the systematic design of medical technological infrastructure [ PhD Thesis].

 Google Scholar

Ng, W., & Nicholas, H. (2015). iResilience of science pre-service teachers through digital storytelling. Australasian Journal of Educational Technology, 31(6), 736–51. https://doi.org/10.14742/ajet.1699

 Web of Science ®Google Scholar