A Pilot Study Assessment of Medical Student Knowledge and System Citizenship Attitudes Pertaining to Health Systems Science

Figures & data

Table 1 Health System Citizenship Scale Virginia Tech Carilion School of Medicine

Item #	Original, 12-Item Attitudinal Version^+*	Revised, 16-Item Version (Part One)*	Revised, 16-Item Version (Part Two)*
	Please Indicate the Degree to Which You Agree with Each of the Following Items	Please Indicate the Extent to Which You Agree You Have Experienced, Observed, or Studied Each of the Following Systems Citizen Components at VTCSOM	Please Indicate the Extent to Which You Agree You are Currently Prepared for Each of the Following System Components
1	An understanding of health systems science impacts how I view the delivery of care to patients	Health systems science impacts on the delivery of care to patients	Contributing to health systems science impacts on the delivery of care to patients
2	Health care is more than a function of the individual physician–patient relationship	Health care as more than a function of the individual physician–patient relationship	Contributing to health care as more than a function of the individual physician–patient relationship
3	The design of a health system has a strong influence on patient outcomes	The design of a health system having a strong influence on patient outcomes	Contributing to the design of a health system having a strong influence on patient outcomes
4	Year 1: A physician should work with others to improve the overall health system/Year 2: A physician should work with others to improve the processes and outcomes of care for specific patient groups	Physicians working with others to improve the processes and outcomes of care for specific patient groups	Working with others to improve the processes and outcomes of care for specific patient groups
5	A physician should be equipped to help patients navigate their way through the health system	Physicians equipped to help patients navigate their way through the health system	Helping patients navigate their way through the health system
6	Physicians should serve in leadership roles in systems thinking and patient safety	Physicians serving in leadership roles in systems thinking and patient safety	Serving in leadership roles in systems thinking and patient safety
7	Physicians should sometimes lead a clinical team but at other times should defer to another health care professional team leader	Physicians sometimes leading a clinical team but at other times deferring to another health care team leader	Leading a clinical team and at times deferring to another health care team leader
8	Physicians should identify and address systems level factors that contribute to ethical dilemmas	Physicians identifying and addressing systems level factors that contribute to ethical dilemmas	Identifying and addressing systems level factors that contribute to ethical dilemmas
9	Physicians should discuss medical errors/unexpected outcomes openly to facilitate learning and patient understanding	Physicians discussing medical errors/unexpected outcomes openly to facilitate learning and patient understanding	Discussing medical errors/unexpected outcomes openly to facilitate learning and patient understanding
10	Year 1: I can apply the concepts of systems thinking to a given patient care scenario/Year 2: The concepts of systems thinking should be applied to each patient care scenario	The concepts of systems thinking applied to each patient care scenario	Applying systems thinking to each patient care scenario
11	Systems thinking is required to re-design health care to improve the quality and safety of care provided	The re-designing of health care using systems thinking to improve the quality and safety of care provided	Using systems thinking to re-design health care to improve the quality and safety of care provided
12	Year 1: When I encounter a problem, I use multiple viewpoints to understand/analyze it/Year 2: When a problem is encountered in the clinical setting, multiple viewpoints should be used to understand/analyze it	The use of multiple viewpoints to understand/analyze when a problem is encountered in the clinical setting	Using multiple viewpoints to understand/analyze a problem encountered in the clinical setting
13	N/A	Physicians demonstrating a duty to be a change agent and help improve the health care system	Demonstrating a duty to be a change agent and help improve the health care system
14	N/A	Physicians demonstrating a duty to ensure that economically disadvantaged people have an equal opportunity to be healthy	Demonstrating a duty to ensure that economically disadvantaged people have an equal opportunity to be healthy
15	N/A	Health systems measuring and monitoring clinical performance related to health equity	Measuring and monitoring clinical performance related to health equity
16	N/A	Physicians as leaders and collaborators in health system transformation	Leading and collaborating in health system transformation

Notes: Rating scale used, both versions: 1=strongly disagree, 2=disagree, 3=neutral/no opinion/unsure, 4=agree, 5=strongly agree. ⁺Used in study years 1 and 2. *Used in study year 2.

Figure 1 Comparisons of HSS exam scores and survey results by student group. *Students had not experienced new HSSIP curriculum. ⁺Students had experienced new HSSIP curriculum.

Table 2 Comparison of Attitudes Toward System Citizenship Fourth-Year versus First-Year Medical Students (Study Year 1, No HSS Curriculum Experience)

Item #	Please Indicate the Degree to Which You Agree with Each of the Following Items	M4 Students (N=24) Mean Rating	M4 Students Standard Dev	M1 Students (N=42) Mean Rating	M1 Students Standard Dev	T-test (2 Tailed) Significance
1	An understanding of health systems science impacts how I view the delivery of care to patients	4.16	0.81	4.46	0.55	p=0.07
2	Health care is more than a function of the individual physician–patient relationship	4.12	1.03	4.65	0.08	p=0.007 Cohen’s d=0.69*
3	The design of a health system has a strong influence on patient outcomes	4.46	0.72	4.74	0.49	p=0.06
4	A physician should work with others to improve the overall health system	4.58	0.50	4.72	0.50	p=0.28
5	A physician should be equipped to help patients navigate their way through the health system	4.50	0.51	4.58	0.54	p=0.55
6	Physicians should serve in leadership roles in systems thinking and patient safety	4.54	0.78	4.58	0.63	p=0.82
7	Physicians should sometimes lead a clinical team but at other times should defer to another health care professional team leader	3.62	1.40	4.37	0.90	p=0.01 Cohen’s d=0.67*
8	Physicians should identify and address systems level factors that contribute to ethical dilemmas	4.41	0.58	4.60	0.58	p=0.21
9	Physicians should discuss medical errors / unexpected outcomes openly to facilitate learning and patient understanding	4.66	0.48	4.65	0.57	p=0.91
10	I can apply the concepts of systems thinking to a given patient care scenario	4.20	0.83	3.81	1.05	p=0.12
11	Systems thinking is required to re-design health care to improve the quality and safety of care provided	4.45	0.59	4.49	0.69	p=0.85
12	When I encounter a problem, I use multiple viewpoints to understand/analyze it	4.37	0.49	4.65	0.65	p=0.07

Notes: Rating scale used, both versions: 1=strongly disagree, 2=disagree, 3=neutral/no opinion/unsure, 4=agree, 5=strongly agree. *Effect size considered moderate.

Table 3 Comparison of Student Perception of Experience with Health Systems Science (Study Year 2, Fourth-Year Students/No HSS Curriculum versus First-Year Students/Had New HSS Curriculum)

Item #	Revised, 16-Item Version (EXPERIENCE Items)	M4 Students (N=28) Mean Rating	M4 Students Standard Dev	M1 Students (N=28) Mean Rating	M1 Students Standard Dev	T-test (2 Tailed) Significance
	Please Indicate the Extent to Which You Agree You Have Experienced, Observed, or Studied Each of the Following Systems Citizen Components at VTCSOM
1	Health systems science impacts on the delivery of care to patients	4.25	0.58	3.64	1.22	P=0.02 Cohen’s d=0.63*
2	Health care as more than a function of the individual physician–patient relationship	4.07	0.54	3.86	1.04	P=0.33
3	The design of a health system having a strong influence on patient outcomes	4.07	0.76	3.82	1.16	P=0.34
4	Physicians working with others to improve the processes and outcomes of care for specific patient groups	4.18	0.67	3.75	1.26	P=0.11
5	Physicians equipped to help patients navigate their way through the health system	3.86	0.80	3.68	1.25	P=0.53
6	Physicians serving in leadership roles in systems thinking and patient safety	4.11	0.68	3.82	1.16	P=0.26
7	Physicians sometimes leading a clinical team but at other times deferring to another health care team leader	4.07	0.54	3.57	1.20	P=0.04 Cohen’s d=0.53*
8	Physicians identifying and addressing systems level factors that contribute to ethical dilemmas	4.07	0.60	3.75	1.24	P=0.22
9	Physicians discussing medical errors/unexpected outcomes openly to facilitate learning and patient understanding	4.11	0.50	3.82	1.16	P=0.23
10	The concepts of systems thinking applied to each patient care scenario	4.00	0.86	3.75	1.17	P=0.37
11	The re-designing of health care using systems thinking to improve the quality and safety of care provided	3.78	0.92	3.87	1.17	P=1.00
12	The use of multiple viewpoints to understand/analyze when a problem is encountered in the clinical setting	4.25	0.64	3.93	1.12	P=0.19
13	Physicians demonstrating a duty to be a change agent and help improve the health care system	4.14	0.65	3.71	1.18	P=0.09
14	Physicians demonstrating a duty to ensure that economically disadvantaged people have an equal opportunity to be healthy	4.03	0.64	3.82	1.16	P=0.39
15	Health systems measuring and monitoring clinical performance related to health equity	4.04	0.84	3.75	1.26	P=0.32
16	Physicians as leaders and collaborators in health system transformation	4.18	0.55	3.89	1.16	P=0.24

Notes: Rating scale used, both versions: 1=strongly disagree, 2=disagree, 3=neutral/no opinion/unsure, 4=agree, 5=strongly agree. *Effect size considered moderate.

Table 4 Comparison of Student Perception of Preparation for Health Systems Science (Study Year 2, Fourth-Year Students/No HSS Curriculum versus First-Year Students/Had New HSS Curriculum)

Item #	Revised, 16-Item Version (PREPARATION Items)	M4 Students (N=28) Mean Rating	M4 Students Standard Dev	M1 Students (N=27) Mean Rating	M1 Students Standard Dev	T-test (2 Tailed) Significance
	Please Indicate the Extent to Which You Agree You are Currently Prepared for Each of the Following System Components
1	Contributing to health systems science impacts on the delivery of care to patients	4.03	0.84	3.52	1.16	P=0.06
2	Contributing to health care as more than a function of the individual physician–patient relationship	4.07	0.76	3.63	1.15	P=0.10
3	Contributing to the design of a health system having a strong influence on patient outcomes	3.96	0.84	3.67	1.14	P=0.27
4	Working with others to improve the processes and outcomes of care for specific patient groups	4.18	0.72	3.74	1.06	P=0.08
5	Helping patients navigate their way through the health system	3.78	1.03	3.44	1.12	P=0.24
6	Serving in leadership roles in systems thinking and patient safety	4.00	0.92	3.55	1.08	P=0.11
7	Leading a clinical team and at times deferring to another health care team leader	4.14	0.65	3.48	1.12	P=0.009 Cohen’s d=0.72*
8	Identifying and addressing systems level factors that contribute to ethical dilemmas	4.07	0.66	3.63	1.08	P=0.07
9	Discussing medical errors/unexpected outcomes openly to facilitate learning and patient understanding	4.18	0.61	3.59	1.08	P=0.02 Cohen’s d=0.66*
10	Applying systems thinking to each patient care scenario	4.00	0.82	3.59	1.08	P=0.12
11	Using systems thinking to re-design health care to improve the quality and safety of care provided	4.03	0.79	3.59	1.05	P=0.08
12	Using multiple viewpoints to understand/analyze a problem encountered in the clinical setting	4.26	0.76	3.74	1.13	P=0.05 Cohen’s d=0.53*
13	Demonstrating a duty to be a change agent and help improve the health care system	4.03	0.74	3.70	1.10	P=0.19
14	Demonstrating a duty to ensure that economically disadvantaged people have an equal opportunity to be healthy	4.25	0.52	3.70	1.10	P=0.02 Cohen’s d=0.63*
15	Measuring and monitoring clinical performance related to health equity	4.14	0.59	3.55	1.05	P=0.01 Cohen’s d=0.69*
16	Leading and collaborating in health system transformation	4.03	0.79	3.70	1.07	P=0.19

Notes: Rating scale used, both versions: 1=strongly disagree, 2=disagree, 3=neutral/no opinion/unsure, 4=agree, 5=strongly agree. *Effect size considered moderate.

Table 5 Comparison of Attitudes Toward System Citizenship Both Study Years, First-Year Medical Students Who Did Not/Did Experience New HSS Curriculum

Item #	Please Indicate the Degree to Which You Agree with Each of the Following Items	M1 Students NO CURRICULUM (N=43) Mean Rating	M1 Students Standard Dev	M1 Students NEW CURRICULUM (N=28) Mean Rating	M1 Students Standard Dev	T-test (2 Tailed) Significance
1	An understanding of health systems science impacts how I view the delivery of care to patients	4.46	0.55	4.00	0.86	p=0.007 Cohen’s d=0.67*
2	Health care is more than a function of the individual physician–patient relationship	4.65	0.53	4.25	0.97	p=0.02 Cohen’s d=0.54*
3	The design of a health system has a strong influence on patient outcomes	4.74	0.49	4.21	0.96	p=0.003 Cohen’s d=0.74*
4	A physician should work with others to improve the overall health system			Not rated^+
5	A physician should be equipped to help patients navigate their way through the health system	4.58	0.54	4.25	0.18	p=0.07
6	Physicians should serve in leadership roles in systems thinking and patient safety	4.58	0.63	4.11	0.96	p=0.01 Cohen’s d=0.61*
7	Physicians should sometimes lead a clinical team but at other times should defer to another health care professional team leader	4.37	0.90	4.25	0.97	p=0.58
8	Physicians should identify and address systems level factors that contribute to ethical dilemmas	4.60	0.58	4.14	0.93	p=0.01 Cohen’s d=0.62*
9	Physicians should discuss medical errors/unexpected outcomes openly to facilitate learning and patient understanding	4.65	0.57	4.21	0.92	p-0.02 Cohen’s d=0.60*
10	I can apply the concepts of systems thinking to a given patient care scenario			Not rated^+
11	Systems thinking is required to re-design health care to improve the quality and safety of care provided	4.49	0.67	4.14	0.97	p=0.08
12	When I encounter a problem, I use multiple viewpoints to understand/analyze it			Not rated^+

Notes: Rating scale used, both versions: 1=strongly disagree, 2=disagree, 3=neutral/no opinion/unsure, 4=agree, 5=strongly agree. *Effect size considered moderate. ⁺Not rated due to wording changes in survey item.