Content validation of Mental Health Literacy Scale (MHLS) for primary health care workers in South Africa and Zambia ─ a heterogeneous expert panel method

Figures & data

Table 1. Demography of participants.

	Professional Research Experts (PREs) (n = 11)	Clinical Experts (CEs) (n = 10)
Country
South Africa	5	-
Zambia	3	10
Germany	2	-
Finland	1	-
Sex
Female	8	8
Male	3	1
(not reported)		1
Age by group
Mean (SD)	48.5 (12.4)	32.5 (8.3)
Health professional category
Registered nurse/midwife		7
Enrolled nurse		1
Psychotherapist	1
Lecturer	6
Research nurse	2
Research psychologist	2
Clinical officer general		1
Art nurse		1
Level of professional education
Certificate		2
Bachelor/Diploma	2	8
Master’s	4
PhD	5
Work experience
Less than 1 year		2
1 to <5 years	1	3
5 to <10 years	1	1
10 to <15 years	1	-
15 years or more	8	4

Table 2. Ratings for 35 items by PREs: items rated 3 or 4 (x) on a 4-point relevance scale [20].

Item	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 6	Expert 7	Expert 8	Expert 9	Expert 10	Expert 11	Number in Agreement	Item CVI (I-CVI-PE)
1	*	x	x	x	x	x	x	x	x	x	–	9	0.90
2	*	x	x	x	x	x	x	x	x	x	x	10	1.00
3	x	x	x	x	x	x	x	x	x	x	x	11	1.00
4	–	x	x	x	x	–	x	x	x	x	x	9	0.82
5	–	x	x	x	x	x	x	x	x	x	–	9	0.82
6	x	x	x	x	x	x	–	x	x	x	x	10	0.91
7	x	x	x	x	x	x	x	x	x	x	x	11	1.00
8	x	x	x	x	x	x	x	x	x	x	x	11	1.00
9	x	x	x	x	x	x	x	–	x	–	x	9	0.82
10	x	x	x	x	x	x	x	–	x	–	x	9	0.82
11	x	x	x	x	x	x	x	x	*	x	x	10	1.00
12	x	x	x	x	x	x	x	–	x	x	x	10	0.91
13	x	x	x	x	x	x	x	x	x	x	x	11	1.00
14	x	x	x	x	x	x	x	x	x	x	x	11	1.00
15	x	x	x	x	x	–	x	x	x	x	x	10	0.91
16	x	x	x	x	x	x	x	x	x	x	x	11	1.00
17	x	x	x	x	x	x	x	x	x	x	x	11	1.00
18	x	x	x	x	x	x	x	x	x	x	x	11	1.00
19	x	x	x	x	x	x	x	x	x	x	x	11	1.00
20	x	x	x	x	x	–	x	x	x	x	x	10	0.91
21	x	x	x	x	x	x	x	x	x	x	x	11	1.00
22	x	x	x	x	x	x	x	x	x	x	x	11	1.00
23	x	x	x	x	x	x	x	x	x	x	x	11	1.00
24	x	x	x	x	x	–	x	x	x	x	x	10	0.91
25	x	x	x	–	x	x	x	x	x	x	x	10	0.91
26	x	x	x	x	x	x	x	x	x	x	x	11	1.00
27	x	x	x	x	x	x	x	x	x	x	x	11	1.00
28	x	x	x	x	x	–	x	x	x	x	x	10	0.91
29	x	x	x	x	x	x	x	–	x	x	x	10	0.91
30	x	x	x	x	x	x	x	–	x	x	x	10	0.91
31	x	x	x	x	x	x	x	x	x	x	x	11	1.00
32	x	x	x	x	x	x	x	x	x	x	x	11	1.00
33	x	x	x	x	x	–	x	x	x	x	x	10	0.91
34	x	x	x	x	x	x	x	x	x	x	x	11	1.00
35	x	x	x	x	x	x	x	x	x	x	x	11	1.00
Mean Item level CVI (S-CVI/Ave)													0.95

* = Unclearly marked by the expert

Table 3. Ratings for 35 items by CEs: items rated 3 or 4 (x) on a 4-point relevance scale [20].

Item	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 6	Expert 7	Expert 8	Expert 9	Expert 10	Number in Agreement	Item CVI (I-CVI-CE)
1	–	x	x	–	x	–	–	x	–	x	5	0.5
2	–	x	x	–	x	x	–	x	–	x	6	0.6
3	–	x	x	x	x	x	–	x	x	–	7	0.7
4	x	x	–	–	x	x	–	x	–	x	6	0.6
5	*	x	x	x	x	–	x	x	–	–	6	0.67
6	x	x	x	–	–	x	x	x	x	x	8	0.8
7	–	–	x	x	x	x	x	x	–	x	7	0.7
8	x	x	x	x	x	x	x	x	–	–	8	0.8
9	–	–	–	–	–	–	–	–	*	x	1	0.1
10	–	–	–	–	–	–	–	–	x	x	2	0.2
11	x	x	x	–	x	–	x	x	x	x	8	0.8
12	–	–	x	–	–	x	x	x	x	–	5	0.5
13	x	x	x	x	x	x	x	x	x	–	9	0.9
14	x	x	x	–	x	x	x	x	x	x	9	0.9
15	x	x	–	–	x	x	–	x	x	–	6	0.6
16	x	x	x	x	x	x	x	x	x	x	10	1.00
17	x	x	x	x	x	x	–	x	x	x	9	0.9
18	x	x	–	x	x	x	x	x	x	x	9	0.9
19	x	x	x	x	x	x	x	x	x	x	10	1.00
20	x	x	x	x	–	–	–	–	x	–	5	0.5
21	x	–	x	–	x	x	–	–	–	–	4	0.4
22	x	x	x	–	x	x	–	–	–	x	6	0.6
23	x	x	*	x	x	x	x	*	–	–	6	0.75
24	x	–	–	–	x	x	–	–	–	–	3	0.3
25	x	x	–	–	x	–	–	–	x	–	4	0.4
26	x	–	x	–	x	x	–	–	x	–	5	0.5
27	x	x	–	–	x	–	–	–	x	–	4	0.4
28	x	x	–	–	–	x	–	–	–	–	3	0.3
29	x	x	–	–	x	x	x	x	–	x	7	0.7
30	x	x	–	–	x	–	x	x	–	–	5	0.5
31	x	x	x	x	x	–	x	x	x	–	8	0.8
32	x	x	x	–	x	–	x	–	x	–	6	0.6
33	x	x	–	–	x	–	x	–	x	–	5	0.5
34	x	x	–	–	x	x	x	–	x	x	7	0.7
35	x	x	–	–	x	x	x	–	x	x	7	0.7
Mean Item level CVI (S-CVI/Ave)												0.62

* = Unclearly marked by the expert

Table 4. Ratings, consensuses and rationale of expert panels on 35 items.

Original Item	Item level CVI (PREs/ CEs/ ALL)	Consensus focus group (PREs/CEs)	Experts’ suggestions/comments	Re-viewed(x)	Rationale	Form of the item after revision
1. If someone became extremely nervous or anxious in one or more situations with other people (e.g., a party) or performance situations (e.g., presenting at a meeting) in which they were afraid of being evaluated by others and that they would act in a way that was humiliating or feel embarrassed, then to what extent do you think it is likely they have Social Phobia	0.90/ 0.50/ 0.70	Relevant/Not clear	One PRE questioned if a responder, a patient or nurse, understands the term ‘social phobia’ in the same context. One CE commented ‘not relevant’ because you can’t be anxious or nervous but overexcited for a party. Two other comments were relevant for social phobia.	x	The term ‘in social gatherings’ was added for clearance	If someone became extremely nervous or anxious in one or more situations with other people (e.g., in social gatherings) or performance situations (e.g., presenting at a meeting) in which they were afraid of being evaluated by others and that they would act in a way that was humiliating or feel embarrassed, then to what extent do you think it is likely they have Social Phobia
2. If someone experienced excessive worry about a number of events or activities where this level of concern was not warranted, had difficulty controlling this worry and had physical symptoms such as having tense muscles and feeling fatigued then to what extent do you think it is likely they have Generalised Anxiety Disorder	1.00/ 0.60/ 0.80	Relevant/Relevant	No specific suggestions	─	─	Retained as original.
3. If someone experienced a low mood for two or more weeks, had a loss of pleasure or interest in their normal activities and experienced changes in their appetite and sleep then to what extent do you think it is likely they have Major Depressive Disorder	1.00/ 0.70/ 0.86	Relevant/Relevant	No specific suggestions	─	─	Retained as original.
4. To what extent do you think it is likely that Personality Disorders are a category of mental illness	0.82/ 0.60/ 0.71	Relevant/Relevant	CE commented as relevant	x	─	Retained as original.
5. To what extent do you think it is likely that Persistent Depressive Disorder (Dysthymia) is a disorder	0.82/ 0.67/ 0.75	Relevant, but needs minor alteration/ Relevant	PREs: ‘Persistent depressive mood’ would be more correct. CE: The word ‘a disorder’ needs to add ‘mental.’	x	Added the term ‘mental,’ as requested and retained the original term ‘disorder’ as suggested by the instrument’s original author.	To what extent do you think it is likely that Persistent Depressive Disorder (Dysthymia) is a mental disorder
6. To what extent do you think it is likely that the diagnosis of Agoraphobia includes anxiety about situations where escape may be difficult or embarrassing	0.91/ 0.80/ 0.86	Relevant/ Relevant	PREs: The term agoraphobia can be hard to understand. ‘Do they [nurses] know?’	x	Added the term “e.g (open market place)” for clarity.	To what extent do you think it is likely that the diagnosis of Agoraphobia includes anxiety about situations (e.g., open market place) where escape may be difficult or embarrassing
7. To what extent do you think it is likely that the diagnosis of Bipolar Disorder includes experiencing periods of elevated (i.e., high) and periods of depressed (i.e., low) mood	1.00/ 0.70/ 0.86	Relevant/ Relevant	PRE: Could add a term, such as ‘extremely’ elevated.	x	Added “extremely elevated” for clarity.	To what extent do you think it is likely that the diagnosis of Bipolar Disorder includes experiencing periods of extremely elevated (i.e., high) and periods of depressed (i.e., low) mood
8. To what extent do you think it is likely that the diagnosis of Substance Abuse Disorder can include physical and psychological tolerance of the drug (i.e., require more of the drug to get the same effect)	1.00/ 0.80/ 0.90	Relevant/ Relevant	No specific suggestions	─	─	Retained as original
9. To what extent do you think it is likely that in general in Australia, women are MORE likely to experience a mental illness of any kind compared to men	0.82/ 0.10/ 0.50	Relevant with minor alternation / Relevant with minor alternation	Clinical experts and some professionals stated that the question is relevant if the location ‘Australia’ would be changed. Some PREs mentioned that there is no epidemiological evidence from Zambia, and that’s why it is probably not relevant in this context. A global perspective would be relevant in absence of an epidemiological study, but specifying a mental illness, such as ‘depression,’ would clear it out.	x	Context changed to a global perspective.	To what extent do you think it is likely that in general women are MORE likely to experience some mental illnesses compared to men
10. To what extent do you think it is likely that in general, in Australia, men are MORE likely to experience an anxiety disorder compared to women	0.82/ 0.20/ 0.52	Relevant/ Relevant with minor alternation	Clinical experts and some professionals stated that the question is relevant if the location ‘Australia’ was changed or left blank. A PRE stated that in Zambia there is no epidemiological information regarding any mental health illness.	x	─	To what extent do you think it is likely that in general, men are MORE likely to experience an anxiety disorder compared to women
11. To what extent do you think it would be helpful for someone to improve their quality of sleep if they were having difficulties managing their emotions (e.g., becoming very anxious or depressed)	1.00/ 0.80/ 0.90	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
12. To what extent do you think it would be helpful for someone to avoid all activities or situations that made them feel anxious if they were having difficulties managing their emotions	0.91/ 0.50/ 0.71	Not clear/ Relevant	PREs: Not very clear, can be understood differently and needs minor alterations. Not ‘helpful information.’ No specific suggestion was made by PREs in terms of minor alterations. CE: Commented as relevant.	x	─	Retained as original.
13. To what extent do you think it is likely that Cognitive Behaviour Therapy (CBT) is a therapy based on challenging negative thoughts and increasing helpful behaviours	1.00/ 0.90/ 0.95	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
14. Mental health professionals are bound by confidentiality; however there are certain conditions under which this does not apply. To what extent do you think it is likely that the following is a condition that would allow a mental health professional to break confidentiality: If you are at immediate risk of harm to yourself or others	1.00/ 0.90/ 0.95	Relevant/ Relevant, but needs minor alteration	CE: The phrase ‘who is in risk of harm’ is not clearclear whether it refers to a patient or a nurse. One said it’s clear. The third said the meaning is more of ‘a professional [nurse].’ The term ‘you’ was substituted with the term ‘patient.’	x	The patient is at immediate risk to oneself or to others.	Mental health professionals are bound by confidentiality; however there are certain conditions under which this does not apply. To what extent do you think it is likely that the following is a condition that would allow a mental health professional to break confidentiality: If a patient is at immediate risk of harm to oneself or others
15. Mental health professionals are bound by confidentiality; however there are certain conditions under which this does not apply. To what extent do you think it is likely that the following is a condition that would allow a mental health professional to break confidentiality: if your problem is not life-threatening and they want to assist others to better support you	0.91/ 0.60/ 0.76	Relevant/ Relevant	As in item 14, the CE commented as relevant.	x	If a patient’s problem is not life-threatening and professionals want to assist others to better support a patient.	Mental health professionals are bound by confidentiality; however there are certain conditions under which this does not apply. To what extent do you think it is likely that the following is a condition that would allow a mental health professional to break confidentiality: If patient’s problem is not life-threatening and professionals want to assist others to better support a patient
16. I am confident that I know where to seek information about mental illness	1.00/ 1.00/ 1.00	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
17. I am confident using the computer or telephone to seek information about mental illness	1.00/ 0.90/ 0.95	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
18. I am confident attending face to face appointments to seek information about mental illness (e.g., seeing the GP)	1.00/ 0.90/ 0.95	Relevant/ Relevant	PREs discussed if these questions are testing knowledge, stigma and attitude, and said ‘all are relevant.’	─	─	Retained as original.
19. I am confident I have access to resources (e.g., GP, internet, friends) that I can use to seek information about mental illness	1.00/ 1.00/ 1.00	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
20. People with a mental illness could snap out if it if they wanted	0.91/ 0.50/ 0.71	Relevant, with minor alteration/ Relevant, but needs minor alteration	PRE: The meaning of ‘snap out’ isn’t clear. The term needs to be revised so it can’t be understood differently. The consensus is for ‘put themselves together’ instead of original term. CE: ‘Snap out’ refers to so many different mental health problems, so the ‘question is clear.’	x	─	People with a mental illness could put themselves together if they wanted
21. A mental illness is a sign of personal weakness	1.00/ 0.40/ 0.71	Relevant/ Relevant, but needs minor alteration	CE: Some of them are, some are not. A word like ‘some’ should be added.	x	─	Retained as original.
22. A mental illness is not a real medical illness	1.00/ 0.60/ 0.81	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
23. People with a mental illness are dangerous	1.00/0.75/ 0.89	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
24. It is best to avoid people with a mental illness so that you don’t develop this problem	0.91/ 0.30/ 0.62	Not clear/ Relevant	A PRE commented ‘meaning which problem?’ Common discussions were about the question not being quite clear. PREs discussed what this question is about, such as attitude or something else. They said, ‘I’m okay,’ ‘then it’s better change’ and ‘you don’t develop this problem.’ CE: This is relevant because there is stigma. ‘You can’t develop mental illness.’	x	─	It is best to avoid people with a mental illness so that you don’t catch their illness
25. If I had a mental illness I would not tell anyone	0.91/ 0.40/ 0.67	Relevant, with minor alternation/Relevant	PREs discussed the meanings of the words anyone, everyone and no one. The suggestion was to change it to ‘I would tell no one.’	x	─	If I had a mental illness I would tell no one
26. Seeing a mental health professional means you are not strong enough to manage your own difficulties	1.00/ 0.50/ 0.76	Relevant/ Not clear	One CE: ‘If you see a professional, it means that you are failing.’ One was unable to assess, and one said ‘not relevant,’ but the others said ‘relevant.’	x	The CE is reflecting personal attitudes rather than evaluating the item.	Retained as original.
27. If I had a mental illness, I would not seek help from a mental health professional	1.00/ 0.40/ 0.71	Relevant/ Relevant	No specific suggestions	x	─	Retained as original.
28. I believe treatment for a mental illness, provided by a mental health professional, would not be effective	0.91/ 0.30/ 0.62	Relevant/ Relevant	No specific suggestions	x	─	Retained as original.
29. How willing would you be to move next door to someone with a mental illness?	0.91/ 0.70/ 0.81	Relevant/ Relevant, but needs minor alteration	CEs: Unable to assess because it depends on what type of mental illness a neighbor has. Considering the ‘stigma,’ it’s relevant or ‘not relevant.’	─	─	Retained as original.
30. How willing would you be to spend an evening socialising with someone with a mental illness?	0.91/ 0.50/ 0.71	Relevant/ Not clear	CE’s: One said it’s ‘not relevant.’ Most said it needs a minor alteration and commented on ‘which type of an illness,’ such as bipolar disorder. One commented that it’s relevant.	x	─	Retained as original.
31. How willing would you be to make friends with someone with a mental illness?	1.00/ 0.80/ 0.90	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
32. How willing would you be to have someone with a mental illness start working closely with you on a job?	1.00/ 0.60/ 0.81	Relevant/ Relevant	CE: ‘What type of a job?’	─	─	Retained as original.
33. How willing would you be to have someone with a mental illness marry into your family?	0.91/ 0.50/ 0.71	Relevant/ Relevant	Most CEs commented that it is relevant.	x	─	Retained as original.
34. How willing would you be to vote for a politician if you knew they had suffered a mental illness?	1.00/ 0.70/ 0.86	Relevant/ Relevant	PRE: ‘I wonder if this really relevant. Your knowledge or your attitude against psychiatry, it’s a personal thing you cannot change.’ A comment was made about if a nurse’s opinion on this topic is really needed. PREs commented that it’s not a matter of a single question but the whole scoring on the scale. CE: ‘Very relevant.’	─	─	Retained as original.
35. How willing would you be to employ someone if you knew they had a mental illness?	1.00/ 0.70/ 0.86	Relevant/ Relevant	No specific suggestions	─	─	Retained as original.
Mean I-CVI (S-CVI/Ave)	0.95/ 0.62/ 0.80

Polit DF, Beck CT. The content validity index: are you sure you know what’s being reported? critique and recommendations. Res Nurs Health. 2006;29:489–497.

 PubMed Web of Science ®Google Scholar