Evaluating shortlisting methods for admission into the social work degree: personal statement versus psychological test

Figures & data

Table 1. SWM and PS measures compared key criteria for good quality psychological scales.

Criteria for good quality	Social Work Match	Personal Statement
Discriminatory power	Continuous measure	Ordinal ranking
Reliability (consistency)	High	Acceptable
Validity (measures intended construct)	Yes (good evidence it does so)	Questionable
Standardised
(On a population)	Yes	Yes
(Admin consistent)	Yes	Questionable

Table 2. Summary of evidence about validity of scores on the PS measure of suitability.

PS scores have: -	Evidence about validity of SW criteria scores using PS
Classic Concepts of validity
Content validity	Face and content validity. On the face of it the four criteria, written communication skills, SW motivation, understanding and values, are appropriate to a measure of suitability for social work even though limited in reach, unresearched and with no scientific basis. The evidence is that the measure is not one-dimensional making validity more difficult to establish—see findings at Table 3. Where construct validity is concerned one must ask ‘What construct is actually being assessed.?’ And then ‘How well has that been done?’ We had surmised that the answer to the first question Is one construct ‘suitability for a career in social work’ but there are concerns about how well it has been assessed. The process of administration, particularly the opportunity afforded to seek advice and guidance on content and style, created uncertainty around authorship and integrity of content. Where was the true voice of the applicant to be found? The marking process exposed further concerns about validity. The criteria are described in broad terms, not clearly defined, leaving room for interpretation, subjective perception and scoring by markers. What is an accurate mark of suitability?
Construct validity
Face validity
Unitary Concept of validity: elements
Credibility	Scoring is indirect inferential and subjective: importance and emotional significance of criteria must be inferred from text judged and scored from 0–6 (skills 0–2).
Fitness	The criteria appear relevant to, suitable for, SW but are described in broad terms leaving much room for interpretation and subjective perception.
Robustness	Procedure and calculations are systematic but simplistic with unrealistic expectations about integrity also error prone and lack robustness due to subjectivity in the scoring process
Reliability	Coefficient of Item Reliability is questionable (α = 0.668). Inter-marker reliability is weakened by broad ill-defined concepts, subjective interpretation and subjective scoring.
Integrity	Free text open response so potentially genuinely honest and moral. The PS seems to be marked as if this were so, blind to the plethora of advice online and offline about content and style with no constraint on its use, unfairly favoring those with greater resources or access to expertise. All this raising uncertainty about the true voice of the applicant
Representativeness	Written communication skills, SW motivation, understanding and values do represent personal characteristics required of competent are social workers
Coherence	The criteria combined do seem to come together as a concept of suitability for SW although factor analysis indicates two ‘hidden’ factors ≈ ‘social work attributes’ and ‘writing skills’.
Transparency	Poor. The PS is presented by the applicant but arrives pre-processed to an unknown extent via online guides and co- authors and is interpreted and scored as if plain truth.

*item analysis assumes a unidimensional measure and evidence suggests two dimensions: value of result is uncertain.

Table 3. Summary of evidence about validity of scores on the SWM measure of suitability.

SWM scores have	Evidence about validity of SW value scores using SWM
Classic concept of validity	Test content and construct validity—values characterizing ‘Suitability for SW’ – is based on a literature search, content analysis, codification, frequency of use count, trialing by SWs, advice and guidance from experienced SWs and a steering committee, re‐affirmed by feedback on its use from student social workers and statistically via item and factor analysis (unidimensional ‘Suitability’).
Content validity
Construct validity
Face validity
Unitary Concept of validity*
Credibility	Direct measure of importance and emotional significance of SW values
Fitness	Values and entities are fit for purpose by design (Classic Concepts)
Robustness	Procedure and calculations are systematic and inherently robust
Reliability	Provide reliable ‘snapshot’ of values at time of response
Integrity	SWM is easy to complete and understand: a genuine response = integrity
Representativeness	By design SWM values represent the domain of SW professional values
Coherence	Concepts are defined by algorithms (equations) for coherent and holistic analysis
Transparency	Data direct from respondent via interface response: automatic analysis/scoring

Table 4. Level 1 T1: Unrotated factor loadings and communalities.

Variable	Factor 1	Factor 2	Communality
VB	0.997	−0.018	0.994
SC	0.943	−0.210	0.934
RES	0.936	−0.294	0.962
CH	0.949	0.112	0.912
RSU	0.916	0.117	0.853
PS	0.916	0.305	0.931
Variance	5.3364	0.2498	5.5862
% Var	0.889	0.042	0.931

Figure 1. A scree plot of eigen values indicating a two factor solution.

Table 5. Level 1 T1: PS unrotated factor loadings and communalities.

Variable	Factor 1	Factor 2	Communality
Q1	0.769	−0.312	0.688
Q2	0.747	−0.213	0.604
Q3	0.879	0.119	0.787
Q4	0.316	0.929	0.964
Variance	2.0228	1.0206	3.0435
% Var	0.506	0.255	0.761

Figure 2. T1 Y1 plot of scores on PS and SWM measure (Vb).

Table 6. Students ranked by % change in VB scores on SWM test: T1 to T2.

VB Score T1 | T2			Change T2	% change	n=23
SUI 46	47.34	66.89	19.55	41.30	UGR
9	58.67	76.95	18.29	31.16	RGR
22	55.61	69.85	14.25	25.61	RGR
29	60.97	74.35	13.37	21.95	UGR
34	58.03	70.63	12.6	21.71	UGR
02	69.51	84.18	14.67	21.10	RGR
25	50.76	60.78	10.03	19.74	UGR
35	52.91	63.19	10.28	19.43	UGR
19	60.63	71.96	11.33	18.69	UGR
23	64.4	71.99	7.59	11.79	UGR
24	70.47	76.98	6.51	9.24	RGR
10	55.26	60.36	5.09	9.23	UGR
33	71.16	75.39	4.23	5.94	UGR
7	53.43	54.83	1.4	2.62	UGR
14	60.32	60.12	-0.2	-0.33	RGR
3	63.71	60.51	-3.2	-5.02	UGR
26	44.73	41.9	-2.82	-6.33	UGR
48	58.12	54.17	-3.95	-6.80	RGR
13	61.08	53.31	-7.77	-12.72	UGR
4	70.16	59.85	-10.31	-14.69	UGR
15	53.1	43.08	-10.02	-18.87	RGR
30	46.49	13.72	-32.77	-70.49	RGR
38	58.8	4.75	-54.06	-91.92	RGR
Mean	58.51	59.6	1.09	1.86
SD	7.59	19.1	11.31	149.01

Table 7. First Year T1: Pearson (r) matrices: scores on themes and value base (VB).

	Value base (VB)*	Professional Standards (PS)	Relationship with SU (RSU)	Character (CH)	Resilience (RES)
PS	.90
RSU	.92	.80
CH	.94	.88	.84
RES	.94	.78	.81	.86
SC	.94	.82	.82	.85	.90

Table 8. First year T1: spearman’s rho matrices: correlation coefficient of scores on PS criteria themes and overall PS score.

n = 24	Overall PS score (Total)	Motivation for social work Q1	Understanding of social work Q2	Social work values Q3
Q1	.80
Q2	.69	.42
Q3	.88	.55	.52
Writing Skill Q4	.28	.02	.07	.31

Figure 3. T1 Y1 plot of scores on PS and SWM measure (VB).

Figure 4. The distribution of scores changes as learning progresses T1 vs T2.

Figure 5. Distinguishability of candidates using the PS and SWM test.