Acceptability of employment readiness measures in youth and young adults on the autism spectrum: a mixed-methods study

Abstract

Purpose

Reliable, valid, and pragmatic measures are essential for monitoring and evaluating employment readiness and comparing the effectiveness of alternative implementation strategies. The Work Readiness Inventory (WRI) and Ansell–Casey Life Skills Assessment (ACLSA) are valid measures of employment readiness in neurotypical populations; however, their acceptability (i.e., user perception of measure as agreeable/satisfactory) for persons on the autism spectrum is not yet known. This investigation assesses the acceptability of the WRI and a modified ACLSA (ACLSA-M) in measuring employment readiness in youth/young adults on the spectrum.

Methods

A concurrent triangulation mixed-methods study design utilizing quantitative pre-post measurement of a community-based employment readiness program alongside qualitative survey assessment was employed to determine concurrent acceptability. For robustness, further explication through peer debriefing of experts evaluated the retrospective acceptability via interview and acceptability-rate assessment.

Results

Findings indicated that both measures are acceptable, although individual- and job-specific item modifications are advised, particularly due to disability-specific needs. Significant change in employment readiness in youth/young adults on the spectrum supports concurrent acceptability. Peer debriefing provided rich data on retrospective acceptability. Acceptability-rates of 0.84 and 0.91 confirm broad acceptability of these measures.

Conclusions

Implications are presented for clinicians and researchers, highlighting the relevance for autism-specific measurement development and acceptability.

Implications for rehabilitation

• Given the lower labor force participation of persons on the autism spectrum, a combination of measures should be used in the assessment of an individual’s employment readiness.

• In youth and young adults on the spectrum, employment readiness can be measured using the Work Readiness Inventory (WRI) and a modified version of the Ansell–Casey Life Skills Assessment (ACLSA-M).

• In clinical practice and research, modifying the contents of these measures may be advised to minimize language complexity, and maximize ease in self report.

• When designing, developing, and testing new measures in rehabilitation practice or research, the intent should be broadened by involving diverse representation from the project outset, by engaging both those on the spectrum and neurotypical populations.

Keywords:

• Autism spectrum disorder
• employment readiness
• youth
• young adults
• measurement

Introduction

The employment rates for working-age individuals identified as having autism spectrum disorder (autism) are the lowest of any disability category in Canada, with only 21.5% of people 15–64 years of age on the spectrum engaged in the labor force [1]. This low labor force participation (working or seeking work) is a trend that is seen internationally with employment rates ranging from 10% to 50% [2,3], despite this neurodiverse population having important skills to contribute to the workforce. The neurodiversity paradigm highlights how neurological differences are valuable and result from normal and natural variation, rather than disease or disorder, and can have desirable and enabling consequences both for the individual and society [4,5].

Persons on the autism spectrum (herein referred to as “the spectrum”) experience significant barriers in the workplace and education, as well as in accessing social services and supports, which may limit labor force participation [6–9]. Unemployment for this group is associated with lower financial security, more limited independence, less community participation, and diminished self-esteem, often resulting in poorer quality of life and well-being [10]. Employment among persons on the spectrum is linked with the broader “ecosystem” comprised of community resources, family support, workplace capacity building (e.g., employer, co-workers), and supportive policy [11]. Thus, supports for employment may be limited if focused only on singular aims such as individual preparedness.

Community-based vocational resources and employment supports can improve employment success for adults on the spectrum [12]; however, what predicts success appears to be a complex combination of factors, including employment readiness [10,13,14]. Understanding employment readiness and how it intersects with other factors leading to meaningful employment is a critical step toward increasing employment outcomes for persons on the spectrum [15–17]. Employment readiness encompasses an individual’s level of employment skills, core life skills (including social and self-management skills), and daily living skills that are all linked closely to an individual’s overall sense of well-being [12,13]. Self-development, occupational focus and action, navigation of work life, and personal well-being and health all contribute to employment readiness [18].

Neurodiverse persons on the spectrum often experience challenges with social interaction and communication [19–22], repetitive behaviors [23], imaginative abilities [5], sensory difficulties [5,24,25], and restricted interests [26]. Thus, persons on the spectrum can benefit from support that fosters the development of relevant and appropriate “soft skills” in areas like relationships and communication, work and study life, self-care, and career and life planning [27]. However, a 2018 evaluation of employment support services indicated that programs appear to generally target specific work preparation tasks (such as resume writing) and not employment-related issues specific to these areas [27].

The assessment of employment support program success and individual employment readiness is often based on the use of participant skill assessments developed for use with neurotypical populations. While numerous measures have been developed and validated to evaluate employment readiness and life skills, many of these, including career-planning and employment readiness measures, have not yet been assessed for use with persons on the spectrum [28,29]. The Work Readiness Inventory (WRI) and Ansell–Casey Life Skills Assessment (ACLSA) are two valid and reliable measures of employment readiness and life skills which were developed for use in neurotypical populations [30,31]. As a hallmark for a strong measure of employment readiness, it should be easy to administer, and present minimal burden on the individual completing it. The measure must also be acceptable such that users perceive it as “agreeable, palatable, or satisfactory,” considering key elements such as content, complexity, and comfort [32,p.67]. Acceptability is a measure of the implementation outcome, which helps determine the quality of use of particular practices [28] as well as a precondition for attaining desired service delivery outcomes [32].

The WRI and a modified version of the ACLSA (ACLSA-M) were used in a community-based vocational intervention called Employment Works Canada (EWC) that was offered to youth and young adults on the spectrum. EWC is a Canada-wide employment readiness program for youth and young adults (15–29 years of age) with autism that focuses on social, communication, and job skills development [33]. Using a broad-based approach, EWC characterizes work readiness within components of self-development, occupational focus and action, navigation of work life, and personal well-being and health [18]. In this context, the ACLSA was modified to specifically capture life skills specific to employment readiness – social skills, well-being (including mental health), and other generalized life skills that may be required for employment of those on the spectrum. While both the ACLSA and WRI are reliable and valid measures in neurotypical populations, the acceptability is unknown among autistic youth. This study aims to answer the following question: are the WRI and ACLSA-M acceptable measures of employment readiness in youth and young adults on the spectrum?

Methods

According to Proctor et al. [32], acceptability should be assessed from participant and/or provider perspectives, considering either knowledge of or direct experience with various dimensions of the practice, or in this case, the measures, implemented. The literature supports both quantitative and qualitative methods of assessing healthcare implementation outcomes, including acceptability [28,34,35]. Further, acceptability can be assessed prospectively (before implementation), concurrently (during implementation), and retrospectively (after implementation) [35]. According to research on evaluating and measuring implementation outcomes such as acceptability, methods ought to be defined by the social purpose of the investigation [34].

Participants

Concurrent acceptability

Participants were included in the quantitative assessment of concurrent acceptability when they met the following eligibility criteria:

• EWC program enrolment. Enrolment criteria for EWC included a diagnosis of autism, being unemployed or underemployed, struggling to get and/or keep a job, seeking an opportunity to build workplace skills, openness to exploring different workplaces, age range of 15–29 years, and willingness to engage in a three-month manualized program for five hours per week.

• Provision of informed consent (including sufficient comprehension to provide informed consent).

• An age-equivalency (AE) score of at least 13 years on the Peabody Picture Vocabulary Test-4 (PPVT-4) as administered and evaluated by EWC facilitators.

Participants were excluded when the WRI and ACLSA-M were not completed at the beginning and end of the intervention.

We randomly selected two participants from those who participated in the quantitative assessment for further qualitative assessment. These participants were identified via criterion sampling [36] using the criteria that their cases indicated improved work readiness on the WRI and improved life skills indicated by the ACLSA-M domain changes. By selecting a small number of individuals with information-rich cases related to employment readiness, the phenomenon of interest, we were able to gain more detail around the nature of these changes to understand relevance to the measures [36,37].

Peer debriefing for retrospective acceptability

The selection of peer debriefers is a key to enhancing trust, credibility, and strength in findings. Of primary importance is the inclusion of participants who are meaningfully engaged in the fields of employment readiness and life skills assessment and are able to purposefully contribute to the validity of the findings [38]. In this case, peer debriefing included assessment of retrospective acceptability of the WRI and ACLSA-M in measuring employment readiness in youth and young adults on the spectrum, through key informant stakeholder interviews and survey assessment. Participants were recruited through purposive and snowball sampling and were selected for inclusion when identified as an expert in the area of employment readiness for youth and young adults on the spectrum. Source of expertise was determined through scholarly publications, research contributions, clinical experience, and/or other recognized and documented contributions in the areas of employment readiness, vocational training, or employment supports for youth and young adults on the spectrum. Peer debriefing participants were recruited through existing researcher networks and the Canadian Autism Spectrum Disorder Alliance (CASDA) 2018 Leadership Summit.

Setting

To understand the concurrent acceptability of these measures in practice, the WRI and ACLSA-M were used as pre-post measures for employment readiness in the EWC program. EWC is a Canada-wide employment readiness program for youth and young adults (15–29 years of age) on the spectrum that focuses on social, communication, and skill development to improve employment readiness, support occupational selection precision, and provide expansive work exposure [39]. Using a broad-based approach, EWC characterizes employment readiness within components of self-development, occupational focus and action, navigation of work life, and personal well-being and health [18].

In combination with mentorship-based training and workplace exposure, programming is tailored to individual goals and offers structured learning opportunity. Capacity building is targeted not only to participants, but also to employers, co-workers/peers, and the community at large. Domains of learning in weekly sessions, as outlined in a manualized curriculum, address the following topics: career exploration and goals, health and financial literacy, communication/socializing skills within the work environment, well-being and self-esteem, adaptive skills, and peer mentor/co-worker autism awareness [11].

Participants were provided the WRI and the ACLSA-M at the beginning and at the end of the EWC program and were asked to complete the measures. Facilitators were available to answer questions or provide guidance for completion if needed.

Instruments

The WRI is a brief self-report that assesses employment readiness in six areas: responsibility, flexibility, skills, communication, self-view, and health and safety with 36 questions [30]. Respondents rate their level of concern in these areas relative to the workplace on a five-point Likert scale, one being “not concerned” and five being “very concerned.” Scores on the WRI range from 36 to 180, with lower scores indicative of less concerns or perceived areas of strength, and higher scores indicative of increased concerns related to employment readiness. Used widely with neurotypical populations, the WRI focuses on “those personal attributes, worker traits, and coping mechanisms needed to not only land a job, but to keep that job” [30,p.4]. The WRI exhibits strong content validity (complete construct-to-item fit, Fleiss’ kappa, K = 1) [40] and concurrent criterion validity (mean Pearson’s r = −.21, p < 0.05) [30], as well as internal consistency (median Spearman–Brown correlation coefficient r = 0.94, p < 0.001) [30] and test–retest reliability (median Pearson’s r = 0.87, p < 0.001) when tested with neurotypical populations [30].

The ACLSA is a strengths-based multidimensional instrument of capabilities, and assesses youth life skills in the areas of daily living, self-care, relationships and communication, housing and money management, work and study, career and education planning, and looking forward [41,42]. The ACLSA is widely used in the identification of developed life skills and for goal setting to acquire new skills [31,43,44]. The ACLSA was modified (ACLSA-M) for use in the EWC program, honing in on generalized life skill abilities that are potential prerequisites to employment for those on the spectrum (and others), and skills directly relevant to the workplace. At the time of writing, the EWC program was the only program, to our knowledge, to use the ACLSA modified in this format. The skills underlying the items chosen for inclusion in the ACLSA-M include employment readiness, occupational focus and action, autism presentation (including social skills and behavior), and well-being (including mental health). The ACLSA demonstrates face, content, construct, and concurrent criterion validity as well as internal consistency (Cronbach’s alpha = 0.97), and test–retest reliability in neurotypical populations [31,45]. The modified ACLSA (ACLSA-M), however, has not been tested for validity or reliability.

Study design

For robustness, our methodology explicitly included triangulation through quantitative and qualitative methodologies amalgamated in a complex mixed-method design selected to assess acceptability both from concurrent perspectives and retrospective perspectives. Using a concurrent triangulation mixed-methods design, we first investigated the concurrent acceptability of these measures in inferring findings about employment readiness quantitatively (using pre-post measurement) alongside qualitative data collection to elaborate on the construct of employment readiness relative to persons on the spectrum. The concurrent triangulation design is the most common and well known mixed-methods study design [46], and its purpose is “to obtain different but complementary data on the same topic” [47,p.122] to best understand the research problem. As represented in Figure 1, the concurrent triangulation research design allows for the collection and analysis both of qualitative data and quantitative data to assess the same conceptual phenomenon simultaneously [48]. While data analysis typically occurs separately, the qualitative results and quantitative results are integrated in the data interpretation stage [46]. Concurrent triangulation designs are useful for attempting to confirm, cross-validate, and corroborate study findings from different methods, often adding to the depth and scope of the findings [46,49].

Figure 1. Visual model for mixed-methods concurrent triangulation design procedures.

Second, to strengthen the validity of the findings, further explication through peer debriefing occurred whereby key informant stakeholders (i.e., employment readiness experts) were interviewed to evaluate the retrospective acceptability of these measures via the acceptability rate of the measures and their components by quantification of survey assessment (Figure 2). Peer debriefing is often used to improve and demonstrate the rigor of qualitative research [47,50–53]. A well-established approach in ascertaining trustworthiness in qualitative inquiry, peer debriefing is methodologically implemented, based on the input of key experts external to the research process in eliciting their views of data resonance [38]. This process further serves to “deepen understanding by collecting a variety of data on the same topic or problem with the aim of combining multiple views or perspectives and producing a stronger account rather than simply achieving consensus or corroboration” [38,p.12–13].

Figure 2. Study methods consisting of a concurrent triangulation design followed by peer debriefing.

The study was reviewed and approved by the University of Calgary’s Conjoint Faculties Research Ethics Board (REB#15-0019, 17-2353). Secondary analysis of the anonymized program evaluation/quality improvement-oriented dataset was completed after identifying features in the data had been removed. Informed consent was received from all interview participants prior to data collection.

Data collection procedures

Concurrent acceptability

To assess concurrent acceptability of the WRI and ACLSA-M in capturing employment readiness quantitatively, participants were provided the WRI and the ACLSA-M at the beginning and at the end of the EWC program and were asked to complete the measures. Facilitators were available to answer questions or provide guidance for completion if needed. To further assess the concurrent acceptability of the WRI and ACLSA-M for use in capturing employment readiness, a review of post-EWC surveys, which contained the perspectives of the participants and their families, was conducted. These post-program surveys were provided to participants and parents and included ratings and free text. During the post-program survey, EWC participants and their parents were asked questions related to their experience in the EWC program, the phenomenon of employment readiness, and the measures utilized to capture changes in employment readiness.

Peer debriefing for retrospective acceptability

First, we utilized semi-structured one-on-one interviews to assess retrospective acceptability. Qualitative assessment was based on stakeholder knowledge of and direct experience with all dimensions of the measures to report on the perceived retrospective acceptability of the measures. Interviews occurred via telephone and were audio-recorded and transcribed verbatim. Participants were asked to discuss the relevance, clarity, and acceptability of the measures as well as the meaning and degree to which individual items represented acceptable content, complexity, and comfort of domains.

Next, participants were also asked to rate each item on the measure as relevant in reflecting the underlying construct of employment readiness using a four-point Likert scale, from not relevant (1), somewhat relevant (2), quite relevant (3), to highly relevant (4). The item rating allowed for the quantification of participant assessment and proportion agreement on acceptability of the measure items.

In addition to the WRI and ACLSA-M, participants were also provided the full unmodified ACLSA and asked to rate each item as they had done previous. The inclusion of the full measure in the assessment of retrospective acceptability led to the determination of acceptability-rates as well as the identification of potential items on the ACLSA relevant to employment readiness for youth and young adults on the spectrum that had not been included in the ACLSA-M.

Data analysis

Concurrent acceptability

The assumption of normalcy was tested both for the WRI data and the ACLSA-M data. To evaluate the concurrent acceptability of the WRI paired sample t-tests were conducted for each domain of the WRI, using a Bonferroni adjusted alpha level of 0.008 to correct for multiple comparisons. For concurrent acceptability of the ACLSA-M, paired samples t-tests were conducted for each domain using a Bonferroni adjusted alpha level of 0.007 to correct for multiple comparisons. During the post-program survey, EWC participants and their parents were asked questions related to their experience in the EWC program, the phenomenon of employment readiness, and the measures utilized to capture changes in employment readiness.

Peer debriefing for retrospective acceptability

Using a six-step framework [54], interview transcripts were inductively analyzed with the support of qualitative data management and analysis software (NVivo 11). This analysis framework included the following stages: step 1: become familiar with the data; step 2: generate initial codes; step 3: search for themes; step 4: review themes; step 5: define themes; and step 6: write-up [54]. Data were coded line-by-line, with the interconnectedness of codes examined and salient themes extracted. Both semantic and latent themes were observed and analyzed, which allowed for recognition of the full diversity of the data.

Likert scale ratings were anonymized and pooled by item. The inter-rater proportion agreement for each item was calculated as the proportion in agreement of acceptability (those giving a rating of either 3 or 4) divided by the total number of experts according to the Lynn method [55]. To address the potential inflation of values due to random chance, we linked these values to a modified kappa statistic [56,57]. The modified kappa was calculated for 10 participants and compared to the standards for evaluating kappa by Fleiss [58] and Cicchetti and Sparrow [59]. After adjusting for chance, any item with a value exceeding 0.79 was deemed acceptable. Once the inter-rater proportion agreement for each item was calculated, the measures themselves were then assessed. The measures were deemed acceptable if the proportion of items from each measure surpassed 0.80 [55,60].

Results

Concurrent acceptability

Between April 2017 and April 2018, a total of 177 participants took part in the EWC program in seven provinces across Canada (British Columbia, Alberta, Saskatchewan, Ontario, Quebec, Newfoundland, and Nova Scotia). Participants were between 17 and 29 years of age (M = 21, SD = 3), 73% were male and 94% indicated their primary language was English. At the beginning of the program, 12% of participants were employed, and 92% indicated that they were interested in obtaining employment. Of this total, 134 participants received a PPVT AE of at least 13. Among these eligible participants, 90% (n = 121) completed the WRI and 84% (n = 112) completed the ACLSA-M at the beginning and at the end of the intervention. Data from both measures were normally distributed. There was a significant decrease in all of the WRI pre- to post-test domain scores indicating increases in employment readiness (p < 0.05) (Table 1). On the ACLSA-M, participants on the spectrum had significantly more positive scores in the domains directly related to employability: (i) relationships and communication, (ii) work and study, (iii) career and education planning, and (iv) looking forward (p < 0.05) (Table 2). Although the focus of this investigation was not on intervention effectiveness, we additionally found that Cohen’s d statistics for the WRI ranged from 0.25 to 0.28, suggestive of a small effect [61]. Cohen’s d statistics on the ACLSA-M ranged from 0.29 to 0.67, suggestive of a small to medium effect [61].

Table 1. Concurrent acceptability pre- and post-test scores on the Work Readiness Inventory (n = 121).

Area (score)	Pre-test		Post-test		p	Cohen's d
	M	SD	M	SD
Responsibility (6–30)	13.1	6.02	11.9	5.59	0.006*	0.26^a
Flexibility (6–30)	15.6	5.58	14.4	5.08	0.003*	0.27^a
Skills (6–30)	14.7	6.23	13.1	5.67	0.003*	0.28^a
Communication (6–30)	14.8	5.73	13.4	5.75	0.003*	0.28^a
Self-view (6–30)	15.6	5.90	14.1	5.63	0.002*	0.28^a
Health and safety (6–30)	12.3	6.15	11	5.24	0.007*	0.25^a

*p < 0.05.

^a Small effect size.

Table 2. Concurrent acceptability pre- and post-test scores on the modified Ansell–Casey Life Skills Assessment (n = 112).

Area (score)	Pre-test		Post-test		p	Cohen's d
	M	SD	M	SD
Daily living (/20)	18.50	2.10	18.60	2.18	0.442	–
Self-care (/20)	17.00	2.53	17.20	2.60	0.248	–
Housing and money management (/5)	4.23	1.08	4.24	1.07	0.929	–
Relationship and communication (/30)	23.9	4.32	25.1	3.96	0.001*	0.33^a
Work and study life (/70)	53.00	9.08	57.20	8.97	0.000*	0.54^b
Career and education planning (/35)	21.40	6.39	25.30	6.34	0.000*	0.67^b
Looking forward (/40)	30.00	6.63	31.50	6.83	0.002*	0.29^a

*p < 0.05.

^a Small effect size.

^b Medium effect size.

Participant 1, an 18-year-old male, had not looked for work in the four weeks preceding the program. He indicated “past unsuccessful job-finding experiences” and “no adequate training/experience” were barriers which discouraged him from actively searching for a job. Upon completion of the program, the participant’s perceptions of his communication abilities and self-view improved. On the ASCLA-M positive changes were noted in the four domains of life skills related to employment readiness: relationship and communication, work and study, career and education planning and looking forward. His parents observed positive changes in his social communication skills and mentioned:

Since doing the program, he is able to speak more confidently to people. At the end of the program, he was able to get up and speak in front of (the) group. Before [doing EWC], he would have never been able to do this. He would never speak or start a conversation, but he is now able to feel comfortable going into a bank for instance and opening an account.

Shifts in the participant’s self-view of his communication were noted on both the WRI and ACSLA-M including an increased ability to speak up for himself and perceived ability to get along with his coworkers. His parents further stated:

With EWC we are both excited and have a new outlook on his life and work. He now has the right tools and confidence he needs to get a job such as interview skills, job portfolio, and what employers expect of him.

On the WRI, the participant felt he was better able to fit in with his co-workers and be successful. At the three-month follow-up, he had obtained a general labor position at a non-profit organization. Of note, his scores appeared consistent with his reported experiences and perspectives related to employment.

The second participant, a 24-year-old female, indicated at the beginning of the program that she had looked for work during the last four weeks and would continue to look for work over the course of the program. All of her scores on the WRI improved indicating that she perceived herself as more work ready at the end of the program. At the end of the program, she indicated she was proud to accept “guidance” and “complete tasks” – items directly explored on the ACLSA-M. She indicated she “learned new skills” and indicated that she was “more prepared to find a job” and her parents stated, “I noticed her being more independent … she is ready to (take) transit to a paid job.” Shifts on the WRI Skills domain and the ACLSA-M work and study life domain reflected these sentiments. At the end of the program, she had obtained a position as a kitchen assistant in a restaurant and remained in this position at the three-month follow-up. Again, as indicated, the measure scores complemented the reported experiences of this individual with autism.

Peer debriefing for retrospective acceptability

Participants included 10 experts from across Canada who were recruited from Alberta (4), Ontario (3), Quebec (2), and Saskatchewan (1). Among participants, 30% were male, 70% female. All had extensive experience in the area of autism and employment for those on the spectrum. Forty percent had affiliation with a university, 60% were involvement in leadership roles or direct service provision with 7–30+ years of experience, and 40% had journal articles published in high caliber peer-reviewed journals.

Thematically, the qualitative data extracted were classified into three main categories, which identified areas for improvement of the two measures: (1) item-specific, (2) individual-specific, and (3) job-specific.

Item-specific

Stakeholders highlighted the relevance of the content and the clarity of item wording (i.e., complexity). They asserted that to be effectively interpreted by youth and young adults on the spectrum, many items needed nuancing and refinement. For example, an item on the ACLSA-M asks respondents to rate the following item, “I know how to use public transportation to get to where I need to go.” A participant wondered: “Are you assessing their ability to navigate using whatever means or assessing their ability to get to a workplace?” Enhancing clarity was recommended.

Items with abstract concepts or lacking specificity and/or context were also flagged as potentially problematic. It is generally more difficult for those on the spectrum to conceptualize any vagueness, and therefore may answer inaccurately. Ultimately, deconstruction of the items was a recurring theme among participants. To illustrate, a participant advised:

Dive deep a little bit more in some of the questions like ‘I can deal with anger without hurting others or damaging things’ like ‘I can recognize when I am feeling angry’ or you know ‘I know how to walk away’ or you know like breaking that down a little more.

Individual-specific

Assessment of individual characteristics like self-view and self-confidence was seen by participants to be important; however, the lack of items related to skills specific to the individual on the spectrum, such as emotional regulation, mental health, and dealing with anxiety, was raised as a concern. These skills were noted to commonly need further development among individuals on the spectrum. The lack of communication skills, social skills, and relational skills of youth and young adults on the spectrum often result in them feeling isolated, alone, or without friends. To illustrate, one participant said: “A lot of folks…if they do get the job the reason they’ll lose it is social.” Another stated the importance of context:

That social communication piece in the workplace…what topics are, you know, workplace safe to talk about, what topics, you know, are not safe to talk about in the workplace because they’re more like, hanging out with your friends topics.

To include items related to social skills, abilities and behavioral responses to new, different, and/or frustrating circumstances were seen as a positive direction in moving forward. Along these lines, a lack of assessing abilities to manage mental health was also a commonly identified need for inclusion.

Job-specific

The majority of participants noted a general need for more focus on the person-job fit, as many of the skills assessed were not necessarily needed for all jobs. One participant wondered: “There will be a lot of folks who try for jobs that they don’t need to create and save and print and send a document. Does this mean they’re not work ready?” There were participants who noted not only the importance of including more focus on the person-job fit, but also unique environmental factors such as group noise, distractions, and the management of job-specific situations.

Overall, all participants agreed that the measures were clear and readable. The content of the assessments was consistently understood. Specificity and clarity around the items being addressed, individual circumstances, and job circumstances were described as important considerations when administering these tools in employment readiness programs for persons on the spectrum. Stakeholders believed that modifications to the tools may improve their ability to meaningfully measure employment readiness in youth and young adults on the spectrum. A common recommendation was for the items in the ACLSA-M and WRI to be more explicit. The lack of specificity in certain items resulted in a number of questions regarding the context of the item or requiring further elaboration, with participants recommending slight alterations to clarify any ambiguity resulting from broad interpretations of various items on the ACLSA-M and/or WRI. According to participants, more granularity would provide enhanced clarity concerning the tangible capacities being assessed. Examples of recommended item insertion include facets of accountability (owning up to mistakes and taking responsibility for one’s actions), job satisfaction (expectations), and personal reflection.

Calculation of acceptability rates allowed a deeper assessment of retrospective acceptability. Based on the pre-defined criteria of a value of 0.80 for measure acceptability, the ACLSA-M and WRI were judged acceptable by participants with acceptability rates of 0.84 and 0.91, respectively (Tables 3 and 4). At the item-level, 88.9% (32) of the WRI items, and 75.0% (33) of the ACLSA-M items were judged to be acceptable with an inter-rater proportion agreement acceptability rate for items greater than or equal to the pre-defined 0.79. While both measures were determined to be acceptable in reflecting employment readiness, participants noted individual items that were less useful (Tables 3 and 4). Critical examination and comparison between the ACLSA and the ACLSA-M provided a number of items reflective of employment readiness (with acceptability rates greater than or equal to 0.79) that were not included at the initial modification of the scale to the ACLSA-M (Tables 3 and 5).

Table 3. Acceptability rate based on inter-rater proportion agreement per item on the Ansell–Casey Life Skills Assessment (ACLSA) and modified Ansell–Casey Life Skills Assessment (ACLSA-M).

Section	Item	Does item appear on modified (M) and/or full (F) assessment?	Acceptability rate	Acceptable?^a
Daily living	I know where to go to get on the internet	M, F	0.7	N
	I can find what I need on the internet	M, F	0.9	Y
	I know how to use my email account	M, F	0.9	Y
	I can create, save, print and send computer documents	M, F	0.9	Y
	I know the risks of meeting someone in person that I met online	F	0.7	N
	I would not post pictures or messages if I thought it would hurt someone’s feelings	F	0.8	Y
	If someone sent me messages online that made me feel bad or scared, I would know what to do or who to tell	F	0.8	Y
	I know at least one adult, other than my worker, who would take my call in the middle of the night if I had an emergency	F	0.8	Y
	An adult I trust, other than my worker, checks in with me regularly	F	0.8	Y
	When I shop for food, I take a list and compare prices	F	0.2	N
	I can make meals with or without using a recipe	F	0.5	N
	I think about how what I eat impacts my health	F	0.6	N
	I understand how to read food product labels to see how much fat, sugar, salt, and calories the food has	F	0.4	N
	I know how to do my own laundry	F	0.6	N
	I keep my living space clean	F	0.7	N
	I know the products to use when cleaning the bathroom and kitchen	F	0.6	N
Self-care	I know how to use a fire extinguisher	F	0.6	N
	I can take care of my own minor injuries and illnesses	F	0.6	N
	I can get medical and dental care when I need it	F	0.9	Y
	I know how to make my own medical and dental appointments	F	0.5	N
	I know when I should go to the emergency room instead of the doctor’s office	F	0.7	N
	I know my family medical history	F	0.3	N
	I know how to get health insurance when I am older than 18	F	0.4	N
	I have at least one trusted adult who would visit me if I were in the hospital	F	0.8	Y
	There is at least one adult I trust who would be legally allowed to make medical decisions for me and advocate for me if I was unable to speak for myself	F	0.7	N
	I know how to use the benefits I am eligible for	F	0.6	N
	I bathe (wash up) daily	M, F	1	Y
	I brush my teeth daily	M, F	1	Y
	I know how to get myself away from harmful situations	M, F	1	Y
	I have a place to go when I feel unsafe	M, F	0.9	Y
	I can turn down a sexual advance	F	0.9	Y
	I know ways to protect myself from sexually transmitted diseases (STDs)	F	0.6	N
	I know how to prevent getting pregnant or getting someone else pregnant	F	0.6	N
	I know where to go to get information on sex or pregnancy	F	0.6	N
Relationships and communication	I can speak up for myself	M, F	1	Y
	I know how to act in social or professional situations	M, F	1	Y
	I know how to show respect to people with different beliefs, opinions and culture	M, F	1	Y
	I can describe my racial and ethnic identity	F	0.4	N
	I can explain the difference between sexual orientation and gender identity	F	0.1	N
	I have friends I like to be with who help me feel valued and worthwhile	M, F	0.9	Y
	I am part of a family and we care about each other	F	0.8	Y
	I can get in touch with at least one family member when I want to	F	0.9	Y
	I have friends or family to spend time with on holidays and special occasions	F	0.8	Y
	I know at least one adult I can depend on when I exit care	F	0.8	Y
	I know an adult who could be a grandparent, aunt or uncle to my children now or my future children	F	0	N
	My relationships are free from hitting, slapping, shoving, being made fun of, or name calling	F	0.9	Y
	I know the signs of an abusive relationship	F	0.8	Y
	I know what my legal permanency is	F	0.2	N
	I have information about my family members	F	0.3	N
	I think about how my choices impact others	M, F	0.9	Y
	I can deal with anger without hurting others or damaging things	M, F	1	Y
	I show others I care about them	F	0.8	Y
Housing and money management	I understand how interest rates work on loans or credit purchases	F	0.2	N
	I understand the disadvantages of making purchases with my credit card	F	0.4	N
	I know the importance of a good credit score	F	0.4	N
	I know how to balance my bank account	F	0.8	Y
	I put money in my savings account when I can	F	0.6	N
	I know an adult who would help me if I had a financial emergency	F	0.9	Y
	I use online banking to keep track of my money	F	0.4	N
	I know the advantages and disadvantages of using a check cashing of payday loan store	F	0.4	N
	I know how to find safe and affordable housing	F	1	Y
	I can figure out the costs to move to a new place, such as deposits, rents, utilities and furniture	F	0.6	N
	I know how to fill out an apartment rental application	F	0.5	N
	I know how to get emergency help to pay for water, electricity, and gas bills	F	0.7	N
	I know what can happen if I break my rental lease	F	0.6	N
	I can explain why people need renters or homeowner’s insurance	F	0.2	N
	I know an adult I could live with for a few days or weeks if I needed to	F	0.9	Y
	There is at least one adult that I have regular contact with, other than my case manager or other professional, who lives in stable and safe housing	F	0.8	Y
	I know an adult I can go to for financial advice	F	0.7	N
	I plan the expenses that I must pay each month	F	0.7	N
	I keep records of the money I am paid and the bills I pay	F	0.6	N
	I know what happens in my state/province if I am caught without car insurance or a driver’s license	F	0.6	N
	I can explain how to get and renew a driver’s license or state/province ID card	F	0.5	N
	I can figure out all the costs of car ownership such as registration, repairs, insurance, and gas	F	0.2	N
	I know how to use public transportation to get where I need to be	M, F	0.8	Y
Work and study life	I know how to develop a resume	M, F	0.7	N
	I know how to fill out a job application	M, F	0.8	Y
	I know how to prepare for a job interview	M, F	0.8	Y
	I know what the information on a pay stub means	M, F	0.4	N
	I know what employee benefits are	M, F	0.6	N
	I know what sexual harassment and discrimination are	M, F	1	Y
	I know the reasons why my personal contacts are important for finding a job	M, F	0.8	Y
	I know how to get the documents I need for work, such as my Social Security/Social Insurance Card and birth certificate	F	0.8	Y
	I know how and when I can see child welfare or juvenile justice records	F	0.2	N
	I know an adult who will go with me if I need to change schools	F	0.6	N
	I know how to get help from my school’s mental health services	F	1	Y
	I know where I can get help with an income tax form	M, F	0.6	N
	I have an adult in my life who cares about how I am doing at school or work	M, F	1	Y
	I can take criticism and direction at school or work without losing my temper	M, F	1	Y
	I know how to prepare for exams and/or presentations	M, F	0.6	N
	I know where I can get tutoring or other help with school work	F	0.7	N
	I look over my work for mistakes	M, F	1	Y
	I get to school or work on time	M, F	1	Y
	I get my work done and turned in on time	M, F	1	Y
Career and education planning	I know how to find work related internships	M, F	0.4	N
	I know where to find information about job training	M, F	0.6	N
	I can explain the benefits of doing volunteer work	M, F	0.6	N
	I have recently talked to an adult who works in a job I would like to have	M, F	0.6	N
	I know what type (college, trade school) of education I need for the work I want to do	M, F	1	Y
	I know how to get into the school, training, or job I want	M, F	0.9	Y
	I know how to find financial aid to help me pay for my education or training	F	0.8	Y
	I have talked about my education plans with an adult who cares about me	F	0.8	Y
	I know an adult who will help me apply for training or education	M, F	0.9	Y
Looking forward	I believe I can influence how my life will turn out	M, F	1	Y
	I can describe my vision for myself as a successful adult	M, F	0.9	Y
	I have a good relationship with a trusted adult I like and respect	M, F	1	Y
	I would like to use my experience to help other youth	M, F	0.2	N
	I believe my relationships with others will help me succeed	M, F	0.8	Y
	I feel I am ready for the next phase of my life	M, F	0.9	Y
	Most days, I am proud of the way I am living my life	M, F	0.9	Y
	Most days, I feel I have control of how my life will turn out	M, F	0.9	Y

^a Item acceptable if acceptability rate >0.79.

Table 4. Acceptability rate based on inter-rater proportion agreement for the Work Readiness Inventory individual items and overall measure.

Item	Acceptability rate	Acceptable?^a
Finding a way to and from work	1	Y
Changing my work schedule	0.9	Y
Acquiring new skills	1	Y
Accepting feedback and criticism	0.9	Y
Feeling too old or young for a job	0.4	N
Complying with a drug-free/smoke-free workplace	0.8	Y
Working up to quitting time	1	Y
Taking on different job duties	0.9	Y
Knowing my own set of abilities	1	Y
Following directions	1	Y
Thinking I lack experience	0.6	N
Wearing/using safety gear or clothing	1	Y
Respecting tools and equipment	1	Y
Accepting changes in my position or job title	0.9	Y
Participating in employer training/retraining	1	Y
Getting along with fellow workers	1	Y
Believing I’ll be a failure	0.8	Y
Maintaining personal hygiene and grooming	1	Y
Following employer policies	1	Y
Relocating to meet employer needs	0.3	N
Identifying my strengths	1	Y
Letting my wants be known	0.9	Y
Having the confidence to do my work	1	Y
Following safety procedures using tools and equipment	1	Y
Meeting quality standards at work	1	Y
Adapting to a dress code or uniform policy	1	Y
Learning more about what I’m doing on my job	1	Y
Checking directions I don’t completely understand	1	Y
Believing I’m not smart enough	0.8	Y
Using proper procedures for lifting and bending	1	Y
Controlling waste and loss	0.7	N
Working overtime	0.9	Y
Feeling qualified to do my work	1	Y
Asking for help with my work	1	Y
Feeling I’ll never fit in	0.9	Y
Keeping physically fit and mentally alert	0.9	Y
Measure overall	0.91	Y

^a Item acceptable if acceptability rate >0.79, measure acceptable overall if acceptability rate >0.80.

Table 5. Items determined to be acceptable (with an acceptability rate >0.79) in assessing employment readiness on the Ansell–Casey Life Skills Assessment (ACLSA) that are not included on the modified Ansell–Casey Life Skills Assessment (ACLSA-M).

Section	Item	Acceptability rate
Daily living	I would not post pictures or messages if I thought it would hurt someone’s feelings	0.8
	If someone sent me messages online that made me feel bad or scared, I would know what to do or who to tell	0.8
	I know at least one adult, other than my worker, who would take my call in the middle of the night if I had an emergency	0.8
	An adult I trust, other than my worker, checks in with me regularly	0.8
Self-care	I can get medical and dental care when I need it	0.9
	I have at least one trusted adult who would visit me if I were in the hospital	0.8
	I can turn down a sexual advance	0.9
Relationships and communication	I am part of a family and we care about each other	0.8
	I can get in touch with at least one family member when I want to	0.9
	I have friends or family to spend time with on holidays and special occasions	0.8
	I know at least one adult I can depend on when I exit care	0.8
	My relationships are free from hitting, slapping, shoving, being made fun of, or name calling	0.9
	I know the signs of an abusive relationship	0.89
	I show others I care about them	0.8
Housing and money management	I know how to balance my bank account	0.8
	I know an adult who would help me if I had a financial emergency	0.9
	I know how to find safe and affordable housing	1
	I know an adult I could live with for a few days or weeks if I needed to	0.9
	There is at least one adult that I have regular contact with, other than my case manager or other professional, who lives in stable and safe housing	0.8
Work and study life	I know how to get the documents I need for work, such as my Social Security/Social Insurance Card and birth certificate	0.8
	I know how to get help from my school’s mental health services	1
Career and education planning	I know how to find financial aid to help me pay for my education or training	0.8
	I have talked about my education plans with an adult who cares about me	0.8

Discussion

This investigation presented a robust and novel assessment of acceptability of employment readiness measures with a specific disability population: youth and young adults on the autism spectrum. Using a concurrent triangulation mixed-methods study design, we determined that the WRI and ACLSA-M are acceptable measures in assessing employment readiness in youth and young adults on the spectrum. We first assessed their concurrent acceptability in measuring employment readiness in this population during a pre-post measurement investigation, which indicated statistically significant increases in employment readiness following the EWC intervention, using both instruments. These findings were elaborated and deepened via qualitative assessment. Peer debriefing enabled us to utilize key informant stakeholders in the assessment of retrospective acceptability of the measures, while also contextualizing results and highlighting the findings. Determination of the acceptability-rate of each measure according to a predefined level of acceptance allowed for consideration of additional items for future measure, something which participants felt was needed. The use of this methodology allowed for a robust exploration of the implementation effectiveness of these measures. Mixing quantitative and qualitative research methodologies results in higher quality inferences [62] and underscores the overarching purpose of the choices for this investigation.

The suggested modifications noted by stakeholders during peer debriefing to both measures would enhance acceptability rates and implementation effectiveness. While the full version of the ACLSA was determined to lack acceptability, assessment at the item-level allowed identification of additional items that should be reconsidered for inclusion on the acceptable modified version. These additional items illustrate the importance of including more comprehensive generalized life skills and appropriate “soft skills” in areas like relationships and communication, work and study life, self-care, and career and life planning in assessing employment readiness in those on the spectrum. Participants advocated for a more granular approach through slight alteration of item language and content on the WRI and ACLSA-M to address the ambiguity that arose in items that were too broad or lacked context. Further, such adjustments were advised for item-, individual-, and job-specific elements. These findings highlight the need and relevance for autism-specific measures to be developed or modified as proposed in this manuscript. These modifications speak to the specific challenges faced by this population in their quest to be “employment ready.” Recognizing that employment readiness exceeds measurement or documentation of a sole outcome measure and to improve confidence in tools assessing employment readiness in youth and young adults on the spectrum, it is recommended that while acceptable in current formats, future modification of these measures occur to include improved social and life skills, including problem-solving, social initiation, and responsiveness. Of note, inherent limitations in empathizing and using pragmatic language consistently emerged.

A heterogeneity of needs among the neurodiverse autism population requires a variety of interventions to improve employment outcomes [2,16]. For youth and young adults on the spectrum, targeted support programs that include components of workplace communication have been shown to improve access to competitive employment over time [63–65], while studies also consistently find that greater independence and enhanced life skills contribute to more positive employment outcomes [16]. Research shows that improved social and communication skills and increased functional independence in personal care, use of public transportation, and social interaction increase access to meaningful employment [16,66]. These identified points were all reiterated by participants as being important to employment readiness.

Improvements in employment readiness for persons on the spectrum are critical as a step toward improving employment outcomes. The UN Convention on the Rights of Persons with Disabilities highlights the need to eliminate barriers and to ensure greater opportunities for people with disability [67]. The UN Human Rights secretariat recently highlighted an urgent need for implementing rights monitoring systems and specifically emphasized the lack of available data regarding youth on the spectrum [68]. This means labor markets, education, and training opportunities should be inclusive and accessible, and we need measures to assess this in education and training opportunities. This study is a first step in examining implementation effectiveness and acceptability among autism-stakeholders for using the WRI and ACLSA-M in measuring employment readiness in youth and young adults on the spectrum. The domains in these measures provide some helpful direction for focused training programs to better prepare these individuals for employment as well as individualizing their programs to address a participant's concerns about his or her work readiness skills. Items from either tool may be used to shape an attainable goal, actions, and strategies for a participant, in addition, items of concern provide an opportunity to explore supports and accommodations that may be helpful. Skills that are not a concern or that are of less of concern to an individual are equally important as they provide a means to recognize an individuals' strengths and/or skills. More broadly, these tools may be of assistance to pre-employment and employment programs as they develop their program content and objectives. These tools focus on skills that are important to employers, such as responsibility, flexibility, communication, but also address important individual attributes like self-efficacy and hopefulness that extend beyond immediate employment to career planning.

This study is not without limitations. First, the number of participants in the qualitative assessment of concurrent acceptability is a potential limitation. Next, the subjective nature of peer debriefer feedback may have introduced potential bias. Efforts were made to minimize bias by purposely recruiting participants with a variety of experience across Canada. The use of the Lynn method for proportion agreement has been criticized for its inability to adjust for chance agreement [57,69]. To address this concern, this study incorporated the use of the modified kappa to adjust for chance agreement and provide information about the degree of agreement beyond chance [56,57]. In addition, although this assessment of acceptability of the WRI and ACLSA-M in youth and young adults on the spectrum shows promise, further research is needed to examine the validity and reliability of the ASLSA-M. While the WRI and ACLSA are both valid and reliable, and the findings of this study suggest that the ACLSA-M is acceptable, research is needed to determine its psychometric properties. Lastly, further robust research is needed to replicate and generalize these findings.

Moving forward, the establishment of other tests and tools as a measure of employment readiness and life skills in persons on the spectrum is necessary to better encapsulate the space with which employment readiness nurtures improvement of employment outcomes for those on the spectrum. The WRI and ACLSA-M, at this time, serve as acceptable measures of employment readiness in this population, and their use in intervention and evaluation should not be overlooked. In addition, our use of peer-debriefing revealed that diverse stakeholders across Canada in the area of employment readiness for youth and young adults on the autism spectrum agreed on the WRI and ACLSA-M being agreeable, palatable, and satisfactory overall, with some suggested changes to content and reduced complexity to maximize self-report comfort. By making additional adjustments to these measures when used with youth and young adults on the spectrum, the WRI and ACLSA-M will arguably comprehensively measure employment readiness outcomes.

Pragmatic measures for youth on the spectrum are essential for monitoring and evaluating their success in alternative population and implementation contexts. Our findings align with others in the need for sensitivity in evaluative instrumentation for youth on the spectrum [70,71] and highlight the importance of clinical utility and implementation effectiveness when measuring employment support program outcomes. These findings can be utilized and applied to other employment support programs using these measures, with or without refinement, to assess employment readiness in youth and young adults on the spectrum.

Acknowledgements

The authors would also like to acknowledge participants as well as staff, facilitators, and participants of EWC Calgary and Edmonton.

Disclosure statement

The authors report no conflicts of interest.

Additional information

Funding

This project was funded by the Sinneave Family Foundation, provider of the EWC program and other services for youth and young adults on the spectrum.