Development of a contextually appropriate, reliable and valid basic Wheelchair Service Provision Test

Abstract

Purpose: Currently, there is no internationally accepted way to measure the competency of wheelchair service professionals. The International Society of Wheelchair Professionals aims to develop a Wheelchair Service Provision – Basic Test as a preliminary step towards establishing a certification process. Method: A team of wheelchair service provision experts developed test questions and conducted alpha and beta testing in order to validate them. Low-performing test items were eliminated. A pilot test was then conducted, which focused on developing a pass score, determining language barriers and validating the test as a measure of competency. Results: 90 participants completed one of three versions of the Wheelchair Service Provision – Basic Test. A pass score of 70% was established and 135 questions were accepted for the final test. Analysis of variance indicated there was a difference in scores based on language (p = 0.001), but not based on experience level. This result motivated translation in to the United Nations’ official languages. Conclusions: The results indicate that the Wheelchair Service Provision – Basic Test is a valid method for measuring basic competency of wheelchair professionals. Additionally, researchers recommend a skills assessment to help to ensure only qualified wheelchair professionals receive the certificate.

Implications for Rehabilitation

• Identify key steps in contextually appropriate test development.

• Describe the components of a contextually appropriate, reliable, and valid test for basic wheelchair service provision.

• Identify methods used for contextually appropriate test validation.

Keywords:

• Clinician
• cultural competence
• knowledge test
• psychometrics
• service provision
• test item construction
• wheelchair

Introduction

Mobility allows for independence in activities of daily living such as dressing, bathing and eating, and is a gateway to societal participation, affecting one’s ability to earn an education or income. Currently, the World Health Organization (WHO) estimates 70 million people around the world are in need of a wheelchair to be mobile.[1] Even for the portion that have a wheelchair, a significant number are likely to use poorly fitting or inappropriate wheelchairs, which put them at risk of secondary injuries and high likelihood of abandoning the technology.[2] To improve wheelchair users’ participation, especially in developing countries, it is important that health care professionals and others providing wheelchairs are well trained and possess knowledge in wheelchair service provision by proof of certification.[3]

The importance of certification is evidenced across nearly all health care domains. Hospitals must be accredited,[4] doctors must be licensed,[5] and nursing assistants must be certified.[6] Certification of health care employees ensures standardization of education and of quality, and establishes suitability for accreditation with entities such as the Joint Commission in the USA.[7] Endorsement from qualified entities then shapes competitive income for health care professionals. For example, one study published in the Journal of the American Pharmacists Association found that pharmacists with specialty certifications reported being more marketable and receiving higher wages after gaining the certifications.[8] Furthermore, knowledge is a foundation for clinical judgment and decision-making.[9] Certification allows for knowledge verification and therefore increases health care quality. Moreover, certification protects the consumer.[10] Clinical knowledge and competence have been identified as essential to mitigate risk of harm. Studies suggest that a large percentage of patient care incidents are attributed to a lack of training, knowledge, skill and competence.[11,12]

Because it is an emerging profession, a standard of certification and training for wheelchair prescription has not yet been established. Wheelchair prescription requires unique skills and experience and may be sparsely included in pre-professional health science programmes. Using occupational therapy (OT) standards as an example, the graduate must be able to train patients “to enhance functional mobility, including … wheelchair management” but does not specify other wheelchair service provision practices such as the initial assessment, the spectrum and type of device, how to incorporate contextual or environmental factors, nor quantity of the content or how it should be delivered.[13] Likewise, there is an analogue with professional certification. Equipment and assistive devices are covered in the National Physical Therapy Exam (NPTE), but not seating and mobility specifically.[14] Thus, existing programmes may not guarantee in-depth seating and mobility training, and in terms of self-study, providers may be less motivated to gain competency in this area since it may not be specifically required. As a consequence, assessment issues may arise. For example, many wheelchair users indicate their mobility needs are unmet.[15] Trainings and guidelines have been developed for wheelchair prescription best practice to mitigate this problem, especially in less resourced environments.[1,16,17] Specific training on wheelchair prescription has been demonstrated to improve users’ performance on a Wheelchair Skills Test developed in Canada [18] and overall satisfaction.[19,20] The Wheelchair Skills Test is a metric that suggests users can improve the efficiency of their wheelchair use by maximizing safety and performance measures outcomes.[19,20] A study incorporating the Wheelchair Skills Training Program into an OT curriculum improved wheelchair skills performance of the students by 25% (p < 0.001) compared to the standard curriculum which increased performance by 9.7% (p = 0.015). The difference in Wheelchair Skills Test performance indicates that OT professionals can significantly benefit from more specific wheelchair prescription training than is currently standard in OT programmes.[20]

However, even if wheelchair service provision training materials were instituted in pre-professional degree programmes worldwide, there is still not a way to consistently assess students or professionals’ knowledge in wheelchair provision. The Rehabilitation Engineering Society of North America (RESNA) Seating and Mobility Specialist (SMS) certification recognizes demonstrated competence in seating and mobility. However, the SMS is focused on North American policies; is only available in the English language; requires previous certification (Assistive Technology Professional) and 1000 h of service; and is cost-prohibitive for people in the developing world ($250). The proprietary nature of the SMS and different focus of the test (e.g., inclusion of power mobility devices) prohibits using this exam as a template for this alternative test. As evidenced by the SMS, though, separating wheelchair prescription into its own specialty can allow for increased attention on the specifics of appropriate fitting.[21] Thus, an interdisciplinary and contextually appropriate certification would benefit professionals, wheelchair users and clinical outcomes around the world by developing a benchmark to drive the improvement in training which vary widely and may have a consequential impact on the quality of service provision.

The following study is aimed at the development of a wheelchair service provision basic test as a preliminary step towards establishing a certification process, one of the goals of the newly formed International Society of Wheelchair Professionals (ISWP).[22] Once established, this process will inform future test development for providers at the intermediate and “trainer” levels. The ISWP will distribute the test to raise awareness of the importance of demonstrated competency, and consequently, the need to improve wheelchair service provision worldwide.

Methods

The test development was composed of six steps: domain selection, question development, alpha testing, beta testing, pilot testing and evaluation.[23] The USAID-funded Leadership, Management & Governance Project (LMG), a team of U.S. experts in test development, managed the development process, recruiting international subject matter experts to form an Assessment Development Task Force (ADTF) and subcontracting psychometricians from The Institute for Performance Improvement (TIfPI) to develop testing domains. The ADTF group, composed of two levels, included international wheelchair provision experts along with basic wheelchair provision trainers who use the WHO’s Wheelchair Service Training Package – Basic Level.[1] Level 1 created items (domains, competencies and questions) while Level 2 reviewed and validated items. The ADTF was trained using three credentialing agencies’ standards: ISO 17024, NCCA National Commission for Credentialing Agencies and International Society for Performance Improvement’s Program Accreditation Standards.[23–25] These standards require that credentialing organizations perform a formal job task analysis to identify required knowledge and skills before writing questions, and for a representation of stakeholders throughout the process. ISWP maintained these standards while conducting additional assessment and deployment.

Domain selection

The ADTF selected the testing domains including assessment (an assessment of the wheelchair user’s needs), prescription (wheelchair specifications, features and modifications), fitting (wheelchair adjustments), production (wheelchair assembly and safety), user training (best methods for transfers, mobility, etc.), process (steps from referral and assessment to funding and ordering) and maintenance and repair (maintenance, repairs and follow-up procedures for the wheelchair) which were elected based on the required steps for basic wheelchair service independent of geographic locations. These domains correspond with WHO’s eight steps to basic wheelchair service provision, although “referral and appointment” and “funding and ordering” [1] are combined in the process domain. Each domain included between 3–9 sub-domains (see Appendix A for sub-domain list in Blueprint).

Question development

The initial questions were drafted by ADTF and TIfPI. TIfPI first provided training on question development and guided ADTF through defining test objectives including how test-takers should be able to use their knowledge, what kinds of questions should be included, how long and how difficult the test should be. The question material included information from several sources and field experiences in order to ensure test-takers were proficient working in settings that provided wheelchair provision services. The ADTF determined the test format (multiple choice) and ensured test questions were unbiased. After questions were drafted, they underwent iterative revisions by ADTF Level 2 to ensure that they were as clear as possible, that only one of the options was correct, and that it conformed to the style rules used throughout the test.

Alpha testing

Alpha testing was completed in December 2013, via Survey Monkey, with three wheelchair providers recruited by LMG. The purpose was to test international Internet access across continents.

Beta testing

For beta testing, a testing platform, test.com, was selected in place of Survey Monkey. This testing platform was selected by the ADTF for its superiority in functionality and analysis capabilities, a friendlier interface for participants, and the ability for users to return to a session if an interruption occurred, especially due to the international context and unreliable Internet access.

LMG asked ADTF Level 1 to suggest names and provide contact information for participants for beta testing. LMG provided a link to one of three versions of the test to 36 novices and 57 experts. The target sample size was 12 novices and 12 experts.

Pilot testing

LMG asked clinic supervisors from 6 continents working in wheelchair provision services to select staff to participant in the pilot test. Unlike the beta test, participants were not selected based on a defined number of novices and experts. Instead, in the demographic information section, participants were asked to identify their category of experience: 1 or less years, 2–3 years, 4–7 years or 8 or more years. The inclusion criterion was that participants be familiar to some degree with wheelchair provision. Table 1 displays demographics.

Table 1. Participant demographics.

Years of experience: Test C	Frequency	Percent of total
1 year or less	23	26
2–3 years’ experience	22	24
4–7 years’ experience	18	20
8 or more years’ experience	27	30
Total	90	100

Participants were assigned to take one of the three test versions on test.com (Test A, Test B and Test C), each consisting of 61 questions from a total of 162 questions. They completed the test online and in English. The number of questions per test was determined by the blueprint developed by LMG (see Appendix A), which was designed by the ADTF. Along with a wider group of stakeholders, a consensus was reached regarding how each domain and subdomain should be represented on the certification test. Participants took the test at a time and place that was convenient to them. Figure 1 shows the test instructions. The participants were asked to take about 2 h to complete the exam, but no time limit was implemented. Test administrators attempted to evenly distribute participants between test forms. Experience level, language and other demographic information was not a factor in distribution. Since testing may require time away from work, the test was designed to be brief while still testing all relevant knowledge. Table 2 shows the number of items included in each test compared to the number of items in the pool. Each test was presented entirely in English.

Figure 1. Instructions given to participants. Test-takers were shown these instructions before beginning the test. The instructions were also translated into Albanian, for participants whose first language was Albanian.

Table 2. Items by domain for each test and in item pool.

Test section	Items in test by domain	Items in the pool
Assessment: Domain 1	15	39
Prescription: Domain 2	10	30
Fitting: Domain 3	8	20
Production: Domain 4	5	10
User Training: Domain 5	12	32
Process: Domain 6	8	22
Follow-up & Maintenance: Domain 7	3	9
Total	61	162

Item analysis

Each item was coded as a 7-digit number where the first 2 numbers corresponded to the domain, the second 2 to the sub-domain and the last 3 to the item number. Each item had 4 response choices, with the correct answer indicated. The frequency of each response choice was recorded, with the percent correct corresponding to the item difficulty score. Figure 2 is a guide on how to interpret the results of the item analysis.

Figure 2. Interpreting the results of the item analysis. The items analysis indicated the correct answer, how many participants chose the correct answer, and the percent of participants who answered correctly. If the correct answer was not selected at least 70% of the time, the question was eliminated.

Evaluation

TIfPI transferred all raw data and the test.com platform to the ISWP who then analysed participant results in relation to demographics, continued individual item analyses, selected all final test parameters and procedures and implemented and distributed the final test. A one-way analysis of variance (ANOVA) was used to compare means by demographics.

Results

Alpha testing

No access issues were identified and the results are included in the beta testing, referenced as Alpha A.

Beta testing

Beta testing was conducted to establish cut-scores and items analysis. However, of the 93 people that were invited by LMG to participate in beta testing, only 25% completed the test. Since an insufficient number of Novice participants (n = 3) responded, a contrasting group could not be firmly established and cut-scores could not be defined. Additionally, there was a large range in the response scores, with the standard deviation being one-third to half the mean. This was too significant to determine cut-scores. The means and standard deviations can be found in Tables 3 and 4.

Table 3. LMG Beta A including Alpha: descriptive statistics.

Test section	Items in section	Respondents	Minimum score	Maximum score	Mean	SD
Assessment: Domain 1	15	15	8.00	15.00	11.4000	1.95667
Prescription: Domain 2	10	15	3.00	10.00	7.1333	2.26358
Process: Domain 6	8	15	2.00	8.00	5.4667	1.88478
Production: Domain 4	5	15	0.00	4.00	2.6000	1.29835
Fitting: Domain 3	8	15	0.00	7.00	3.9333	2.18654
Follow-up & Maintenance: Domain 7	3	15	0.00	3.00	1.9333	1.03280
User Training: Domain 5	12	15	0.00	10.00	6.6667	3.03942
Total	61	15	19.00	54.00	39.1333	10.88818

Table 4. LMG Beta C: descriptive statistics.

Test section	Items in section	Respondents	Minimum score	Maximum score	Mean	SD
Assessment	15	9	7.00	14.00	12.0000	2.34521
Prescription	10	9	2.00	10.00	7.3333	2.73861
Process	8	9	5.00	8.00	6.7778	0.97183
Production	5	9	2.00	4.00	3.1111	0.60093
Fitting	8	9	1.00	6.00	3.6667	1.65831
Follow-up & Maintenance	3	9	1.00	3.00	2.1111	0.78174
User Training	12	9	1.00	11.00	6.0000	2.64575
Total	61	9	24.00	53.00	41.0000	8.76071

Data from the 24 completed exams were useful in eliminating underperforming questions. Question difficulty was determined from the percentage of participants that correctly answered the question, with lower numbers indicated a greater difficulty of the question. Therefore, if a question difficulty was below 50%, the question was eliminated from the item pool for pilot testing. Significant problems were found with the Test B version. The problems were due to errors in test.com that told participants they had completed the test when they had not. The errors were located and eliminated. The next round of testing is detailed in the current report.

Pilot testing

Participants

The pilot test was made available to participants between 18 July 2014 and 22 August 2014. During that time, a total of 90 participants took Test A (n = 37), Test B (n = 28) or Test C (n = 25). A total of 3 participants took Pilot B, but 2 of those participants also took another test. In this case, the higher score was recorded. Including the one participant that only took Pilot B, there were a total of 91 participants. Participant demographics can be found in Appendix B.

Test scores

The total mean score for each test was calculated along with the mean score by experience level (see Table 5). Test C had the highest mean of M = 0.753 and Test B had the lowest of M = 0.668. The lowest recorded score was 0.17 (Test A) and highest 0.90 (Tests B and C).

Table 5. Mean scores by test and by experience level.

	Test A	A – Exp Group	Test B^a	B – Exp Group	Test C	C – Exp Group
Number completing test	37		28		25
Difficulty level: % getting correct
Mean % for test	67.3%		66.8%		75.3%
Mean % for experience = 0–1 year	67.7%	10	60.2%	7	73.0%	6
Mean % for Experience = 2–3 years	64.7%	9	68.0%	8	77.7%	5
Mean % for experience = 4–7 years	70.7%	8	74.2%	3	78.0%	7
Mean % for experience = 8 or more years	71.4%	10	67.1%	10	72.4%	7
High score for test	88.5		90.2		90.2
Low score for test	17.0		49.2		49.2
Passing at 70% or higher	22		14		20
Passing at Mean or higher	26		18		15^b

a Test B includes 2 test-takers from Pilot B, which had one less question; however, both passed with high scores.

b Test C had a mean score significantly higher than 70% and the mean scores of the other two tests.

Additional mean scores organized demographically can be found in Tables 6 and 7. ^{Table B1 Table B2 Table B3 Table B4 Table A1}There was a significant difference between mean scores based on language ability (p = 0.001) as displayed in Table 6. Table 7 demonstrates there was no significant difference between mean scores based on experience level (p = 0.448). A majority of participants completed the tests in less than 2 h (80%).

Table 6. Analysis of mean score by language ability.

Language ability	Average score: overall	SD	N
English spoken at home	76	9.7	15
English fluency (not spoken at home)	70	10.8	57
No English fluency reported	61	13.7	18
ANOVA			p = 0.001

Table 7. Analysis of mean score by experience level.

Experience level	Average score: overall	SD	N
1 year or less	67	9.36	23
2–3 years	69	11.62	22
4–7 years	74	10.89	18
8 or more years	70	14.61	26
ANOVA			p = 0.448

Item analysis

Items that were ± 1.2 SD (0.42; 0.98) the mean item difficulty (M = 0.70) were removed, resulting in a total of 135 unique questions over 7 domains. This cutoff was selected in order to ensure that a sufficient number of questions remained in the pool and to remove questions that 100% of the initial test-takers got correct, indicating the item was too obvious or easy, and have a lower boundary equivalent to the upper boundary. In order to match the original representation in the blueprint, 75 questions were selected for the final test. Each test-taker will therefore have a different set of questions, increasing the security of the test. Item performance will continue to be monitored, and questions added to increase the item pool to mitigate question recall among participants who need to take the test multiple times.

Discussion

This pilot validates the test items for a Wheelchair Service Provision – Basic Test. The creation of this test will aid clients, organizations and health professionals in finding qualified wheelchair professionals since, currently, no standard exists to unite wheelchair service organizations and professionals internationally under guiding principles. This pilot and the resulting test provide a foundation for training wheelchair professionals and support standardized, freely accessible wheelchair services anywhere in the world.

Experience level

Experience level did not appear to have a strong correlation with test performance. It is unclear why this occurred. Perhaps, the test content is not based on experience or practical skills but rather is more fundamental. Since test-takers with more experience are likely to have trained several years ago, they may rely less on the textbook answer and more on their skills and experience. If this is the case, a supplementary portion of the certificate process may need to be developed to test wheelchair skills at the basic level, which should account for this perceived discrepancy.

Pass score

Since experience level did not have a strong correlation with test performance, there was no clear contrast group in this study to determine an objective pass score. Therefore, a subjective pass score was established for the test based on the normalized distribution (mean and median = 0.7) observed for each test. A pass score higher than 70% would also detract from the main objective of professionalizing wheelchair provision: to develop capacity, recognize basic knowledge and eventually recognize proficiency. The pass score may be further amended after collection of long-term data.

English literacy

English literacy was shown to relate to the participant’s score significantly (p = 0.001). ANOVA analysis indicates that there is a significant difference between native English-speaking and non-native English-speaking test-takers (see Table 6). This was the expected result, since the test was administered in English. In the future, videos and images may help alleviate low literacy difficulties. There are also current efforts to translate the questions into other languages including Albanian and Khmer (available as of early 2015) and Arabic, Chinese, French, Mongolian, Portuguese, Russian, Spanish and Vietnamese (available by end of 2015).

Lapse time

Most participants completed the exam in less than 2 h (80%). However, some appeared to have left their computers on well after completing the test. To prevent this issue, the final test featured a time-out feature after 105 min (the average after removing outliers). Over time with more test-takers, the test time may need to be extended for equitable access for non-native English speakers.

Study limitations

There was a challenge in enrolling participants at all levels, possibly due to time constraints because some potential participants do not have regular computer access. Therefore, this validation may have limitations affecting analysis of question difficulty, English literacy, pass scores and time allotted. Despite these possible limitations, researchers are confident that these preliminary data are sufficient to establish an initial pass score and launch of the test. Continued psychometric analyses will be conducted and necessary changes will be made.

Conclusion

The pilot of the Wheelchair Service Provision – Basic Test validated test items for the primary launch of the test. As a next step, researchers will need to determine the feasibility of integrating this competency measure internationally.

In the future, the test will be translated into languages other than English as described above. All tests will continue to be monitored for trends that may impact pass score and questions will be added, removed and revised. Additionally, a wheelchair skills assessment will be considered. The goal for this test is to be a metric for certification in basic level wheelchair service provision in a broad range of settings. This test and the development process will also serve as a model for an intermediate level test, which will be piloted in 2015.

Funding information

Funding for the International Society of Wheelchair Professionals (ISWP) was provided by USAID (Agreement No.: AIDOAA-A-12-00047, Subaward No.: APC-GM-0068).

Acknowledgements

The authors thank USAID; Advancing Partners & Communities; JSI Research & Training Institute, Inc.; Leadership, Management, and Governance Project at Management Sciences for Health; The Institute for Performance Improvement.

Disclosure statement

The authors report no declarations of interest beyond the grant award received.

Appendix A:

Blueprint.

Table A1. Blueprint.

Domains	Subdomains	Single	Domain total	Percent of total
Assess (gathering information in order to prescribe)	Client goals	2	16	28
	Lifestyle and environment (2)
	>Lifestyle	2
	>Environment	2
	Support systems	2
	Physical assessment (3)
	>Pressure risk	2
	>Size (measurement)	2
	>Posture	2
	Functional abilities	2
Fitting	Posture	2	8	14
	Pressure	3
	Size and adjustment	2
	Mobility	1
Follow-up (Maintenance & Repair)	Follow-up (2)		2	3
	>Timing	0.5
	>Reason for	0.5
	Maintenance and repair	1
Prescription	Size	3	9	16
	Type and features	3
	Cushions	3
Process	Documentation	3	7	12
	Service steps (2)
	>WHO guidelines	1
	>Proper equipment and facilities	1
	Professionalism	1
	Confidentiality	1
Product Preparation	Assembly and adjustments	1	5	9
	Safe and ready	1
	Basic modifications	2
	Cushion	1
User Training	Wheelchair skills (5)		11	19
	>Handling (folding, set-up, etc.)	1
	>Propel	1
	>Slope	1
	>Barriers	1
	>Transfer	1
	Maintenance	2
	Basic health	2
	Troubleshooting	1
	Caregiver skills	1
Total		58	58	100

Table A1 shows the number of questions per domain on each version of the test (A, B or C) along with the weight given to each domain in terms of the percentage of the total questions. The symbol “>” is used to designate categories in a subdomain. See Table 2 for the available number of questions per domain and domain number designations.

Appendix B:

Participant demographics by test.

Table B1. Participant demographics: language ability.

Language ability	Frequency	Percent of total
Test A
English spoken at home	4	11
English fluency (not spoken at home)	27	73
No English fluency reported	6	16
Total	37	100
Test B
English spoken at home	6	21
English fluency (not spoken at home)	13	47
No English fluency reported	9	32
Total	28	100
Test C
English spoken at home	5	20
English fluency (not spoken at home)	17	68
No English fluency reported	3	12
Total	25	100

Table B2. Participant demographics: experience level.

Years of experience	Frequency	Percent of total
Test A
1 year or less	10	27
2–3 years	9	24
4–7 years	8	22
8 or more years	10	27
Total	37	100
Test B
1 year or less	7	25
2–3 years	8	28
4–7 years	3	11
8 or more years	10	36
Total	28	100
Test C
1 year or less	6	24
2–3 years	5	20
4–7 years	7	28
8 or more years	7	28
Total	25	100

Table B3. Participant demographics: average age by test.

Test	Average age
A	33.75 years
B	36.5 years
C	36 years

Table B4. Participant demographics: gender distribution by test.

Gender	Test A	Test B	Test C	Pilot B	Total
Male	25	9	10	1	45
Female	12	19	15		46
Total	37	28	25		91