Development of a Cyber/Information Technology Knowledge Test for Military Enlisted Technical Training Qualification

1-4338-1903-1

Abstract

An Armed Services Vocational Aptitude Battery (ASVAB) Review Panel, with expertise in personnel selection, job classification, psychometrics, and cognitive psychology developed recommendations for changes to the military enlistment test battery. One recommendation was to develop and evaluate a test of cyber/information and communications technology literacy to supplement current ASVAB content. This article summarizes a multiphased Cyber Test development process: (a) a review of information/computer technology literacy definitions and measures, (b) development and pilot testing of a cyber knowledge measure, (c) validation of test scores against final school grades (FSGs) for selected technical training courses, (d) development of an operational reporting metric and subgroup norms, and (e) examination of construct validity. Results indicate the Cyber Test has predictive validity versus technical training school grades and incremental validity comparable to the ASVAB technical knowledge tests when used with the ASVAB Armed Forces Qualification Test (AFQT) verbal/math composite as a baseline.

• cyber
• information and communications technology
• technical knowledge
• selection and classification
• Armed Services Vocational Aptitude Battery

Notes

1 ASVAB tests include Arithmetic Reasoning (AR), Assembling Objects (AO), Auto and Shop Information (AS), Electronics Information (EI), General Science (GS), Math Knowledge (MK), Mechanical Comprehension (MC), Paragraph Comprehension (PC), and Word Knowledge (WK). The verbal tests (PC and WK) are combined into a verbal (VE) composite. VE and the math tests (AR and MK) are combined into the Armed Forces Qualification Test (AFQT) composite, which is used by all U.S. military Services for enlistment qualification. Each Service develops its own composites to qualify applicants for technical training.

2 The CAT-ASVAB is a computerized adaptive testing platform for administering the ASVAB at the Military Entrance Processing Stations (MEPS).

3 Nontraditional formats such as multiple response or matching may violate assumptions of the item response theory (IRT) model used in CAT-ASVAB, such as local independence. Polytomous scored items also present a challenge for integration with CAT-ASVAB that uses only dichotomously scored items using the three parameter logistic model (3PL; Lord & Novick, 1968). IRT models appropriate for polytomously scored items (e.g., Muraki, 1997) are available, and mixing of models is not problematic within the IRT framework. Nevertheless, the current CAT-ASVAB infrastructure is configured to work with the 3PL model only, and revising it to include other models would require substantial changes to the current system.

4 The 14 biodata items were multiple response format, representing 79 discrete items.

5 The Assembling Objects test is a nonverbal reasoning test that requires examinees to determine how an object will appear when its parts are put together.

6 Test information is an index of measurement precision.