Official position of the American Academy of Clinical Neuropsychology on test security

Abstract

Objective:

To provide education regarding the critical importance of test security for neuropsychological and psychological tests, and to establish recommendations for best practices for maintaining test security in forensic, clinical, teaching, and research settings. Previous test security guidelines were not adequately specified.

Method:

Neuropsychologists practicing in a broad range of settings collaborated to develop detailed and specific guidance regarding test security to best ensure continued viability of neuropsychological and psychological tests. Implications of failing to maintain test security for both the practice of neuropsychology and for society at large were identified. Types of test data that can be safely disclosed to nonpsychologists are described.

Results:

Specific procedures can be followed that will minimize risk of invalidating future use of neuropsychological and psychological measures.

Conclusion:

Clinical neuropsychologists must commit to protecting sensitive neuropsychological and psychological test information from exposure to nonpsychologists, and now have specific recommendations that will guide that endeavor.

Keywords:

• Test security
• neuropsychological tests
• psychological tests
• forensic practice
• litigation

Test security: Concept and importance

Psychological and neuropsychological tests offer an objective means of evaluating strengths and weaknesses of performance capacities, as well as characteristics of symptom reporting. Norm-based psychological and neuropsychological tests have normative values collected from individuals with no prior knowledge or exposure to the specific tests. Thus, valid and effective use of these tests in subsequent clinical practice requires that an examinee’s exposure at the time of testing be comparable to this standard. That is, no individual should have gained access to item content or test-taking strategy that would allow unfair advantage, whether that advantage be manipulating results to artificially increase or decrease performance capabilities or to misrepresent symptom report. In this context, test security refers to the broad and clear expectation that in order to maintain valid use and effectiveness, the contents and key operational characteristics of psychological and neuropsychological tests must be protected from inappropriate disclosure to non-psychologists.

Test security is a fundamental and critical issue, which, without meticulous adherence, threatens the entire enterprise of psychological and neuropsychological testing. The importance of this issue has been identified for many years in the neuropsychological community, as noted generally in practice guidelines of the American Academy of Clinical Neuropsychology (AACN; Board of Directors, 2007), and elaborated in an official joint organizational position statement of AACN, Division 40 of the American Psychological Association, and Association of Postdoctoral Programs in Clinical Neuropsychology (disclosure of test data versus test materials; Attix et al., 2007), as well as recommendations promulgated by the National Academy of Neuropsychology (NAN; 2000a, 2003). Related guidelines have been published by AACN (third party observers, 2001), NAN (third party observers, 2000b; opposing advance notice of test selection, 2018), and recently, an interorganizational paper issued by AACN, NAN, and the American College of Professional Neuropsychology (third party observers; Glen et al., 2021). The topic of inappropriate disclosure of test information has also been the subject of legal case review (Kaufmann, 2009). Moreover, originally noted in a 2009 AACN consensus statement on validity testing (Heilbronner et al., 2009), a recent consensus statement update has again highlighted the importance of test security as unconditionally necessary in maintaining the ability to verify the validity of all psychological and neuropsychological test results (Sweet et al., 2021). These examples demonstrate the ongoing serious concerns of testing specialists. Nevertheless, prior statements have not provided an integrated and comprehensive perspective with clear specific guidance for practitioners. In this regard, the APA Ethics Code offers only rudimentary, and somewhat contradictory, instruction regarding maintaining test security, and the recent APA Guidelines for Psychological Assessment and Evaluation (March, 2020) provide minimal guidance on the topic, thereby adding further impetus for the development of detailed test security guidelines.

While neuropsychological organization position papers have advocated strongly for test security, with increasing digital access to information, as well as more aggressive demands for protected test data in forensic settings, it is critical to revisit this issue in order to provide more specific guidelines regarding the establishment and maintenance of test security. It is clear that test security is a broad consideration, with multiple subtopics, for which the present position statement provides comprehensive coverage, ranging from specific practical guidance, when possible, to preferences that can strengthen protection of tests, even in instances when complete protection may not be possible.

Statement of the problem: failure to ensure test security jeopardizes test effectiveness resulting in negative impact to society

Neuropsychologists are increasingly asked to provide services in a manner that raises concerns regarding test security. This is particularly true for forensic applications of testing, in which neuropsychologists sometimes encounter demands that observers be allowed at neuropsychological examinations; that test takers and/or their legal counsel be allowed to record by audio or video the entire examination, including test administration procedures; and that test item responses and test materials from neuropsychological exams be provided to non-psychologists who do not have adequate training in neuropsychological testing and are under no ethical obligations to protect the tests (e.g., attorneys, physicians, examinees). Beyond forensic applications, test security lapses additionally occur when psychologists and neuropsychologists reproduce test instructions and procedures in their reports; researchers describe test paradigms in their peer-reviewed publications, books, or other platforms accessible by non-psychologists; evaluators allow patients to complete self-report inventories in unsupervised settings that could permit reproduction of test items; examiners divulge test answers to examinees; psychologists give lectures to non-psychologist colleagues regarding psychological and neuropsychological test procedures and test content; authors provide detailed descriptions of tests and illustrations of test stimuli in textbooks; professors and instructors post psychological/neuropsychological test slides and lecture materials intended for psychology graduate students on the internet; and practitioners do not dispose of obsolete or unneeded tests in a way that ensures test security. Most recently, the COVID-19 pandemic, which led to widespread restrictions on social interactions and required use of masks, fostered attempts to develop “remote” or telehealth neuropsychological evaluations (conducted via video-platforms). Recommended consent procedures include that the telehealth neuropsychological evaluation will not be recorded by either party. Despite this proactive attempt to ensure test security, use of a telehealth platform raises prominent concerns regarding test security in that it may not be knowable whether test instructions and stimuli are being surreptitiously recorded and/or whether observers are present during such exams.

The present position paper on test security is based on a fundamental premise: Allowing non-psychologists (whether other professionals or examinees) to have access to test questions and stimuli or access to key operational characteristics of specific tests (e.g., details regarding identification of invalid responding) directly undermines the effectiveness and validity of psychological and neuropsychological tests, which is ultimately a detriment to society.

A. Damage to test effectiveness

The effectiveness and accuracy of psychological and neuropsychological test results is contingent on (1) examinee naivete to the test materials and procedures, (2) full cooperation of examinees in producing genuine symptom reporting and best performance responses (Sweet et al., 2021), and (3) tests being administered and scored according to standardized test instructions. To the extent that psychological and neuropsychological test questions and procedures are provided in advance to examinees, tests lose their ability to measure what they were intended to measure. If provided test questions, test stimuli, or strategic information regarding interpretation prior to a neuropsychological exam, examinees can score higher than their actual skill level or, conversely, they can decide how best to adjust their performance to depict deficits that they do not have. The reporting of, or denial of, key symptoms on self-report inventories can also be manipulated. Moreover, because psychological and neuropsychological tests were developed and validated on individuals who were naïve to the tests (i.e., participants in studies that established test normative values did not have knowledge of the tests prior to test administration), allowing examinees to have awareness of test stimuli and procedures before testing represents a violation of standardized test administration procedures. Departures from standardized test administration can render normative data or other patient comparison data inappropriate. For example, classification statistics of performance and symptom validity tests (e.g., true positive and false positive rates) are calculated in known groups studies in which examinees were not aware of the test procedures and stimuli prior to testing. The resulting classification statistics will not apply to individuals who are inappropriately informed regarding the tests prior to test administration.

Psychological and neuropsychological tests require years of test development at considerable cost and major investment of professional time. Once test effectiveness and validity are compromised (through pre-testing exposure to materials, instructions, or test-taking strategies), test authors, publishers, and practicing neuropsychologists have lost the ability to utilize the very tools upon which the psychological and neuropsychological assessment enterprise is based.

B. Detriment to society

There has been extensive study and documentation of diagnostic inaccuracy resulting from over reliance on subjective information, such as examinee self-report and even information from family members, which is often obtained via brief, unstructured interviewing of examinees and collaterals (cf. Andersson et al., 2019; Edmonds, Delano-Wood, Galasko, Salmon, & Bondi, 2014; Goldberg, 2017; Pavlova & Uher, 2020). Even key historical facts, such as self-reported injury characteristics, which might presumptively provide a basis for prior or current diagnoses, may not be recalled accurately (e.g., Don & Carragee, 2009; McKinlay et al., 2016). Alternatively, psychological and neuropsychological tests can provide objective bases for diagnostic determination. In fact, these tests were specifically developed with the recognition that over reliance on self-report is fraught with problems and can result in diagnostic errors. For example, differential diagnosis involving cognitive dysfunction after traumatic brain injury can be erroneous without testing that can identify the confounding effects of factors such as anxiety, depression, stress, negative life events, secondary gain context, and litigation (e.g., Eckerström et al., 2016; Gardner et al., 2017; Hromas et al., 2021 advance epub; Vos et al., 2020).

• Impact to Public Safety: Some occupations involve public safety, and psychological and neuropsychological test data are used to confirm that individuals in these occupations, or who aspire to these occupations, are fit to perform these jobs. For example, the Federal Aviation Administration (FAA) requires cognitive testing of pilots who have experienced medical, neurologic, psychiatric, and substance abuse conditions to ensure that pilots are cognitively capable of flying safely. Similarly, individuals desiring to become police officers are required to undergo psychological testing to ensure that they do not have problematic personality characteristics or other psychiatric conditions that would compromise their ability to safely function as officers. If pilots and police academy candidates were to obtain psychological and cognitive test information prior to undergoing testing, they would be able to “study up” regarding how to perform well on the tests, thus rendering the tests ineffective in identifying individuals who might represent a public safety risk. In a similar vein, in 2018, the President of the United States underwent a dementia screening exam (i.e., Montreal Cognitive Assessment; MoCA), triggering concerns expressed in JAMA Neurology (Haghbayan et al., 2018) that the test questions revealed by the media could impact future effectiveness of this cognitive screening procedure. Indeed, in the days after the media announced that the MoCA was used to evaluate the President, dozens of articles in the lay press presented readers with all items from which the total score is calculated, either in the body of the article or linking to it. Providing such information compromises subsequent use of such a measure to screen for cognitive impairment.

• Impact to Judicial Decisions: Within the judicial system, judges and juries rely on accurate psychological and neuropsychological test data to make informed judgments on such issues as psychological and cognitive damages/injuries in personal injury and medical malpractice cases, competency to stand trial, and insanity defenses and mitigation (i.e., sentence reductions) for criminal offenses (some of which might even occur in the context of death penalty litigation; see Denney & Fazio, 2021). To the extent that examinees in civil and criminal litigation become aware of the psychometric procedures used to determine symptom severity and validity, they would be in a position to adjust their test performances to effectively reflect more severe psychological and cognitive dysfunction than is actually the case, which in this context could result in incorrect judicial decisions (for example, Chafetz et al., 2015).

• Impact to the Educational System: Primary, secondary, and post-secondary educational institutions, and academic testing companies rely on accurate information regarding presence of learning disorders, attention-deficit/hyperactivity disorder, slow processing, and other conditions in determining whether test takers qualify for testing accommodations (e.g., more time to take classroom exams). If examinees are provided with key test information regarding the psychometric methods used to document academic challenges and determine validity of performance results, they can learn to successfully exaggerate nonexistent or lesser impairments and circumvent the tests and methods used to detect feigning, thereby obtaining accommodations in the classroom or on high-stakes tests (e.g., Scholastic Aptitude Test [SAT], American College Testing [ACT], Law School Administration Test [LSAT], Medical College Admission Test [MCAT]) to which they are not truly entitled (Marshall et al., 2016; Wallace et al., 2019). Alternatively, when high ability or knowledge-based test scores confer advantages, if examinees learn of test content prior to testing, they can attain higher scores than actually reflect their skill level, thereby receiving opportunities and benefits to which they are not truly qualified, such as gifted programs. Examples exist of parents gaining access to IQ test questions in order to help ensure their children qualify for high aptitude programming, which in certain cases can have substantial socioemotional costs for children who are unable to succeed in demanding academic environments. The 2020 highly publicized “college cheating scandal” concerning college admission testing (e.g., https://knowledge.wharton.upenn.edu/article/college-admissions-scandal/) provides another strong example of the high degree of motivation among some individuals to manipulate test results for personal advantage.

• Impact to Medical Care System: In routine treatment of patients, neuropsychologists and physician colleagues typically assume that symptom report is accurate and that cognitive test performances are valid when making diagnostic and treatment decisions. If test takers learn of the psychological and cognitive assessment methods that document whether reports of medical symptoms (e.g., pain) and cognitive conditions (e.g., attention-deficit/hyperactivity disorder) are valid, they could learn to “game” the tests, thereby allowing them to obtain medications (such as opiates and stimulants) to which they should not have access (Schroeder & Martin, 2021).

• Impact to Public and Private Services and Resources: The cost of feigned mental disorders within the Social Security Disability program as of 2011 was estimated at $20 billion (Chafetz & Underhill, 2013). The Rand Corporation (Carroll et al., 1996) judged that 34% to 40% of motor vehicle injury costs submitted to insurers were “excessive,” and added $13 to $16 billion to the country’s total automobile insurance bill, which is an average of $100 per individual policy. These excess claims were also associated with $4 billion in additional health care utilization. Based on estimates by the National Insurance Crime Bureau (NICB), Workers’ Compensation fraud is a $30 billion problem annually in the United States (California Department of Insurance; http://www.insurance.ca.gov/0300-fraud/0100-fraud-division-overview/25-wc-conv/). Neuropsychological and psychological testing, particularly when performance validity tests are included, is able to accurately detect when individuals are misrepresenting the extent of cognitive and psychiatric dysfunction. When these tests are compromised, the ability to detect feigned conditions markedly declines, at a substantial cost to society and citizens.

Thus, it is essential that psychological and neuropsychological test security be tightly maintained so that the tests continue to provide the critical information they are designed and validated to measure.

Failure to maintain test security can result in “coaching”

Empirical research shows that when examinees are “coached” as to the symptoms of neuropsychological conditions, and/or the measures used to determine performance and symptom validity, test takers are better able to adjust their test responses to match target conditions (Lamb et al., 1994; DiCarlo et al., 2000; Rose et al., 1998; Rüsseler et al., 2008; Suhr & Gunstad, 2007); such coaching can enable and facilitate fraud.

As discussed below, available data show that test takers can be, and have been, coached on psychological/neuropsychological tests when test information is possessed by non-psychologists who are not mandated to protect the tests. In a survey conducted by Wetter and Corrigan (1995), approximately half of attorneys and a third of law students believed their clients should always or usually be informed about validity scales in psychological tests to be given in an assessment. Youngjohn (1995) additionally described a case in which a Worker’s Compensation attorney admitted on the record to the Administrative Law Judge at the Industrial Commission of Arizona that he had coached and educated his client prior to a neuropsychological exam, and Youngjohn relayed that he was told by another attorney that it would be unethical for an attorney not to coach his client prior to a forensic neuropsychological evaluation.

Subsequently, survey data obtained by Essig et al. (2001) from members of the Association of Trial Lawyers of America revealed that 75% “typically spend up to an hour preparing their clients for neuropsychological evaluations and commonly cover test content, detection of malingering, and brain injury symptoms” (p. 271), 41% request names of specific tests to be administered in advance of the exam, and 29% review the MMPI-2 with their clients prior to neuropsychological examination. The authors note that “although only 8% appear to specifically instruct their clients how to respond to neuropsychological tests, advance knowledge of test content may be sufficient to allow the plaintiff to respond in a manner that does not reflect his or her current level of cognitive functioning and thus alter the expert’s conclusions” (p. 284). More recently, Spengler et al. (2020), in a survey of practicing attorneys, documented that over 50% endorsed providing clients with information regarding MMPI-2 validity scales.

An April 2018 Motion for the Appointment of a Special Investigator pertaining to the National Football League Concussion Settlement stated that “fraud discovered in the Program so far is deep and widespread” (p. 2; see https://mdl.law.uga.edu), including that “a law firm representing more than 100 Settlement Class members coached retired players on how to answer questions during their neuropsychological evaluations” (p. 2) and “text messages and other communications reveal a disturbing pattern of a claims service provider coaching players to ‘beat’ the neuropsychological tests” (p. 3).

Clearly, failures of test security can invalidate test results. The above examples demonstrate that coaching of examinees regarding how to manipulate psychological/neuropsychological test responses has been occurring for at least the past 25 years. Coaching can be expected to continue unless better procedures are in place to limit access to these measures by non-psychologists.

Endorsement of test security by psychological organizations, judicial rulings, state laws and regulations, test publishers, and other professions

The need to maintain test security to protect psychological test instruments is recognized in the Ethical Principles of Psychologists and Code of Conduct (American Psychological Association, 2002/2010, Principle 9.11, Maintaining Test Security), the Standards for Educational and Psychological Testing (American Educational Research Association et al., 1999, Standard 5.7; and 2014, Standards 10.10 and 10.18), the Statement on the Use of Secure Psychological Tests in the Education of Graduate and Undergraduate Psychology Students (American Psychological Association Committee on Psychological Tests and Assessment, 1994), and the International Test Commission (2014) Guidelines on the Security of Tests, Examinations, and Other Assessments. The importance of test security is further discussed in various position statements for psychological organizations, including the American Psychological Association (1999), and the National Association of School Psychologists (2000).

The United States Supreme Court first addressed the security of psychological tests in Detroit Edison Co. v. National Labor Relations Bd. (1979), ruling against the utility union that demanded the release of test scores, materials, and manuals in a dispute about breadth of the National Labor Relations Board (NLRB) authority. Kaufmann (2005) argued that Edison created a narrow, implied nondisclosure privilege that imposes a duty on psychologists to safeguard test materials from wrongful disclosure. The High Court noted the “strong public policy” of test security, and the rationale for a privilege to protect psychological tests was first extended to a clinical case in Chiperas v. Rubin (1998). Subsequent state courts recognized “the psychology profession’s legitimate interest in the security of tests” (p. 776, Florida DOT v. Piccolo, 2007). Kaufmann (2009) identified a series of additional federal court and NLRB decisions that uniformly recognize that discovery of psychological tests is restricted under Edison.

In the past decade, courts have heard numerous arguments and issued a range of rulings on the psychologist nondisclosure privilege and duty to safeguard psychological tests from wrongful disclosure. Consistent with Edison, many states have enacted at least some protections for psychological (including neuropsychological) test materials and content (see Kaufman, 2009; Shapiro, 2021). After carefully considering the Health Insurance Portability and Accountability Act (HIPAA) access and privacy requirements, Maine (2013) enacted a model statute for protecting standardized psychological and neuropsychological tests from wrongful disclosure (see Appendix 1).

Publishers of psychological and neuropsychological tests consider these products to be trade secrets in that information pertaining to test content derives independent economic value from not being generally known to the public and is the subject of reasonable efforts by the publisher to maintain its secrecy (e.g., see Terms and conditions of sale and use of Pearson products on the Pearson Assessments website; https://www.pearsonassessments.com/footer/terms-of-sale). Such methods include restricting sale of the tests and test materials to individuals who are qualified by education and training to administer and interpret the tests, who are bound by professional ethical standards to maintain test security, and who agree to follow federal copyright laws prohibiting reproduction of tests. The development and refinement of items and norms for individual intelligence, personality, and neurocognitive psychometric assessment instruments typically require many years of research, and extensive investment of professional expertise and time, as well as considerable capital investment by test publishers. Improper disclosure of test materials can result in damage to the parties with ownership interest in these tests, and to the professions of psychology and neuropsychology.

Other professions have similar test protection concerns and zealously maintain test security. For example, in the spring of 2018, the National Conference of Bar Examiners (NCBE) created a new position of Director of Test and Information Security to “further minimize the security risks that threaten to undermine the integrity of the bar admissions process” (https://thebarexaminer.org/article/spring-2018/test-and-information-security-centralizing-security-initiatives-at-ncbe/). It was noted that the NCBE “closely safeguards the security of its exam questions. The security of the questions is important before exam day to ensure that no examinee has an unfair advantage by having gained advance knowledge of the questions…NCBE strictly prohibits copying, reproducing, or disclosing any NBE (National Bar Exam) questions or answers, whether via electronic, telephonic, written oral or other means, to any party or to any public forum during or after the exam.” At the October 2018 Conference on Test Security (COTS), representatives of the NCBE were in attendance, as well as members of organizations involved in administration of the LSAT, SAT, and Graduate Record Examination (GRE), and testing conducted in Kindergarten through 12^th grade levels. In this meeting (Albanese, Zhang, & Hill; Test Security: A Meeting of Minds, The Bar Examiner, Winter 2018-2019), test security was defined as “protecting test materials from being…compromised long before exam day as well as during and after the exam. Test security also includes ensuring that examinees…do not bring any impermissible materials and technology devices into the exam to inflate their performances or record the exam questions, and that they do not reproduce or share any exam content at any point even for the benefit of others.” They concluded “the importance of maintaining test security cannot be overemphasized, because cheating, regardless of which form it takes, erodes the validity of the interpretations of test scores and then undermines the legitimacy of decisions based on those scores.” The same protections that are demanded for academic and professional licensure tests should also be afforded psychological and neuropsychological tests.

The American Psychological Association (2002) Ethics Code and test security

The APA Ethics Code was changed in 2002 to incorporate separate Standards for “test materials” versus “test data” (Standards 9.04 and 9.11; see Appendix 2). In excellent reviews of the changes from the 1992 APA Ethics Code to the 2002 APA Ethics Code, Bush and Martin (2006) and Bush et al. (2020) describe how the 2002 Standards 9.04 and 9.11 are in conflict; Standard 9.11 instructs psychologists to maintain the security of “test materials” (manuals, instruments, protocols, and test questions or stimuli; 9.11) while Standard 9.04 indicates that in response to patient release forms, psychologists are to turn over “test data” (raw and scaled scores, client/patient responses to test questions or stimuli, and psychologists’ notes and recordings concerning client/patient statements and behavior during an examination) to patients or their representatives. Bush and colleagues (Bush & Martin, 2006; Bush et al., 2020) correctly note that the distinction between “test materials” and “test data” is artificial in that many patient responses are the same as the test stimuli (e.g., reproduction of line drawings, repeating of word list learning tasks, etc.), and they assert that “If it is important to safeguard a list of words or a complex figure before it is revealed to the examinee, then it should be just as important to safeguard them after they have been repeated or reproduced by the examinee” (Bush & Martin, 2006; p. 119). They indicate that the APA Ethics Code Task Force apparently incorrectly perceived that 1996 HIPAA requirements requiring release of health information to clients/patients included test data because “HIPAA does not recognize the protection of test materials (e.g., test security) as a legitimate reason to withhold test data…” (Fisher, 2003, p. 7). However, Bush and Martin (2006) and Bush et al. (2020) point out that Richard Campanelli, Director of the Office for Civil Rights at the U.S. Department of Health and Human Services, the Office responsible for the administration of the HIPAA of 1996, stated:

Any requirement for disclosure of protected health information pursuant to the Privacy rule is subject to Section 1172(e) of HIPAA, ‘Protection of Trade Secrets.’ As such, we confirm that it would not be a violation of the Privacy Rule for a covered entity to refrain from providing access to an individual’s protected health information, to the extent that doing so would result in a disclosure of trade secrets. (Pearson Assessments, 2018)

Bush and Martin (2006) conclude:

Although the changes to the Ethics Code relevant to the handling of requests for raw test data (e.g., Standard 9.04) seem to represent an effort to make the Code more consistent with the anticipated implications of HIPAA…, HIPAA does not require psychologists to release trade secrets in the form of test data. Thus, Standard 9.04 does not seem to have met the goal for which the Code was radically changed. Rather, Standard 9.04 conflicts with the other relevant section of the Code (Standard 9.11), with general bioethical principles, and with other sources of ethical authority (with the ultimate effect of increasing confusion for many neuropsychologists). (p. 119)

They also noted that HIPAA specifically indicates that information relevant to civil, criminal, and administrative proceedings is not subject to the same rights of review and amendment as other health care information, and “thus, HIPAA does not prohibit neuropsychologists in forensic contexts from withholding test data…” (p. 120).

Bush and Martin (2006) further assert that the Ethics Code includes the principle of Nonmaleficence, which underlies Ethical Standard 9.11 on test security:

…the importance of safeguarding raw test data is based on the principle of nonmaleficence. That is, neuropsychologists have an obligation to avoid harming individuals and society at large. Releasing raw test data to individuals not qualified to interpret them and not bound by the ethical mandates to safeguard them may have harmful consequences for examinees and the validity of future examination results. To the extent that the release of raw test data may result in substantial harm to individual or society, neuropsychologists should refrain from releasing. (p. 117)

We concur that neuropsychologists are in compliance with HIPAA when withholding psychological and neuropsychological test data if release would compromise test security. This position is also mirrored by psychological test publishers, such as Pearson Assessments, who have asserted that “customers may not disseminate copies of test record forms or protocols to persons who erroneously claim that they are entitled to copies under HIPAA. As the HHS [Department of Health and Human Services] has now confirmed, HIPAA does not require any person to disclose any trade secret materials, and all restrictions on the dissemination of test record forms and protocols remain in effect” (Pearson Assessments website as of December 19, 2021).

Managing competing interests regarding access to psychological tests by non-psychologists

The admonition that protected psychological and neuropsychological testing information be limited to licensed psychologists has some important exceptions; for example, when psychometrists, interpreters, trainees, and research assistants assist in psychological and neuropsychological examinations. Psychometrists have a Code of Ethics for Certified Specialists in Psychometry through the Board of Certified Psychometrists, which indicates that they are “professionally obligated to maintain test security” (Section D:6). Interpreters, trainees, and research assistants have no formal requirement to protect psychological/neuropsychological tests; therefore, it is necessary to instruct them regarding the importance of maintaining test security, and it may be appropriate as routine practice to request that they sign attestations that they will protect test security.

In school-based settings, the relevant law pertaining to access to records is the Family Education Rights and Privacy Act (FERPA), which allows parents of minor students and students once they reach age 18 or begin attending a post-secondary institution to inspect and review educational records. Thus, a psychologist in the school setting who has conducted testing may need to allow access to the student’s test data. However, this can and should be done in a way that minimizes risks to test security, such as only providing scores, percentiles, and characteristics of comparison groups used for test interpretation. Psychologists should not allow a reproduction of any test materials (e.g., by permitting a photograph) or give a copy of any test materials to a non-psychologist. A copy of the testing records and data sheets could be sent to a psychologist of the family’s choosing, who is bound ethically to maintain test security.

In legal settings, attorneys may argue that they need access to psychological and neuropsychological tests and test-related materials in order to best represent their clients; specifically, that they need to understand the assessment measures employed with their clients to expose flaws, bias, and other irregularities in the assessment process. Thus, the legal and psychological community have competing interests vis a vis disclosure of psychological/neuropsychological tests and materials, and in this context, what is the approach that best protects all parties?

Information can be provided regarding psychological and neuropsychological tests that allows attorneys to adequately represent their clients without compromising test security. In Daubert v. Merrell Dow Pharmaceuticals, Inc., the U.S. Supreme Court suggested that the following factors be considered when allowing expert scientific testimony: (1) Has the technique been tested in actual field conditions (and not just in a laboratory)? (2) Has the technique been subject to peer review and publication? (3) What is the known or potential rate of error? (4) Do standards exist for the control of the technique’s operation? (5) Has the technique been generally accepted within the relevant scientific community?

• Regarding whether the technique has been “field tested,” attorneys have a right to know how many validation studies are available regarding specific techniques, and the type of methodology used to test/establish validity. For example, in validation of performance validity tests (PVTs), some studies involve simulators (participants instructed to feign) while others utilize “known groups” (actual credible and noncredible test takers); to the extent that tests are validated solely through simulators, questions could be raised as to whether the technique has been “field tested.”

• Regarding peer-review and publication, attorneys should have access to information regarding the number of peer-reviewed publications pertaining to particular tests.

• Regarding the error rate of the technique, attorneys are entitled to have the following information regarding psychological and neuropsychological tests:

  • Classification accuracy (e.g., cut-off specificity and sensitivity) so that the false positive and false negative rates can be evaluated.

  • Characteristics of the validation studies, including sample size and sample demographic information that might impact the error rate in general, and for specific patients. For example, small sample sizes in validation studies can result in an increased error rate due to reduced reliability of the data, and, to the extent that demographic characteristics of validation samples deviate from the demographic characteristics of particular patients, use of a test with individuals not represented in validation samples can increase the error rate.

• Regarding standards for control of the technique, attorneys should be given access to information as to whether tests have standardized administration methods (although the actual administration methods should not be disclosed). Determination as to whether standard procedures have been followed can be provided by psychology testing experts on behalf of any interested parties.

• Regarding acceptance of the technique within the field, attorneys should have access to information as to whether the tests are in common use in the psychological and neuropsychological community, such as is documented in published surveys of test usage.

Conversely, release of additional test information that would compromise future use only serves to undermine the tests, rather than fairly inform. Specifically, test stimuli, test procedures and instructions, and scoring methods, including score reports, provide no useful information to non-psychologists attempting to gauge test accuracy, whereas disclosure of this information will compromise future use of the tests. That is, to the extent that test takers obtain advance knowledge regarding key test information, tests can be rendered ineffective for the purposes of accurately measuring neurocognitive skills and determining performance/symptom validity. For example, if a PVT were to involve shoe tying, knowledge of the test paradigm would not allow an attorney to evaluate the effectiveness of the test, but this information could allow coaching of future clients to ensure passing this portion of the validity testing within a given test battery (e.g., “When you are asked to tie your shoes, perform well!”). Moreover, attorneys do not have training in psychological/neuropsychological testing, and thus are not able to judge whether testing procedures have been accurately followed. Requests that attorneys be allowed to observe testing and/or have access to tests, test data sheets, and score reports thus appear to not accomplish a purported reason for access, namely, critique of the test results; accurate judgment of proper administration, scoring, and interpretation procedure requires the assistance of a retained expert. Professional psychological organization position papers (see National Academy of Neuropsychology, 1999; American Psychological Association, 1999) and test security recommendations provided by test publishers (e.g., Pearson Assessments) recommend that in litigation, protected psychological test information be conveyed directly to psychologists retained by opposing counsel; these receiving psychologists can scrutinize test data sheets, score reports, and recordings to determine if tests have been appropriately administered, scored, and interpreted.

Cross-examination questions at times ask for test items and instructions, scoring methods, and other sensitive test information. To the extent possible under the law, neuropsychologists should decline to provide specific information that compromises test security in their oral answers, just as they withhold it in paper/written format. Neuropsychologists can explain that the purpose of withholding sensitive test information is to uphold the mandate of our profession regarding maintaining test security. In this manner, we are attempting to educate the court that a critical test security issue has emerged, thereby providing the court with the opportunity to intervene on behalf of our tests. If ultimately instructed to answer such questions by the judge, neuropsychologists can request that the problematic testimony be removed from publicly accessible records.

Despite the option of having test materials forwarded directly to retained experts, attorneys may still demand direct access to protected psychological test information. As a last resort in managing demands by non-psychologists for sensitive psychological and neuropsychological information, psychological organizations and test publishers have recommended that protective orders be used to guard the security of tests. Protective orders typically prohibit the copying of test materials, require that the test materials be returned to the psychological professional at the conclusion of the legal proceeding, stipulate that only individuals directly involved in the litigation can have access to the test materials, and indicate that the materials are not to become part of the publicly available records from the legal proceeding. Such protective order requirements have been supported by some test publishers (cf., Pearson; pearsonclinical.com/legal.html).

Unfortunately, even when ordered by a judge, the execution of the protective order may not be fully enforced. With current technology, it would take only a violation by a single individual to result in widespread digital access of test materials by non-psychologists. In fact, the U.S. Supreme Court commented on protective orders in a case addressing psychological tests (Edison), and expressed uniform skepticism of such orders, with the majority registering concern about intentional violations and the minority noting problems with inadvertent disclosure; the Supreme Court intimated that it may be problematic for psychologists to release raw data and test materials to nonpsychologists (Bush et al., 2020). Also, even when fully enforced, protective orders do not “erase” newly acquired knowledge of tests from the minds of non-psychologists. For example, “punchlines” and rationales of tests (e.g., the significance of below chance responding) can be easily retained and recalled by attorneys, and used in educating future clients.

Alternative to, and/or in addition to, use of protective orders, some practitioners may opt for other methods of maintaining test security, such as when faced with attorney demands for access to audio recordings of testing and test data sheets that show questions and answers, they proactively adjust test batteries and materials to protect tests. For example, when under a judge’s order to allow audio recording of testing by plaintiff, a neuropsychologist may opt to only administer visual-type tests that are not unduly compromised by an audio recording (e.g., WAIS-IV Perceptual Reasoning and Processing Speed subtests, visual memory tests, self-administered paper and pencil vocabulary and verbal reasoning measures, and PVTs that involve visual processing and visual recall/recognition). Testing reports can indicate that the battery was abbreviated due to the necessity for maintaining test security given the imposed conditions of the exam. If test data sheets are ordered to be turned over to opposing attorneys, neuropsychologists could opt, prior to conducting the exam, to use reconfigured test data sheets that do not contain test stimuli, test instructions, or scoring algorithms, and that also do not violate test form copyrights. Further, practitioners can superimpose large font, diagonally placed, non-removable labeling across test data sheets to indicate that they were released under protective order for a specified case, and that paper copies (which are more difficult to circulate) be forwarded in preference to digital versions.

When faced with a judicial order that a neuropsychologist believes undermines test security, the practitioner can choose to withdraw from the case, and can also opt to document, through canvassing of other local neuropsychologists, that the broader neuropsychology community refuses to conduct exams under invasive parameters that threaten the validity of the assessment process. This latter action may demonstrate to the court that the imposed conditions are not reasonable and not necessary. For example, in 2012 and 2013 all Illinois neuropsychologists board-certified by the American Board of Clinical Neuropsychology signed an affidavit refusing third party observations and recording of evaluations. This document was then made available in professional forums for use by anyone facing such requests (cf. https://theaacn.org/wp-content/uploads/2015/10/third-party-observer-affidavit-completed-2013-all-abcn-illinois-practicing-psychologists.pdf). Other locales have taken similar actions.

In some states, such as California (Carpenter v. Yamaha Motor), neuropsychologists may be required to disclose to plaintiff counsel the list of tests to be administered prior to the exam. However, advance notice of the specific tests to be administered allows pre-evaluation searching on the internet, thereby facilitating learning about the tests prior to the examination. In Ragge v. MCA/Universal Studios, 165 F.R.D. 605 (C.D. Cal. 1995) the court concluded under FRCP 35 that no purpose would be served by requiring defendant’s examiner “to select, and disclose, the specific tests to be administered in advance of the examination,” and that the order under FRCP 35 can be satisfied by identifying the “types of tests” that will be employed (e.g., memory, attention, problem-solving, etc.). The latter approach satisfies concerns regarding test security, while informing the interested parties of the general nature of the examination process.

In a similar vein, in 2018, NAN affirmed that third parties, who lack appropriate training in neuropsychological assessment, should not determine which specific tests are to be included or excluded in a forensic or clinical neuropsychological exam. However, there are some unique situations in which pre-exam specification of tests may be required, such as baseline and in-season testing of athletes, litigation related to professional sports (e.g., the National Football League class action lawsuit), and requirements by academic institutions regarding test-specific documentation needed for test accommodations. In such exceptional circumstances, in which test batteries are predetermined prior to participation of the neuropsychologist in the assessment process, there may be no alternative, if the practitioner chooses to participate. Practitioners are encouraged to weigh the pros and cons of conducting evaluations in such circumstances. While no doubt arising from a desire to impose uniformity and equity, such practices can, unfortunately, also inadvertently undermine the validity of test results by identifying tests in advance.

It is preferable that test forms visible to examinees not contain test names; allowing examinees to see the names of tests as they are being administered provides a learning opportunity as to which procedures correspond to particular tests. Such information could alter test behavior in re-evaluations. Conversely, recall of test procedures without knowing which tests they are linked to has less potential for compromise of test security. It would be optimally proactive if the field of neuropsychology and clinical psychology shifted toward development, and routine use, of test data sheets (and associated score reports) that do not contain protected test information, and test materials that do not display test names.

Practice errors that result in compromise of test security often occur with novice practitioners who are not experienced with the legal system. These individuals may reflexively provide test information requested in subpoenas and other written requests from attorneys, as well as in response to deposition and cross-examination questions, under the mistaken impression that they have no other option. When served with demands and requests to provide protected psychological/neuropsychological test information to non-psychologists, neuropsychologists should decline, citing commitment to the test security requirements of our field as memorialized in the Ethics Code for psychologists and various positions papers. Instead, practitioners should provide alternative options, such as offering to forward the information to other licensed psychologists. It is also recommended that practitioners consult with retaining or personal attorneys in regard to responding to subpoenas or other orders, and to develop declarations and related materials to educate the court as to the necessity of maintaining test security. Additional guidance for handling subpoenas is contained within the American Psychological Association’s (2016) “Strategies for private practitioners coping with subpoenas or compelled testimony for client/patient records or tests data or test materials.” More experienced neuropsychologists can also be contacted for advice and sample declarations, and neuropsychological organizations are becoming aware of the need to develop and compile such materials for access and use by neuropsychologists when navigating this difficult terrain. An introductory summary of basic information for neuropsychologists who are having their first encounters with the legal system is available (Sweet & Klipfel, 2021).

Concluding guidance for protection of psychological and neuropsychological tests

• Information that in any manner undermines current or future use of a psychological or neuropsychological test is not to be divulged or made accessible to non-psychologists. Such information includes test rationales and paradigms, instructions and procedures, stimuli (including test questions and answers), and scoring methods. This prohibition applies to all assessment settings, including evaluations conducted in the context of civil and criminal forensic litigation, disability claims, academic admissions or accommodations, and patient care and research.

• Test materials are to be retained under the direct control and supervision of licensed psychologists. Practitioners employing remote administration methods for personality inventories and other tests as offered by test publishers should ensure test security through use of onsite trained proctors or remote proctoring per the procedures described by Corey and Ben-Porath (2020), which include real-time teleconference observation of the test taker and the testing site, and prohibition of observers and screenshots, photographing, or other recording of test materials. Protection of test security in telehealth administration of neurocognitive tests is more problematic than for personality measures that are read and answered silently by test takers. Neuropsychological exams involve oral instructions, questions, and answers, in addition to presentation of pictures, designs, and other visual test stimuli, and prohibition of surreptitious audio recording is more difficult to enforce. Therefore, test security concerns currently preclude administration of many, if not most, neuropsychological tests via telehealth neuropsychology platforms. One argument for introduction of telehealth neuropsychology services is that they could provide previously unavailable access to care for marginalized and underserved populations. The COVID-19 pandemic forced long overdue consideration of telehealth procedures for neuropsychological testing, and an important arena for the future will be to develop telehealth neuropsychology methods that are robust to test security issues and coaching, but which unfortunately are not readily available at the present time.

• An exception to the admonition to limit access to protected test materials to psychologists would be during collaborative work with psychometrists, interpreters, trainees, and research assistants. Interpreters, trainees, and research assistants should be instructed on the importance of test security prior to the commencement of exams, and it is preferable that they sign attestations indicating that they will not divulge protected test information.

• The routine practice of describing or reproducing test stimuli, instructions, and scoring in books, research publications, and other methods of disseminating research and practice information that are accessible to non-psychologists, should be discontinued. Practitioners and researchers should instead refer readers to test manuals and other test development materials, unless inclusion of such information in publications is critical for understanding research findings and/or translating research results into clinical practice. It is recommended that journal editors and reviewers implement procedures for publications on psychological/neuropsychological tests whereby authors confirm that no information that could compromise test security has been included, or if included, a justification for such. Alternatively, and preferably, rather than including protected psychological test information in publications, such information could be moved to a category of supplemental material, with access regulated by authors, professional organizations, and/or journals and publishers, and with release only to licensed psychologists or others with appropriate log-in credentials.

•  Disclosure of test cut-offs is a unique consideration. Publications are the most direct and rapid method of circulating up-to-date cut-offs to practitioners. However, investigators should routinely consider development of cut-offs which, if accessed by non-psychologists via publications, do not compromise future use of tests. Examples of cut-offs that are more robust against coaching efforts include discriminant functions, combination scores and equations incorporating various types of test data (e.g., accuracy and time scores), standard scores, T-scores, percentiles, etc. Release of such information provides no “road map” as to how to adjust future test performances. Even disclosure of raw accuracy and time scores is not necessarily compromising of test security if publications accessible to the general public do not report total number of items on tests or the exact sections of tests that are timed.

• In academic settings, protected test information should not be provided in lectures, posted in on-line course materials, or distributed in written materials to which non-psychologists are in attendance or have access. Only trainees who are at a graduate-level or above in psychology education, en route to licensure as a psychologist, should have access to such materials when deemed a necessary part of their training. Test materials used for training purposes should be stored in secure locations, and trainees should receive careful instruction regarding test security when test materials are in their custody.

• Information relevant to test effectiveness and error rate, including classification statistics, validation study methodology, and reliability and applicability of the validation studies to particular patients, can be provided to non-psychologists. Similarly, information regarding frequency of test usage in the neuropsychological community, extent of peer-reviewed publications relevant to applications of the test, and whether test methods employ standardized administration procedures can also be provided.

• In the context of testing performed in potentially adversarial circumstances (e.g., civil litigation), the preferable manner of conveying protected test information is from one licensed psychologist to another, with the receiving psychologist assuming the responsibility for test security. Protective orders may not completely ensure test security. If protective orders are used, it is recommended that they be buttressed by additional protective methods, such as by using redacted test data forms or tailoring batteries to use instruments that are least likely to be affected by dissemination of recordings of the exam.

• As a general principle, the names of specific psychological/neuropsychological tests to be administered should not be provided in advance of neuropsychological exams, and neuropsychologists should be the professionals who determine which tests are to be included in any specific test battery. For the limited exceptions that require predesignated tests (e.g., testing for educational accommodations), practitioners should weigh the pros and cons related to attainment of valid results as they decide whether to participate in such an evaluation.

• Neuropsychological and psychological tests that are no longer needed or have become obsolete should be disposed of in an appropriate manner that guarantees no breach of test security. Examples of appropriate disposition of unneeded tests include turning them over to other psychologists, redistribution to under-resourced regions via professional organizations, or destroying the tests. Selling of older or unneeded tests on on-line platforms is discouraged unless individuals purchasing the tests are vetted for appropriate credentials.

Closing comments

Establishing and maintaining policies and procedures that protect psychological and neuropsychological tests from inappropriate disclosure to non-psychologists is essential, and such guidelines likely require regular updating, given rapid development in technology and emerging legal precedents. Otherwise, there is a clear risk that the future use of such tests will not produce valid results. The range of academic/training, clinical, and forensic scenarios in which there is risk of improper disclosure of test information is broad. When related to academic/training and clinical scenarios, access to test information is primarily under the control of the psychologist examiner, with the exception of testing for educational accommodations. It is primarily within forensic scenarios that practitioners may encounter energetic opposition to maintaining test security from attorney advocates. This latter scenario requires special consideration, in order to balance the legal perspective of a ‘level playing field’ among individuals with opposing legal interests versus the need to guard against the invalidation of the very tools that test developers and publishers spend considerable time and money creating, and without which psychologists who provide testing services cannot practice. With regard to respecting the rights of all interested parties in forensic scenarios, practitioners consider the best means of maximizing test security concerns, while facilitating legal and appropriate access to test information. Maintaining psychological and neuropsychological test security is essential for protecting the profession’s ability to serve the needs and safety of society.

Executive summary

Psychological and neuropsychological test security is a critical issue which, without meticulous adherence by practitioners, threatens the entire enterprise of psychological and neuropsychological testing. If test questions and stimuli, instructions, scoring methods, and other sensitive information become available to non-psychologists, test result accuracy will be sacrificed, at considerable risk and cost to society, as well as to the viability of clinical neuropsychology as a specialty.

Objective psychological and neuropsychological testing requires that examinees have no access to test questions and answers in advance of their examination. Unfortunately, despite previous position papers in neuropsychology and psychology on test security, exact procedures to be followed to maintain test security have largely not been specified. In addition, the evolving digitization of information can provide for ready uploading of test materials, leading to mounting threats to test security.

Requests and, indeed demands, which are particularly problematic for neuropsychologists include: allowing observers at neuropsychological examinations; allowing test takers and/or their legal counsel to record by audio or video the entire examination, including test administration procedures; and providing test item responses and test materials from neuropsychological exams to non-psychologists (e.g., attorneys, physicians, examinees), who do not have adequate training in interpretation of neuropsychological testing and are under no ethical obligations regarding test security.

With regard to respecting the rights of all interested parties in forensic scenarios, practitioners consider the best means of maximizing test security concerns, while facilitating legal and appropriate access to relevant test information. This AACN position paper provides comprehensive guidelines to maximize test security in all venues in which neuropsychologists and psychologists are involved, such as forensic, educational, and clinical assessments; teaching and training settings; and research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Appendix 1.

State of Maine test protection statute

Whereas, current law does not protect from disclosure neuropsychological and psychological testing materials; and

Whereas, disclosure of neuropsychological and psychological testing materials and distribution to even just one person who is the subject of testing or to many persons who may be the subjects of the testing will compromise and invalidate such testing; and

Whereas, maintaining the integrity of the testing materials is critical to test results and to the functioning of the system of neuropsychological and psychological testing in this State and requires the immediate action of the Legislature; and

Whereas, in the judgment of the Legislature, these facts create an emergency within the meaning of the Constitution of Maine and require the following legislation as immediately necessary for the preservation of the public peace, health and safety; now, therefore,

Be it enacted by the People of the State of Maine as follows:

A. Except as provided in paragraph B, neuropsychological or psychological test materials and neuropsychological or psychological test data, the disclosure of which would compromise the objectivity or fairness of the evaluation methods or process, may not be disclosed to anyone, including the person who is the subject of the test, and are not subject to disclosure in any administrative, judicial or legislative proceeding.

B. A person who is the subject of a neuropsychological evaluation or psychological evaluation is entitled to have all records relating to that evaluation, including neuropsychological or psychological test materials and neuropsychological or psychological test data, disclosed to any neuropsychologist or psychologist who is qualified to evaluate the test results and who is designated by the person. A neuropsychologist or psychologist designated to receive records under this paragraph may not disclose the neuropsychological or psychological test materials and neuropsychological or psychological test data to another person.

Additional information can be found at the following web site: http://www.mainelegislature.org/legis/bills/bills_126th/chapters/PUBLIC353.asp

Appendix 2.

American Psychological Association (2002) ethical principles of psychologists and code of conduct, standards 9.04 versus 9.11

9.04 Release of Test Data. (a) The term test data refer to raw and scaled scores, client/patient responses to test questions or stimuli, and psychologists’ notes and recordings concerning client/patient statements and behavior during an examination. Those portions of test materials that include client/patient responses are included in the definition of test data. Pursuant to a client/patient release, psychologists provide test data to the client/patient or other persons identified in the release. Psychologists may refrain from releasing test data to protect a client/patient or others from substantial harm or misuse or misrepresentation of the data or the test, recognizing that in many instances release of confidential information under these circumstances is regulated by law. (See also Standard 9.11, Maintaining Test Security.)

(b) In the absence of a client/patient release, psychologists provide test data only as required by law or court order.

9.11 Maintaining Test Security. The term test materials refers to manuals, instruments, protocols and test questions or stimuli and does not include test data as defined in Standard 9.04, Release of Test Data. Psychologists make reasonable efforts to maintain the integrity and security of test materials and other assessment techniques consistent with law and contractual obligations, and in a manner that permits adherence to this Ethics Code.