Multidimensional Measurement Within Adult Protective Services: Design and Initial Testing of the Tool for Risk, Interventions, and Outcomes

Abstract

This study describes the development, field utility, reliability, and validity of the multidimensional Tool for Risk, Interventions, and Outcomes (TRIO) for use in Adult Protective Services (APS). The TRIO is designed to facilitate consistent APS practice and collect data related to multiple dimensions of typical interactions with APS clients, including the investigation and assessment of risks, the provision of APS interventions, and associated health and safety outcomes. Initial tests of the TRIO indicated high field utility, social worker “relevance and buy-in,” and inter-rater reliability. TRIO concurrent validity was demonstrated via appropriate patterns of TRIO item differentiation based on the type of observed confirmed abuse or neglect; and predictive validity was demonstrated by prediction of the risk of actual APS recurrence. The TRIO is a promising new tool that can help meet the challenges of providing and documenting effective APS practices and identifying those at high risk for future APS recurrence.

KEYWORDS:

• elder abuse and neglect
• Adult Protective Services (APS)
• risk assessment
• APS recurrence
• measure development
• reliability
• validity

INTRODUCTION

Although many people aged 65 and older maintain healthy and fulfilling lives, some are not as fortunate. Chronic illness, inadequate access to care, cognitive decline, loneliness and isolation, fixed incomes, and decreased mobility to seek services and social connections represent some of the challenges that place older adults at risk for abuse and neglect (Acierno et al., 2010; Dyer, Goodwin, Pickens-Pace, Burnett, & Kelly, 2007; Gorbien & Eisenstein, 2005). With the population aged 65 and over expected to increase from 40 million in 2010 to over 70 million by 2030 (Vincent & Velkoff, 2010), addressing elder mistreatment represents a significant and growing public health concern (Mosqueda & Dong, 2011).

The specific definitions may vary, but elder mistreatment is generally recognized to occur in a variety of forms, including psychological, financial, physical, and sexual abuse, as well as self-neglect and neglect by someone else (Anetzberger, 2005). A nationally representative study of cognitively intact persons aged 60 and over indicated that more than 1 in 10 (11.4%) experienced at least one form of mistreatment within the past year (Acierno et al., 2010). Unfortunately, mistreated elders often experience more than one form of abuse or neglect (Dyer et al., 2007), and the mistreatment rates for persons with cognitive impairments are even higher (Cooper, Manela, Katona, & Livingston, 2008; Schillerstrom, Salazar, Regwan, Bonugli, & Royall, 2009).

The negative personal and social repercussions of elder abuse and neglect are substantial. In addition to the immediate trauma, pain, and/or financial costs experienced by the individual, elder abuse and self-neglect are associated with increased nursing home placements (Lachs, Williams, O’Brien, & Pillemer, 2002), hospitalizations (Dong, Simon, & Evans, 2012), poor physical health outcomes (Dong, 2005), and increased mortality (Dong et al., 2009; Lachs, Williams, O’Brien, Pillemer, & Charlson, 1998). While the causal order is not always clear, mistreatment is often associated with poor mental health, particularly depression (Dyer, Pavlik, & Murphy, 2000).

To help address elder abuse and neglect, states were authorized by the federal government to create agencies that provide adult protective services (APS) as part of the 1974 Title XX Social Security Act amendments (Teaster, Wangmo, & Anetzberger, 2010). The primary tasks of APS agencies include receiving reports of suspected abuse and neglect, investigating the validity of those reports, and providing a range of services to eliminate or reduce instances of abuse and neglect. However, in contrast to child welfare agencies, the federal government has not played a substantial role in structuring or funding APS agencies or providing leadership for the field (Mixson, 2000; U.S. Government Accountability Office, 2011). This has contributed to wide definitional and service variations across geographic jurisdictions (Goodrich, 1997; Mixson, 2010), as well as a perception that the APS field overall has an underdeveloped knowledge base and a need for greater use of evidence-based tools and practices (Anetzberger, 2005; Anthony, Lehning, Austin, & Peck, 2009). This situation inhibits the ability of individual APS staff members and agencies, as well as the loosely defined national APS “system,” to adequately meet the growing need for high-quality services over the coming decades (U.S. Government Accountability Office, 2011).

Existing measurement tools related to elder mistreatment primarily consist of either screening tools utilized to detect potential abuse and neglect within particular populations, such as patients in health care settings (e.g., Bomba, 2006), or assessment protocols, often used by APS workers, to guide allegation investigation and the determination of abuse and neglect risk level (Anthony et al., 2009). In an effort to address the need for additional data and tools to support APS practice, we undertook an initiative to design and implement a new multidimensional tool, the Tool for Risk, Interventions, and Outcomes (TRIO). One specific goal was to develop an instrument that could be used to document and collect data related to multiple dimensions of a typical APS episode, including: (a) the investigation and assessment of allegations, (b) the identification of abuse and neglect risk factors, (c) the delivery of a range of potential interventions, (d) the achievement of specific outcomes, and (e) an assessment of the prognosis for future APS involvement at case closure. Ultimately, the TRIO was designed to allow for a comprehensive description of APS response to and involvement with clients and the resulting client outcomes. Collecting this multidimensional data in a systematic manner is expected to facilitate feedback to APS social workers, supervisors, and administrators regarding aspects of each dimension independently, as well as the patterns and cross-dimensional linkages between the components of typical APS episodes. Such information is intended to help guide individual APS social worker practice decisions, as well as provide local APS systems and the field of elder abuse and neglect more generally with a better understanding of the relationship between client risk characteristics, APS services received, and the achievement of client outcomes. The data resulting from use of the TRIO will help in the effort to identify promising APS practice strategies and increase the capacity of APS systems to demonstrate service effectiveness. In the following sections we report on the development and initial testing of the field utility, reliability, and validity of the TRIO.

METHODS

Instrument Development

Elder abuse and neglect are “wicked problems” (Anetzberger & Balaswamy, 2010), in that each case of elder abuse has unique characteristics, frequently represents symptoms of other problems, and the definition of the problems and associated solutions may vary substantially between relevant stakeholders. Many individuals and organizations play vital roles in the prevention, identification, and amelioration of elder abuse and neglect. However, the APS system is uniquely and specifically situated to respond to such needs. Therefore, it is important to provide APS social workers, supervisors, and administrators the tools to effectively achieve organizational and client goals in an often changing and ambiguous environment. The development of the TRIO was prompted by the desire of APS administrators and social workers to create a tool that would help guide APS social work practice and provide better data regarding risks, interventions, and outcomes, and their relationships to one another.

The TRIO was designed by APS administrators and social workers from Ventura County, California, a midsized county with approximately 825,000 residents, of whom 11.7% are 65 years old and older (U.S. Census Bureau, 2010). In California, APS is a county-administered, short-term system, with most cases closing in less than 90 days. Figure 1 presents the Practice-based Model of Adult Protective Services (PMAPS) to illustrate the key components and operations of typical APS systems. As discussed above, jurisdictional variations may contribute to some differences in how the work of specific APS agencies is organized and carried out, but the core features shown in Figure 1 are anticipated to be relevant to most APS systems.

FIGURE 1 Practice-based model of Adult Protective Services (PMAPS).

The model highlights the dependence of APS systems on receiving allegations of abuse or neglect from outside of the APS system by mandated reporters or voluntarily from concerned community members. Once the APS system receives the allegation, an investigation is initiated to assess for risk factors indicative of past/current abuse and neglect, as well as factors that may contribute to the future likelihood of abuse and/or neglect. This risk investigation process results in an official determination of whether some form of abuse/neglect has already occurred. Based on the official abuse/neglect findings, as well as the identification of other known risk factors, it is expected that APS social workers will initiate relevant interventions in an effort to eliminate the area(s) of concern. Through the course of delivering any APS-related interventions, APS workers continue to assess for additional risk factors (represented in Figure 1 by the arrow flowing back from APS interventions to the investigation/assessment of risk factors). This may prompt further APS interventions and/or official determinations regarding abuse/neglect in an iterative loop.

The final component of the proposed APS practice model focuses on the outcomes of the APS episode. In the proposed model, outcomes are not limited to “process” indicators such as the number of client contacts or whether the case closed on time, but more importantly they include measures of what was (or was not) accomplished as a result of involvement with the APS system. A range of potential APS-related outcomes are possible at the conclusion of APS system involvement, including an indicator of the extent to which abuse/neglect concerns were resolved, as well as indicators specifically related to an area(s) of concern (such as improvements in health, safety, finances, etc.). For some APS clients, one potential outcome of particular concern is that they may return back into the APS system via a future allegation (represented in Figure 1 via the dotted line linking outcomes back to the initial step of the APS practice model, allegations from the community).

While the model discussed above may generally reflect APS practice experiences and expectations, limited data and research exist regarding the specific relationships between risks, interventions, and outcomes within the context of APS service delivery. Consistent with recommendations from elder abuse and/or neglect research (e.g., National Research Council, 2003), and motivated by the need to generate better practice level knowledge, the TRIO was developed to collect and integrate information related to multiple dimensions of typical client encounters with APS, including the investigation and risk assessment functions, the types of interventions provided, and the outcomes achieved.

Initial tool development efforts focused on generating extensive lists of potentially relevant items to include on the instrument related to the three primary dimensions: risks, interventions, and outcomes. The risk indicators were selected from a review of existing standardized APS assessment risk instruments that has been endorsed by a survey of APS workers (Anthony et al., 2009; Bay Area Social Services Consortium, 2007). The TRIO risk indicators covered common factors associated with specific types of abuse and/or neglect (e.g., physical, sexual, and psychological abuse, exploitation, or neglect) as well as indicators of more generalized vulnerabilities (e.g., environmental concerns or resource limitations). A detailed list of the individual risk, intervention, and outcome items from the TRIO is provided in the Appendix. The list of intervention items was developed through an examination of the full range of services that APS workers typically provided or could initiate with external partners. The identified interventions reflected basic social work practices (e.g., establishing trust, client acceptance that a problem exists, etc.) as well as best/promising practices within APS (e.g., consultation with multidisciplinary teams, in-home assessment by health/mental health professionals). These interventions ranged from less intensive activities like providing education about health and safety topics to more complex interventions like initiating guardianship/conservatorship. All risk and intervention items were measured dichotomously to indicate the presence or absence of a particular item.

Outcome items were identified through an iterative process of consultation between APS administrators and social workers and informed by the overall APS practice goals of promoting safety, health, and improved quality of life. Through these deliberations it was determined that three categories of outcomes would be beneficial to include on the TRIO: (a) a range of potential health and safety outcomes achieved via involvement with APS staff (e.g., dichotomous indicators that indicate the achievement of specific outcomes such as recouped financial losses, linked to housing, improved nutritional status, etc.), (b) a measure to record whether the original protective issue was eliminated, reduced, unresolved, or found to be not evident, and (c) a measure of the expected prognosis for nonrecurrence back to APS that is completed at case closure (measured on a six-point Likert-type scale ranging from “poor” to “excellent”). The approach for assessing outcomes with the TRIO was purposefully intended to model the foundational social work practices of establishing goals and evaluating progress toward those goals. Given that APS is a voluntary program and involvement is often required to be a short-term interaction (e.g., often less than 90 days from initial investigation to case closure), it may not always be possible to fully eliminate or reduce the protective issue with clients, many of whom may be experiencing multiple challenges.

Change is often difficult, and persons with the capacity for self-determination may require time to contemplate and act upon the level of change needed to completely resolve the health or safety issue. Therefore, it is important to acknowledge progress toward the ultimate goal of establishing safe living situations for clients, while utilizing the social worker’s knowledge of the case to assess prognosis for future involvement with APS. Measuring the social worker’s assessment of prognosis for future APS involvement is intended to identify who would likely benefit from long-term case management services were these resources available. All TRIO items were structured so that each can be used with either elder adult (age 65 and over) or dependent adult (age 18–64) APS clients. For the empirical analyses reported below we focus only on elder adult TRIO results. To promote consistent usage of the TRIO, detailed definitions of each item on the TRIO were established and refined with input and feedback from the APS staff (i.e., social workers, supervisors, administrators; definitions available upon request from the authors). In this way, the TRIO combines a standardized and comprehensive approach to investigation and risk assessment with the social worker skills, knowledge, and judgment central to its design. It guides the practice of investigation, assessment, and interventions but does not explicitly prescribe the decisions for the social worker.

From a practice perspective, the TRIO was designed to advance a cohesive and comprehensive approach to investigation and assessment of the client’s situation and needs within a dynamic and often complicated context. It was intended to promote and cultivate a perspective that each case unfolds as a process comprising the interrelationships between the clusters of risk indicators, interventions, and measurable outcomes. By completing the TRIO, data is systematically collected for each episode that records the identified risks, the interventions provided, and the outcomes achieved. Such information is regularly utilized by APS practitioners, supervisors, and administrators for case planning and consultation. Academic–practitioner partnerships as in the present study are critical for conducting the more extensive analyses of the interrelationships between TRIO items and dimensions of inter-relationships, with the ultimate goal that increased knowledge and understanding of these associations and pathways will enable the individual social worker to deliver the right intervention at the right time with an increased likelihood of achieving positive outcomes for the client.

Field Testing and Initial Implementation of the TRIO

Ventura County APS receives approximately 2,500 referrals annually, with about 85% meeting criteria for investigation and assessment by an APS social worker. Approximately 50% of the Ventura County APS social workers had bachelor’s level training and 50% had a master’s degree in social work or a related field. The social workers maintained an active caseload of about 30 to 35 clients during this time period. The majority of clients were White (68%), followed by Hispanic (22%), other (6%), and unknown (4%). Most of the clients served by Ventura County APS (69%) were at least 65 years old (47% were aged 65–84, and 22% were 85 years old and older). Approximately 1/3 (31%) were aged 18–64. Staff was trained on and began field testing an initial version of the TRIO in 2008. Over an approximately 12-month period, an iterative process of implementation, feedback, and refinement was utilized to improve the fit of the tool with practice realities. To promote consistency in the usage and completion of the TRIO, the initial definitions for each component of the TRIO were revised based on the field tests. In the fall of 2009 the TRIO was fully implemented into practice. Training and ongoing supervision were provided to all staff utilizing the TRIO. During field testing and initial implementation the TRIO was used in a hard copy, paper format as shown in Figure 2.

FIGURE 2 Tool for Risk, Interventions, and Outcomes (TRIO).

To facilitate ease of completion in the field and in the office, the TRIO was subsequently converted to an electronic version hosted on a secure network. Of particular interest, the social workers requested that the visual representation of the TRIO curve be continued in the electronic version of the TRIO since it provided a useful conceptualization of the process that APS workers follow when assisting clients. Similar to the “Jellinek Curve” used in the field of substance abuse and addiction (Blume, Rudisill, Hendricks, & Santoya, 2013), working “down” the curve for risk assessment was consistent symbolically with the likely negative health and safety trends associated with the unchecked accumulation of risk factors. In contrast, the implementation of APS interventions was perceived as reversing that trajectory and turning it “up” toward more favorable client health and safety outcomes.

Pilot Study

Following the development of a new measure it is important to assess the field utility, reliability, and validity of the tool. Explicitly examining these constructs can support subsequent implementation and usage as well as the identification of any aspects that may benefit from further refinement. In the sections below we describe the methods used for assessing each of these constructs.

Field Utility

Tools are only useful to the extent that the frontline staff tasked with using them are able and willing to complete them and make the results a part of case planning and outcome assessment. Important factors to examine related to field utility are the relative burden and benefits associated with tool completion. To help evaluate these considerations, an anonymous survey was developed and administered, independently from the TRIO developers, by the Ventura County Quality and Improvement Department in the Office of Strategic Management. The survey was completed by all APS social workers implementing the TRIO (n = 12). The survey was first administered in March 2011, corresponding to when the TRIO was in a hard copy, paper format, and then again in October 2011 following the conversion of the TRIO to an electronic format. The same APS social workers completed the survey at each time point, and the response rate at both times was 100%. A 4-point Likert-type scale was used at both times to assess the level of respondent agreement with a series of statements (strongly disagree to strongly agree). The APS social workers completing the field utility survey had been involved in the earlier phases of TRIO development and refinement and had utilized draft versions of the tool with clients in order to provide feedback regarding individual items. Key field utility survey findings are reported in the results section below.

Reliability

Given the multidimensional structure of the TRIO as an index of multiple potential risk, intervention, and outcome items related to a complex phenomenon, traditional measures of scale reliability or internal consistency (e.g., Cronbach’s alpha and factor scores) were not applicable (Streiner, 2003; Terwee et al., 2007). However, consistency among the various APS staff completing the TRIO remained an important and testable consideration. To assess the reliability of the TRIO across practitioners, an inter-rater reliability (IRR) study of the TRIO was conducted. The IRR study was accomplished by constructing three different detailed client vignettes (available upon request), having all the social workers complete a TRIO for each vignette, and then assessing the extent to which the TRIO responses were the same. A total of nine social workers reviewed the three different client vignettes and completed a TRIO for each one. In addition, the developer of the vignettes completed a master version of the TRIO for each vignette to establish a standard for how the TRIO should have optimally been completed. The text of the vignettes and completion of the TRIO using the vignettes was pilot tested with individuals not participating in the IRR study who were familiar with APS practice. Based on this feedback, additional revisions to the vignettes and the master TRIO response key were made prior to the IRR study with the actual APS social workers.

In our IRR study, we examined two forms of rater agreement. First, we assessed the overall level of agreement between each rater and every other rater. Then we compared each APS social worker to the master version to determine the extent to which the raters typically agreed with the “correct” TRIO response. As discussed in more detail below, the type of rater agreement being assessed (all-to-all vs. comparison to master), and the level of measurement of the TRIO items (categorical vs. ordinal), determined which specific empirical test was used to measure rater agreement. Kappa is a commonly used test statistic that measures the degree of similarity in the responses across raters for categorical variables that accounts for the level of agreement expected by chance alone (Fleiss, Levin, & Paik, 2003). The value of kappa typically ranges from 0 to 1, with 0 indicating low or no agreement between raters and 1 indicating complete agreement between raters. While specific cut points are arbitrary, Landis and Koch (1977) have proposed the following general standards for identifying measures with substantial agreement (kappa = 0.61–0.80) and almost perfect agreement (kappa = 0.81–1.0).

For the dichotomous (i.e., Yes/No) indicators from the risk, intervention, and outcome sections of the TRIO we computed Fliess’s kappa to assess the simultaneous comparison of multiple raters and Cohen’s kappa to compare the agreement between how each social worker compared to the “master” TRIO (Fleiss, 1971). We reported the average of the Cohen’s kappa for the social worker to master version comparisons. We report kappas for each of the risk, intervention, and outcome sections comprising the relevant TRIO indicators as well as an overall kappa combining all dichotomous risk, intervention, and outcomes indicators of the TRIO. For descriptive purposes and to supplement the primary assessment of reliability, we also list the percent agreement between the raters and the master TRIO for each individual risk, intervention, and outcome indicator in the Appendix.

For individual TRIO indicators comprising ordinal or ranked information (i.e., protective outcome and prognosis) we computed weighted kappas and intraclass correlations (ICCs) to assess the comparison between all raters, as well as the level of agreement between the social workers and the master TRIO (Cohen, 1968; Streiner, 1995). As with Fliess’s and Cohen’s kappa, values of the weighted kappa and ICC that are closer to 1 indicate higher levels of agreement across raters.

Validity

Assessing the validity of a multidimensional checklist or index instrument such as the TRIO is challenging in that there are no independent “gold standard” measures to compare against that evaluate whether or not each individual TRIO item was truly present or absent for a given client. However, we utilized available data sources to develop two criterion-based approaches to objectively assess the validity of key aspects of the TRIO. First, concurrent validity was assessed through a comparison of the pattern of risk, intervention, and outcome indicators found for two distinct groups of APS clients: (a) those with an official confirmation of financial abuse-by-other, and (b) those with an official confirmation of self-neglect (clients with both forms of confirmed abuse were removed from the analyses). A small but growing literature has consistently found that while certain factors are nearly universally evident (e.g., low social support), client characteristics and risk factors often differ based upon the type of abuse and/or neglect experienced (Acierno et al., 2010; Jackson & Hafemeister, 2011; Payne & Gainey, 2005). As with the “known-groups” validation approach (Hattie & Cooksey, 1984), we purposefully selected two groups of APS clients that had experienced very different forms of abuse and neglect (i.e., self-neglect and financial abuse by others) and examined whether the pattern of risk, intervention, and outcome items selected on the TRIO were generally consistent with expectations. For example, whereas characteristics such as poor physical health, unclean living environment, and psychiatric conditions such as hoarding are particularly associated with self-neglect (Pavlou & Lachs, 2008), unique factors related to financial abuse-by-other include difficulties managing financial resources and vulnerabilities for exploitation (Triebel & Marson, 2012). Therefore, validly completed TRIOs should reflect significant and substantial differences in the prevalence of individual risk, intervention, and outcome indicators anticipated to be strongly associated with a specific form of abuse or neglect. Conversely, we should not find substantial differences between the two groups in the prevalence of TRIO items related to more universal risk factors such as cognitive decline and depression (Bomba, 2006; Heath, Brown, Kobylarz, & Castaño, 2005) and the receipt of more generalized APS interventions such as education/information sharing and building a positive social relationship between the APS worker and the client.

We used chi-square tests to statistically compare the prevalence of all TRIO risk, intervention, and outcome items for the two groups of APS clients. Additionally, we report the relative difference in prevalence between the two groups (calculated as the higher percentage prevalence divided by the lower percentage prevalence) to illustrate the magnitude of difference between the two populations in the relative prevalence of that TRIO item. The analysis used in this portion of the validation study was based on 589 older adults (age 60 or older) who had either a confirmed allegation of self-neglect or financial abuse-by-other episode that started between November 1, 2009, and June 30, 2011. These 589 older adults experienced a total of 669 episodes (471 confirmed self-neglect episodes and 198 confirmed financial abuse-by-other) during the study period. Approximately two-thirds (63.2%) of the 589 clients were female. The racial/ethnic background of the sample was 73.3% Caucasian, 15.3% Hispanic, 2.5% Asian, 2.0% African American, with 6.8% having unknown or missing racial/ethnic information. The average client age at the time of their index episode was 79.7 years (SD = 8.0).

The second criterion-based approach for assessing TRIO validity evaluated the capacity to accurately predict actual APS recurrence based on the TRIO item measuring the prognosis for nonrecurrence. Given that the TRIO was designed to guide and facilitate APS practice from the investigation, intervention, and outcome phases, the prognosis item completed at case closure represents a cumulative assessment of all that has been learned about the client, their environment, and their response to APS involvement. Therefore, prognosis for nonrecurrence incorporates many aspects of the TRIO, along with the expertise of the social worker, to allow for a final determination of the likelihood for APS recurrence. While not a direct measurement of the validity of each specific risk, intervention, and outcome indicator for a given client, successful prediction of APS recurrence provides evidence that APS social workers can validly evaluate the overall circumstances of APS clients at case closure and record that information on the TRIO.

Prognosis for nonrecurrence was evaluated on a 6-point scale ranging from “poor” to “excellent,” with standard definitions provided for each category (e.g., “Poor: Less than 50% chance of nonrecurrence and is very likely to have a recurrence of a protective issue in the near future/12 months”). To measure actual APS recurrence in a consistent manner, we established a 180-day observation period following the end date of an APS episode and then documented whether clients had another APS episode within the next 180 days. The recurrence analysis used in the validation study was based on 588 older adults (age 60 or older) who had a confirmed abuse and/or neglect allegation episode that started and ended between November 1, 2009, and March 31, 2011. Individuals who returned within 180 days with another confirmed abuse and/or neglect allegation were considered to have experienced subsequent APS recurrence. These 588 older adults experienced a total of 648 confirmed episodes of abuse and/or self-neglect during the study period. Approximately two-thirds (65.3%) of the 588 clients were female. The racial/ethnic background of this sample was 70.2% Caucasian, 19.6% Hispanic, 2.6% Asian, and 1.7% African American, with 6.0% having unknown or missing racial/ethnic information. The average client age at the time of their index episode was 79.6 years (SD = 8.1).

We used two methods to assess the capacity to predict actual 180-day APS recurrence using the “prognosis for nonrecurrence” indicator on the TRIO. First, we tested for a linear trend in the proportion of clients experiencing actual recurrence across the ordinal prognosis categories using the linear chi-square test. We then conducted a receiver-operating characteristic (ROC) analysis to determine whether the area under the curve (AUC) was significantly different from that expected by random chance. The AUC is commonly used to evaluate the predictive validity of measures such as diagnostic medical tests, or for predicting likely candidates for recidivism in criminal justice settings (Zweig & Campbell, 1993). The AUC indicates the probability that two randomly selected persons with differing outcomes will have scores on the tool that are consistent with their outcomes (e.g., the person who actually has the disease will have a higher score on the diagnostic tool than someone without the disease). An AUC value of 1 represents perfect sensitivity and specificity (i.e., no false negative or false positive predictions), and a value of .50 corresponds to the equivalent of using a coin toss (random chance) for prediction. These techniques provide an assessment of the extent to which prognosis can accurately identify clients most likely to experience APS recurrence. All quantitative analyses were conducted using release version 20 of the IBM SPSS statistical software package (IBM Corp, 2011). The Institutional Review Board at the University of California, San Diego reviewed and approved the protection of human subjects for this study.

TABLE 1 APS Social Worker Survey Results Regarding TRIO Utilization

	Percent Agree/Strongly Agree
	Time 1 (n = 12)	Time 2 (n = 12)
With proper training the TRIO is easy to use	100.0%	100.0%
The TRIO is worth the time to complete	91.7%	100.0%
The TRIO increases ability to assess a case more thoroughly	91.7%	100.0%
The TRIO risk indicators guide referrals and interventions	91.7%	91.7%
Using the TRIO increases work satisfaction	75.0%	91.7%

RESULTS

Field Utility

As shown in Table 1, the results of the anonymous survey of APS social workers (n = 12) indicated favorable perceptions regarding the use of the TRIO at both survey time points. All respondents agreed or strongly agreed that the TRIO was easy to use following proper training, and the overwhelming majority thought that the benefits of the TRIO made any associated burdens worthwhile. Additional qualitative feedback from social workers suggested that the electronic version further improved ease of use over the paper version. The TRIO was nearly universally viewed as a tool that helped the APS social workers assess clients more thoroughly (91.7% at Time 1 and 100% at Time 2) and guide referrals and interventions (91.7% at Time 1 and Time 2). Almost all APS social workers indicated that using the TRIO also helped to increase work satisfaction, particularly at Time 2 (75.0% at Time 1 and 91.7% at Time 2).

Reliability

The results from the IRR study presented in Table 2 indicated high levels of inter-rater agreement for all components of the TRIO, including risks, interventions, outcomes, protective outcome, and prognosis. These results were evident in both the analyses assessing agreement among all raters and the analyses of each rater compared to the “correct” or master TRIO response key. Following the recommended interpretation of Landis and Koch (1977), the section with the lowest scores, outcomes, was still considered to have “substantial” agreement between the raters in both sets of analyses (.73 and .78, respectively). The other individual TRIO components and the overall assessment of the TRIO all had values above .80, which suggests “almost perfect” agreement between raters.

TABLE 2 TRIO Inter-Rater Reliability Test Results

	TRIO	Agreement Among All			Agreement With Master
TRIO Section	Items (n)	FK	WK	ICC	CK	WK	ICC
Risks	68	.823			.861
Interventions	27	.900			.939
Outcomes	20	.726			.777
Risks, interventions, and outcomes	105	.837			.875
Protective outcome	1		.985	.998		.990	.995
Prognosis	1		.828	.980		.840	.913

Note. FK = Fliess’ kappa; WK = Weighted kappa; CK = Cohen’s kappa; ICC = Intraclass correlation.

TABLE 3 Prevalence of Select TRIO Items for Older Adults With Either Confirmed Self-Neglect or Confirmed Financial Abuse Episodes (n = 669)

TRIO Dimension	TRIO Item	Uniquely Associated With One Group	Confirmed Self-Neglect (n = 471)	Confirmed Financial Abuse (n = 198)	Sig.	Relative Difference in Prevalence
Risk	Confusion	No	39.3%(185)	30.8%(61)	*	1.3
Risk	Feels depressed	No	12.1%(57)	9.1%(18)		1.3
Risk	Passive behavior	No	9.1%(43)	7.1%(14)		1.3
Risk	Underweight/Frail	Yes	31.2%(147)	7.1%(14)	***	4.4
Risk	Hoarding	Yes	11.9%(56)	1.0%(2)	***	11.9
Risk	Unclean environment	Yes	25.9%(122)	2.5%(5)	***	10.4
Risk	Evidence of exploitation	Yes	1.5%(7)	67.2%(133)	***	44.8
Risk	Misuse of money	Yes	7.6%(36)	30.3%(60)	***	4.0
Intv.	Accepts education/information	No	61.6%(290)	53.0%(105)	*	1.2
Intv.	Established bond with social worker	No	72.6%(342)	72.2%(143)		1.0
Intv.	Nursing assignment	Yes	33.8%(159)	9.6%(19)	***	3.5
Intv.	FAST team	Yes	0.2%(1)	4.5%(9)	***	22.5
Outcome	Self-advocacy	No	27.8%(131)	28.3%(56)		1.0
Outcome	Improved mental health	No	9.1%(43)	8.6%(17)		1.1
Outcome	Financial stability	Yes	3.6%(17)	20.2%(40)	***	5.6
Outcome	Improved functioning	Yes	19.1%(90)	3.5%(7)	***	5.5

*p < .05; ***p < .001.

Validity

All TRIO items were utilized to allow for the identification of diverse patterns of risk factors regardless of initial allegation type.

Table 3 shows the prevalence of select TRIO items for two types of older adult APS episodes, those with confirmed self-neglect allegations and those with confirmed financial abuse-by-other allegations (results for all TRIO items are available on request). The findings indicated statistically significant and substantial differences in prevalence for TRIO items that are expected to be uniquely associated with one of the types of episodes being examined (e.g., TRIO items such as “Hoarding” and “Misuse of Money”). In contrast, TRIO items related to characteristics that are anticipated to be more universally applicable (e.g., TRIO items such as “Feels Depressed, Established Bond with Social Worker” or “Increased Self-Advocacy”) did not demonstrate substantial differences in prevalence. While some of these TRIO items exhibited statistically significant differences (e.g., “Confusion” or “Accepts Education/Information”), the relative difference in prevalence (as measured by the higher prevalence divided by lower prevalence) was still close to 1. This pattern of findings was evident among indicators from each dimension of the TRIO.

Predictive validity was assessed through an examination of the relationship between prognosis for recurrence determined at case closure and actual APS recurrence. Of the 648 episodes included in the analyses, 9.3% had a poor prognosis, 13.9% had a guarded prognosis, 25.6% had a fair prognosis, 35.0% had a good prognosis, 12.3% had a very good prognosis, and 3.9% had an excellent prognosis. The overall 180-day confirmed episode recurrence rate for the 648 episodes included in the analysis was 12.7%. The linear chi-square test indicated a statistically significant linear trend between prognosis for nonrecurrence and actual 180-day recurrence (χ²_linear = 31.2, df = 1, p < .001). The specific 180-day recurrence rate for each prognosis category is presented in Figure 3. Of those with a “poor” prognosis at case closure, 25.0% returned with a confirmed episode within 180 days. The observed recurrence rate declined for each successive prognosis category, culminating in a 0.0% rate for those with an “excellent” prognosis.

FIGURE 3 180-day APS recurrence prognosis category at case closure.

As shown in Figure 4, the ROC curve analysis indicated that the prognosis measure significantly improved the capacity to predict future recurrence (p < .001) with an AUC value of .69 (95% confidence interval from .63 to .74). We replicated the ROC analyses using alternative configurations of the prognosis variable with fewer classification categories (e.g., combined the “poor” and “fair” prognosis categories together) and obtained nearly identical values for AUC, the 95% confidence interval, and statistical significance. The AUC results indicated that 69% of the time a randomly selected person who returned back to the APS system would have had a lower prognosis level than a randomly selected person who did not return. The substantial presence of “tied” data (i.e., randomly selected recurrence and nonrecurrence cases within the same prognosis level) suggest that an AUC value of .69 is likely a conservative estimate of the true AUC (Zweig & Campbell, 1993).

FIGURE 4 ROC curve for 180-day APS recurrence prognosis at case closure.

DISCUSSION

The TRIO was designed to provide APS workers with a practice-oriented tool that would promote consistency and guide interactions with clients. Additionally, the TRIO purposefully documents data regarding client risks, interventions delivered, and outcomes achieved to facilitate ongoing analysis and empirically informed APS service improvement. The findings from the initial tests of TRIO feasibility, reliability, and validity were promising.

Field Utility

Overall, the results of the brief, anonymous APS social worker survey regarding their perceptions of using the TRIO with their clients were very favorable. Social workers nearly universally supported the use of the TRIO and felt that it contributed positively to their work activities (e.g., better client risk assessment, improved tailoring of interventions to clients, etc.) while not creating extensive burdens. The findings suggest a good fit between the TRIO and the needs of the APS social workers. These are important considerations, since a tool that is perceived to be not helpful or too difficult to use will be hard to implement in APS settings. The extensive and collaborative development and field testing process that relied upon and incorporated much social worker feedback likely contributed to such a positive view of the TRIO by the social workers. Their participation in the development process ensured that the resulting product was tailored to their needs while also likely creating a sense of ownership and commitment to the tool over time. Additional field utility testing in APS sites that were not directly involved in the development of the TRIO will be needed to further inform the degree to which the TRIO can be successfully implemented into general APS practice.

Reliability

The inter-rater reliability rating findings indicated a high level of consistency in the completion of all sections of the TRIO (risks, interventions, outcomes, protective outcome, and prognosis) by different APS workers. This is an important finding for two related reasons. First, consistent tool completion by different persons is crucial for any tool to yield meaningful results and not be entirely subject to the skill or biases of the completer. Second, APS work is inherently complicated and nuanced, so one of the goals of the TRIO was to bring a greater element of standardization and definitional specificity to their activities. The results of the inter-rater reliability study demonstrate that, at least for simulated cases, the TRIO can be consistently completed, which suggests a high level of uniformity in interpreting and recording client risks and identifying appropriate goals and interventions for the client. Social worker and administrator feedback indicated that the implementation has promoted greater use of “a common language” to evaluate and communicate information regarding a client’s situation and get all social workers focused on the over-arching goal of eliminating risk for the APS clients. The TRIO is intended to minimize variations across social workers, so that all clients receive high-quality investigation, assessment, and services. The reliability results are important in that APS workers were able to complete the TRIO similarly, but just as important is the implication that the TRIO itself may have fostered greater order and consistency of APS actions in situations that are often challenging. The detailed training provided to the social workers regarding the appropriate completion of the TRIO and the development and dissemination of the explicit definitions for the TRIO items likely contributed to the high levels of inter-rater reliability achieved. Certainly, “real world” conditions may create tendencies for greater variation across practitioners than the structured reliability assessment reported on above. To mitigate these concerns, it is important to emphasize the need for providing thorough training and ongoing support and supervision regarding the appropriate use of the TRIO with actual APS clients. The inter-rater reliability test results coupled with the detailed training and guidance indicate that the TRIO can likely be reliably utilized in other settings.

Validity

The multidimensional nature of the TRIO and the limited availability of previously established “gold standard” tools presented challenges for evaluating overall measure validity. Given this situation, we assessed certain aspects of the TRIO to provide general evidence of the extent to which the TRIO appears to accurately measure what it is intended to measure. First, as discussed above, the process utilized to develop the TRIO suggests a high level of face validity, since the TRIO was grounded in prior research on appropriate items to include in APS risk assessment tools (Anthony et al., 2009; Bay Area Social Services Consortium, 2007), as well as a practice-based review of recommended and typically provided APS interventions and the likely range of achievable APS outcomes.

Additionally, we utilized two criterion-based approaches to empirically assess TRIO validity. In our first criterion-based assessment of TRIO validity we examined the prevalence rates of TRIO items for two very different types of older adult APS client episodes, those with confirmed self-neglect allegations compared to those with confirmed financial abuse-by-other allegations. Our results demonstrated significant and substantial differences in the prevalence of TRIO items uniquely associated with only one of the types of episodes. For example, hoarding and unclean environments were much more prevalent in self-neglect episodes, whereas evidence of exploitation and improved financial stability were more commonly found in financial abuse episodes. Furthermore, TRIO items related to more universally applicable characteristics generally did not exhibit significant or substantial differences in their prevalence. These findings, differences where anticipated, and no substantial differences where none were strongly expected, does not by itself prove the validity of the TRIO results, but this pattern of findings is consistent with what one would expect for a tool that is validly and truly reflecting the characteristics and experiences of the APS clients.

The results of the predictive validity evaluation indicated that APS social workers were able to utilize the TRIO to successfully identify clients most at risk of return back into the APS system. Those with a worse prognosis for nonrecurrence at episode closure were most likely to return to APS within 180 days. Fully 25% of those with a poor prognosis returned within 180 days, and the return rates would likely increase substantially given a longer observation period, such as 12–18 months. This suggests that using the TRIO would enable social workers to accurately determine who would most likely benefit from longer-term support services if they were available. The use of the ROC curve analysis and the AUC to evaluate the predictive validity of a recurrence risk identification measure is a relatively new approach within the APS field. The AUC results from the TRIO were favorable and consistent with those found in a prior study of the capacity to predict APS recurrence (Johnson, Bogie, Wagner, & Park, 2010).

CONCLUSIONS AND LIMITATIONS

The TRIO offers a new tool for APS systems that can help standardize core activities of APS social workers, such as investigation and assessment, as well as providing additional structure to guide intervention selection and the identification and documentation of desired and achievable APS-relevant outcomes. In addition to episode-level guidance for the individual APS social worker and facilitation of their case consultation with APS supervisors, the resulting TRIO data can be used by APS administrators to better understand client characteristics, the types of interventions typically provided, and importantly, what outcomes have been achieved while APS was involved with the client. The TRIO data can also facilitate academic-practice partnerships, as evidenced by the relationship between UCSD investigators and the Ventura County APS system. Such partnerships can help advance the field of APS by contributing to a more detailed understanding of the multidimensional aspects of APS client episodes, such as characteristics and risk factors, interventions provided, and outcomes achieved, as well as more nuanced examinations into the intersections of these components. Assessing the nature of the connections between these multiple dimensions represents an important next step for research utilizing the TRIO. For example, we need to identify the extent to which certain interventions are associated with specific measurable outcomes, and whether client characteristics or clusters of characteristics may influence such outcomes. Future research will also need to examine variations within the APS populations (e.g., by allegation type, age, etc.) to determine if certain characteristics are associated with the capacity to accurately predict recurrence. Such analyses can further refine the evaluation of prognosis by APS social workers and target scarce resources to the group of clients who would mostly likely benefit from ongoing or longer-term involvement with APS or other support systems. Additional testing of the predictive results in other geographic contexts and future time periods is needed to confirm the preliminary results reported in this study. By systematically collecting multidimensional “real world” APS system data, the TRIO provides the means to advance APS practice and knowledge for the individual social worker and the field of APS more generally.

While the TRIO offers much promise, a limitation of the TRIO validity assessment was the lack of additional independent and objective methods to measure the “true” presence or absence of the many individual items collected on the TRIO. As with many APS tools (Anthony et al., 2009), there is great reliance upon the skill of social workers to accurately investigate and document their interactions with clients. In order to address this concern, the TRIO developers have manualized the TRIO and established specific definitions for each item and protocols for how to accurately complete each TRIO item (available upon request). Such training activities are important for improving both the validity and reliability of the TRIO data. Given the extensive training and the involvement of the APS social workers in the development and refinement of the TRIO, it is unclear to what extent the reliability and validity test results would generalize to other APS agencies and social workers. Implementing the TRIO in other settings as part of an academic–practitioner partnership would help identify the organizational and training requirements for successful utilization of the TRIO.

Additionally, there are two content areas that were not emphasized on the TRIO: (a) a detailed section regarding potential perpetrator information, and (b) documentation of specific “strengths” of the client or their environment. While these aspects are important to APS practice, the TRIO developers decided to focus initially on standardizing core activities of APS social workers, including investigation and assessment. As part of the environmental assessment, select perpetrator information is captured by the TRIO, and client strengths are considered during typical APS social worker interactions with clients, but due to concerns about tool length, these areas were not included as separate, detailed sections. These may represent potential TRIO refinements to be considered following additional research and evaluation of the tool.

The TRIO was designed to help facilitate the multidimensional APS social work practice and to systematically collect data regarding the interaction between client risk characteristics, APS services received, and client outcomes, with the overall goal of promoting empirically informed APS practice improvement and increasing the capacity to demonstrate APS effectiveness. The implementation and utilization of the TRIO is a first step in this process. As the TRIO data are continually collected and assessed, we expect to identify associations between risks, interventions, and outcomes suggestive of promising practices that can inform future, more rigorous testing of specific APS service approaches for APS clients overall and/or unique subgroups. A model like the TRIO that standardizes the approach to risk assessment and connects that data with interventions and outcomes has the capacity to promote consistent, high-quality service provision and provide vital information to APS direct service practitioners and policy makers. Even if APS systems do not adopt the TRIO as part of their internal standardization of practice, documentation, and data collection systems, we hope that TRIO-based research can promote introspection of existing APS data management systems and prompt movement toward greater use of and reporting with systematic, multidimensional data collection tools that include measurement of risks, interventions, outcomes, and how these dimensions intersect with each other. To ensure that APS systems can meet the challenges of the growing population of older adults over the coming decades, we need to be able to know if, when APS responds to a case, we make a difference in the lives of the people we serve and can measure it empirically. The TRIO represents one tool to help us move closer to answering that question.

FUNDING

The development and testing of the TRIO was generously supported by a grant from the Administration on Aging (#90MA0042/01).

APPENDIX TRIO Item Information

TRIO #	TRIO Item	TRIO Section	Percent Agree^a
1.	History of referral	Risk: Precursor	96.3
2.	Lives alone	Risk: Precursor	100.0
3.	No regular physician	Risk: Precursor	100.0
4.	Lacks social support, isolation	Risk: Precursor	100.0
5.	Refuses help from others	Risk: Precursor	88.9
6.	Lack of resources or homeless	Risk: Precursor	100.0
7.	Marital, family conflict, codependency	Risk: Precursor	74.1
8.	Prior or current mental health services	Risk: Precursor	100.0
9a.	Severe physical disabilities	Risk: Precursor	74.1
9b.	Dependent for all ADLs	Risk: Precursor	92.6
10.	History of violence, abuse, neglect	Risk: Precursor	96.3
11a.	Alcohol/substance abuse by client	Risk: Precursor	96.3
11b.	Alcohol/substance abuse by caregiver	Risk: Precursor	100.0
11c.	Alcohol/substance abuse environment	Risk: Precursor	96.3
12.	Evidence of psychiatric disorder	Risk: Precursor	92.6
13a.	Economically dependent adult in the home	Risk: Precursor	100.0
13b.	Economically dependent transients in and out of home	Risk: Precursor	100.0
14.	Confusion or evidence of cognitive impairment	Risk: Precursor	96.3
15.	Poor or impaired judgment, poor decision making	Risk: Precursor	100.0
16.	Evidence of high-risk behavior	Risk: Precursor	88.9
17.	Unclean physical appearance, poor hygiene	Risk: Biological	100.0
18.	Odor of feces or urine	Risk: Biological	96.3
19.	Vague references to sexual assault or unwanted advances	Risk: Biological	92.6
20.	Vague or illogical explanation for injury	Risk: Biological	100.0
21.	Under or overuse or confusion about prescriptions or OTC medications	Risk: Biological	100.0
22.	Repetitive hospital admissions due to probable failure of health care surveillance	Risk: Biological	100.0
23.	Failure to respond to warning of obvious disease	Risk: Biological	100.0
24.	Painful body movements, limping, trouble sitting or standing (not illness related)	Risk: Biological	100.0
25.	Underweight, frail, or weak; frequent falls	Risk: Biological	100.0
26.	Inadequate food or meal preparation supplies in the home	Risk: Biological	96.3
27.	Various stages of healing of any bruising or fractures	Risk: Biological	96.3
28a.	Evidence of injury—admitted	Risk: Biological	100.0
28b.	Evidence of injury—unexplained bruising, welts, wounds, broken bones or sprains	Risk: Biological	100.0
29.	Presence of dehydration or decubiti	Risk: Biological	100.0
30.	Alert, oriented declaration by elder adult of physical, sexual abuse	Risk: Biological	100.0
31.	Self-blame for current situation or makes excuses for partner or caregiver behavior	Risk: Psychological	96.3
32a.	Feelings of shame, guilt, fear, or loneliness	Risk: Psychological	85.2
32b.	Feelings of depression	Risk: Psychological	100.0
33.	Sense of resignation and hopelessness with vague reference to mistreatment	Risk: Psychological	96.3
34a.	Appears anxious	Risk: Psychological	100.0
34b.	Appears clingy	Risk: Psychological	96.3
34c.	Appears afraid of someone or something	Risk: Psychological	100.0
35a.	Behavior that is passive	Risk: Psychological	92.6
35b.	Behavior that is helpless	Risk: Psychological	96.3
35c.	Behavior that is withdrawn	Risk: Psychological	100.0
36.	Hoarding	Risk: Psychological	100.0
37.	Alert oriented declaration of psychological abuse	Risk: Psychological	100.0
38.	Poorly maintained animals, odor of feces or urine	Risk: Social	100.0
39.	Communication cut off from family or friends	Risk: Social	96.3
40.	Lack of access availability or reliability of medical care or home health care	Risk: Social	85.2
41a.	Unclean or unsafe environment	Risk: Social	96.3
41b.	Environment infestation	Risk: Social	100.0
42.	Inadequate utilities; lack of heat, cooling, water, electricity, toilet facilities	Risk: Social	100.0
43.	Lack of access, availability, or reliability of transportation	Risk: Social	100.0
44.	Dependent on alleged perpetrator or vice versa for care of finances	Risk: Social	85.2
45.	Left alone in unsafe environment for extended periods of time without adequate support	Risk: Social	100.0
46.	Evidence of exploitation by others	Risk: Social	100.0
47.	Precipitous withdrawal of care by caregiver without adequate alternate arrangements	Risk: Social	100.0
48.	Overpayment for goods or services	Risk: Social	100.0
49.	Misuse of money	Risk: Social	100.0
50.	Reports of demands for goods in exchange for services	Risk: Social	100.0
51.	Unexplained changes in power of attorney, wills, or other legal documents	Risk: Social	100.0
52.	Neglect of household finances; unpaid bills; unopened mail	Risk: Social	100.0
53a.	Inability to account for money, property, utility shut off	Risk: Social	100.0
53b.	Eviction	Risk: Social	100.0
54.	Alert oriented declaration by elder adult of exploitation	Risk: Social	100.0
55.	Alert oriented declaration by elder adult of neglect by others	Risk: Social	100.0
56.	Alert oriented declaration by elder adult of abandonment	Risk: Social	100.0
1.	Accepts education, information	Intervention	92.6
2.	Establishing bond of trust, engage with social worker	Intervention	100.0
3.	Client accepts that a problem exists	Intervention	96.3
4a.	Referral to services	Intervention	100.0
4b.	Linking to services	Intervention	88.9
5.	Tangible support used to purchase necessary items	Intervention	100.0
6.	Client support system works with APS	Intervention	100.0
7.	Client agrees to case management services	Intervention	96.3
8.	Presentation to Rapid Response Team (MDT)	Intervention	100.0
9.	Consultation with mental health expert	Intervention	92.6
10.	Consultation with medical expert	Intervention	100.0
11.	In-home nursing assessment completed	Intervention	100.0
12.	In-home medical evaluation by a physician completed	Intervention	92.6
13.	In-home mental health, psychological evaluation completed	Intervention	96.3
14.	Presentation to Financial Abuse Specialist Team	Intervention	100.0
15.	Restraining order sought	Intervention	100.0
16.	Conservatorship sought	Intervention	100.0
17.	Recoup financial loss or property loss	Outcome	100.0
18.	Financial stability	Outcome	70.4
19.	Client demonstrates self-advocacy	Outcome	74.1
20.	Linked to housing	Outcome	100.0
21.	Stable and safer home environment	Outcome	63.0
22.	Restraining order obtained	Outcome	100.0
23.	Conservatorship obtained	Outcome	100.0
24a.	Short-term care placement	Outcome	100.0
24b.	Long-term care placement	Outcome	100.0
25.	Safety net services in place	Outcome	92.6
26.	Decrease in hospital use	Outcome	100.0
27.	Improved functional status	Outcome	70.4
28.	Improved nutritional status	Outcome	100.0
29.	Improved physical health, medical condition	Outcome	88.9
30.	Improved mental health	Outcome	77.8
31.	Reduce, eliminate use of alcohol, substance abuse	Outcome	100.0
32.	Accepts case closure	Outcome	100.0
33.	Agreed to partial services	Outcome	88.9
34.	Refused all services	Outcome	100.0
35.	Linked to public assistance	Outcome	96.3

^aAverage percent agreement between raters and master TRIO in the inter-rater reliability study.