Establishing clinician reliability using the therapy outcome measure for the purpose of benchmarking services

Abstract

Benchmarking is one approach to quality improvement by comparing best practice, which requires appropriate process and outcome indicators. A benchmarking study to identify best practice was designed to investigate the use of Therapy Outcome Measure (TOM) as an outcome indicator. To ensure comparison of like with like, one study objective was to establish the reliability of the speech and language therapists (SLTs) using the TOM. This article describes this important aspect. One hundred and twenty-five SLTs from eight services, spanning seven National Health Service Trusts, participated in both the TOM training and interrater reliability assessments, where they independently rated 10 cases using composite case histories and videotape recordings. The acceptable level of reliability for participating in this benchmarking study was set as substantial (ˇ- 0.61) using an interclass correlation and was met on 52/53 TOM dimensions. On well-being, one adult team's reliability was below the required level, but this was resolved by an additional training session and reliability check. Following this, of the 53 domain results, 36 (68%) were almost perfect, and 17 (32%) were satisfactory. The reliability study found that SLTs could be trained to be consistent in their use of the TOM for benchmarking purposes. It was found that training was important for consistency and to eradicate bias; additionally, the method of assessing reliability, and the adequacy of the available information, all affected reliability.