Identifying performance indicators to measure overall performance of telephone triage – a scoping review

Figures & data

Figure 1. The PRISMA flowchart used in the scoping review.

Figure 2. Indicator groups based on the IHI triple aim dimensions.

Figure 3. Performance framework for telephone triage services, based on the scoping review.

Table 1. Indicators of the triple aim dimension: Patient experience of care.

Performance dimension: Patient experience of care	Frequency	Indicator category	Classifying variable	Performance measurement indicator	References
Service accessibility
Call arrival time (start and end time of call; time in relation to own health centre)	22	Process	x		[20,21,25,29,34–51]
Total volume of calls	23	Process	x		[1,20–22,29,36,37,39,41,43,45,46,52,54–61]
Call duration	9	Process		x	[5,20–22,40,41,45,59,62]
Type of call (callbacks, direct calls, administrative contacts, medication-related calls)	4	Structural	x		[4,45,61,62]
Waiting times	4	Process		x	[25,35,51,63]
Access	1	Process		x	[64]
Patient opinion on the quality of service
Patient satisfaction	11	Outcome		x	[2,25,38,41,45,50,51,63–66]
Communication (professional communication skills and quality of communication)	8	Structural	x		[1,2,4,5,21,22,38,52]
Complaints (rate and type)	1	Process	x		[67]
Patient’s experience	3	Outcome		x	[2,18,38]
Professional competence	10	Structural	x		[4–6,22,29,46,58,60–62]

Table 2. Indicators of the triple aim dimension: Health outcomes.

Performance dimension: Health outcomes	Frequency	Indicator category	Classifying variable	Performance measurement indicator	References
Effectiveness and safety
Reason for encounter (as in ICPC-2)	27	Process	x		[1,3,5,21,22,25,28,29,34,38,42–47,49,53,59–61,65,68,69,71]
Accuracy of triage decision (Percentage of accurately triaged patients, percentage of over- and under-triaged patients, identified life-threatening signs)	11	Outcome		x	[3,5,20,28,29,35–37,42,45,46,50,65,66,69,71]
Severity and urgency of the symptoms	13	Outcome		x	[3,25,27,28,42,46,55,60,61,66,69,72]
Duration and development of patient symptoms	10	Process	x		[3,20,35,40,42,46,47,65,66,69]
Co-morbidity	6	Structural	x		[1,34,42,46,55,66]
Type of current medical problem	3	Structural	x		[41,45,69]
Cause (possible) and consequence of symptoms	1	Structural	x		[35]
Patient-relevant outcomes
Triage response (follow-up advice type and/or after-triage contact type, appropriate urgency and level of care)	36	Outcome		x	[1,3,5,20,21,25,27–29,34,36,37,41,42,44,46,47,49,50,53,55,56,58–70,72]
Patient compliance with advice given	8	Outcome		x	[21,27,49,52,59,62,67,68]
ICD-10 diagnoses	3	Structural	x		[34,60,71]
Mortality	1	Outcome	x		[70]
Patient medical history and medications	2	Structural	x		[5,59]

Table 3. Indicators of the triple aim dimension: Reducing the per capita cost of healthcare.

Performance dimension: Reducing the per capita cost	Frequency	Indicator category	Classifying variable	Performance measurement indicator	References
Costs per case treated
Follow-up healthcare service use (patient guided to ED percentage per hour; utilization of PC and ED visits)	10	Outcome		x	[34,39,41,43,50,57,59,61,70,72]
Number of calls to telephone triage per patient by specific time	5	Process	x		[26,36,37,50,52]
Hospitalization after the call (frequency, type and duration of follow-up hospitalization)	5	Process		x	[1,47,57,70,73]
Service effects on healthcare utilization (number of office visits to primary care clinics/ED)	4	Structural	x		[39,43,47,70]
ED visit outcome (clinical outcomes, diagnosis and hospitalization rate of ED visit)	4	Outcome		x	[42,52,55,64]
Costs (private and public costs, running costs of service)	3	Outcome		x	[48, 64,71,72]

Graversen DS, Huibers L, Christensen MB, et al. Communication quality in telephone triage conducted by general practitioners, nurses or physicians: a quasi-experimental study using the AQTT to assess audio-recorded telephone calls to out-of-hours primary care in Denmark. BMJ Open. 2020;10(3):e033528. doi: 10.1136/bmjopen-2019-033528.

 PubMed Web of Science ®Google Scholar

Morgan JI, Muskett T. Interactional misalignment in the UK NHS 111 healthcare telephone triage service. Int J Med Inform. 2020;134:104030. doi: 10.1016/j.ijmedinf.2019.104030.

 PubMed Web of Science ®Google Scholar

Ebert JF, Huibers L, Lippert FK, et al. Development and evaluation of an “emergency access button” in Danish out-of-hours primary care: a study protocol of a randomized controlled trial. BMC Health Serv Res. 2017;17(1):379. doi: 10.1186/s12913-017-2308-y.

 PubMedGoogle Scholar

Schoenmakers B, Delmeiren L, Pietermans S, et al. The implementation of the nationwide out-of-hours phone number 1733 in Belgium: analysis of efficiency and safety. Prim Health Care Res Dev. 2021;22(1733):e7. doi: 10.1017/S1463423621000098.

 PubMedGoogle Scholar

Gamst-Jensen H, Gamst-Jensen H, Frischknecht Christensen E, et al. Self-rated worry is associated with hospital admission in out-of-hours telephone triage – a prospective cohort study. Scand J Trauma Resusc Emerg Med. 2020;28(1):53. doi: 10.1186/s13049-020-00743-8.

 PubMed Web of Science ®Google Scholar

Anderson A, Roland M. Potential for advice from doctors to reduce the number of patients referred to emergency departments by NHS 111 call handlers: observational study. 2015;1–4.

 Google Scholar

Elliott AM, McAteer A, Heaney D, et al. Examining the role of Scotland’s telephone advice service (NHS 24) for managing health in the community: analysis of routinely collected NHS 24 data. BMJ Open. 2015;5(8):e007293. doi: 10.1136/bmjopen-2014-007293.

 PubMed Web of Science ®Google Scholar

Zinger ND, Blomberg SN, Lippert F, et al. Satisfaction of 30 402 callers to a medical helpline of the emergency medical services Copenhagen: a retrospective cohort study. BMJ Open. 2019;9(10):e029801. doi: 10.1136/bmjopen-2019-029801.

 PubMed Web of Science ®Google Scholar

Haraldseide LM, Sortland LS, Hunskaar S, et al. Contact characteristics and factors associated with the degree of urgency among older people in emergency primary health care: a cross-sectional study. BMC Health Serv Res. 2020;20(1):345. doi: 10.1186/s12913-020-05219-0.

 PubMed Web of Science ®Google Scholar

Nørøxe KB, Huibers L, Moth G, et al. Medical appropriateness of adult calls to Danish out-of-hours primary care: a questionnaire-based survey. BMC Fam Pract. 2017;18(1):34. doi: 10.1186/s12875-017-0617-1.

 PubMedGoogle Scholar

Gamst-Jensen H, Frishknecht Christensen E, Lippert F, et al. Impact of caller’s degree-of-worry on triage response in out-of-hours telephone consultations: a randomized controlled trial. Scand J Trauma Resusc Emerg Med. 2019;27(1):44. doi: 10.1186/s13049-019-0618-2.

 PubMedGoogle Scholar

Kaboli PJ, Augustine MR, Rose DE, et al. Call center remote triage by nurse practitioners was associated with fewer subsequent face-to-face healthcare visits. J Gen Intern Med. 2021;36(8):2315–2322. doi: 10.1007/s11606-020-06536-0.

 PubMed Web of Science ®Google Scholar

Roivainen P, Hoikka MJ, Raatiniemi L, et al. Telephone triage performed by nurses reduces non-urgent ambulance missions: a prospective observational pilot study in Finland. Acta Anaesthesiol Scand. 2020;64(4):556–563. doi: 10.1111/aas.13542.

 PubMed Web of Science ®Google Scholar

Graversen DS, Christensen MB, Pedersen AF, et al. Safety, efficiency and health-related quality of telephone triage conducted by general practitioners, nurses, or physicians in out-of-hours primary care: a quasi-experimental study using the Assessment of Quality in Telephone Triage (AQTT) to assess audio-recorded telephone calls. BMC Fam Pract. 2020;21:84.

 PubMed Web of Science ®Google Scholar

Wallace E, Cronin S, Murphy N, et al. Characterising patient complaints in out-of-hours general practice: a retrospective cohort study in Ireland. Br J Gen Pract. 2018;68(677):E860–e868. doi: 10.3399/bjgp18X699965.

 PubMed Web of Science ®Google Scholar

Katayama Y, Kitamura T, Hirose T, et al. Characteristics and outcome of patients triaged by telephone and transported by ambulance: a population-based study in Osaka, Japan. Acute Med Surg. 2020;7(1):e609. doi: 10.1002/ams2.609.

 PubMed Web of Science ®Google Scholar

Lu AD, Junge M, Garber J, et al. Feasibility of a telemedicine urgent care program to address patient complaints on first contact. Emerg Med Int. 2020;2020:8875644–8875644. doi: 10.1155/2020/8875644.

 PubMed Web of Science ®Google Scholar

Ernesäter A, Engström M, Winblad U, et al. Telephone nurses’ communication and response to callers’ concern-a mixed methods study. Appl Nurs Res. 2016;29:116–121. doi: 10.1016/j.apnr.2015.04.012.

 PubMed Web of Science ®Google Scholar

Smits M, Keizer E, Ram P, et al. Development and testing of the KERNset: an instrument to assess the quality of telephone triage in out-of-hours primary care services. BMC Health Serv Res. 2017;17(1):798. doi: 10.1186/s12913-017-2686-1.

 PubMedGoogle Scholar

Siddiqui N, Greenfield D, Lawler A. Calling for confirmation, reassurance, and direction: investigating patient compliance after accessing a telephone triage advice service. Int J Health Plann Manage. 2020;35(3):735–745. doi: 10.1002/hpm.2934.

 PubMed Web of Science ®Google Scholar

McKenzie R, Williamson M, Roberts R. Who uses the “after hours GP helpline”? a profile of users of an after-hours primary care helpline. Aust Fam Physician. 2016;45(5):313–8.

 PubMedGoogle Scholar

Smits M, Colliers A, Jansen T, et al. Examining differences in out-of-hours primary care use in Belgium and The Netherlands: a cross-sectional study. Eur J Public Health. 2019;29(6):1018–1024. doi: 10.1093/eurpub/ckz083.

 PubMed Web of Science ®Google Scholar

Thilsted SLB, Egerod I, Lippert FK, et al. Relation between illness representation and self-reported degree-of-worry in patients calling out-of-hours services: a mixed-methods study in Copenhagen, Denmark. BMJ Open. 2018;8(9):e020401. doi: 10.1136/bmjopen-2017-020401.

 PubMed Web of Science ®Google Scholar

Graversen DS, Pedersen AF, Carlsen AH, et al. Quality of out-of-hours telephone triage by general practitioners and nurses: development and testing of the AQTT–an assessment tool measuring communication, patient safety and efficiency. Scand J Prim Health Care. 2019;37(1):18–29. doi: 10.1080/02813432.2019.1568712.

 PubMed Web of Science ®Google Scholar

Tranberg M, Vedsted P, Bech BH, et al. Factors associated with low patient satisfaction in out-of-hours primary care in Denmark – a population-based cross-sectional study. BMC Fam Pract. 2018;19(1):15. doi: 10.1186/s12875-017-0681-6.

 PubMedGoogle Scholar

Knowles E, O’Cathain A, Turner J, et al. Effect of a national urgent care telephone triage service on population perceptions of urgent care provision: controlled before and after study. BMJ Open. 2016;6(10):e011846. doi: 10.1136/bmjopen-2016-011846.

 PubMed Web of Science ®Google Scholar

Vilstrup E, Graversen DS, Huibers L, et al. Communicative characteristics of general practitioner-led and nurse-led telephone triage at two Danish out-of-hours services: an observational study of 200 recorded calls. BMJ Open. 2019;9(6):e028434. doi: 10.1136/bmjopen-2018-028434.

 PubMed Web of Science ®Google Scholar

Ebert JF, Huibers L, Christensen B, et al. Giving callers the option to bypass the telephone waiting line in out-of-hours services: a comparative intervention study. Scand J Prim Health Care. 2019;37(1):120–127. doi: 10.1080/02813432.2019.1569427.

 PubMed Web of Science ®Google Scholar

Sexton V, Dale J, Atherton H. An evaluation of service user experience, clinical outcomes and service use associated with urgent care services that utilise telephone-based digital triage: a systematic review protocol. Syst Rev. 2021;10(1):25. doi: 10.1186/s13643-021-01576-x.

 PubMed Web of Science ®Google Scholar

Gamst-Jensen H, Huibers L, Pedersen K, et al. Self-rated worry in acute care telephone triage: a mixed-methods study. Br J Gen Pract. 2018;68(668):e197–203–e203. doi: 10.3399/bjgp18X695021.

 Web of Science ®Google Scholar

Huibers L, Moth G, Carlsen AH, et al. Telephone triage by GPS in out-of-hours primary care in Denmark: a prospective observational study of efficiency and relevance. Br J Gen Pract. 2016;66(650):e667–73–e673. doi: 10.3399/bjgp16X686545.

 PubMed Web of Science ®Google Scholar

Morreel S, Philips H, Colliers A, et al. Performance of a new guideline for telephone triage in out-of-hours services in Belgium: a pilot study using simulated patients. Health Serv Manage Res. 2020;33(4):166–171. doi: 10.1177/0951484820921809.

 PubMed Web of Science ®Google Scholar

Li L, Lake R, Raban MZ, et al. Medication-related calls received by a national telenursing triage and advice service in Australia: a retrospective cohort study. BMC Health Serv Res. 2017;17(1):197. doi: 10.1186/s12913-017-2135-1.

 PubMedGoogle Scholar

Howell T. ED utilization by uninsured and medicaid patients after availability of telephone triage. J Emerg Nurs. 2016;42(2):120–124. doi: 10.1016/j.jen.2015.08.015.

 PubMed Web of Science ®Google Scholar

Holt TA, Fletcher E, Warren F, et al. Telephone triage systems in UK general practice: analysis of consultation duration during the index day in a pragmatic randomised controlled trial. Br J Gen Pract. 2016;66(644):e214–8–e218. doi: 10.3399/bjgp16X684001.

 PubMed Web of Science ®Google Scholar

McKenzie R, Williamson M. The league of extraordinary generalists: a qualitative study of professional identity and perceptions of role of GPs working on a national after hours helpline in Australia. BMC Health Serv Res. 2016;16(1):142. doi: 10.1186/s12913-016-1387-5.

 PubMedGoogle Scholar

Sen B, Clay H, Wright J, et al. Impact of emergency medicine consultants and clinical advisors on a NHS 111 clinical assessment service. Emerg Med J. 2019;36(4):208–212. doi: 10.1136/emermed-2017-207335.

 PubMed Web of Science ®Google Scholar

Varley A, Warren FC, Richards SH, et al. The effect of nurses’ preparedness and nurse practitioner status on triage call management in primary care: a secondary analysis of cross-sectional data from the ESTEEM trial. Int J Nurs Stud. 2016;58:12–20. doi: 10.1016/j.ijnurstu.2016.02.001.

 PubMed Web of Science ®Google Scholar

Katayama Y, Kitamura T, Nakao S, et al. Association of a telephone triage service for emergency patients with better outcome: a population-based study in Osaka city, Japan. Eur J Emerg Med. 2022;29(4):262–270. doi: 10.1097/MEJ.0000000000000902.

 PubMed Web of Science ®Google Scholar

Moth G, Huibers L, Christensen MB, et al. Out-of-hours primary care: a population-based study of the diagnostic scope of telephone contacts. Fam Pract. 2016;33(5):504–509. doi: 10.1093/fampra/cmw048.

 PubMed Web of Science ®Google Scholar

Mulcahy D, O’Callaghan C, Hannigan A. Nurse triage in an Irish out-of-hours general practice co-operative. Ir Med J. 2017 Mar 10;110(3):530.

 PubMedGoogle Scholar

Li L, Georgiou A, Xiong J, et al. Healthdirect’s after hours GP helpline – a survey of patient satisfaction with the service and compliance with advice. Stud Health Technol Inform. 2016;227:87–92.

 PubMedGoogle Scholar

Graversen DS, Pedersen AF, Christensen MB, et al. Factors associated with undertriage and overtriage in telephone triage in Danish out-of-hours primary care: a natural quasi-experimental cross-sectional study of randomly selected and high-risk calls. BMJ Open. 2023;13(3):e064999. doi: 10.1136/bmjopen-2022-064999.

 PubMed Web of Science ®Google Scholar

Njeru JW, Damodaran S, North F, et al. Telephone triage utilization among patients with limited English proficiency. BMC Health Serv Res. 2017;17(1):706. doi: 10.1186/s12913-017-2651-z.

 PubMed Web of Science ®Google Scholar

Warren FC, Calitri R, Fletcher E, et al. Exploring demographic and lifestyle associations with patient experience following telephone triage by a primary care doctor or nurse: secondary analyses from a cluster randomised controlled trial. BMJ Qual Saf. 2015;24(9):572–582. doi: 10.1136/bmjqs-2015-003937.

 PubMed Web of Science ®Google Scholar

Gibson A, Randall D, Tran DT, et al. Emergency department attendance after telephone triage: a population-based data linkage study. Health Serv Res. 2018;53(2):1137–1162. doi: 10.1111/1475-6773.12692.

 PubMed Web of Science ®Google Scholar

Gamst-Jensen H, Lippert FK, Egerod I. Under-triage in telephone consultation is related to non-normative symptom description and interpersonal communication: a mixed methods study. Scand J Trauma Resusc Emerg Med. 2017;25(1):52. doi: 10.1186/s13049-017-0390-0.

 PubMed Web of Science ®Google Scholar

Griffin E, McCarthy JP, Thomas F, et al. New Zealand healthline call data used to measure the effect of travel time on the use of the emergency department. Soc Sci Med. 2017;179:91–96. doi: 10.1016/j.socscimed.2017.02.035.

 PubMed Web of Science ®Google Scholar

Gamst-Jensen H, Jensen AN, Christensen EF, et al. Socioeconomic inequality in telephone triage on triage response, hospitalization and 30-day mortality. Eur J Public Health. 2021;i:1–3.

 Google Scholar

Tran DT, Gibson A, Randall D, et al. Compliance with telephone triage advice among adults aged 45 years and older: an Australian data linkage study. BMC Health Serv Res. 2017;17(1):512. doi: 10.1186/s12913-017-2458-y.

 PubMedGoogle Scholar

Lake R, Georgiou A, Li J, et al. The quality, safety and governance of telephone triage and advice services – an overview of evidence from systematic reviews. BMC Health Serv Res. 2017;17(1):614. doi: 10.1186/s12913-017-2564-x.

 PubMedGoogle Scholar

Henricson J, Ekelund U, Hartman J, et al. Pathways to the emergency department – a national, cross-sectional study in Sweden. BMC Emerg Med. 2022;22(1):58. doi: 10.1186/s12873-022-00619-3.

 PubMedGoogle Scholar

Rebolho RC, Raupp FM. Implementation costs of telephone nurse triage service custos de implantação do serviço de triagem de enfermagem por telefone costos de implantación del servicio de triaje telefónico de enfermería. Cad Saude Publica. 2023;39(3):e00095522. 2023;39(3).

 Google Scholar

Data sharing statement

No unpublished data are available following this study.