Instruments to assess patient satisfaction after teleconsultation and triage: a systematic review

Figures & data

Table 1 Characteristics of the identified publications

Author	Year	Country	Setting	Consultation done by	Type of administration	Participants (receiving teleconsultation)	Time of administration after encounter
Campbell et al^Citation18,b	2007	UK	Out-of-hours service	Health professional	Mail	570a	14 days
Campbell et al^Citation38,c	2009	UK	Out-of-hours service	Health professional	Mail	570	14 days
Kelly et al^Citation39,c	2010	UK	Out-of-hours service, 3 accident and emergency units, teleconsultation service	Physician or nurse	Mail	353	14–28 days
Dehours et al^Citation30,b	2012	France	Teleconsultation service for French ships worldwide	Physician	Email	165	4–16 months
Dixon and Williams^Citation19,b	1988	UK	Air Call Medical Services, out-of-hours service	Physician	Mail	3,296	3 working days
Dixon and Sthal^Citation22,b	2009	USA	Family doctor teleconsultations	Physician	Written	152	Immediately after consultation
Garratt et al^Citation31,b	2010	Norway	Out-of-hours service	Unclear	Mail	225	NR
Hicks et al^Citation23,b	2003	USA	Dermatology teleconsultation service	Physician	Written	483	After each visit
López et al^Citation45,c	2011	Colombia	Teleconsultation at rural first-level outpatient institution	Physician	Telephone	121	NR
Keatinge and Rawlings^Citation29,b	2005	Australia	Out-of-hours service	Nurse	Telephone	101	NR
McKinley et al^Citation20,b	1997	UK	Out-of-hours service	Physician	Interview or written	163	1–5 days
Salisbury^Citation32,c	1997	UK	Out-of-hours service	Physician	Mail	686	7 days
Shipman et al^Citation33,c	2000	UK	Out-of-hours service	Physician	Mail	513	7 days
McKinley and Roberts^Citation34,c	2001	UK	Out-of-hours-service	Physician	Written	1,402	5 days
McKinley et al^Citation35,c	2002	UK	Out-of-hours service	Physician	Mail	2,000	NR
Glynn et al^Citation36,c	2004	Ireland	Out-of-hours service	Physician	Mail	134	7 days
Thompson et al^Citation37,c	2004	Northern Ireland	Out-of-hours service	Physician	Mail	1,075	5 days
McKinstry et al^Citation6,b	2002	UK	Management of same-day appointments	Physician	Mail	186	1 day
Mekhjian et al^Citation24,b	1999	USA	Prison	Physician	Written	221	Immediately after consultation
Moll van Charante et al^Citation27,b	2006	Netherlands	Out-of-hours service	Nurse	Mail	2,352	2 days
Giesen et al^Citation40,c	2007	Netherlands	Out-of-hours service	Nurse	Mail	2,583	2 days
Smits et al^Citation41,c	2012	Netherlands	Out-of-hours service	Nurse	Mail	1,565	2–4 days
Moscato et al^Citation25,b	2003	USA	Teleconsultation service	Nurse	Mail	87	NR
Beaulieu and Humphreys^Citation42,c	2008	USA	Teleconsultation service	Nurse	Telephone	20	3–5 days
Reinhardt^Citation43,c	2010	USA	Teleconsultation service	Nurse	Mail	865	NR
Rahmqvist et al^Citation17,b	2009	Sweden	Teleconsultation service	Nurse	Mail	660	14–21 days
Rahmqvist et al^Citation44,c	2011	Sweden	Teleconsultation service	Nurse	Mail	273	14–21 days
Salisbury et al^Citation21,b	2005	UK	Out-of-hours service	Physician or nurse	Mail	342	7 days
McKinstry et al^Citation46,c	2010	UK	Family doctor teleconsultations	Physician	Mail	46	NR
Ström et al^Citation26,b	2011	Sweden	Teleconsultation service	Nurse	Mail	517	14–21 days
Van Uden et al^Citation28,b	2005	Netherlands	Out-of-hours service	Physician’s assistant	Mail	908	21 days

Notes:

a Total number of respondents, including recipients of face-to-face services;

b development study;

c outcome study. The indented names in the “Author” column are referred to in the above-named reference.

Abbreviation: NR, not reported.

Table 2 Instrument content (related to teleconsultation)

Author	Year	Access to service	Attitude of health professional	Attitude of patient	Communication	Individual informationa	Management	Overall satisfaction	Professional skills	Telephone advice	Other	Number of domains covered per study	Total number of items
Campbell et al^Citation18	2007	1	3	1	5	15	4	3	2	3		9	37
Dehours et al^Citation30	2012	0	0	1	3	5	2	1	0	1	Diagnostics (8), training of staff (3)	6	24
Dixon and Williams^Citation19	1988	0	0	0	1	0	0	0	1	1		3	3
Dixon and Sthal^Citation22	2009	0	1	0	2	0	0	1	1	0		4	5
Garratt et al^Citation31	2010	0	4	0	1	0	4	0	1	0		4	10
Hicks et al^Citation23	2003	1	1	2	0	0	0	3	0	0	Technical aspects (1)	4	8
Keatinge and Rawlings^Citation29	2005	0	0	4	1	3	1	1	0	2	Alternative to teleconsultation (1)	6	13
McKinley et al^Citation20	1997	2	1	1	6	0	0	3	0	7		6	20
McKinstry et al^Citation6	2002	0	0	1	3	0	0	0	0	1		3	5
Mekhjian et al^Citation24	1999	0	0	5	4	0	0	1	1	0	Technical aspects (3)	4	14
Moll van Charante et al^Citation27	2006	1	2	0	5	0	1	0	1	3	Access to pharmacy (1)	6	14
Moscato et al^Citation25,b	2003	0	1	3	2	2	0	3	2	1	Alternative to teleconsultation (1)	7	15
Rahmqvist et al^Citation17	2009	0	0	0	3	0	1	1	1	1		5	7
Salisbury et al^Citation21	2005	1	1	0	0	0	2	1	0	2		5	7
Ström et al^Citation26	2011	2	2	1	3	0	1	1	2	2		8	14
Van Uden et al^Citation28	2005	2	1	2	2	0	5	4	0	6		7	22
# of studies that covered a certain domain with at least one question		7	10	10	14	4	9	12	9	12		5.4	13.6 mean

Notes:

a Sociodemographics; result of teleconsultation;

b revised version as published by Beaulieu and Humphreys.⁴²

Table 3 Descriptive information of the instruments

Author	Year	Development process	Piloting	Rating mode
Campbell et al^Citation18	2007	Literature review, consultation with experts (no further specification)	Yes	5-point Likert scale
Dehours et al^Citation30	2012	Consensus of the working group	Yes	Yes/no, categorical, open-ended
Dixon and Williams^Citation19	1988	NR	Yes	Yes/no
Dixon and Sthal^Citation22	2009	NR	Yes	Numerical rating scale 1–5
Garratt et al^Citation31	2010	Literature review, consultation with experts, interview with patients	Yes	Unclear
Hicks et al^Citation23	2003	NR	NR	7-point Likert scale
Keatinge and Rawlings^Citation29	2005	NR	Yes	Categorical
McKinley et al^Citation20	1997	Literature review, focus group meetings with patients recruited from general practice registers and community groups led by a nonclinician	Yes	5-point Likert scale
McKinstry et al^Citation6	2002	NR	NR	Numerical rating scale 0–3
Mekhjian et al^Citation24	1999	Literature review	NR	5-point Likert scale
Moll van Charante et al^Citation27	2006	Literature review, interview of stakeholders	Yes	Numerical rating scale 1–10
Moscato et al^Citation25	2003	Qualitative interviews with adults who had received phone advice	Yes	5-point Likert scale and check-off options
Rahmqvist et al^Citation17	2009	NR	NR	7-point Likert scale
Salisbury et al^Citation21	2005	Literature review, use of McKinley questionnaire as a basis, development of draft short version	Yes	5-point smiley faces (very dissatisfied to very satisfied)
Ström et al^Citation26	2011	Multidisciplinary expert group decision, interview with patients	Yes	Visual analog scale 0–10
Van Uden et al^Citation28	2005	Literature review, interview of general practitioner’s managers	NR	5-point Likert scale

Abbreviation: NR, not reported.

Table 4 Recruitment strategy and handling of nonresponders

Author	Year	Inclusion patient characteristics	Consecutive patients	Exclusion patient characteristics	Response rate	Nonresponse analysis
Campbell et al^Citation18	2007	Out-of-hours contact	Yes	Age 12–16 years, residing in nursing home, caravan park or hospital ward, recorded/expected death, end-stage terminal illness, serious/life threatening disease or distressing condition (eg, miscarriage), ‘Special message’ (eg, potentially violent patient, homeless)	46%	Respondents were older and more affluent than nonresponders, no difference in sex
Garratt et al^Citation31	2010	Out-of-hours contact	Unclear	NR	42%	NR
McKinley et al^Citation20	1997	Out-of-hours contact	Yes	NR	96%	NR
Mekhjian et al^Citation24	1999	Participation in teleconsultation	Unclear	NR	74%	NR
Moll van Charante et al^Citation27	2006	Out-of-hours contact	Yes	Death	52%	Higher response rate in men, age groups 5–14 years and 45–74 years, and privately insured. No differences in response were found for type of contact, trauma, reason for consultation and part of the day. The main reasons for nonresponse were “forgotten/not interested” (n=160; 34.6%) and “too ill” (n=83; 17.9%); 30 patients (6.5%) stated dissatisfaction as reason for nonresponse
Beaulieu and Humphreys^Citation42 (reporting on revised instrument by Moscato et al^Citation25)	2008	Out-of-hours contact, willlingness to be contacted, English speaking	Yes	NR	100%	NR
Salisbury et al^Citation21	2005	Patients who were transferred to a GP cooperative	Yes	Questionnaire would cause distress or had already been sent to same household, incomplete address	46%	NR
Ström et al^Citation26	2011	Phone call with nurse advice center	Yes	Seriously ill or person called on behalf of another person (except children)	61%	NR
Van Uden et al^Citation28	2005	Out-of-hours contact	Yes	Wrong address, death, previous participation	42%	55.6% were male; 44.5%, female; 40% had forgotten to return the questionnaire; 6.7% were not interested; 6.7% did not find it needful; 46.7% gave other reasons (no time, too difficult, lost questionnaire)

Abbreviations: NR, not reported; GP, general practitioner.

Table 5 Psychometric properties

Author	Year	Item nonresponse	Factor structure	Reliability	Validity
Campbell et al^Citation18	2007	Instrument with 45 items, 37 related to teleconsultation: • 0.5% of participants had >50% missing items • Confusion about the appropriate sections for answering (ie, related to point of care • Missing items for not management-specific items 1.2%–20.4% • 4 items (which health professional conducted the consultation; ethnic group; occupation; were you happy with [the final management of your call]) had missing values exceeding 10% of responses (maximum)	Exploratory principal components analysis: • 2 Factors accounting for 68% of variance • Factor 1: interaction (10 items, 55% of variance) • Factor 2: consultation satisfaction (9 items, 13% of variance) • Overlap of loadings for 4 items	Cronbach’s alpha: • Consultation satisfaction, 0.96 • Entry access, 0.82 Inter item correlation: • Consultation satisfaction, 0.63–0.89 • Entry access, 0.45–0.86 Item total correlation: • Consultation satisfaction, 0.77–0.90 • Entry access, 0.56–0.73 Test/retest correlation: • Consultation satisfaction, 0.76 • Entry access, 0.60	Convergent validity: • Scores for consultation satisfaction and entry access subscale were moderately correlated (r=0.43) Construct validity: • Higher levels of global satisfaction, higher age, and longer duration of consultation were correlated with higher total and subscale scores
Garratt et al^Citation31	2010	Instrument with 24 items, 10 items related to teleconsultation: • Missing items, 3.2%–11.7%, mean, 7.2%	Principal component analysis: • 4 Factors accounting for 79% of variance • Factor 1: doctor services (7 items) • Factor 2: telephone contact and organization (10 items) • Factor 3: nursing services (6 items) • Factor 4: single item “unanswered questions” • Only factor 2 related to teleconsultation	Cronbach’s alpha: • Telephone contact, 0.91 • Doctor contact, 0.90 • Nurse contact, 0.93 • Organization, 0.82 Item total correlation: • Telephone contact, 0.76–0.82 • Doctor contact, 0.73–0.83 • Nurse contact, 0.78–0.89 • Organization, 0.66–0.71	Construct validity: • Difficulties to contact by phone correlated to lower scores in telephone contact subscale • Short waiting time for teleconsultation and treatment large amount of information, high levels of overall satisfaction, fulfillment of expectations and absence of poor treatment were correlated to higher scores of all subscales
McKinley et al^Citation20	1997	Instrument with 32 items, 20 related to teleconsultation: • Median completion rate, 96.5% (interquartile range, 95.7%–97.1 %)	Principal component analysis: • 6 Factors • Factor 1: satisfaction with communication and management (7 items; 29% of Variance) • Doctor’s attitude (5 items; 9% of variance) • Continuity of care (4 items; 8% of variance) • Delay until visit (3 items; 6% of variance) • Access to out-of-hours care (3 items; 5% of variance) • The initial contact person (2 items; 4% of variance) • Telephone advice (4 items; variance NR) • Overall satisfaction (4 items; variance NR)	Cronbach’s alpha: • Communication and management 0.88 • Doctor’s attitude, 0.87 • Continuity of care, 0.69 • Delay until visit, 0.65 • Access to out-of-hours care, 0.61 • Initial contact person, 0.72 • Telephone advice, 0.79 • Overall satisfaction, 0.77 Test/retest correlation: • Communication and management 0.86 • Doctor’s attitude, 0.82 • Continuity of care, 0.72 • Delay until visit, 0.81 • Access to out-of-hours care, 0.76 • Initial contact person, 0.62 • Telephone advice, NR • Overall satisfaction, 0.82	Construct validity: • All subscales correlated to overall satisfaction
Mekhjian et al^Citation24	1999	Instrument with 14 items: • 26% of questionnaires were excluded due to nonresponse	Common factor analysis: • 2 Factors accounting for 47% of variance • Factor 1: information exchange (7 items; 40% of variance) • Factor 2: patient comfort (7 items; 47% of variance)	Cronbach’s alpha: • Information exchange, 0.88 • Patient comfort, 0.81	NR
Moll Van Charante et al^Citation27	2006	Instrument with 66 items, 14 items related to teleconsultation: • Missing items 4.8%–16.5% (Mean 12.6%)	Principal component analysis: • 3 Factors • Factor 1 (77% of variance) • Factor 2 (72% of variance) • Factor 3 (83% of variance)	Cronbach’s alpha: • Telephone nurse, 0.95 • Organization, 0.74 • Overall, 0.81 Item total correlation: • Telephone nurse, 0.84–0.92 • Organization, 0.53–0.59 Test retest correlation: • Telephone nurse, 0.85 • Organization, 0.92 • Overall, 0.79	NR
Beaulieu and Humphreys^Citation42 (reporting on revised instrument by Moscato et al^Citation25)	2008	Instrument with 19 items: • Missing items, NR	NR	Cronbach’s alpha: • Total scale, 0.70	NR
Salisbury et al^Citation21	2005	Instrument with 7 items: • Missing items 54.1%–56.l% (Mean 55.7%)	NR	Cronbach’s alpha: • Total scale, 0.94	Construct validity: • Each item correlated to overall satisfaction • Increasing age, female sex and transfer to face-to-face visit correlated to higher scores Concurrent validity: • Intraclass correlation coefficient between short scale and long scale 0.38–0.54
Ström et al^Citation26	2011	Instrument with 14 items: • Missing items, NR	Explorative factor analysis: • 3 Factors • Factor 1: interaction (8 items; 34% of variance) • Factor 2: service (3 items; 50% of variance) • Factor 3: product (3 items; 63% of variance)	Cronbach’s alpha: • Total scale, 0.88 • Interaction, 0.90 • Service, 0.80 • Product, 0.45	NR
Van Uden et al^Citation28	2005	Instrument with 35 items, 22 items related to teleconsultation: • Missing items, NR	Principal component analysis: • 6 Factors • Accessibility by phone (3 items) • Doctor’s assistant’s attitude (5 items) • Questions asked by assistant (2 items) • Advice given by assistant (5 items) • Urgency of complaint (2 items) • Overall satisfaction (5 items)	Cronbach’s alpha: • Accessibility by phone, 0.72 • Doctor’s assistant’s attitude, 0.91 • Questions asked by assistant, 0.64 • Advice given by assistant, 0.93 • Urgency of complaint, 0.86 • Overall satisfaction, 0.93	Construct validity: • Higher total scores correlated with overall satisfaction

Abbreviation: NR, not reported.

Figure 1 Flowchart.

Table S1 Medline search algorithm

1. Patient satisfaction MeSH heading: exp, topic	5. Questionnaire MeSH heading: exp, topic	9. Triage MeSH heading: exp, topic
2. User satisfaction, topic	6. Instrument, topic	10. Hotline, topic
3. Consumer satisfaction, topic	7. Measurement, topic	11. Telephone, topic
4. or\1–3	8. or\5–7	12. Remote consultation, topic
		13. Telemedicine, topic
		14. Telecommunication, topic
		15. Telehealth, topic
		16. Teletriage, topic
		17. Referral, topic
		18. Teleconsultation, topic
		19. Telenurse, topic
		20. Helpline, topic
		21. Healthline, topic
		22. Telecare, topic
		23. Callcenter, topic
		24. or\9–23
		25. and\4, 8, 24

Abbreviation: MeSH, medical subject heading.

CampbellJLDickensARichardsSHPoundPGrecoMBowerPCapturing users’ experience of UK out-of-hours primary medical care: piloting and psychometric properties of the Out-of-hours Patient QuestionnaireQual Saf Health Care200716646246818055892

 PubMedGoogle Scholar

CampbellJRolandMRichardsSDickensAGrecoMBowerPUsers’ reports and evaluations of out-of-hours health care and the UK national quality requirements: a cross sectional studyBr J Gen Pract200959558e8e1519105911

 PubMed Web of Science ®Google Scholar

KellyMEgbunikeJNKinnersleyPDelays in response and triage times reduce patient satisfaction and enablement after using out-of-hours servicesFam Pract201027665266320671002

 PubMed Web of Science ®Google Scholar

DehoursEValléBBounesVUser satisfaction with maritime telemedicineJ Telemed Telecare201218418919222604271

 PubMed Web of Science ®Google Scholar

DixonRAWilliamsBTPatient satisfaction with general practitioner deputising servicesBMJ19882976662151915223147058

 PubMed Web of Science ®Google Scholar

DixonRFStahlJEA randomized trial of virtual visits in a general medicine practiceJ Telemed Telecare200915311511719364890

 PubMed Web of Science ®Google Scholar

GarrattAMDanielsenKForlandOHunskaarSThe Patient Experiences Questionnaire for Out-of-Hours Care (PEQ-OHC): data quality, reliability, and validityScand J Prim Health Care20102829510120433404

 PubMed Web of Science ®Google Scholar

HicksLLBolesKEHudsonSPatient satisfaction with teledermatology servicesJ Telemed Telecare200391424512641892

 PubMed Web of Science ®Google Scholar

LópezCValenzuelaJICalderónJEVelascoAFFajardoRA telephone survey of patient satisfaction with realtime telemedicine in a rural community in ColombiaJ Telemed Telecare2011172838721139016

 PubMed Web of Science ®Google Scholar

KeatingeDRawlingsKOutcomes of a nurse-led telephone triage service in AustraliaInt J Nurs Pract200511151215610339

 PubMedGoogle Scholar

McKinleyRKManku-ScottTHastingsAMFrenchDPBakerRReliability and validity of a new measure of patient satisfaction with out of hours primary medical care in the United Kingdom: development of a patient questionnaireBMJ199731470751931989022436

 PubMed Web of Science ®Google Scholar

SalisburyCPostal survey of patients’ satisfaction with a general practice out of hours cooperativeBMJ19973147094159415989186172

 PubMed Web of Science ®Google Scholar

ShipmanCPayneFHooperRDaleJPatient satisfaction with out-of-hours services; how do GP co-operatives compare with deputizing and practice-based arrangements?J Public Health Med200022214915410912552

 PubMedGoogle Scholar

McKinleyRKRobertsCPatient satisfaction with out of hours primary medical careQual Health Care2001101232811239140

 PubMedGoogle Scholar

McKinleyRKStevensonKAdamsSManku-ScottTKMeeting patient expectations of care: the major determinant of satisfaction with out-of-hours primary medical care?Fam Pract200219433333812110550

 PubMed Web of Science ®Google Scholar

GlynnLGByrneMNewellJMurphyAWThe effect of health status on patients’ satisfaction with out-of-hours care provided by a family doctor co-operativeFam Pract200421667768315528288

 PubMed Web of Science ®Google Scholar

ThompsonKParahooKFarrellBAn evaluation of a GP out-of-hours service: meeting patient expectations of careJ Eval Clin Pract200410346747415304147

 PubMed Web of Science ®Google Scholar

McKinstryBWalkerJCampbellCHeaneyDWykeSTelephone consultations to manage requests for same-day appointments: a randomised controlled trial in two practicesBr J Gen Pract20025247730631011942448

 PubMed Web of Science ®Google Scholar

MekhjianHTurnerJWGailiunMMcCainTAPatient satisfaction with telemedicine in a prison environmentJ Telemed Telecare199951556110505370

 PubMed Web of Science ®Google Scholar

Moll van CharanteEGiesenPMokkinkHPatient satisfaction with large-scale out-of-hours primary health care in The Netherlands: development of a postal questionnaireFam Pract200623443744316641129

 PubMed Web of Science ®Google Scholar

GiesenPMoll van CharanteEMokkinkHBindelsPvan den BoschWGrolRPatients evaluate accessibility and nurse telephone consultations in out-of-hours GP care: determinants of a negative evaluationPatient Educ Couns200765113113616939708

 PubMed Web of Science ®Google Scholar

SmitsMHuibersLOude BosAGiesenPPatient satisfaction with out-of-hours GP cooperatives: a longitudinal studyScand J Prim Health Care201230420621323113756

 PubMed Web of Science ®Google Scholar

MoscatoSRDavidMValanisBTool development for measuring caller satisfaction and outcome with telephone advice nursingClin Nurs Res200312326628112918650

 PubMedGoogle Scholar

BeaulieuRHumphreysJEvaluation of a telephone advice nurse in a nursing faculty managed pediatric community clinicJ Pediatr Health Care200822317518118455066

 PubMed Web of Science ®Google Scholar

ReinhardtACThe impact of work environment on telephone advice nursingClin Nurs Res201019328931020601639

 PubMed Web of Science ®Google Scholar

RahmqvistMHusbergMEffekter av sjukvårdsrådgivning per telefon: En analys av rådgivningsverksamheten 1177 i Östergötland och JämtlandCMT Rapport20093

 Google Scholar

RahmqvistMErnesäterAHolmströmITriage and patient satisfaction among callers in Swedish computer-supported telephone advice nursingJ Telemed Telecare201117739740221983224

 PubMed Web of Science ®Google Scholar

SalisburyCBurgessALattimerVDeveloping a standard short questionnaire for the assessment of patient satisfaction with out-of-hours primary careFam Pract200522556056915964865

 PubMed Web of Science ®Google Scholar

McKinstryBHammersleyVBurtonCThe quality, safety and content of telephone and face-to-face consultations: a comparative studyQual Saf Health Care201019429830320430933

 PubMed Web of Science ®Google Scholar

StrömMBaigiAHildinghCMattssonBMarklundBPatient care encounters with the MCHL: a questionnaire studyScand J Caring Sci201125351752421338380

 PubMed Web of Science ®Google Scholar

van UdenCJAmentAJHobmaSOZwieteringPJCrebolderHFPatient satisfaction with out-of-hours primary care in the NetherlandsBMC Health Serv Res200551615651997

 PubMed Web of Science ®Google Scholar