https://orcid.org/0000-0001-7739-777X
Abstract
General practitioners (GPs) in the Region of Southern Denmark were randomly allocated to a range of interventions to optimize their use of Vitamin D tests over one year. The aim of the current survey study was to investigate GPs assessment of the interventions. Using REDCap web-platform, we invited 638 GPs to participate in a survey about their experiences of guidelines, feedback reports, non-interruptive alerts, and interruptive alerts. The questions were customized for the different interventions. We received responses from only 131 GPs (21%), but no differences in gender, age, or type of GP clinic were observed between responders and invited GPs. Approximately half of the GPs found that guidelines were helpful, and a similar proportion of GPs read the feedback reports ‘often’ or ‘always′. The pop-up alerts were accepted when used for maximum three months for often-used tests. In contrast, alerts were accepted for long periods for rarely-used tests. The groups that were exposed to the interruptive alert found it ‘problematic’ that it appeared every time vitamin D was requested. Guidelines and feedback reports on tests numbers were accepted, but it was previously found, that they had little effect on improving the use of biochemical tests. Pop-up alerts in the requesting IT system can produce alert fatigue. Future research should focus on developing feedback reports that – when possible - also include relevant clinical information, and pop-up alerts should for often used tests be displayed only for weeks or a few months, but can be repeated.
Introduction
In their role as gatekeepers in the healthcare system, general practitioners (GPs) need to be aware of both common and rare diseases, their diagnosis and treatment, and the many healthcare services on offer [Citation1]. Several studies have investigated how to improve the practice patterns of GPs [Citation2,Citation3], but few studies have addressed interventions regarding GPs requisition of laboratory tests [Citation4].
We recently published the results of a large randomized controlled trial [Citation5] in which we evaluated the effect of six combinations of four interventions applied to the process of requesting biochemical tests: 1) Information with guidelines alone or in combination with either 2) a non-interruptive alert or 3) an interruptive alert in the requesting IT system, all of these combined with 4) feedback reports sent out every fourth week or alternatively no feedback. The interventions were randomly applied over one year to 313 primary care clinics in the Region of Southern Denmark (RSD), using vitamin D as a model test (detailed information can be found elsewhere [Citation5]). We found that some of the interventions resulted in fewer tests being ordered, but that there were no signs of improved appropriateness of test utilization. The latter was evaluated from the ratio between pathological and normal results, which remained unchanged, meaning that fewer patients with clinically relevant results were identified. These findings highlighted the need for further exploration of GPs’ perspectives on such initiatives.
The aim of this cross-sectional survey study was to investigate how GPs in the Region of Southern Denmark viewed the different interventions applied over the previous year for optimizing their use of the Vitamin D test.
Methods
The design of the survey
A cross-sectional electronic survey was conducted among GPs working in the Region of Southern Denmark, using the REDCap web-platform for building and managing the survey [Citation6]. A standardized survey [Citation7] was designed for the study and was then reviewed by two GPs experienced in undertaking qualitative studies (one of them was author JS) and six clinical biochemists. The revised survey was thereafter completed and commented on by three GPs who were members of the project’s advisory board [Citation5].
Invitation to the survey
On February 22, 2021 a letter was sent either by e-mail or regular mail to the 271 GP clinics that participated in the randomized controlled trial [Citation5], thus inviting 638 GPs to participate in the study. The letter included the link that led the participants to the electronic survey. A reminder was sent out after four weeks. The GPs were offered financial compensation for the time spent answering the survey.
The GPs in the six intervention groups were mainly asked questions related to the interventions they were exposed to. To obtain a broader view of the interventions, some questions were also asked in hypothetical form about interventions the responders were not exposed to (the survey questions are available in Supplementary material 1).
The survey
All GPs were asked questions on their gender, age, years of experience as a GP, and type of clinic in which they worked. Further questions focused on which category of employees was primarily responsible for actually requesting laboratory tests in the requesting system (meaning the person that physically made the request in the system), and whether the Vitamin D test was usually requested as part of a requesting profile or as a single test (Supplementary material 1).
The guidelines
Using a 6-point Likert response scale (ranging from ‘Very high degreè to ‘Not at all′), all GPs were asked if they assessed the guidelines on the vitamin D test helpful when they were considering the indication for the test. Afterwards, they were asked if they were interested in receiving guidelines for other tests in the future (binary answer). If they answered ‘Yes’ to this question, they were prompted to indicate which types of tests they were interested in receiving guidelines for (classified as ‘New tests′, ‘Tests with new indications′, ‘Expensive tests′, ‘Rarely used tests′, ‘Other′); if ‘Other’ was selected, they were requested to specify what ‘Other’ meant (free text) (Supplementary material 1, items 1-2-3-4).
The feedback reports
Using a 5-point Likert scale (ranging from ‘Always’ to ‘Never′), GPs who had received feedback reports were asked if they read the reports and whether the graphics presented were understandable (binary answer). If the answer was ‘Nò, they were asked to suggest alternative presentations (free text). Furthermore, all GPs (including those in the ‘no feedback report’ group - who received the same question formulated as a hypothetical question) were asked if they were interested in receiving feedback reports for other tests in the future (binary answer). If the answer was ‘Yes′, they had to select which types of tests (classified as ‘New tests′, ‘Tests with new indications′, ‘Expensive tests′, ‘Rarely used tests′, or ‘Other′); if they selected ‘Other′, they were asked to specify what ‘Other’ meant (free text) (Supplementary material 1, items 5-6-7-8-9-10).
The pop-up alerts
The non-interruptive alert
The GPs in the groups ‘non-interruptive alert’ and ‘interruptive alert’ (‘the non-interruptive alert’ constituted the first part of the ‘interruptive alert’) were asked whether they considered the non-interruptive alert displayed in their requisition system during the previous year to be helpful or not (binary answer) (The pop-up alerts are illustrated in Supplementary material 2). If the answer was ‘Nò, they were asked to elaborate on the previous question (free text). Moreover, they were asked whether it was problematic that a non-interruptive alert was displayed every time they requested a Vitamin D test (binary answer).
The GPs in the ‘guidelines group’ had a corresponding hypothetical question instead. They were asked whether it might be helpful to receive guidelines in the form of a pop-up alert when they requested vitamin D (binary answer).
Finally, using a 7-point Likert scale ranging from ‘Never’ to ‘Always′, all GPs were asked to respond as to how long a pop-up alert should be displayed—for both a frequently used test and a rarely used test—for it to be perceived as being helpful (Supplementary material 1, items 11-12-13-14-15-16).
The interruptive alert
The GPs in the ‘interruptive alert’ group were first asked whether the interruptive alert had made them consider the relevance of the vitamin D test, using a 6-point Likert scale ranging from ‘Very high degreè to ‘Not at all′. They were then asked how often they had inserted the specific/correct indication for the test when they were asked to do so, with answers from ‘Always’ to ‘Never’ (Supplementary material 1, items 17 -18).
Overview question
All the GP groups (except the GPs in Group 2, who only received guidelines as intervention, and therefore could not provide meaningful comparisons) were asked which of the interventions they had experienced had worked best. Possible answers were customized based on the interventions the GPs had been exposed to. Finally, all GPs were asked whether they had further comments (free text) (Supplementary material 1, items 19-20).
Statistical analysis
Statistical tests on demographic characteristics and answers to the survey were applied to data from the groups combined according to intervention. However, for results of question number 19 (see survey, Supplementary material 1), the analysis was performed on the six individual groups.
Quantitative survey data were summarized using descriptive statistics. Pearson’s chi-squared test was used to compare the responders to the electronic survey vs. invited GPs group who participated in the earlier randomized controlled trial.
To compare GPs’ characteristics across the intervention groups, we used Pearson’s chi-squared test, ANOVA, or Fisher’s exact test. For the questions on the different interventions, we used Fisher’s exact test. In questions where multiple answers could be given, a test was performed for each possible answer, indicating whether the individual answer was chosen or not. For questions with only one possible answer, an overall test was performed. For ‘the overview’ question, we reported a direct percentage of the rating of the interventions for each individual group.
Stata/BE-Basic Edition 17.0 [Citation8] was used for the calculations, and a p-value <0.05 was considered significant for all statistical tests.
Free text comments are summarized under each subsection in the Results.
Ethics
Under Danish regulations, permission from an ethical committee was not required as the project was a quality study. All respondents agreed to take part in the survey. The study was recommended by the Multi-practice Committee from the Danish College of General Practitioners (DSAM) and the GPs’ professional organization.
Results
Demographic characteristics and requesting patterns
Of the 638 GPs who received the survey, 131 completed and returned it, yielding a response rate of 21%. As shown in , there were no significant differences between the respondents and the invited GPs in terms of gender, age, and type of clinics.
Table 1. Characteristics of responders to the electronic survey vs. the group of GPs invited in the study.
As shown in , the GPs in the different intervention groups had similar characteristics. No significant differences were found for gender, age, years of experiences as a GP, or type of clinic. The mean number of GPs per clinic ranged from 2.9 to 3.5 in the intervention groups.
Table 2. GP characteristics in the intervention groups. Absolute numbers and percentage of GPs answers.
As shown in , Vitamin D was primarily requested by doctors in almost all clinics, followed by nurses and other staff categories .The same pattern was seen when participants were asked to specify the one type of employee who most often requested the test. More than half of the clinics responded that vitamin D was requested as part of a profile (‘often’ and ‘sometimes’ were chosen by 54% to 65% in each group).
Table 3. Survey questions regarding clinic working routines. Absolute numbers and percentage of GPs answers.
The guidelines
As shown in , approximately half of the participants reported that guidelines were helpful (to ‘high degreè or ‘some degreè). A significant difference was found between the intervention groups due to the different numbers answering ‘low degreè, ‘not at all’ and ‘do not know’ regarding the value of guidelines (p-value 0.003). No significant difference was found between the two feedback groups according to whether they considered guidelines helpful or not.
Figure 1. a) the percentage of GPs who found the guidelines helpful to ‘high’ or ‘somé degree; b) the percentage of GPs who read the feedback reports ‘always’ or ‘often’; c) the percentage of GPs who found it problematic that the pop-up alert was displayed every time a test of Vitamin D was requested.
All groups were interested in receiving guidelines in the future (ranging from 67% to 82% without significant differences between groups), including the 21 GPs who answered ‘do not know’ to the question whether guidelines were considered helpful (6 out of 10 in the guidelines group and 10 out of 11 in the non-interruptive alert group).
GPs preferred to receive guidelines for use of ‘New tests’ (ranging from 60% to 76%) followed by ‘Tests with new indications’ (58% to 73%), ‘Expensive tests’ (53% to 62%), and ‘Rarely used tests’ (29% to 41%). There were no significant differences between groups (Supplementary material 3)
Overall, GPs gave positive comments about receiving guidelines. Some GPs noted that it was important to keep guidelines up-to-date and as simple as possible. It was also noted, that GPs receive many guidelines from different sources, which might cause problems.
The feedback reports
Half of the GPs who received feedback reports stated that they read the reports (always or often) () and that the reports were easy to understand.
Most of the GPs (52% to 69%) were open to receive feedback reports in the future (including the groups who did not receive feedback reports as an intervention). GPs preferred to have feedback reports on ‘Expensive tests’ (46% to 67%), ‘Tests with new indications’ (40%-57%), ‘New tests’ (31%-48%), and ‘Rarely used tests’ (17%-37%) (Supplementary material 3).
Some GPs noted that the feedback reports were difficult to understand and suggested making the graphics more clear in the future. However, no specific suggestions for how to improve the layout or the graphic were provided.
The pop-up alerts
The non-interruptive alert
The non-interruptive alert was generally found helpful (48% to 63%), and for half of the GPs in the ‘non-interruptive alert’ group, it was not a problem that the alert was shown every time the test was requested (47% said yes) ().
Overall, GPs indicated that a pop-up alert might be displayed for 3 months—and less often for frequently used tests. However, the groups who received the pop-up were more likely to indicate a shorter time, while in the guidelines group – who did not experience the alert themselves - the answers were spread over all options, resulting in a significant difference between the groups (p-value 0.002). GPs agreed that for ‘Rarely used tests′, the pop-up might be displayed for long periods—more than half of all GPs chose the answer ‘Always’ to this question (Supplementary material 3).
Two-thirds of GPs in the guidelines group reported that having pop-ups in the future might be helpful (62% said yes).
The interruptive alert
The GPs in the interruptive alert group reported that having the pop-up alert displayed every time they requested the test was problematic (67% said yes) (), resulting in a significant difference between the non-interruptive and the interruptive alert groups (p = 0.009). No significant differences were seen between the two feedback groups to this question (Supplementary material 3).
Most GPs in the interruptive alert group answered that having to specify an indication for each vitamin D requisition made them reconsider the use of the test (to some degree: 39% to 35%; to low degree: 32% to 35%). Participants stated that they always (61% to 50%) or often (50% to 25%) specified the correct indication for each patient. No significant differences were seen between the feedback groups to this question.
Some GPs commented that ‘having a pop-up alert that is displayed every time the test is requested is irritating/annoying′, while others thought that ‘the pop-up was helpful and make the doctors avoid requesting unnecessary tests′. One participant suggested to ‘display the information in an ‘information box’ that a GP could open when they felt it necessary (irritation could then be avoided) ‘.
The overview
shows the rating of each GP group for the interventions. The guidelines + feedback reports group suggested that ‘feedback reports’ was the intervention that worked best, while all the other groups favored the ‘non-interruptive alert’ (54% to 77%). ‘The guidelines’ intervention was selected by 8% to 35% of GPs, ranking it as the second-best intervention in all the groups.
Table 4. Overview question. The rating of the interventions reported by each group of GPs.
Discussion
This study showed that GPs generally accepted interventions in the requisition process for biochemical tests, although there were clear messages as to the kind of intervention preferred and the period for which it should be applied. Interruptive alerts were found to be annoying when they were used for long periods and when they appeared every time a frequently used test was requested.
Previous studies [Citation9,Citation10] reported that educational interventions are appreciated by GPs, which was confirmed by the present study as approximately half of the GPs found guidelines to be helpful, and a similar proportion of GPs read the feedback reports ‘often’ or ‘always′. However, despite the evaluation of GPs, we recently demonstrated that the use of guidelines alone or combined with feedback reports did not significantly change the number of tests requested or improved the ratio of clinically interesting results when compared to the control group [Citation5].
Twenty one GPs answered ‘do not know’ to the question of whether the guidelines on vitamin D were helpful, but most of them (16 GPs) still wanted to receive guidelines for other tests in the future. A possible explanation for these contradictory answers might be that the GPs did not actually see the guidelines themselves, and thus were unaware whether the guidelines had been helpful, but they still considered it worthwhile to receive guidelines in the future.
Eleven GPs found the feedback reports difficult to understand, but five of them would still like to receive feedback reports in the future, especially for expensive tests. As also reported by Horn et al. [Citation11], doctors are highly interested in knowing the cost of the laboratory tests, but when information on test costs is made available as a method to optimize test usage, the results have been inconsistent [Citation4,Citation12,Citation13].
Pop-up alerts have not been widely used in primary care in the context of test ordering [Citation4], and to our knowledge, collecting GPs views using an electronic survey has not previously been reported.
Investigation of methods to optimize GPs’ use of laboratory tests should be designed in collaboration with representatives of the GPs, as we and others have done [Citation5,Citation14,Citation15]. Another important aspect of this type of research is to ask the requesters about how the interventions were assessed by the users and how the interventions functioned in daily practice.
The pop-up alerts were explicitly designed to be used by the doctors, but the survey revealed that GPs are not the only personnel category that request tests in the electronic requisition system, as ‘Nursè and ‘Other’ request almost as often as the GPs themselves. It is not easy for other health care professionals than doctors to answer the interruptive alert in the electronic system. This might explain why some clinics in the interruptive alert group reported the reason for the test was ‘because the doctor ordered the test′.
Significantly more GPs found that interruptive alerts were ‘problematic’ compared to non-interruptive alerts when both type of alerts appeared every time the test was requested. This may reflect the extra time needed to answer the questions about indication for the test and whether it was ordered for diagnostic or monitoring purposes. It cannot be excluded that the alert fatigue observed in the interruptive alert group was caused or increased by the relatively long period (one year, which was chosen due to the seasonal variation of Vitamin D) of exposure to the alert. Most GPs exposed to the interruptive alert considered it should be used ‘never’ or maximally for ‘3 months’ with frequently used tests. In comparison, more than half of the responders considered that alerts could be used ‘always’ for rarely used tests.
This study has some limitations. Although we considered the questionaire easy to access and to fill out, only 21% of the potential responders actually did so, raising questions about the representativeness of the study sample. It is possible that the responders were more motivated or had stronger opinions - positive or negative - about the interventions under evaluation compared to those who did not to respond, potentially biasing the results. However, we received responses from 131 GPs who were evenly distributed among the intervention groups and found no statistical differences in gender, age or type of GP clinic between the groups of GPs. Unfortunately, we were not able to compare responders to non-responders in terms of length of experience as a GP due to absent data for the non-responders. As the responders did not differ significantly from the entire group of GPs, we interpret the differences found in the answers as related to the interventions.
It is possible that the ‘real’ response rate is higher than 21% as some answers might have been given by one GP on behalf of all GPs in an individual clinic, which might mitigate this issue. However, this of course can’t be counted on.
The survey was customized to the interventions that each group received, but some questions were asked in a hypothetical way to the groups that did not receive that specific intervention. This might add to variation of answers. On the other hand, the hypothetical questions increases the number of GPs expressing their opinion on the issues under consideration, which is positive considering the low response rate.
Conclusions and future research
The current study demonstrates that feedback reports are communication tools with a potential high visibility as they were read by half of the GPs. Non-interruptive alerts in the requesting IT system are well tolerated for shorter periods with often used tests and for longer periods for rarely used tests.
In the randomized controlled study, the alerts significantly reduced the number of tests requested, but the fraction of clinically interesting results was not increased [Citation5]. In fact, the clinical impact of the test was lower when the alerts were introduced into the system. When guidelines and/or feedback reports were applied, there was no reduction in the number of tests requested and no increase in the proportion of clinically useful tests [Citation5]. Thus, no matter which type of intervention is considered for the requisition process, other changes are also necessary to improve the clinical effectiveness of the tests that are requested.
Future research in this area should focus on supportive facilities that are considered helpful and relevant by the users in terms of improving use of the biochemical tests. This probably is best achieved through a high degree of user involvement in developing the supportive interventions. According to the survey responses, we suggest to display pop-up alerts for short periods, but to repeat them with relevant intervals. Furthermore traditional feedback reports could be improved, by showing not only the numbers of test requested, but also clinical relevant information whenever possible.
Supplemental Material
Download PDF (385.2 KB)Disclosure statement
No potential conflict of interest was reported by the authors.
References
- Rotenberg DK, Stewart-Freedman B, Søgaard J, et al. Similarities and differences between two well-performing healthcare systems: a comparison between the Israeli and the Danish healthcare systems. Isr J Health Policy Res. 2022;11(1):14. doi:10.1186/s13584-022-00524-x.
- Chauhan BF, Jeyaraman MM, Mann AS, et al. Behavior change interventions and policies influencing primary healthcare professionals’ practice-an overview of reviews. Implement Sci. 2017;12:3.
- Khalil H, Bell B, Chambers H, et al. Professional, structural and organisational interventions in primary care for reducing medication errors. Cochrane Database Syst Rev. 2017;10(10):CD003942. doi:10.1002/14651858.CD003942.pub3.
- Lillo S, Larsen TR, Pennerup L, et al. The impact of interventions applied in primary care to optimize the use of laboratory tests: a systematic review. Clin Chem Lab Med. 2021;59(8):1336–1352. doi:10.1515/cclm-2020-1734.
- Lillo S, Larsen TR, Pennerup L, et al. A randomized controlled study of biochemical tests in primary care: interventions can reduce the number of tests but usage does not become more appropriate. Clin Chem Lab Med. 2022;60(3):343–350. doi:10.1515/cclm-2021-1138.
- RedCap 13.1.33. 2023 https://www.project-redcap.org/.
- Crawford IM. 1997. Marketing research and information systems. Food & Agriculture Org. ISBN 9251039054, 9789251039052
- StataCorp. 2021. Stata statistical software: release 17. College Station TSL.
- Mindemark M, Larsson A. Long-term effects of an education programme on the optimal use of clinical chemistry testing in primary health care. Scand J Clin Lab Invest. 2009;69(4):481–486. doi:10.1080/00365510902749123.
- Vrijsen BEL, Naaktgeboren CA, Vos LM, et al. Inappropriate laboratory testing in internal medicine inpatients: prevalence, causes and interventions. Ann Med Surg (Lond). 2020;51:48–53. doi:10.1016/j.amsu.2020.02.002.
- Horn DM, Koplan KE, Senese MD, et al. The impact of cost displays on primary care physician laboratory test ordering. J Gen Intern Med. 2014;29(5):708–714. doi:10.1007/s11606-013-2672-1.
- Silvestri MT, Bongiovanni TR, Glover JG, et al. Impact of price display on provider ordering: a systematic review. J Hosp Med. 2016;11(1):65–76. doi:10.1002/jhm.2500.
- Goetz C, Rotman SR, Hartoularos G, et al. The effect of charge display on cost of care and physician practice behaviors: a systematic review. J Gen Intern Med. 2015;30(6):835–842. doi:10.1007/s11606-015-3226-5.
- Salinas M, Lopez-Garrigos M, Asencio A, et al. Laboratory utilization improvement through a computer-aided algorithm developed with general practitioners. Clin Chem Lab Med. 2015;53:1391–1397.
- Salinas M, López-Garrigós M, Flores E, et al. Laboratory intervention to improve the request of urinary albumin in primary care patients with arterial hypertension and financial implications. Clin Biochem. 2019;69:48–51. doi:10.1016/j.clinbiochem.2019.04.012.