https://orcid.org/0000-0002-7491-517X
ABSTRACT
Introduction
There is limited access to life-saving antenatal ultrasound in rural and low-resource settings largely due to shortages in skilled staff. Studies have shown healthcare practitioners can be upskilled in PoCUS through focused training, offering a viable solution to this deficit. However, standards for training and competency assessment are unclear and regulation surrounding practice is lacking. We aimed to review published literature examining antenatal PoCUS training programs, comparing teaching approaches and study methodologies.
Methods
A search of electronic databases EMBASE, MEDLINE and Google Scholar was conducted. Original research articles evaluating antenatal PoCUS training of healthcare professionals worldwide were identified for analysis. Articles with limited detail on the PoCUS training intervention and those describing comprehensive diagnostic training programs were excluded. Evaluations were compared against the Kirkpatrick Evaluation Framework (KEF).
Results
Twenty-seven studies were included from an initial search result of 484 articles. There was considerable heterogeneity between the PoCUS training programs described. Course duration ranged from 3 hours to 2 years, with 11 of the 27 studies delivering obstetric-exclusive content. 44% trained multidisciplinary groups of health professionals. Long-term follow-up training and skills assessments were lacking in over half of the reviewed studies. Study quality and reporting detail varied, but overall beneficial outcomes were reported with 3/4s of the studies reaching upper KEF levels 3 and 4.
Conclusion
PoCUS performed by upskilled healthcare professionals offers an attractive solution to the problem of inequitable access to antenatal ultrasound. A review of available literature highlighted a paucity of comparable high-quality studies needed to establish a stronger evidence base for antenatal PoCUS, and a need to standardise training and competency assessment. This review may inform educators, researchers and policy-makers on existing training formats and methodologies to assist in establishing best practice antenatal PoCUS training methods for safe service delivery by remote healthcare professionals.
Introduction
Antenatal ultrasound is the primary imaging modality in pregnancy [Citation1,Citation2], routinely used to estimate due dates, monitor fetal growth and well-being, detect anomalies, and guide specialist referral [Citation3,Citation4]. It can also facilitate the early detection of life-threatening complications such as ectopic pregnancy, fetal malpresentation, multiple pregnancies, placenta praevia and placental abruption [Citation5–8]. The International Society of Ultrasound in Obstetrics and Gynaecology (ISUOG) have published guidelines recommending women receive two antenatal US examinations during a normal low-risk pregnancy [Citation9]. However, studies into service accessibility in rural, remote and low-resource settings around the world indicate women are not receiving this care [Citation10–12]. The WHO estimates most maternal deaths are preventable, with over 90% occurring in low-resource settings. Approximately 86% of estimated global maternal deaths in 2017 were attributed to the developing nations of sub-Saharan Africa and Southern Asia [Citation13]. The majority of neonatal mortality also occurred in these regions [Citation14]. A 2015 survey of healthcare providers in South America, Africa and Asia reported the primary reason for not making ultrasound available to pregnant women was a lack of suitable education [Citation15]. Skill shortages are also evident in developed nations like the USA, Canada and Australia, where many remote medical centres have no onsite sonographer and rely on visiting professionals available as infrequently as one day per month [Citation10,Citation16]. The recent COVID-19 pandemic has increased uncertainties in travel and logistics, impacting locum staffing in rural areas and highlighting the importance of trained remote healthcare workers. Accurate estimation of due dates and early detection of potentially life-threatening complications are crucial for remotely located women, who may need days of travel to access specialist obstetric care [Citation7,Citation10,Citation17]. It is in these low-resource settings that antenatal Point-of-Care ultrasound (PoCUS) can offer substantial benefits.
Modern portable ultrasound machines capable of producing high-quality images are affordable and have helped establish PoCUS in many medical fields [Citation1,Citation3]. Performed and interpreted at the bedside by the healthcare provider, PoCUS allows for focused studies to assist procedures or direct care and referal [Citation18]. As a highly skilled, operator-dependent modality, PoCUS requires appropriate and ongoing training of experienced healthcare professionals [Citation3]. It takes years of study and training to produce qualified sonographers, and once trained it is challenging to entice these professionals to relocate and remain in rural locations [Citation16,Citation19,Citation20]. There is growing evidence indicating PoCUS training programs can effectively teach the skills necessary to allow for task-shifting of focussed ultrasound examinations from sonographers to doctors, nurses and midwives [Citation21–26]. Ultrasound training is being increasingly incorporated into undergraduate medical curricula and on the job training [Citation26–28], but it is less well established in non-physician (nursing and midwifery) programs [Citation29,Citation30]. In most developing countries and low-resource settings, antenatal care is provided primarily by midwives and nursing staff, which presents an opportunity to task-shift and upskill these essential workers [Citation21–26].
lists general PoCUS workshop requirements and methods for assessing competency [Citation31–33]. The WHO recommends a standardised curriculum and competency assessment be adopted by all countries for antenatal PoCUS training [Citation5]. However, training guidelines and standards to ensure a minimum level of competency for safe practice vary between countries, with PoCUS remaining largely unregulated globally [Citation34–36]. In many countries, health practitioners may perform PoCUS with little or no training, and without formal accreditation, leading authorities to call for reform and regulation of its use [Citation16,Citation37]. Ultrasound performed by untrained clinicians may represent a higher risk of misdiagnoses. Overlooked health conditions may lead to delayed diagnoses and treatment (‘false negatives’), while misinterpretations and incidental findings (‘false positives’) can cause considerable patient anxiety and unnecessary follow-up investigations, increasing the economic burden on the healthcare system [Citation38–40].
Table 1. General PoCUS workshop requirements and methods for assessing competence
Endorsement for outreach training programs have been provided by the Australasian Society for Ultrasound in Medicine (ASUM), ISUOG, World Federation for Ultrasound in Medicine and Biology and RAD-AID, indicating a global effort to address the skill shortage [Citation35]. This literature review examines the training and evaluation methods being employed to teach antenatal PoCUS to medical and allied health professionals.
Methods
This review investigated international literature on antenatal PoCUS education from 2000 to January 2021, focusing on publications that evaluate the efficacy of training models. It has adopted the ‘Preferred Reporting Items for Systematic Reviews and Meta-analysis’ (PRISMA) guidelines and was formerly registered with the international prospective register of systematic reviews (PROSPERO), registration number CRD42021230267.
A team of three researchers from the University of South Australia with experience in research, tertiary education and clinical practice (including ultrasound and PoCUS training in low-resource settings) conducted the review and reached consensus on the eligibility criteria, search strategy and terms, final article inclusion, data extraction and quality assessment.
Search strategy
A systematic search of electronic databases EMBASE and MEDLINE was conducted through OVID for original research literature, performed on the 7th of January 2021. An experienced librarian was consulted to assist in the design of search terms and strings, which were then reviewed by all members of the research team. The search was limited to ‘Human’ and ‘English’ language only. No restrictions were set regarding the publication year. Search terms used for both databases were grouped into main four areas and combined using terms synonymous with pregnancy, ultrasound, point-of-care, and training. A grey literature search was conducted through Google Scholar (5 pages, 50 results) using the key search terms (pregnancy/antenatal, point-of-care ultrasound/ultrasound, and training/education). Connected papers (https://www.connectedpapers.com/) was searched to canvas for additional relevant articles (see Appendix : Search strategy).
Eligibility criteria
For inclusion, an original research study must have described and evaluated an ultrasound training program intended for point-of-care or bedside application on antenatal patients. Articles involving PoCUS obstetric training as part of a broader training curriculum were eligible. All medical and allied health specialties were included as the training participant population, and pre-graduate students from all health disciplines. No restrictions were placed on study/training setting.
Studies evaluating advanced training (complex and interventional scanning) or formal diagnostic ultrasound programs leading to qualification as a sonographer were excluded. Articles with limited descriptions of the training program provided were excluded, as were conference abstracts/reviews, editorials, commentaries and letters (see : Eligibility criteria).
Table 2. Eligibility criteria
The database search was performed by the primary author. Duplicates were removed prior to the initial title and abstract screening conducted independently by two reviewers, who then performed full-text reviews. Non-consensus at both initial title/abstract screening and later full-text screening was decided by a third independent reviewer. The citations of all identified articles included in the review were searched, with title/abstract then full-text screening performed by two independent reviewers.
Data extraction and synthesis
All three reviewers discussed themes and agreed on data points to be extracted, and reviewed these pre-determined categories in a Microsoft Excel spreadsheet prior to data extraction. Data was extracted to the spreadsheet and collated by the primary author and examined by a second reviewer. Meta-analysis was not possible due to the heterogeneous teaching and assessment methodologies used, thus an integrative approach to data synthesis was employed. Emerging themes were discussed amongst all three reviewers and a narrative response was composed. Comprehensive tables summarising the reviewed studies’ training and evaluation methods, and key investigated outcomes are included to facilitate comparison.
Quality assessment
A modified Medical Education Research Quality Index (MERSQI) tool (see Appendix ) was used to assess the quality of the included articles. The MERSQI is a validated assessment instrument used in medical education research to measure the quality of experimental, quasi-experimental and observational studies [Citation41,Citation42]. The tool was modified to include two categories: 1) ‘Number of trainees’ (score of 0.5 to 1.5), and 2) ‘Follow-up of training’ (score of 0 to 3). These domains were considered valuable to this review given the variation of recruited participant numbers between studies (impacting study power), and the importance of follow-up training, assessment and ongoing support of trainees for learning PoCUS and ensuring safe practice. The percentage response rate was omitted as this measure was not applicable to the vast majority of the studies. Each domain was scored out of 3. The primary author performed the quality assessment, grading all articles within a potential score range of 5 to 21. A risk of bias assessment was also conducted relating to five key areas of educational development: Underpinning bias, Resource bias, Setting bias, Content bias, Educational/Development bias (see Appendix : Risk of bias assessment tool). However, assessment of quality and risk of bias did not restrict article inclusion in this review’s final synthesis, ensuring the inclusion of a broad cross-section of literature representative of the range of study quality and methodologies in published circulation.
Each study reviewed was compared against The Kirkpatrick Evaluation Framework (KEF), a validated four level model designed to evaluate and classify training and development programs [Citation43,Citation44]. The four levels described (Level 1- Reaction; Level 2- Learning; Level 3- Behaviour; Level 4- Results) by the Kirkpatrick model represent a continuum of complexity and value in evaluation measures. Level 4, representing the highest evaluation measure (healthcare or patient-related outcomes), assesses impact and aligns with the MERSQI ‘Outcomes’ domain.
Results
Twenty-seven studies were included, from an initial 484 articles retrieved in the OVID database search during the identification stage (see : PRISMA framework). Of the 27 articles, 16 were identified through EMBASE and MEDLINE, with a further 11 obtained through citation searches. Google Scholar and Connected papers searches yield no additional eligible articles.
Figure 1. PRISMA framework – Search results.
The identified studies spanned two decades; the oldest study published in the year 2000. Three-quarters were published after 2014 reflecting the growing interest in PoCUS research. Most were single arm interventional studies using convenience samples of health professionals and patients. Four of the studies were conducted by the same research team, Shah et al. [Citation45–48]. Twenty studies were conducted in developing countries, of which only five were conducted in urban settings. Seven studies included developed countries (five in the USA, all urban areas with one intended for rural deployment; one in Australia and one in Denmark). Two of these studies used training sites in both developed and developing countries; one [Citation49] employed a clustered randomised control trial design that was carried out over five clinical sites located in Africa, Asia and America, and the other [Citation50] compared six training sites and formats conducted in rural and urban settings in Australia, Timor Leste and Indonesia. Key outcomes investigated and the main findings of the studies are summarised in .
Table 3. Key outcomes investigated and findings
Training methods and delivery
A summary of the included studies’ teaching methods, including duration and location, trainee and instructor demographics, curriculum and practical skills taught, and follow-up training/support, is provided in Appendix : Training methods and delivery.
Curricula
Course duration ranged from 3 hours to 2 years. Obstetric-exclusive curricula were delivered in 11 of the 27 studies (see ). Of the studies that covered multiple-organ systems, several delivered their content in as few as 1–2 days [Citation21,Citation22,Citation51,Citation52], with either refresher sessions or additional online learning modules provided to support the intensive practical training. When designing intensive courses with limited delivery time, other resources to supplement face-to-face training, such as digital and written learning modules/resources, and online/distance teaching and feedback are important. Many of the studies employed these supportive measures (see Appendix ). Maintaining a focussed curriculum for specific predefined clinical indications and remote technical assistance was recommended [Citation51].
Figure 2. Of the included studies in the review a. Curriculum delivered (Multiple organ systems taught or Obstetrics and gynaecology only); b. Number of professionals trained (total recruited and trained per study); c. Ultrasound experience of trainees prior to undertaking the training; d. Discipline/role of the trainees (Multidiscipline/Physicians/Nurses/Midwives).
The course topics and practical skills taught by each study are listed in of the Appendix; these ranged from basic assessments like fetal lie/presentation to more complex fetal biometry, heart assessment and the detection of anomalies such as ventriculomegaly, anencephaly, hydronephrosis, and spina bifida. A training needs assessment was conducted by five of the studies [Citation24,Citation26,Citation46,Citation47,Citation53] which informed the choice of ultrasound applications that were most relevant for program inclusion. Specific skills were taught if relevant to endemic needs; for example, three studies [Citation51,Citation54,Citation55] conducted in Tuberculosis prevalent regions included ultrasound assessment of HIV/Tuberculosis (FASH) in their multisystem curriculum.
Over two-thirds of the studies detailed ultrasound physics and instrumentation in their curriculum, and four specifically mentioned ‘safety’ [Citation30,Citation45,Citation49,Citation56], consistent with the WHO’s recommendations that trainees are able to monitor mechanical and thermal indices on equipment and understand safety concepts underpinning ultrasound exposure of the patient and fetus [Citation57]. Trainees should understand the limitations of their focussed training and seek assistance with image interpretation and patient management decisions from an experienced sonographer and/or clinician where necessary. Appropriate documentation of the PoCUS exams findings and patient referral is also important. Three of the studies listed ‘documentation’ in their curriculum [Citation29,Citation30,Citation58], one specifically including ‘liability and risk reduction strategies’ [Citation29].
Trainees’ previous ultrasound experience
Participant trainee numbers in any single study ranged from 3 to 162, with a total of 903 trainees across all included studies. Over half of the studies trained fewer than 20 trainees, limiting study power (see ). Of the studies that reported trainees’ prior ultrasound experience, nine trained participants with mixed ultrasound experience, six enrolled trainees with limited but similar ultrasound exposure and seven restricted eligibility to trainees with no prior ultrasound training or experience (see ). Delivering a program to suit trainees with varied ultrasound education and proficiency was an acknowledged challenge [Citation22,Citation51]. Shokoohi et al. [Citation51] raised concerns that more experienced trainees may not have received the same benefit from their curriculum, particularly in the introductory session. Grouping trainees and designing specific curricula and objectives catered to each groups’ experience and skill level could mitigate this problem.
Multidisciplinary training groups
The future of healthcare and its education is moving towards a more cooperative interdisciplinary culture [Citation59], demonstrated in the multidisciplinary participant groups trained together in almost half of the reviewed studies. Ten (33%) studies exclusively trained medical physicians and only 5 (19%) were dedicated to nurses and midwives (see ). Several authors [Citation29,Citation50] described unique challenges to interprofessional teaching, where individual needs and preferences can vary significantly between trainees. This scenario, however, provides the opportunity to use trainees’ unique experiences and individual strengths to enhance course design and foster collaborative practice. Shaw-Battista et.al [Citation29]. reported enhanced inter-professional collaboration as a benefit of their multidisciplinary PoCUS training initiative, stating it fostered better communication, coordination of care and understanding of other professionals overlapping and unique scopes of practice. Providing clear objectives, varying curricula in breakout groups tailored to participant’s experience level, and the opportunity for peer-led teaching in mixed skills/experience groups would benefit a multidisciplinary PoCUS training cohort.
Instructors’ experience
Instructor-to-trainee ratios were reported in over half of the studies, mostly for practical training sessions, and ranged from 1:1 to 1:6; guidelines have recommended a maximum ratio of 1:5 [Citation32,Citation60]. The instructors’ experience ranged from second year medical students (aimed at providing low-cost outreach training in developing countries [Citation54,Citation55]) to trained sonographers and medical specialists. Some courses benefited from a multidisciplinary teaching team with specialised content delivered by experts in their field (cardiac, paediatric and obstetric specialists). Several articles included principles of teaching in their curriculum [Citation45,Citation61], and reported scenarios where trainees went on to teach ultrasound to colleagues on return to work [Citation51,Citation61,Citation62]. Training the trainer initiatives, where trainees are taught teaching methods to pass on learning to colleagues, is a useful and potentially cost-saving option in settings with limited resources and access to training. Care should be taken using this teaching model with PoCUS courses of short duration, given the complexities of learning ultrasound and the often unsupervised work environments trainees return to. Instructor qualifications and teaching ratios are provided in Appendix .
Ethical considerations
Few papers mentioned the ethics surrounding the use of pregnant patients for ultrasound training purposes. Ideally, practical training would utilise both simulated and real-life patients with strictly limited times placed on scanning pregnant volunteers and a heavier reliance on phantom models and virtual/simulation technologies in early training [Citation50,Citation63]. In most studies, only healthy models were used during training, which precludes the demonstration of pathology [Citation29,Citation55]. Simulation can be beneficial in this respect and offers the advantage of learning in a safe, patient-free environment. They have a particular utility in obstetric ultrasound training, but the cost of implementing high-fidelity simulated systems would be prohibitive in the majority of settings reviewed [Citation64,Citation65]. Westerway [Citation50] reported using commercial and handmade phantoms and Vinayak et al. [Citation63] used ‘scanning phantoms’ in the initial week of training, but no high-fidelity simulation systems were used in any of the reviewed studies. Shaw-Battista et al. [Citation29] discussed the ethics of using pregnant models, stating their intention to introduce equipment to simulate first-trimester ultrasound in their next course iteration. This would reduce reliance on pregnant volunteers and provide the opportunity to scan simulated first trimester pregnancies, commonly lacking in training courses due to the early gestation of the fetus and risk of identifying an unexpected abnormality in pregnant volunteers who are yet to receive formal scanning.
Follow-up training and support
Follow-up training is beneficial to reinforce learning and provides the opportunity to assess knowledge retention, which is important for continued safe practice. Follow-up training sessions, in either face-to-face or online format, were offered by half the studies, with periods varying from 3 months to 2 years. Appendix summarises the follow-up training and support provided by the reviewed studies. Of those who did not report/provide additional follow-up training sessions, five offered assistance through personal telecommunications feedback and image review. Telemedicine was investigated by Kolbe et al. [Citation26] who provided remote real-time scanning and image review. Vinayak et al. [Citation63] used an asynchronous method, where trainee images and interim report was sent for specialist review while the patient waited. Other studies used telecommunications via email and various messaging platforms to remotely assist trainees, provide feedback and review images, but only Kolbe et al. [Citation26] used remote real-time scanning supervision.
Training evaluation and assessment methods
The approach and method used to evaluate the courses and trainees varied widely across studies. A summary of the evaluation methods reported including knowledge and practical assessment, expert image review, frequency and application of scanning, patient outcomes and trainee feedback/survey is provided in : Trainee & course evaluation.
Table 4. Trainee & course evaluation
Fourteen studies performed knowledge assessment of the trainees. Nine of these conducted pre- and post-training tests, and of these, five administered identical exams before and after training. Utilising the same exam allows for a quantifiable measure of trainee improvement but can introduce bias. This risk may be mitigated by randomising question order, not informing trainees the test would be re-administered or discussing test results. Trainees from five studies were required to pass a written test following online self-directed learning before proceeding to practical training. Pre-course learning and testing saves face-to-face time for hands-on learning and can ensure trainees have similar base knowledge on course entry. Six studies reported performing follow-up (knowledge retention) assessment in addition to any immediate post-course testing. Practical assessments or Objective Structured Clinical Examinations (OSCE) were conducted in 16 of the studies, of which half performed consecutive testing allowing for improvement measures. Overall, half the studies performed expert image review (remotely or during training) for quality assurance and to assess competence, and for some, to guide feedback and the necessity for refresher training. This is a useful competence measure where direct supervision is not possible, as it may be performed asynchronously and remotely. Written or practical assessments were not described by seven of the studies. Of these, expert image review was performed by four studies as an indirect evaluation method.
Almost half the studies investigated patient outcomes, several going further to ascertain if PoCUS changed the patient diagnosis and if this impacted their management/treatment. The clinical application was evaluated in nine studies, and in some cases, the frequency of scanning following training was used as an evaluation measure [Citation21,Citation24,Citation30,Citation51,Citation53,Citation62,Citation66,Citation67]. Other measures useful for quality control and course improvement are post-course evaluation surveys, conducted by seven [Citation21,Citation29,Citation45,Citation50,Citation51,Citation54,Citation55] of the reviewed studies, and self-reported post-course scanning confidence that was evaluated by three [Citation45,Citation54,Citation56]. Several studies [Citation48,Citation63] used ‘time to complete scans’ as an evaluation measure. In Vinayak et al. [Citation63] scan times halved after 30 completed examinations with consistent image quality. While not the best measure of competence, speed is an important consideration in time poor, resource-limited settings where extensive scanning times could be prohibitive to PoCUS examination during antenatal consultation. Shah’s et al. [Citation48] study demonstrated that an entire focussed ultrasound assessment (fetal heart rate, head position and estimated gestational age) could be completed by trainees in under 5 minutes, important when facing time-sensitive decisions in an emergency caesarean delivery.
With the aim of revising policy and training to align with the credentialing requirements of their site/country, Lathrop et al. [Citation58] investigated the introduction of a learner portfolio (documented evidence of didactic learning, teaching resources, logged cases and images) and evaluation rubric to demonstrate competence. This approach is consistent with other accrediting authorities’ requirements of scanning logs and evidence of completing a pre-set number of studies. Lathrop et al. [Citation58] was the only study whose participants all progressed to formal accreditation in the use of antenatal PoCUS.
It should be noted many of the reviewed studies were outreach projects that aimed to maximise training opportunities in low-resource settings. While minimal trainee assessment may have been undertaken by some, it is possible that the methods reported were not the only means of assuring trainee competence. Ideally, some form of competence assessment should be performed and support options provided before trainees perform unsupervised clinical scanning. All studies concluded positively regarding the PoCUS training intervention investigated. The outcomes investigated by each study and their main findings are summarised in .
Barriers to PoCUS following training
Longer-term follow-up training and skills assessments, essential to building confidence and ensuring competence and retention of learning [Citation36], was lacking in over half of the reviewed studies. Insufficient onsite supervising experts on return to clinical practice was also a recurrent theme [Citation21,Citation29,Citation51,Citation56,Citation58,Citation66,Citation67]. This problem is further compounded in some locations by poor telecommunications access, which impedes off-site assistance [Citation22,Citation51,Citation66]. In such cases, telehealth, which is emerging strongly in the wake of the COVID pandemic, would be ineffective as a tool for real-time support of trainees. Several articles in this review offered off-site asynchronous expert image review for quality assurance and feedback [Citation26,Citation45–47,Citation51]. This solution is inadequate in cases where technical hands-on correction is required or immediate image review is needed to guide patient management in an emergency.
Another common barrier included access to quality ultrasound equipment following training. Henwood et al. [Citation53] reported some trainees did not have routine access to ultrasound and the ability to save images from completed examinations, and in Westerway [Citation50], not all trainees scanned patients on return to work due to no or poor/faulty equipment. Busy departments allowing little time for scanning or supervision was another reported barrier. Half of the midwives surveyed in Kimberly et al. [Citation62] reported difficulty finding time to perform ultrasound due to heavy clinical workloads and raised concerns over neglecting other clinical obligations. Almost half of the participants surveyed by Shokoohi et al. [Citation51] listed ‘lack of time to scan’ as the main perceived challenge integrating PoCUS into patient care.
Limitations and quality of the reviewed studies
There was a distinct lack of pedagogy described by the reviewed studies. Only Westerway [Citation50] provided a detailed discussion describing the New World Kirkpatrick training evaluation model. Elements of pedagogy were present in other studies’ designs. For example, a constructivist, flipped classroom approach was utilised by a number of training programs [Citation22,Citation51,Citation52,Citation63], but limited description of the principles and foundations of this model were provided.
Generally, the studies suffered from limitations and biases inherent in research conducted in remote settings, including small study designs- low participant numbers, convenience samples, and loss to follow-up. A lack of longer-term follow-up of trainee outcomes was a reported limitation of many of the reviewed studies [Citation46,Citation49,Citation62,Citation66,Citation67]. This is likely the result of geographic isolation, finite funding and an overburdened and transient health workforce [Citation68,Citation69]. Of those that did follow trainee progress, comparison of participants was confounded for some by inconsistent ultrasound exposure on their return to different workplaces, with varying onsite assistance and supervision between sites. Eleven of the studies performed expert review of trainee images, most asynchronously with the expert unable to perform concurrent scanning to verify trainees’ findings [Citation49,Citation58,Citation63]. Blinded image review was specified by several authors [Citation45,Citation70] to mitigate this shortcoming. Poor and worsening participant compliance with patient data recording and image logs was reported by several studies [Citation62,Citation66], some basing image review and quality control on scans selected for uploading by the trainees, potentially biasing results [Citation49].
The overall quality of studies and their evidence varied, with the MERSQI assessment scores ranging from 7.5 to 18 out of 21 (mean score of 12.9). The review lacked randomised, controlled studies necessary to achieve a top MERSQI score. The upper KEF levels 3 and 4 were reached by 3/4s of the included studies. Studies reaching KEF levels 1–2 had an average MERSQI score of 11.2 and KEF Levels 3–4 averaged 13.5. illustrates the KEF level reached by the included studies and corresponding average MERSQI score. of the Appendix presents the MERSQI evaluation of each reviewed study inclusive of corresponding KEF level and colour coded bias assessment ranking. The limited use or reporting of conceptual frameworks and models underpinning the development of the educational programs reviewed is reflected in the generally low MERSQI validity scores and underpinning bias ratings; categories commonly underreported in medical education research [Citation71,Citation72]. However, it should be noted that ‘limited reporting’ increases the risk of bias but does not necessarily mean the educational development is of poor quality [Citation72].
Figure 3. Kirkpatrick evaluation level reached by the reviewed studies.
Discussion
This review identified 27 observational studies of moderate to low quality within the specified eligibility criteria despite an inclusive search strategy (obstetric or obstetric inclusive PoCUS training of professionals from any healthcare discipline globally). Description of theoretical concepts or pedagogy underpinning the training programs was generally lacking. Substantial heterogeneity in training formats was found between the reviewed studies, and half lacked follow-up support, training and assessment important to the safe ongoing practice of PoCUS (see - Training methods and delivery). The studies’ findings were generally positive, reporting improved knowledge (where pre- and post-assessments were conducted), and competence being attained despite the substantial variation in course durations (3 hours to several years). Variation in competence assessment and levels of evidence was also observed (see - Trainee and course evaluation), with 11 of the 27 studies not surpassing KEF levels 1 or 2 of assessing immediate reaction to training and knowledge gained. This is a recognised trend in which medical education researchers commonly cease evaluation at the lower KEF levels, finding longer-term investigations required for evaluations at levels 3 and 4 difficult to accomplish [Citation73,Citation74]. This is especially the case for PoCUS training research, which is predominantly conducted in low-resource settings where additional logistical challenges exist.
We identified several recently published systematic reviews of PoCUS training [Citation36,Citation75–77]. All were limited to the instruction of medical physicians, excluding allied health professionals. Other differences included setting, with Rajamani et al. [Citation36] (2020) and Andersen et al. [Citation76] (2019) focusing on general practice and critical care, respectively. Dickson et al. [Citation77] (2017) included studies of trainees with no prior formal ultrasound training only. Consistent with this review’s findings, variation in teaching curriculum and assessment methods, and an overall low standard of study quality were unanimously reported.
Rajamani et al. [Citation36] reported a distinct lack of high-quality evidence on PoCUS competence, with two-thirds of the reviewed studies failing to describe important details on how assessments were conducted, and very few utilising bias minimisation strategies important to observational study designs [Citation78]. None of the 42 studies identified by Rajamani et al. performed follow-up repeat assessments essential to assessing learning retention and safety to practice, and most of the educational programs reviewed failed to follow recommended processes for assessing PoCUS competence.
Anderson et al. [Citation76] found a ‘great variety of pedagogic approaches’ and substantial disparity in training durations (2 to 320 hours) between their included studies, which were reported to be of ‘low quality… mainly because of issues with design and reporting’. Assessment methods also varied but focussed PoCUS scans were found to require less training. Higher diagnostic accuracy and frequency of clinical use was also reported for obstetric indications, consistent with the findings in this review. These outcomes, and the need to use non-ionising imaging modalities in pregnancy highlight the utility of antenatal PoCUS in the hands of appropriately trained healthcare providers.
Strengths and limitations of this review
This literature review has relevance to medical educators, researchers, clinicians and policymakers interested in developing curriculum and translating PoCUS safely into clinical practice. To the authors’ knowledge, this is the first published systematic review of worldwide antenatal PoCUS training of multidisciplinary healthcare clinicians. It provides comprehensive tables summarising the reviewed studies PoCUS teaching and evaluation methods. The variable training and evaluation methods described and limitation in reporting made direct comparison of study results for metanalysis unfeasible, meaning effective and ineffective training approaches could not be confidently discerned. Whether one approach to training and assessment was superior to another could not be reliably established, limiting this review. Such heterogeneity, while complicating synthesis of evidence, can offer the advantage of examining the consistency of findings and generalizability of interventions across studies, assessing the relative feasibility of different educational approaches [Citation71,Citation79]. While the level of evidence and detail in reporting was lacking in many of the reviewed studies, a strength of this review is evident in the identification of this gap and the onus for more comprehensive and comparable longer-term studies with which to establish a stronger evidence base for antenatal PoCUS. An inclusive search string with broad criteria ensured a wide cross-section of antenatal PoCUS studies could be scrutinised. However, only two peer reviewed medical databases (MEDLINE and EMBASE) were searched and ‘English only’ articles included. Published studies from non-English-speaking countries in particular would have been missed.
Recommendations for future research and practice
Robust clinical studies demonstrating the efficacy of training models, and the clinical impact of trainee operated antenatal PoCUS on quality of care and maternal/fetal outcomes are needed. Economic analyses investigating the cost-effectiveness of PoCUS training and implementation would be valuable to justify and inform future programs.
The use of Telehealth with antenatal PoCUS for real-time scanning assistance (Teleultrasound) in the clinical setting also merits further investigation, along with Artificial Intelligence systems that have the capacity to assist minimally trained operators in unsupervised clinical environments [Citation80]. Advancements in Teleultrasound systems now make it possible for the remote clinician to view the ultrasound monitor, images and probe position, communicate via live video and text message with the operator, and even take control of the ultrasound machines functions and demonstrate findings to the patient, all in real time [Citation81]. Many of the reviewed studies are unlikely to have the resources and infrastructure (quality internet/broadband) to support such advanced systems, but they do offer considerable advantages for remote supervision of ultrasound trainees and may see greater utilisation in the future as the technology becomes more affordable.
Only one article in this review reported on the patient experience. Vinayak et al. [Citation63] surveyed patients, reporting they felt trainee performed PoCUS during antenatal care was safe, convenient & reassuring, providing a better antenatal visit experience, increased confidence in care delivery and increased spouse attendance. Considering the propensity for cultural minorities to live in remote communities and the underutilisation of Antenatal care services in these regions [Citation82–84], investigating the patient and partners’ perspective on trainee provided antenatal ultrasound would be beneficial to inform curriculum development with an aim to providing culturally sensitive patient centred care.
Conclusion
PoCUS is an increasingly utilised diagnostic tool that can enhance the physical exam and guide clinical decision making. It has particular utility in rural clinical practice and developing countries, where advancements in ultrasound equipment and telemedicine are opening new avenues for its establishment. Its lack of regulation and rapid expansion into most specialties underscores the need to establish standards in PoCUS training, competency and on-the-job scanning, ensuring providers are safe to practice. Acknowledging the urgent need for these skills and the difficulty accessing training in remote areas, regulation must be implemented carefully to preserve the time and financial advantages offered by PoCUS training.
Quality education programs require careful and informed consideration in their initial course design, with ongoing review of curricula, training resources, knowledge/practical skills assessment, availability of expert trainers, follow-up support and evaluation. Overall, findings from this review support intensive PoCUS training courses for task-shifting and upskilling of the medical and allied health workforce. However, significant heterogeneity in training, evaluation and research methodologies in the included studies was observed. Quality longitudinal studies with comparable evidence are needed to help frame policy guidelines and inform validated antenatal PoCUS training programs, ensuring the safe implementation of this valuable healthcare resource.
Authorship declaration
The authorship listing conforms with the journal’s authorship policy. All authors (AB, EB, NP) contributed to the review’s design, data acquisition, inclusion and interpretation, and were involved in drafting, revision and final approval of the manuscript.
Disclaimer
The views expressed in the submitted article are the authors own and not an official position of the institution or funder.
Systematic review registration
This systematic review is registered with the International Prospective Register of Systematic Reviews (PROSPERO registration number CRD42021230267).
Acknowledgments
We thank the Hospital Research Foundation for their funding and support of our research- The Healthy Newborn Project, which has provided outreach antenatal PoCUS training for remote Australian clinicians.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Data availability statement
Additional data referred to but not included in the main manuscript is provided in the Appendix as supplementary material (‘Additional Tables’).
Additional information
Funding
References
- Campbell S. A short history of sonography in obstetrics and gynaecology. Facts Views Vis Obgyn. 2013;5(3):213–36. PMCID: PMC3987368.
- Australian Department of Health. Clinical Practice Guidelines: pregnancy care 2019 edition. Canberra: Australian Government, 2019, viewed 11 May 2021. <https://www.health.gov.au/sites/default/files/pregnancy-care-guidelines_0.pdf>.
- Moore CL, and Copel JA. Point-of-care ultrasonography. N Engl J Med. 2011;364(8):749–757.
- Nicolson M. Imaging and imagining the fetus: the development of obstetric ultrasound. Baltimore: Johns Hopkins University Press; 2013.
- World Health Organization (WHO). WHO recommendations on antenatal care for a positive pregnancy. 2016, World Health Organization, Geneva, viewed 25 march 2020, <https://www.who.int/publications/i/item/9789241549912>.
- Whitworth M, Bricker L, and Mullan C. Ultrasound for fetal assessment in early pregnancy. Cochrane Database Syst Rev. 2015;7:1–57. DOI:10.1002/14651858.CD007058.pub3.
- Wiafe YA, Odoi AT, and Dassah ET. The Role of Obstetric Ultrasound in Reducing Maternal and Perinatal Mortality. In: Minin IV, and Minn OV, editors. Ultrasound imaging - medical applications. London: IntechOpen; 2011.
- Murugan VA, Murphy BOS, Dupuis C, et al. Role of ultrasound in the evaluation of first-trimester pregnancies in the acute setting. Ultrasonography. 2020;39(2):178–189. DOI:10.14366/usg.19043.
- Salomon LJ, Alfirevic Z, Da Silva Costa F, et al. ISUOG practice guidelines: ultrasound assessment of fetal biometry and growth. Ultrasound Obstetrics Gynecol. 2019;53(6):715. DOI:10.1002/uog.20272.
- Community Affairs References Committee Availability and accessibility of diagnostic imaging equipment around Australia. Australian Parliament House, Canberra, 2018, viewed 20 March 2020, <https://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Community_Affairs/Diagnosticimaging/Report>.
- Hofmeyr GJ, Haws RA, and Bergström S, et al. Obstetric care in low-resource settings: what, who, and how to overcome challenges to scale up? Int J Gynaecol Obstet. 2009;107 Suppl 1:S21-44, s44-25 DOI:10.1016/j.ijgo.2009.07.017.
- McClure EM, Nathan RO, Saleem S, et al. First look: a cluster-randomized trial of ultrasound to improve pregnancy outcomes in low income country settings. BMC Pregnancy Childbirth. 2014;14(1):73. DOI:10.1186/1471-2393-14-73.
- World Health Organization (WHO). Maternal mortality. 2020, World Health Organization, viewed 25 March 2021, <https://www.who.int/news-room/fact-sheets/detail/maternal-mortality>.
- World Health Organization (WHO). Newborns: improving survival and well-being. 2020, World Health Organization, viewed 9 April 2021, <https://www.who.int/news-room/fact-sheets/detail/newborns-reducing-mortality>.
- Shah S, Bellows BA, Adedipe AA, et al. Perceived barriers in the use of ultrasound in developing countries. Crit Ultrasound J. 2015;7(1):1–5. DOI:10.1186/s13089-015-0028-2.
- Australasian Society of Ultrasound in Medicine (ASUM). Minimum education & training requirements for ultrasound practitioners. Australas J Ultrasound Med. 2017;20(3):132–135. DOI:10.1002/ajum.12061.
- Fentress M, Heyne TF, Barron KR, et al. Point-of-care ultrasound in resource-limited settings: common applications. South Med J. 2018;111(7):424–433. DOI:10.14423/SMJ.0000000000000827.
- Australasian Society of Ultrasound in Medicine (ASUM). Discussion paper: definition of Point of Care Ultrasound (POCUS), The Australasian Society of Ultrasound in Medicine (ASUM), viewed 28 March 2020, <https://www.asum.com.au/files/public/RealTime/2017/ASUM-Discussion-Paper-Definition-of-POCUS.PDF>.
- Australian Sonographer Accreditation Registry (ASAR). Sonographer accreditation. 2020, Australian Sonographer Accreditation Registry (ASAR), viewed 25 March 2020, <https://www.asar.com.au/sonographer-info/accredited-medical-sonographer/>.
- Australasian Sonographers Association (ASAR). The Australasian sonographers association 2019–20 Australian government pre-budget submission. 2020, The Australasian Sonographers Association, Melbourne, viewed 24 March 2020, <https://www.sonographers.org/news-item/1997/the-asa-2019-20-australian-government-pre-budget-submission>.
- Wanjiku GW, Bell G, and Wachira B. Assessing a novel point-of-care ultrasound training program for rural healthcare providers in Kenya. BMC Health Serv Res. 2018;18(1):607. <https://doi.org/10.1186/s12913-018-3196-5>.
- Bell G, Wachira B, and Denning G. A pilot training program for point-of-care ultrasound in Kenya. Afr J Emergency Med. 2016;6(3):132–137. DOI:10.1016/j.afjem.2016.03.002.
- Vinayak S, and Brownie S. Collaborative task-sharing to enhance the Point-Of-Care Ultrasound (POCUS) access among expectant women in Kenya: the role of midwife sonographers. J Interprof Care. 2018;32(5):641–644. <https://dx.doi.org/10.1080/13561820.2018.1470499>.
- Rominger AH, Gomez GAA, Elliott P. The implementation of a longitudinal POCUS curriculum for physicians working at rural outpatient clinics in Chiapas, Mexico. Crit Ultrasound J. 2018;10(1). DOI:10.1186/s13089-018-0101-8
- Dalmacion GV, Reyles RT, Habana AE, et al. Handheld ultrasound to avert maternal and neonatal deaths in 2 regions of the Philippines: an iBuntis intervention study. BMC Pregnancy Childbirth. 2018;18(1):32. DOI:10.1186/s12884-018-1658-8.
- Kolbe N, Killu K, Coba V, et al. Point of care ultrasound (POCUS) telemedicine project in rural Nicaragua and its impact on patient management. J Ultrasound. 2015;18(2):179–185. DOI:10.1007/s40477-014-0126-1.
- Smith A, Parsons M, Renouf T, et al. A mixed-methods evaluation of a multidisciplinary point of care ultrasound program. Med Teach. 2019;41(2):223–228. DOI:10.1080/0142159X.2018.1461202.
- Hoppmann RA, Rao VV, Poston MB, et al. An integrated ultrasound curriculum (iUSC) for medical students: 4-year experience. Crit Ultrasound J. 2011;3(1):1–12. DOI:10.1007/s13089-011-0052-9.
- Shaw-Battista J, Young-Lin N, Bearman S, et al. Interprofessional obstetric ultrasound education: successful development of online learning modules; case-based seminars; and skills labs for registered and advanced practice nurses, midwives, physicians, and trainees. J Midwifery Women’s Health. 2015;60(6):727–734. DOI:10.1111/jmwh.12395.
- Stolz LA, Muruganandan KM, Bisanzo MC, et al. Point-of-care ultrasound education for non-physician clinicians in a resource-limited emergency department. Trop Med Int Health. 2015;20(8):1067–1072. DOI:10.1111/tmi.12511.
- Australasian Society of Ultrasound in Medicine (ASUM). A guide to providing an ultrasound workshop. 2016, Australasian Society of Ultrasound in Medicine (ASUM), viewed 11 May 2021, <https://www.google.com/search?client=firefox-b-d&q=A+Guide+to+Providing+an+Ultrasound+Workshop#>.
- Australasian Society of Ultrasound in Medicine (ASUM). Guidelines for course providers seeking CCPU course accreditation. 2015, Australasian Society of Ultrasound in Medicine (ASUM), viewed 6 May 2021, <https://www.google.com/search?client=firefox-b-d&q=Guidelines+for+Course+Providers+Seeking+CCPU+Course+Accreditation>.
- Kumar A, Kugler J, and Jensen T. Evaluation of trainee competency with Point-of-Care Ultrasonography (POCUS): a conceptual framework and review of existing assessments. J Gen Intern Med. 2019;34(6):1025–1031. DOI:10.1007/s11606-019-04945-4.
- Leonardi M, Murji A, and D’Souza R. Ultrasound curricula in obstetrics and gynecology training programs. Ultrasound Obstet Gynecol. 2018;52(2):147–150. DOI:10.1002/uog.18978.
- Wright J, Noriega O, and Ho H. The application of hand-held ultrasound scanner in teaching of telemedicine and rural medicine. 2014;8(1):87–91. DOI:10.5005/jp-journals-10009-1340.
- Rajamani A, Shetty K, Parmar J, et al. Longitudinal Competence Programs for Basic Point-of-Care Ultrasound in Critical Care: a Systematic Review. Chest. 2020;158(3):1079–1089. DOI:10.1016/j.chest.2020.03.071.
- Collins K, Collins C, and Kothari A. Point-of-care ultrasound in obstetrics. 2019;22(1):32–39. DOI:10.1002/ajum.12133.
- Torloni MR, Vedmedovska N, Merialdi M, et al. Safety of ultrasonography in pregnancy: WHO systematic review of the literature and meta-analysis. Ultrasound Obstetrics Gynecol. 2009;33(5):599–608. DOI:10.1002/uog.6328.
- Abramowicz JS. Benefits and risks of ultrasound in pregnancy. Semin Perinatol. 2013;37(5):295–300. <https://doi.org/10.1053/j.semperi.2013.06.004>.
- Miller DL. Safety assurance in obstetrical ultrasound. Semin Ultrasound CT MR. 2008;29(2):156–164. DOI:10.1053/j.sult.2007.12.003.
- Reed DA, Cook DA, and Beckman TJ, et al. Association between funding and quality of published medical education research. JAMA. 2007;298(9):1002–1009. DOI:10.1001/jama.298.9.1002.
- Reed DA, Beckman TJ, Wright SM, et al. Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to jgim’s medical education special issue. J Gen Intern Med. 2008;23(7):903–907. DOI:10.1007/s11606-008-0664-3.
- Kirkpatrick DL. Techniques for evaluating training programes. 1979;33:78. ISSN 0041-0861.
- Kirkpatrick DL. Evaluating training programs the four levels/Donald L. Kirkpatrick, James D. Kirkpatrick. San Francisco CA: Berrett-Koehler; 2006.
- Shah S, Santos N, Kisa R, et al. Efficacy of an ultrasound training program for nurse midwives to assess high-risk conditions at labor triage in rural Uganda. PLoS ONE. 2020;15(6):e0235269. DOI:10.1371/journal.pone.0235269.
- Shah S, Noble VE, Umulisa I, et al. Development of an ultrasound training curriculum in a limited resource international setting: successes and challenges of ultrasound training in rural Rwanda. Int J Emerg Med. 2008;1(3):193–196. DOI:10.1007/s12245-008-0053-z.
- Shah SP, Epino H, Bukhman G, et al. Impact of the introduction of ultrasound services in a limited resource setting: rural Rwanda 2008.(Research article)(Report). BMC Int Health Hum Rights. 2009;9(1):4. DOI:10.1186/1472-698X-9-4.
- Shah S, Adedipe A, Ruffatto B, et al. BE-SAFE: bedside sonography for assessment of the fetus in emergencies: educational intervention for late-pregnancy obstetric ultrasound. West J Emerg Med. 2014;15(6):636–640. DOI:10.5811/westjem.2014.7.18480.
- Nathan RO, Swanson JO, and Swanson DL, et al. Evaluation of focused obstetric ultrasound examinations by health care personnel in the democratic Republic Of Congo, Guatemala, Kenya, Pakistan, and Zambia. Curr Probl Diagn Radiol. 2017;46(3):210–215. DOI:10.1067/j.cpradiol.2016.11.001.
- Westerway SC. Comparing the effectiveness of training course formats for point-of-care ultrasound in the third trimester of pregnancy. Australas J Ultrasound Med. 2019;221:45–50. DOI:10.1002/ajum.12125.
- Shokoohi H, Raymond A, Fleming K, et al. Assessment of point-of-care ultrasound training for clinical educators in Malawi, Tanzania and Uganda. Ultrasound Med Biol. 2019;45(6):1351–1357. DOI:10.1016/j.ultrasmedbio.2019.01.019.
- Lindgaard K, and Riisgaard L. Validation of ultrasound examinations performed by general practitioners. Scand J Prim Health Care. 2017;35(3):256–261. <https://dx.doi.org/10.1080/02813432.2017.1358437>.
- Henwood PC, Mackenzie DC, Liteplo AS, et al. Point-of-care ultrasound use, accuracy, and impact on clinical decision making in Rwanda Hospitals. J Ultrasound Med. 2017;36(6):1189–1194. DOI:10.7863/ultra.16.05073.
- Dornhofer K, Farhat A, and Guan K, et al. Evaluation of a point-of-care ultrasound curriculum taught by medical students for physicians, nurses, and midwives in rural Indonesia. J Clin Ultrasound. 2020;48(3):145–151. DOI:10.1002/jcu.22809.
- Lee JB, Tse C, Keown T, et al. Evaluation of a point of care ultrasound curriculum for Indonesian physicians taught by first-year medical students. World J Emerg Med. 2017;8(4):281–286. DOI:10.5847/wjem.j.1920-8642.2017.04.006.
- Kotagal M, Quiroga E, Ruffatto BJ, et al. Impact of point-of-care ultrasound training on surgical residents’ confidence. J Surg Educ. 2015;72(4):e82–e87. DOI:10.1016/j.jsurg.2015.01.021.
- World Health Organization (WHO). WHO recommendations on antenatal care for a positive pregnancy experience: ultrasound examination: highlights and key messages from the World Health Organization’s 2016 global recommendations. 2018, World Health Organization, viewed 20 April 2021, <https://apps.who.int/iris/bitstream/handle/10665/259946/WHO-RHR-18.01-eng.pdf;jsessionid=DDAF84D516491D79FAD4E8AC33222AFD?sequence=1>.
- Lathrop A, and Blackburn M. Learner portfolios and hands-on workshop to facilitate and evaluate nurses’ learning in obstetric ultrasound. J Obstet Gynecol Neonatal Nurs. 2011;40(5):654–661. DOI:10.1111/j.1552-6909.2011.01287.x.
- Engum SA, and Jeffries PR. Interdisciplinary collisions: bringing healthcare professionals together. Collegian. 2012;19(3):145–151. <https://doi.org/10.1016/j.colegn.2012.05.005>.
- American College of Emergency Physicians (ACEP). Ultrasound guidelines: emergency, point-of-care and clinical ultrasound guidelines in medicine. Ann Emerg Med. 2016;69:e27–e54. DOI:10.1016/j.annemergmed.2016.08.457.
- Baltarowich OH, Goldberg BB, and Wilkes AN, et al. Effectiveness of ‘teaching the teachers‘ initiative for ultrasound training in Africa. Acad Radiol. 2009;16(6):758–762. DOI:10.1016/j.acra.2008.12.023.
- Kimberly HHM A:, Mennicke M:, and Liteplo A:, et al. Focused maternal ultrasound by midwives in rural Zambia. Ultrasound Med Biol. 2010;36(8):1267–1272. <https://dx.doi.org/10.1016/j.ultrasmedbio.2010.05.017>.
- Vinayak S, Sande J, Nisenbaum H, et al. Training midwives to perform basic obstetric point-of-care ultrasound in rural areas using a tablet platform and mobile phone transmission technology-A WFUMB COE project. Ultrasound Med Biol. 2017;43(10):2125–2132. DOI:10.1016/j.ultrasmedbio.2017.05.024.
- Chalouhi GEB, Ville Y, and Ville Y. Ultrasound simulators in obstetrics and gynecology: state of the art. Ultrasound Obstetrics Gynecol. 2015;46(3):255–263. DOI:10.1002/uog.14707.
- Tolsgaard MG. Assessment and learning of ultrasound skills in obstetrics & gynecology. Dan Med J. 2018;65(2, B5445 29393042).
- Adler D, Mgalula K, Price D, et al. Introduction of a portable ultrasound unit into the health services of the Lugufu refugee camp, Kigoma District, Tanzania. Int J Emerg Med. 2008;1(4):261–266. DOI:10.1007/s12245-008-0074-7.
- Mandavia DP, Aragona J, Chan L, et al. Ultrasound training for emergency physicians–a prospective study. Acad Emerg Med. 2000;7(9):1008–1014. DOI:10.1111/j.1553-2712.2000.tb02092.x.
- Willis-Shattuck M, Bidwell P, Thomas S, et al. Motivation and retention of health workers in developing countries: a systematic review. BMC Health Serv Res. 2008;8(1):247. DOI:10.1186/1472-6963-8-247.
- Siriwardhana C. Promotion and reporting of research from resource-limited settings. Infect Dis (Auckl). 2015;8:25–29. DOI:10.4137/IDRT.S16195.
- Vyas A, Moran K, Livingston J, et al. Feasibility study of minimally trained medical students using the Rural Obstetrical Ultrasound Triage Exam (ROUTE) in rural Panama. World J Emerg Med. 2018;9(3):216–222. DOI:10.5847/wjem.j.1920-8642.2018.03.009.
- Reed D, Price EG, Windish DM, et al. Challenges in systematic reviews of educational intervention studies. Ann Intern Med. 2005;142(12_Part_2):1080–1089. DOI:10.7326/0003-4819-142-12_Part_2-200506211-00008.
- Gordon M, Patricio M, Horne L, et al. Developments in medical education in response to the COVID-19 pandemic: a rapid BEME systematic review: BEME Guide No. 63. Med Teach. 2020;42(11):1202–1215. DOI:10.1080/0142159X.2020.1807484.
- Yardley S, and Dornan T. Kirkpatrick’s levels and education evidence. Med Educ. 2012;46(1):97–106. DOI:10.1111/j.1365-2923.2011.04076.x.
- Moreau KA. Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox? Med Teach. 2017;39(9):999–1001. DOI:10.1080/0142159X.2017.1337874.
- Moses A, Weng W, and Orchanian-Cheff A, et al. Tteaching point-of-care ultrasound in medicine: a scoping review. 2020;15(2):13–29. DOI:10.22374/cjgim.v15i2.368.
- Andersen CA, Holden S, and Vela J, et al. Point-of-care ultrasound in general practice: a systematic review. Ann Fam Med. 2019;17(1):61–69. DOI:10.1370/afm.2330.
- Dickson R, Duncanson K, and Shepherd S. The path to ultrasound proficiency: a systematic review of ultrasound education and training programmes for junior medical practitioners. Australas J Ultrasound Med. 2017;20(1):5–17. DOI:10.1002/ajum.12039.
- Labrecque JA, and Kaufman JS. Commentary: can a Quasi-experimental Design Be a Better Idea than an Experimental One? Epidemiology. 2016;27(4):500–502. DOI:10.1097/EDE.0000000000000485.
- Mulrow C, Langhorne P, and Grimshaw J. Integrating heterogeneous pieces of evidence in systematic reviews. Ann Intern Med. 1997;127(11):989–995. DOI:10.7326/0003-4819-127-11-199712010-00008.
- Abramowicz JS. Obstetric ultrasound: where are we and where are we going? Ultrasonography. 2021;40(1):57–74. DOI:10.14366/usg.20088.
- Loria K. Remote access: How ultrasound in telemedicine is changing education, training, and patient care. Radiology today magazine. 2021, viewed 6 May 2021, <https://www.radiologytoday.net/archive/rt0818p24.shtml>.
- Rurangirwa AA, Mogren I, Nyirazinyoye L, et al. Determinants of poor utilization of antenatal care services among recently delivered women in Rwanda: a population based study. BMC Pregnancy Childbirth. 2017;17(1):142. DOI:10.1186/s12884-017-1328-2.
- Jacobs C, Michelo C, and Moshabela M. Why do rural women in the most remote and poorest areas of Zambia predominantly attend only one antenatal care visit with a skilled provider? A qualitative inquiry. BMC Health Serv Res. 2018;18(1):409. DOI:10.1186/s12913-018-3212-9.
- Rumbold AR, Bailie RS, Si D, et al. Delivery of maternal health care in Indigenous primary care services: baseline data for an ongoing quality improvement initiative. BMC Pregnancy Childbirth. 2011;11(1):16. DOI:10.1186/1471-2393-11-16.
- Swanson JO, Kawooya MG, Swanson DL, et al. The diagnostic impact of limited, screening obstetric ultrasound when performed by midwives in rural Uganda. J Perinatol. 2014;34(7):508–512. DOI:10.1038/jp.2014.54.
APPENDIX
Table A1. Search strategy
Table A2. Modified Medical Education Research Study Quality Instrument (MERSQI) [Citation41,Citation42]
Table A3. Risk of bias assessment tool [Citation72].
Table A4. Training methods and delivery
Table A5. Quality assessment and bias ranking of included articles using a modified Medical Education Research Study Quality Instrument (MERSQI) [Citation41,Citation42]