Abstract
This article describes the development of educational material by the European CanCer Organisation. The ‘oncovideos project’ developed an e-learning tool to improve the skills of oncologists throughout Europe. The development process is reviewed from the planning phase of the project to the production of the e-learning tool. The strengths, weaknesses, opportunities and threats that were encountered are discussed. The final product can be consulted at www.ecco-org.eu/oncovideos. The project was supported by the Lifelong Learning Programme of the European Union.
Introduction
Continuing medical education (CME) is an important element in oncology and aims to ensure optimal diagnosis, patient care and treatment. Oncology experiences an important increase in knowledge as the result of innovative research leading to new treatments and treatment techniques. To implement these changes in clinical practice, several formats have been used such as teaching sessions, conferences, guidelines and other means.
In Europe, doctors aiming to specialise in oncology after their postgraduate training need to acquire several skills before being qualified as oncology specialists in one of the oncology disciplines. Postgraduate training should be part of a CME programme that takes place during their daily clinical activity in a training institute. However, health professionals in Europe sometimes lack opportunities to participate in clinical teaching sessions that are essential to be qualified to treat cancer patients. Therefore, oncology specialists in-training may lack the opportunity to acquire some ‘know how’ and ‘best practices’ during their education. New formats of teaching by experienced expert oncologists to improve knowledge and skills that are approved and endorsed by European Oncology Societies are thus necessary.
In this article, we report on the experience of developing a learning programme to improve skills in oncology by videos on specific issues encountered in oncology. The project was supported by a European Union (EU) grant ‘Leonardo da Vinci multilateral project (development of innovation)’ and resulted in the production of a library of 25 videos of procedures and techniques in oncology.
Aim of the project
The aim of the project was to design and develop an educational tool to assist in the delivery of CME to health professionals who need to acquire certain skills and competencies to qualify as oncology specialists. The tool was conceived as an on-line library of 25 videos with practical oncology sessions showing standard clinical techniques and topics related to the various oncology disciplines. The videos had to be accompanied by practical comments made by experts in the specific subject area of each session and the ability to evaluate the acquired knowledge of the trainee to gain CME credits. The topics were determined by different oncology societies, based on a needs assessment.
Realisation of the project
Initial phase: From idea to project proposal
The project was initiated in June 2008 and finalised in March 2012 (). It was approved by the European CanCer Organisation (ECCO) Board of Directors. The total cost of the project was 396,000 Euros, which was covered by the European Commission. Approximately, 150,000 Euros was spent on IT and audio-visual subcontracting. The other part of the grant was used to cover the fees of the experts in the partner institutions and the management fees of the Coordinator.
Table 1. Timeline of the project.
The ECCO Education Committee was given the task to develop an e-learning programme for ECCO, and the video project was a part of this initiative. Discussions within the ECCO Education Committee on the project included the title of the project, the partnership and how to select the topics of the videos, the interactivity, the editorial control and the output of the project.
Participation was solicited among the member societies of ECCO namely the European Society for Medical Oncology (ESMO), the European Society for Surgical Oncology (ESSO), the European Society for Radiotherapy and Oncology (ESTRO), the European Society of Paediatric Oncology Europe (SIOPE) and the European Oncology Nursing Society (EONS). The aim of the project was explained in a letter, and they were asked to participate with their education committees defining relevant topics for the videos in their oncology disciplines, to provide names of experts in these topics together with the names of cancer institutes; to act as an editorial board for the videos; and to collaborate in the dissemination and validation of the project among oncology specialists in their discipline. Other member societies of ECCO could join if they were interested in the project.
From the start, it was intended to involve cancer institutes/centres to underline that the project was aiming at individuals already at work rather than postgraduate students. These institutions could be the institutes/centres of the members of the ECCO Education Committee or of the members of the other societies’ Education Committees.
The European Association of Neuro-Oncology (EANO), ESMO, ESSO, ESTRO and SIOPE were willing to participate in the project, and they submitted topics, partners and partner institutions (). The suggested partners and partner institutions were contacted, and the project was explained and their participation solicited.
Table 2. Partner institutions and topics produced on suggestion of the cancer societies.
The project could only be realised if the necessary funding was available. Therefore, application for an EU grant was suggested by the members of the ECCO office. As official partners, the partner institutes would receive part of the funding for the work of the health professionals/experts involved in the project. Because of the intention to submit the project for funding to the EU, at least three organisations had to be located in the eligible countries – EU Member States and EFTA countries + candidate (Turkey). The project included 18 partner cancer institutions in nine EU countries. Each partner institution submitted a budget plan for the production of the ‘oncovideos’ based on local costs and requirements.
The proposal was submitted to the EU in February 2009 and was selected for funding in September 2009 to start in January 2010 for a 2-year period.
Development phase
Contracting partners and partner institutions. The different partners and partner institutions were contacted for the administrative procedures in relation to the EU contract. Each partner institution had to sign a contract stipulating the costs of individual items (e.g. staff, subcontractors) before starting any project procedure. A timesheet was provided in order to claim financial support for the time each partner contributed to the project.
Definition of the topics of the videos. The participating organisations (EANO, ESMO, ESSO, ESTRO, SIOPE) were asked to suggest topics for the videos. To this aim, the education committee of each organisation was asked to prioritise learning objectives based on their oncology curricula. Many European Professional Oncology Societies (e.g. ESSO, ESMO, ESTRO, SIOPE, EONS) have developed a European curriculum specifying the requirements for health professionals to be qualified as oncology specialists. These curricula outline a number of practical skills and competencies that should be gained through practical sessions and training in a teaching institution/department. The proposed videos should represent ‘standard’ procedures included in a specific curriculum so that a large number of oncologists in-training throughout Europe would benefit from the direct access to practical on-line sessions to acquire additional knowledge and skills to qualify as oncology specialists. The list of topics is given in .
Definition of the e-learning product. The e-learning product should be a library of ‘oncovideos’ performing and explaining a particular procedure in the various oncology disciplines, namely, surgical oncology, radiation oncology, medical oncology, paediatric oncology and neuro-oncology. The video should last at maximum 30 minutes and should be accredited by the Accreditation Council of Oncology in Europe (ACOE) to enable users to collect CME credits and encourage continued development in oncology specialisation. It would have to provide a user-friendly interface freely available to healthcare professionals upon completion of a simple registration. The target users are young oncologists in the various oncology disciplines.
Definition of the development process. From the start it was intended to involve a peer-review process of the proposed content of the ‘oncovideos’ to ensure the highest possible quality. To this aim, the partners were asked to submit their ‘oncovideos’ proposal before the actual production of the material.
According to specific guidelines, the partners had to provide the objectives and learning aims of the videos, a script explaining all the steps to be involved in the slide presentation and the video in texts and graphics, points of attention in relation to problems and difficulties as well as solutions, conclusion of the video, a summary that would be translated in six different languages (FR, NL, SI, PL, IT, DE) by a professional subcontractor, biographies of the authors and contributors to the video, information on the partner institutions, acknowledgements, and five multiple choice questions with 4 bullet-points with one correct or one wrong answer for CME purposes. The partners were also asked to provide 10 keywords for the search engine.
Information on how to write the scripts was developed by the ECCO office and distributed to the partners. The actual production of the material started in June 2010. The partners were asked to submit their scripts and slide presentation by early September 2010.
Definition of the format. The format of the ‘oncovideos’ was a slide presentation with the aim and objectives of the video, a description of the procedure shown and a video of the procedure incorporated in the slide presentation.
The slide show was added with a voice-over, so the script had to include the text of each slide. The partners were asked to provide a professional voice-over.
An example of the script and how the final product had to look were produced by the ECCO office and distributed among the partners.
Web-based provider. From the start, an IT company was involved to ensure that the ‘oncovideos’ would be integrated in the ECCO website. The deliverables and functionalities discussed were the introductory page, the procedure for user registration that should be as simple as possible, a waiver in relation to the use of the videos, the possibility to stop/pause the slide presentation and to restart after re-entering at a specific time point in the slide presentation, the availability of a search engine, a CME questionnaire and accreditation process for participants, a registration system of users to extract user statistics, the home page interface, the template of the slide presentation, and the technical instructions for the participating institutions for the deliverables (dedicated website to upload materials, slides, video qualifications).
Dissemination plan. A preliminary draft dissemination plan was developed and circulated to the participants with a list of specific promotional tasks that ECCO could perform. Partners were asked to provide a list of tasks which could be done by their institution to promote the ‘oncovideos’ project as a whole and not just the videos they are producing. Several ideas were provided by the partners such as contacting national societies and physicians specialising in oncology and internal medicine, promotion by the ECCO member societies, promotion in scientific journals, promotional activities during meetings and congresses and the production of a USB stick with the oncovideo ‘how to deliver bad news to patients’ with a link to the ‘Oncovideos’ website that could be distributed at the European Multidisciplinary Cancer Congress in Stockholm in September 2011.
Production phase At the start of the production phase, the cancer societies were informed about the process and their involvement in the peer review board. Peer review was performed by the education committees of the different societies and comments were provided to the partners for adaption of the script and slide presentation.
After the peer review of the script and slide presentation, the videos could be produced according to predefined production criteria. It was agreed that for each video showing patients, an informed consent had to be obtained from the patients or from the legal representatives in case of children. In addition, in case of children, faces were made not recognisable.
A lay-out of the homepage of the ‘Oncovideos’ was produced and was discussed with the partners. To this aim, a link with the draft of the homepage was sent to the partners and some minor corrections were made and links to the ECCO Cancer Societies were integrated.
When the videos and the slide presentations were produced, they were uploaded to a dedicated web space and adapted according to the format of the template by the provider. The partners had to review and give their consent to the final version of their ‘oncovideo’ before it was placed on the ECCO website.
After the revision by the partners, the slide presentations were activated ‘on-line’ as soon as possible and in April 2011, the first four videos were accessible to the professional community. A rating system was to be included inviting users to rate the videos by a simple rating system.
It was decided to post the videos during a period of 2 years and that there should be a revision by the partners at this time or earlier if there was an important change in information relating to the content of the videos.
The number of individual viewers per oncovideo from February until May 2012 is shown in . The CME accreditation test was taken by 606 users until mid-May 2012.
Table 3. Viewers of the oncovideos per video in the first months of 2012.
Evaluation of the project
Strengths
The strengths of the ‘oncovideos’ project were the endorsement by the ECCO board, the participation of the different cancer societies with their network of partners and partner institutions, the dedicated persons involved at the ECCO office, the financial support by the EU and the quality of the product.
The ECCO board decided to invest in an e-learning programme at the time where most oncology societies had no such programme in place. Because there was a real need at that time, the motivation to produce these ‘oncovideos’ was high. The project was endorsed by several ECCO members, and they were willing to participate by involvement of their education committees to provide topics, serve in peer review of the scripts and to use their network for the dissemination of the ‘oncovideos’. The partners and partner institutions brought their expertise in the different oncology fields and their infrastructure and video production facilities. They also provided meaningful information in relation to the dissemination of the videos.
There was a dedicated person at the ECCO office (F. van Hemelryck) for the ‘oncovideos’ project who was responsible for support during the development of the project, the financial negotiations with the funding organisation, the IT provider, the partners and partner institutions, the practical organisation of the many phone conferences and face-to-face meeting of the partners. She was also responsible for the production of the final EU report.
The project was able to proceed because of the funding from the EU. Such financial support had previously been available for other ECCO projects, and this experience facilitated the funding process.
The peer review process by the different cancer societies ensured that the quality of the final product was high and had an educational value.
The IT-based submission of the material and the possibility to review and adapt the material by the partners had an added value.
The contact among the partners was ensured by regular phone conferences taking place every 4–6 weeks enabling partners to raise their issues and exchange experiences between themselves. These regular teleconferences very much helped in giving a sense of partnership and tracking progress. In addition, the possibility for the partners to get personal information and help by the ECCO office created an added value for all the participants.
Weaknesses
There were many partners and partner institutions (19) each with his/her/its own financial and legal departments, and specific adaptions were necessary to individual contracts among the different legal entities.
The beginning of the project was initially scheduled for January 2010 but because some changes were made on the budget breakdown (moving budget from staff to subcontracting), approval was needed by the funding agency, which was received in May 2010. This was caused by the fact that not all participating institutions had video facilities and had to hire a subcontractor, which was not in their original budget proposal.
Two partners cancelled their participation: one partner because his institution was no longer performing the medical procedure of the proposed video, and the other because they had no staff to replace the expert going on maternity leave. A third partner asked for a change of institutional name keeping the same staff involved in the project.
Since there were many partners and partner institutions, it was difficult to get everybody's presence at all meetings and this lead to some misunderstanding on the format (slide presentation with video versus video with slides) although minutes and examples were distributed to all partners.
Although all partners and partner institutions signed a contract, one partner did not deliver a final product and had to be replaced.
The budget was divided into different parts, and the partners and partner institution had to pre-finance the costs.
Some of the IT appliances were not working correctly all the time and this lead to some difficulties in evaluating the product.
Opportunities
The project showed that it is possible within a set time frame to produce this kind of educational material according to a peer-reviewed process. The experience of the partners and partner institutions makes it possible to produce other ‘oncovideos’ based on the template and experience gained during the project. Other societies can benefit from this experience to produce their own educational e-learning material.
Threats
The project had a large number of partners and partner institutions, and this made the overview a challenge and the possibility of drop-outs a real threat. In addition, the financial support by the EU was based on time sheets, and sometimes, it was difficult to obtain these time sheets in time to ensure the necessary funding.
The timeline had to be changed due to contractual issues. Maintaining a strict timeline is difficult if many partners are involved since it is difficult to ensure that deadlines are kept.
User evaluation
Evaluation of the use of the videos is performed through different approaches. A statistical tool was embedded in the programme enabling the website administrator to track numbers of users watching the videos. The statistics show a steady progression over time ().
In addition, users from the different oncology disciplines were asked to complete a form inviting them to score on a scale between 1 and 4 (1 = poor/requires improvement; 2 = average; 3 = good, 4 = excellent) the different aspects of the e-learning tool such as the educational value as well as the user friendliness of the website, the registration and CME test functionalities. This evaluation resulted in an average score of 3.25 meaning that users consider this as a good e-learning tool.
CME accreditation
All videos were accredited by the European Accreditation Council of Oncology in Europe (ACOE) enabling users to collect CME credits. As part of this process, authors of the videos provided five multiple choice questions with four bullet-points with one correct or one wrong answer. Users taking the CME test are required to obtain 80% of correct answers in order to pass the test. Upon completion of an evaluation form, they are able to download their CME certificate with 1 CME credit per video.
Conclusion
This ‘oncovideos’ project shows that it is possible to produce high-quality peer-reviewed CME material within a set time frame. The videos are available at the ECCO website (http://www.ecco-org.eu/oncovideos/) and are freely accessible for healthcare professionals.
Declaration of interest
Funding
The oncovideos project was supported by a grant of the Lifelong Learning Programme of the European Union.
Author(s) Financial Disclosure
Declaration of financial/other relationships: no financial support for the authors except the grant of the Lifelong Learning Programme of the European Union.
Peer Reviewers Financial Disclosure
Both Reviewers have disclosed that they have no relevant financial relationships.