ABSTRACT
As an education charity, it is vital for ecancer to understand how effective its educational videos are as educational tools. This includes understanding what effect the funding of the production of the video has, whether it was funded through an unrestricted educational grant from industry or it was funded by ecancer itself through its charitable funds. In this article, we have looked at four viewer engagement and satisfaction ratings in order to examine any differences. One hundred and twenty-three videos were examined in total recorded at six separate conferences over two years.
KEYWORDS:
Introduction
Major conferences are seen as a preferred source of medical education; however, the lack of time, financial support or travel difficulties can impact clinicians’ ability to attend these international meetings. Digital education from Independent Medical Education (IME) providers is increasingly important as a resource for clinicians and other healthcare professionals who are unable to attend these meetings. ecancer is a UK charity with a global mission to raise the standards of care for cancer patients across the world through education. As such, ecancer attends these major conferences to produce educational interviews with the leading experts who are presenting data, and it is vital to be able to demonstrate the impact of these videos through the effective measuring of educational outcomes.
ecancer attends around 30–40 conferences to produce an average of 947 videos per year. These videos receive over 940,00 views with 16.2% produced with support from industry funding through unrestricted educational grants. All other videos are funded by the charity itself and there is currently a total of 8,353 available to view in English and Spanish [Citation1]. All industry-supported videos cover disease areas agreed with the funder involving leading experts discussing key developments and are independent, peer-reviewed programmes.
The educational videos produced by ecancer are in three formats.
To Camera Interview
One expert talks directly to the camera and is interviewed by an oncology journalist.
Expert to Expert Interview.
Two experts interview each other on a specific topic.
Expert Panel Discussion
A panel of four experts interview one another to discuss the latest developments in a particular area.
It is vital to ecancer as an organisation that our content is seen to be trustworthy, reliable and as independent from any bias related to funding as possible. We therefore used the outcome measures described in this article to assess if there were differences in the measured outcomes between the different video formats and industry-funded vs. non-industry-funded content.
The videos contain interviews with leading experts discussing data and developments in the treatment of cancer. They discuss their interpretation of conference presentations and any other key developments with a focus on what this means for the latest understanding of best clinical practice and how oncologists and other healthcare professionals should be reacting to these data when they next treat their patients. The focus is on practical guidance for healthcare professionals.
Measures of Engagement and Educational Outcomes
In order to measure the viewer engagement and educational outcomes for these videos, ecancer uses a number of different metrics. Each of these metrics gives a different measure of viewer engagement and is used to determine the educational effectiveness of each video.
The key measures that are the focus of this research are:
Total number of views
The total number of views is straightforward to measure with a counter that is activated by a viewer pressing play on the video
Percentage of video watched
The percentage of the video watched is measured by dividing the average time that each video was viewed by the length of that video (See Appendix A)
Net promoter score[Citation2] (NPS) (a measure of how likely viewers are to recommend a video to a colleague – see Appendix B)
The NPS is measured using the viewer rating which is a five-star system in response to the statement “I would recommend this video to my colleagues”. The NPS is the percentage of promoters (viewers who rate the video 5 stars) minus the percentage of detractors (viewers who rate the video 1, 2 or 3 stars)
Likely impact on clinical practice as rated by viewers (Clinical practice measure – see Appendix C)
The likely impact on clinical practice is measured using the viewer rating in response to the statement “The information in this video is likely to improve my clinical practice or research”
We analysed data from six congresses as part of this analysis:
2019 ESMO Congress
2019 ASCO Annual Meeting
2019 Genitourinary Cancer Symposium (ASCO GU)
2018 ESMO Congress
2018 ASCO Annual Meeting
2018 Genitourinary Cancer Symposium (ASCO GU)
Looking at the outcome measures in videos covering lung and prostate cancer.
10 Expert to Expert Interviews
9 Expert Panel Discussions
104 To Camera Interviews
73 funded by ecancer
31 industry supported by three pharmaceutical companies (including Janssen) through unrestricted educational grants.
Results
The statistics cover a period from publication, which varies for each video depending on the date of the conference, to 4 September 2020. This represents a total of 229,582 views in total with 20,328 viewer ratings. The mean value for each video format is displayed below.
Views
Views per video are very similar for the To camera format videos with only a small difference between the videos which were supported by industry compared to those which were supported by ecancer. Expert to expert interviews and Expert panel discussions have far higher levels views with an average of 149% increase between To camera interviews and the other formats.
Percentage of Video Watched
This data shows that the highest percentage of the video watched is for To Camera Interviews (industry-supported) videos. To Camera Interviews (ecancer supported) videos are next highest with a 3.4% reduction, with a total of 12.9% reduction to Expert Interviews and 18.5% total reduction to Expert Panel Discussions you can see that the To Camera Interviews have higher percentages. One reason which could explain this difference is that To Camera Interviews are shorter in total length.
NPS
The NPS score shows that viewers of Expert to Expert Interviews and Expert Panel Discussions indicate that they are more likely to recommend the video to a colleague. It is noticeable that To Camera Interviews in both industry-supported and ecancer-supported formats achieve lower NPS scores than Expert to Expert Interviews and Expert Panel Discussions. Through discussions with healthcare professionals who use ecancer as an educational tool, one possible reason for this is that learners find the formats more trustworthy. Users of ecancer have said that, because there is more than one expert present, there is a kind of in person peer-review happening while the video is being recorded and the experts involved could be more careful that they are being accurate with their statements knowing that there is another expert present.
Clinical Practice Measure
Clinical practice measure indicates whether viewers believe it is likely that watching the video will improve their clinical practice or research. The above graph shows that the score is lowest for To Camera Interviews (industry-supported) with a fairly consistent rise by format up to the Expert Panel Discussions which score 0.8 higher. As mentioned in the NPS score, anecdotal evidence is that the more experts present during the filming, the more trustworthy the content, this data reinforces that hypothesis.
Conclusion
The purpose of this research was to examine whether there is a difference in viewer engagement and ratings between industry supported content when compared to content support by ecancer as well as looking at whether there is a difference between formats. In order to answer the first part of the research question, we compare the data for To Camera Interviews (Industry supported) with the To Camera Interviews (ecancer supported). The format in both groups of videos is the same. Therefore, any difference is likely to be because of the different funding source. The differences in all four scoring methods between the two types of To Camera Interviews is lower than the differences between video formats. This potentially indicates that the difference in format is larger than the funding method as a measure of viewer engagement and satisfaction.
The differences in the format are larger in the number of views than the other engagement and satisfaction measures. A larger percentage of the To Camera Interviews were watched than the Expert to Expert Interviews and Expert Panel Discussions which may be due to the shorter length of these videos and the NPS and Clinical Practice Measure scores for the formats with multiple experts are higher when more experts are involved in the production and filming.
Overall, the Expert Panel Discussions and Expert to Expert Interviews score higher on the educational satisfaction ratings and there appear to be small differences between industry supported and ecancer supported viewer engagement and ratings.
Disclosure Statement
Danny Burke: An employee of ecancer who received unrestricted educational grants from Janssen, Amgen, Takeda, Teva, MSD, Pfizer and Astra Zeneca.
Eva Hofstädter-Thalmann: An employee of Janssen, The Pharmaceutical Companies for Johnson & Johnson.
Chair of EFPIA Medical Education Working Group, Chair of iPACME
Prof Gordon McVie
Leadership: Ellipses Pharma.
Honoraria: BiovelocITA, Beta Glue.
Consulting or advisory: Roche.
Chair: ecancer.
Additional information
Funding
References
- Averaged annual total videos from 2016–2019. ecancer video library. [cited 2020 Sep 4]. Available from: https://ecancer.org/en/video
- Reichheld FF. One number you need to grow. Harvard Business Review.2003 December
Appendices Appendix A
Percentage of video watched is calculated by sourcing the average amount of time each viewer viewed the video from our video hosting software then using the following calculation:
(Average viewing time in seconds ÷ total length of video in seconds) × 100 = Percentage of video watched.
Appendix B
The NPS score is a measure developed by Frederick F Reichheld and published in the Harvard Business Review where it states “The best predictor of top-line growth can usually be captured in a single survey question: Would you recommend this company to a friend? When customers (viewers) act as references, they do more than indicate they’ve received good economic value from a company; they put their own reputations on the line”
The NPS score is used by ecancer as a measurement of how likely our viewers are to recommend a video to a colleague which is seen by Frederick F Reichheld as the ultimate endorsement of a product (or educational resource in this case).
As developed by Frederick F Reichheld, the NPS score is calculated using the following method:
NPS = (percentage of 5-star scores) – (percentage of 1-3-star scores) = NPS score
Appendix C
The “Likely impact on clinical practice” is measured from viewer responses on a Likert scale from 1 to 5 stars. The response mechanism is available to viewers if they pause the video within ecancer.org or if they come to the end of the total viewing time. The rating system pops up alongside the video as seen here: