141
Views
1
CrossRef citations to date
0
Altmetric
Original Research

Assessing the Accuracy, Quality, and Readability of Patient Accessible Online Resources Regarding Ocular Gene Therapy and Voretigene Neparvovec

ORCID Icon, , , &
Pages 3849-3857 | Published online: 17 Sep 2021

Abstract

Purpose

To evaluate the accuracy, quality, and readability of online information regarding the Food and Drug Administration (FDA) approved ocular gene therapy voretigene neparvovec (Luxturna, Spark Therapeutics, Philadelphia, PA, USA).

Methods

Ten online resources about voretigene neparvovec were assessed in this cross-sectional study. A novel 25-question assessment was created to evaluate the information most relevant to patients. Each article was assessed by independent graders using the assessment and the DISCERN instrument. An online readability tool, Readable, was used to assess readability. Accountability was evaluated using the Journal of the American Medical Association (JAMA) benchmarks.

Results

The average questionnaire score for all the articles was 33.93 (SD 11.21, CI 95% ±6.95) out of 100 possible points with significant variation in the content accuracy and quality between the articles (P=0.017). EyeWiki achieved the highest score and MedicineNet the lowest. The mean reading grade for all articles was 12.88 (SD 1.93, CI 95% ±1.19) with significant variation between articles (P=0.001). Wikipedia was the most readable, and the FDA website was the least. None of the articles achieved all four JAMA benchmarks, and only one of the ten articles, EyeWiki, achieved three of the four JAMA benchmarks.

Conclusion

The information available online regarding this FDA-approved ocular gene therapy is generally of low quality, above the average reading level of the general population, and varies significantly between sources. The articles provide incomplete information that is not entirely accurate or easy to read, and as a result, the material would not support patients adequately in their medical decisions and questions about this new therapeutic option.

Introduction

Gene augmentation therapy is a technique in which genetic material is introduced into a patient’s cells to treat or prevent disease.Citation1 Gene therapy was introduced in human subjects in the early 1990s, and currently, it is being recognized as a promising treatment option for multiple diseases.Citation2 Several trials have been conducted or are ongoing to evaluate the efficacy and safety of gene therapy, including but not limited to hematopoietic stem cell based gene therapy for β-thalassemia major, liver-directed gene therapy for hemophilia B, and T-cell immunotherapy for acute lymphoblastic leukemia.Citation2,Citation3

Gene therapy has been of particular interest in ophthalmology. It has been studied in years of experimentation to determine its use to treat several ocular conditions including neovascularization, retinitis pigmentosa, Stargardt disease, retinoblastoma, and more.Citation4 In 2017, voretigene neparvovec (Luxturna, Spark Therapeutics, Philadelphia, PA, USA) became the third gene therapy to be approved by the Food and Drug Administration (FDA).Citation5 Voretigene neparvovec was unanimously approved by the FDA for patients with the biallelic RPE65 gene mutation form of hereditary retinal dystrophy.Citation5 In the pivotal study, patients with this inherited retinal dystrophy were treated with subretinal injections of voretigene neparvovec and showed improvement in their functional vision which persisted for at least 4 years.Citation6,Citation7

With the approval of voretigene and the subsequent press coverage, patients with eye conditions are likely to learn about treatment options on the internet. Millions of Americans search health-related information and seek medical advice online every day.Citation8 Previous research shows about 62% of Internet users search the internet for health information.Citation8 The content acquired online influences patients’ health decisions.Citation9 The content patients encounter would ideally be easy to comprehend. Most patients read at an eighth grade level; however, most sources of medical information are written at a tenth grade level or higher.Citation10 Low health literacy has been associated with poor medication adherence and adverse outcomes such as uncontrolled chronic disease and increased hospitalizations.Citation11 For these reasons, it is critical that the information patients obtain is accurate, complete, and easy to understand. As such, the goal of this study is to assess the accuracy, quality, and readability of online information regarding ocular applications of this gene therapy.

Methods

Article Selection and Content Analysis

The keywords “eye gene therapy” or “voretigene neparvovec” or “Luxturna” were entered in Google.com and in major medical websites such as the American Academy of Ophthalmology (AAO) website, and relevant articles were selected for analysis from the top search results with exclusion of similar results from the same organization. Preference was given to articles of formal as well as commercial organizations trusted by ophthalmologists and those believed to be regularly accessed by patients such as Wikipedia. The chosen articles included ones from the AAO,Citation12 American Optometric Association (AOA),Citation13 EyeWiki,Citation14 US Food and Drug Administration (FDA),Citation15 Spark Therapeutics,Citation16 MedicineNet,Citation17 National Eye Institute (NEI),Citation18 Novartis,Citation19 WebMD,Citation20 and Wikipedia.Citation21 A twenty-five question grading tool was designed by two vitreoretinal surgeons (JS and AEK) to assess the accuracy and quality of patient-relevant information in each article (). The questions were chosen to match presumed typical questions a patient or patient family member would ask and potential topic discussion when considering treatment with voretigene neparvovec. The evaluation of each website was completed independently by two vitreoretinal surgeons (JS and AEK) and one vitreoretinal surgery fellow (NY). A grading scale from 0–4, with 4 as a maximum, was utilized to evaluate each question as detailed in . A score of 0 was given if there was no information provided regarding the question. A score of 1 indicated the response was unclear, had inaccurate information, omitted significant information, and was very poorly organized. A score of 2 was for partial answers that somewhat addressed the concept but had gaps in the information and organization. A score of 3 meant the article provided essential elements to answer the question, addressed the most relevant points, and was focus and organized. A maximum score of 4 indicated the answer was accurate and thorough; the article provided a clear answer that was well explained, focused, and organized. Each article was independently graded by each observer, and interobserver reproducibility was assessed with a Spearman correlation. The average score between the three graders was used to compare the quality of the articles.

Table 1 Questionnaire to Assess Online Resources Describing Ocular Gene Therapy with Mean and Total Scores

The articles were further evaluated by two graders (JS and NY) using the DISCERN instrument. DISCERN is widely used to determine the quality of health information provided to patients.Citation22 Each article was independently graded using the 16-question tool. Each question is rated using a 5-point scale ranging from “No” to “Yes.” A score of 1 is given if the answer to the question is “No.” A score of 5 is given if the answer is a definite “Yes.” Scores of 2 through 4 are given if the article meets the criterion of the question partially. The interobserver reproducibility was determined with a Spearman correlation. The average score between the two graders was used to compare the quality of the articles.

Accountability Analysis

The accountability of each article was evaluated using the Journal of the American Medical Association (JAMA) benchmarks.Citation23 Each article was assessed for the 4 standards: Authorship, attributions, disclosure, and currency. Each article should include the authors and contributors along with their affiliations and relevant credentials to meet the authorship requirements outlined by JAMA. Attributions, or references, should be reported, and disclosures and currency, or date of update, should be specified.

Readability Analysis

The online tool, Readable, was used to analyze readability of the articles.Citation24 Each article was assessed using the Flesch Reading Ease Score (FRE), Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and Simple Measure of Gobbledygook (SMOG) Index. The Flesch Reading Ease Score awards each article a score from 0–100 based on total words, sentences, and syllables in the text. Higher scores translate to better readability. A score between 70 to 80 is comparable to an eighth-grade reading level. The Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and SMOG Index are separate readability formulas which report the reading level required to understand the text. Each index states the US grade level of education reflected by the article. The mean score of these indices corresponds to the numerical US grade level. Therefore, a mean score of 12 indicates a 12th grade reading level while a score above 12 corresponds to a college reading level.

Statistical Analysis

Statistical analysis was completed using IBM SPSS Statistics for Mac, version 25.0, released 2017 (Armonk, NY: IBM Corp). Content analysis using the 25-question assessment and the DISCERN instrument was performed by treating the data as ordinal variables and analyzing with the Kruskal–Wallis test. A post-hoc Dunn-Bonferroni test was used to determine pairwise comparisons. Statistical significance was set at P≤0.05 for the main comparisons and the pairwise comparisons. The readability analysis was conducted using a Kruskal–Wallis test to compare the mean reading grade level for each article. A post-hoc Dunn-Bonferroni test was used to determine pairwise comparisons. A Spearman correlation test was carried out to evaluate correlation between accuracy and readability. Statistical significance was set at P≤0.05 for the main comparisons, Spearman correlation, and pairwise comparisons.

Results

Article Selection and Content Analysis

Ten articles were analyzed for the study. The inter-observer reproducibility was statistically significant between JS and AEK (r=0.66, P=0.038) and between JS and NY (r=0.84, P=0.003) and approached statistical significance between AEK and NY (r=0.62, P=0.058). The average questionnaire score for all the articles was 33.93 (SD 11.21, CI 95% ±6.95) out of 100 possible points. There was a statistically significant difference in the content accuracy and quality between the articles (P=0.017). The top scoring article was EyeWiki with an average score of 51.33 points. MedicineNet was the lowest scoring article with an average score of 16.67 points (). There was a statistically significant difference in the articles’ scores on each of the 25 questions (P<0.001).

Table 2 Mean and Total Scores for Grading Online Resources Describing Ocular Gene Therapy

For the DISCERN instrument, the inter-observer reproducibility was statistically significant between the two graders (r=0.89, P<0.001). The average score for all of the articles with this grading tool was 45.20 (SD 8.40, CI 95% ±5.21) out of 80 possible points. Results of the instrument also showed a statistically significant difference in the content quality between the articles (P=0.043), with EyeWiki scoring highest with an average score of 60.00 points and MedicineNet scoring lowest with 30.00 points (). There was a statistically significant correlation between the articles’ scores on the 25-question grading scale and on the DISCERN instrument (r=0.90, P<0.001).

Table 3 Mean and Total Scores for Online Sources Describing Ocular Gene Therapy Graded Using the DISCERN Instrument

Accountability Analysis

None of the articles achieved all 4 JAMA benchmarks. One of the 10 (10%) articles, EyeWiki, achieved 3 of the 4 JAMA benchmarks (). Currency was the most displayed benchmark followed by attribution and authorship. There was no correlation between the accuracy of the articles and JAMA benchmarks (r=−0.229, P=0.524).

Table 4 JAMA Benchmarks Achieved by Online Sources Regarding Ocular Gene Therapy

Readability Analysis

The mean Flesch Reading Ease Score for all the articles was 39.51 (SD 9.98, CI 95% ±6.19). The mean reading grade for all websites was 12.88 (SD 1.93, CI 95% ±1.19). There was no significant correlation between the FRE score and mean reading grade level (r=−0.340, P=0.336). There was a significant difference between the mean reading grade level of the websites (P=0.001). Wikipedia had the lowest mean reading grade level of 9.63, and the FDA website had the highest mean reading grade level of 16.73. Wikipedia (P=0.005) and the Spark Therapeutics product website (P=0.015) had significantly lower reading levels than the FDA website (). There was no significant correlation between website accuracy and the mean reading grade (r=−0.444, p=0.199).

Table 5 Readability Analysis of Online Sources Describing Ocular Gene Therapy

Discussion

The results () demonstrated that there was significant variation in the quality, accuracy, and readability of the information available to patients about the ocular gene therapy voretigene neparvovec. EyeWiki provided the most complete and accurate information of the articles analyzed and achieved the most benchmarks of all the articles fulfilling 3 of the 4 benchmarks. However, EyeWiki had a mean reading grade level of 12.85, and is difficult to read for a layperson, as it is primarily designed for ocular provider education rather than patient consumption. Still, given that it is freely available and accessible by the public, it was included in the analysis.

Table 6 Summary Table of Grading Indices Used to Assess Online Sources Describing Ocular Gene Therapy

The second-highest scoring resource on both grading tools was the Spark Therapeutics patient informational website regarding voretigene neparvovec; it was the second most readable article as well with a mean reading grade level of 10.40. Nevertheless, it scored only 54.50 out of 80 points on the DISCERN grading tool and below 50 out of 100 points on average on the 25-question assessment. It did not achieve any of the JAMA benchmarks; moreover, despite being the highest scoring resource designed for reading by patients, it is not an ideal source of reference given the potential inherent conflict of interest present in the manufacturing company’s discussion of its own product.Citation25

In this study, the information available from formal organizations such as the FDA, AAO, and NEI was of overall low quality. Ideally, these major organizations would have accurate and complete information that is easy to understand as many patients rely on these institutions for medical knowledge. Prior studies have shown that educated patients have greater trust in federal agencies and community organizations as sources of medical information.Citation26 Overall, patients prefer non-commercial websites especially from established organizations such as the FDA and NEI over other sources.Citation27,Citation28 The FDA website received the highest content and DISCERN average score of these three resources; however, it had a mean reading grade level of 16.73, the highest level of all the articles. Although the AAO and NEI websites were easier to read with mean reading levels of 11.45 and 13.03, respectively, both resources scored poorly on both scales. Furthermore, the AAO and FDA websites only achieved 1 of the 4 JAMA benchmarks while NEI achieved 2 of the 4 benchmarks demonstrating the overall poor accountability of the articles, although one could argue that the organization as a whole may be taking accountability for the content, which would not be identified by the JAMA benchmark scoring.

This study was limited in several aspects. The 25-question assessment was based on two retina providers' experience and assumptions regarding typical patient questions and was not validated in a patient population. Furthermore, the inter-observer correlation between two pairs of the graders resulted in moderate strength of correlation demonstrating variability in interpretation of the articles or of the grading scale. To further test the online resources being evaluated, the standardized DISCERN instrument was also incorporated, and there was a significant and strong positive correlation between the DISCERN scores and the content assessment scores. Inter-observer reproducibility had greater strength in the DISCERN instrument as well. Patients may use search engines or keywords that vary from the ones used to determine websites included in this study. Some resources included in this study were found beyond the second results page on Google, but patients often only use the first two results pages for their research.Citation28 These resources were still included given their importance as prominent, public organizations (eg NEI, FDA). The resources evaluated were non-peer reviewed sources of varied websites including commercial, government, and non-government organizations. These articles were chosen as they were readily available during search, but patients may prefer to read peer reviewed sources or information from other organizations. Resources were not directly assessed by patients; future studies could further study online resource quality and content by directly evaluating patients’ comprehension.

In conclusion, this study demonstrated that patient accessible online resources regarding the ocular gene therapy voretigene neparvovec are generally incomplete, difficult to read, and of poor quality. The available material varied significantly by source and did not provide adequate information to aid patients in their medical decisions. This indicates an unfilled need to create patient-friendly online content regarding this ocular gene therapy as available research and therapeutic options continue to evolve. Future articles can be improved by assuring accuracy of the information using peer reviewed research papers as reference and by maintaining a reading level sufficient for patients’ understanding.

Disclosure

Dr Nicolas A Yannuzzi is advisor for Alcon, Novartis, Genentech, and Alimera Sciences, outside the submitted work. Dr Jayanth Sridhar reports personal fees from Alcon, Dorc, Genentech, and Regeneron, outside the submitted work. The authors report no other conflicts of interest in this work.

Additional information

Funding

This work was supported by the NIH under Center Core Grant P30EY014801, Research to Prevent Blindness Unrestricted Grant. The sponsor or funding organization had no role in the design or conduct of this research.

References