Excellence in PhD dissertations in health professions education: Toward standards and expectations

Abstract

The authors of this perspective contribution have served two terms (2014 and 2016) in the Jury of the biennial Best PhD Dissertation Award committee of the Netherlands Association for Medical Education. During this period, the committee reviewed 32 dissertations. Based on discussions among the jury regarding elements of an award winning dissertation and existing literature, we propose seven criteria to evaluate PhD dissertations: size, breadth of research skills exhibited, coherence of studies, relevance to field, validity, style, communicative power and ethics, and impact of the work. We anticipate these may not only assist similar committees but also provide criteria of excellence for future doctoral work in health professions education.

Introduction

Two decades ago, structured Master’s and PhD programs in health professions education (HPE) were rare. Now, advanced academic degrees in HPE have become popular. In 2013, Tekian et al. identified 121 master’s programs (Tekian et al. 2013) and in 2014, 24 PhD programs (Tekian 2014) in HPE. In addition, many European universities provide opportunities to obtain a PhD degree without enrollment in a structured program in the sense of a formal course with tuition, thus the total number of PhD dissertations in health professions defended is likely larger than those resulting from the identified 24 programs. The interest in an advanced degree in HPE may stem from the desire of individual faculty to establish a scholarly career in education and a degree may ease this pathway (Sethi et al. 2016). Additionally, educational institutions may wish to advance the academic rigor of their programs and hire faculty with such advanced degrees (Blanchard et al. 2015). Academic leaders should therefore be interested in the skills expected to come with HPE degrees. While health professions educators may rejoice in this rapid increase in HPE degrees, some may wonder whether the quality and significance of these scholarly advances keeps pace with its numbers (Pugsley et al. 2008).

So what should a HPE scholar with a doctoral degree bring in addition to the skills of excellent teachers and Masters in Health Professions Education (HPE)? A dissertation should arguably be one important reflection of this added value.

The aim of this perspective is to clarify expectations for excellence in PhD dissertations in HPE. The authors mentor doctoral students in HPE and are responsible for HPE doctoral programs at their own institutions. In addition, on two occasions (2014 and 2016), they participated in a jury to review PhD dissertations for the biennial prize of the Netherlands Association for Medical Education. While there is no “objective” standard for a PhD dissertation, and countries and universities have different habits and rules, our agreement on criteria for excellence reflects at least the vision of some experienced PhD mentors with an international view, possibly adding to a benchmark.

Expectations of a PhD in HPE

There are two clear expectations of a PhD in the HPE: quality research and contributions to the literature. The dissertation should provide insight into the quality of work a PhD in HPE demonstrates. Most importantly, we believe that a researcher with an HPE doctoral degree should have the potential to add to the existing body of knowledge in HPE through significant educational innovation and research. This too might be present in the dissertation.

These expectations align with the objectives for PhD degrees as developed for the European Union, which include that PhD candidates (a) have demonstrated a systematic understanding of a field of study and mastery of the skills and methods of research associated with that field, (b) have demonstrated the ability to conceive, design, implement and adapt a substantial process of research with scholarly integrity, (c) have made a contribution through original research that extends the frontier of knowledge by developing a substantial body of work, some of which merits national or international refereed publication, (d) are capable of critical analysis, evaluation and synthesis of new and complex ideas, (e) can communicate with their peers, the larger scholarly community and with society in general about their areas of expertise, and (f) can be expected to be able to promote, within academic and professional contexts, technological, social, or cultural advancement in knowledge-based society (Bologna_Working_Group_on_Qualification_Frameworks 2005). These features align well with Boyer’s criteria of scholarship, as adapted by Glassick (2000), stipulating how experienced scholars use clear goals, adequate preparation, appropriate methods, significant results, effective presentation, and reflective critique, as a habit of professional conduct (Crites et al. 2014).

Standards for high quality PhD dissertations in the HPE

While the PhD degree is one measurement of knowledge and skills of a PhD candidate, and many of the above qualifications must also be observed in practice, in most jurisdictions the PhD degree is primarily awarded based on the evaluation of the dissertation by an independent assessment committee.

The Organization for PhD Education in Biomedicine and Health Sciences in the European System (ORPHEUS) (Deckert et al. 2012) provides useful specifics regarding dissertations. They should contain, according to ORPHEUS, at least “three in-extenso papers” published in internationally recognized, peer-reviewed journals, including a review of the literature relevant to the themes in the papers, and a full account of the research aims, methodological considerations, results, discussion, conclusions, and further perspectives of the PhD project. We propose, as a measure of excellence, the expectation of at least four rather than three published papers in HPE dissertations. We acknowledge that other jurisdictions support single monographs but recognize a trend toward widely available publications.

We propose seven distinct facets of evaluation (size, breadth, coherence of studies, relevance, validity of conclusions, style of writing, and impact). Except for “size”, these criteria, elaborated below, relate to all objectives of PhD degrees as mentioned above (see Table 1).

Table 1. Criteria for PhD dissertations related to characteristics of PhD degrees.

	a	b	c	d	e	f
1. Size
2. Breadth or research skills exhibited	X	X
3. Coherence of studies	X	X	X			X
4. Relevance	X		X	X
5. Validity of conclusions		X		X
6. Style, communicative power and ethics		X			X	X
7. Impact			X		X	X

a. Systematic understanding of a field of study.

b. Ability to conceive, design, implement research with scholarly integrity.

c. Contributing knowledge through original work, meriting (inter)national publication.

d. Capable of critical analysis, evaluation and synthesis of new and complex ideas.

e. Communicating with their peers, the larger scholarly community and society in general about their areas of expertise.

f. Able to promote, within academic and professional contexts, technological, social or cultural advancement in a knowledge based society.

We have applied these criteria to evaluate dissertations for the NVMO biennial Best Dissertation Awards in 2014 and 2016. The Netherlands Association for Medical Education first announced this award in 2008 and awarded its first prize in 2010. Every two year a call is sent out to solicit applications for this award endorsed by the primary supervisor. Dissertations completed and defended at a Dutch university in the previous two calendar years are eligible. Applicants must be members of NVMO, as well as at least one promotor. A core Award Jury is appointed every four years, i.e. for two terms, every term supplemented with a recent award winner, for which the following criteria were used.

Size or extent

The dissertation should contain a series of chapters, at least four of which should be published, accepted for publication, or at least submitted in the established international peer reviewed literature. We and others argue that the process of peer-reviewed publication is an essential indicator of a research skill that PhD in HPE students must master (Kamler 2008; Jackson 2013). We believe that the publication process not only creates an essential skill but also provides a doctoral graduate’s career with output and connection to a research community. All publications should have the candidate as a first author. While four is adequate, five or more publications can form an excellent set. However, the number of publications is always to be weighed against the nature and impact of the papers. Quantity alone should not be a prevailing criterion. We favor this model as opposed to writing an unpublished monograph, which is still the standard in several countries.

As a rule, the dissertation should contain two additional chapters that typically are not published papers—an introductory chapter justifying the relevance of the topic and outlining pertinent research questions, and a general discussion chapter that puts the findings in a wider context of the specific area of study. In addition to the required set of published papers, one or two may have a first author other than the candidate, but only if the candidate has made a significant contribution and the chapter adds to the coherence of the dissertation as a program of research.

Breadth of research skill exhibited

The expectation is that the PhD will be acquainted with a range of research methods. HPE researchers should be familiar with various research designs (e.g. observational, qualitative, quasi-experimental, and experimental), data collection methods (e.g. surveys, knowledge and skills tests, observations, interviews, and focus groups), forms of quantitative and qualitative analysis (e.g. coding and analysis of transcripts, descriptive and inferential statistics), and literature review strategies. A candidate who demonstrates a variety of skills in the dissertation studies is perceived as demonstrating excellence provided that the publications are of high quality, and the candidate has developed the skills rather than relying extensively on others in conducting the studies.

We recognize that all of these approaches are unlikely in a single dissertation. Studies in the dissertation may predominantly draw from one research paradigm or another; the studies may use qualitative data collection and analysis methods (interviews, focus groups, and document analysis), or quantitative methods and descriptive or inferential statistics. Excellent dissertations show a variety of methods. We expect all doctoral dissertations to contain a review of the literature in the area of interest. This review can be a systematic or other type of review (McGaghie 2015) and published as a stand-alone paper or included in the introduction or discussion chapter, or both. Regardless of paradigm, all candidates should demonstrate the application and/or development of theoretical models to support the understanding of concepts. The dissertation studies may typically expand the field by adding to existing theory or development of new theoretical insights. Health professions education is not a basic science, so applicable products may be legitimate part of the dissertation, such as guidelines, curriculum development, instructional strategies, and methods of assessment.

Coherence of studies

A high quality dissertation is more than a collection of studies—it represents a program of research. It should explore and expand a domain of interest through multiple coherent investigations. This should show in the logical sequence of studies, and should be evident in the introductory and discussion chapters of the dissertation. The introduction should provide insight into the PhD’s knowledge through a general overview of the state of the art in a research domain, should identify major gaps in the existing literature, in need of investigation. The discussion chapter weaves the findings of prior chapters in a coherent fashion. An excellent dissertation shows how studies naturally build upon each other to arrive at answering the bigger questions set out in the introductory chapter.

Relevance

A strong dissertation addresses important gaps and questions that potentially have a major impact on theory and practice of HPE. As an applied science, practical implications are important in HPE research. The better the candidate articulates the goal of addressing a gap in the introductory or discussion chapters, the higher value has this body of work. In an excellent dissertation, innovative routes of inquiry are chosen to add to the existing body of knowledge.

Validity of conclusions

Naturally, the validity of conclusions within chapters and inferences from chapters, stated as general conclusions in the dissertation are of utmost importance. The methodological quality of the individual studies must be high, and their study designs must be appropriate. While the work of an assessment committee is supported by papers that have been accepted for publication in peer reviewed journals, critical analysis may still reveal issues of validity, as journal peer review is not necessarily perfect. In the HPE domain, researchers should be familiar with modern views of validity in social and educational sciences for both design and assessments, which should be reflected in the presentation of validity support for their arguments (Downing & Haladyna 2009; AERA/APA/NCME 2014; O’Brien et al. 2014; Tavakol & Sandars 2014a, 2014b; Cook et al. 2015). The validity of conclusions should also be grounded in adequate educational of psychological theory.

Style, communicative power and ethics

A high quality dissertation should be well written and of interest to colleagues acquainted with the domain being investigated. This is not primarily for esthetic reasons. Conducting HPE research is primarily a communication exercise leading to the exchange of arguments with a relevant community of scholars about proposed truths and foundations for education and potentially for policy decisions. Research is not a contribution if not published or communicated with others, or if it has no impact on the practice of education. For that reason, research findings must be communicated in a clear and transparent way, in as much detail as allowing for thorough evaluation and replication by others. Clear, informative titles, concise and informative abstracts, logical delineation of information in the respective IMRAD sections (introduction, methods, results and discussion) of research reports are important. Transparency also belongs to the research ethics that have become standard in any domain of research. The ethical guidelines that respected journals adhere to (see http://publicationethics.org/) should hold true for the review of doctoral dissertations too.

Impact

Impact, finally, is the purpose of research and should be the ambition of any researcher. While some impact can only be judged over time, a short-term impact can be estimated by evaluating the uptake of a study published in the literature. High impact journals (see Thomson’s http://admin.webofknowledge.com/SessionError.cgi?CSID=&DestApp=JCR&Error=SESSION+NOT+ACTIVE) enhance the chance that publications will be read. Up-to-date listings of citations to a publication are available to evaluate work (e.g. https://scholar.google.com/intl/en/scholar/about.html) and some journals offer statistics about access rates and downloads of papers. Due to electronic publication, this type of impact can happen in a very short time period. However, as in biomedical science, journal impact factors should not be the ultimate sign of impact (Ioannidis 2016); as said, impact in terms of uptake in educational programs and curricula may be viewed as significant impact as well.

Another proxy to obtain a sense of the impact is the variety of journals and audiences that were chosen for publication and communication. Variation is beneficial, as long as the journals are of high standing. Papers predominantly published in a single journal are bound to reach a narrow readership. In HPE, education journals and medical specialty journals may add to the impact of findings.

Quantifying PhD dissertation reviews

Quality of scientific work cannot easily be expressed in simple numbers, because of the large variety of facets to be evaluated and the expert judgment that is necessary to do so. The authors, however, have been charged with ranking dissertations for an award and used these seven criteria to arrive at consensus. We use the criteria elaborated above, and found they we applicable. The following procedure was used in two different years (the example is taken from the 2016 group). All dissertations were reviewed by a majority of jury members. Those personally involved with a dissertation were excluded from review; former NVMO Best dissertation award winners were recruited to augment the jury as needed. For all reviews, the jury rated each of the seven criteria with a 5-point scale (inadequate—acceptable—good—excellent—outstanding), resulting in sum scores per reviewer per dissertation. These were averaged across raters for each thesis, first leading to identify rank of dissertations, and in the second round to a identifying the top-10 ranked dissertations. The dissertations received average scores of 22.3 (SD 3.7). Intraclass correlation coefficients across the 7 criteria ranged from 0.34 to 0.94 (mean 0.69). Five ICCs could not be calculated because of negative covariances within a small range. While the score rankings provided guidance for final decisions, the panel did negotiate consensus to confirm their agreement with the rank of the top three dissertations.

Discussion

This perspective paper was written after a process of reviewing doctoral dissertations for the purpose of awarding the “best PhD dissertation”. More importantly, by doing so, we arrived at criteria that may be used more widely in PhD programs, to inform PhD candidates, supervisors and those who consider starting such a trajectory. We acknowledge that different countries and specific institutions may hold other criteria that may be just as valuable, but we found working with these criteria and the scoring format to lead to a workable procedure and acceptable conclusion across diverse institutions, with a reasonable ICC agreement on most dissertations. While the jury had to comply with Dutch expectations of PhD dissertation, the aim was to generalize the criteria internationally. Among the jury members, two were Dutch, one was from Belgium, and two from the USA. Of the 22 award candidates in 2016, three were from countries other than the Netherlands. The authors, who were among the members of the jury, aimed to describe standards for excellence that could hold internationally, while not disavowing any other standards universities apply for dissertations. Our standards, as the title says, refer to a level of excellence, not to minimum criteria.

Evaluating a graduating PhD in HPE by evaluating a written dissertation has its limitations. One is that dissertation work is a product of collaboration. We have stressed the importance of experience with all aspects for real world scientific research, which mostly is collaborative. However, evaluating only a dissertation does not show how much this is the work of others. In extreme cases, the candidate may have had an assistance with the literature review, a statistician doing data analyses, a supervisor writing substantial parts of articles and a copy editor shaping language. While this only may sound exceptional, such cases exist; they limit the value of a dissertation as a measure of a graduate’s qualities. All of this support is not visible to an outside reviewer. An oral defense of the dissertation may compensate for this only to a limited extent, both because interrogation time is limited and because the defense may have a more ceremonial nature rather than being an examination that can lead to rejection.

So far, we have not given any value statements about our findings. We were pleasantly surprised with the majority of the submissions. Of course, these were submitted because supervisors supported their nominations. Additionally, the goal of the criteria was to generate a spread amongst excellent dissertations to enable allocating one final award. We found that about half of the submissions clearly exceeded the expectations of an adequate PhD dissertation, and a number were excellent, as would be expected. This foreshadows excellence in the future of HPE research.

Do HPE educators with a doctoral degree add to the quality education for the health professions? While we firmly believe this, it is not easy to prove it. Medical education research originated roughly in the 1950s and the volume of high quality studies in HPE has steadily increased since. Knowledge of relevant theories, application of sophisticated research methods, and new avenues of research development characterize the HPE domain more than other areas of higher education, and it has been suggested that the existence of PhD programs has contributed to that distinction (Jaarsma et al. 2013).

This contribution focused on the quality of PhD dissertations. Obtaining the PhD degree is usually supplemented by a satisfactory defense. As countries and universities differ in their rules concerning the nature, timing, examination committee, interrogation, consequences of unsatisfactory defenses, and other critical elements of HPE defense ceremonies, it would be worth providing such an overview in a future publication. To our knowledge that does not exist.

We believe that sharing our criteria for dissertations is useful even to those who do not have the same expectations as universities in The Netherlands have. We hope that our work may stimulate the quality of doctoral efforts, and that it adds to the quality of doctoral dissertations, to the skills of HPE researchers and eventually to the quality of education in the health professions.

Practice points

• Scholars with advanced academic degrees in Health Professions Education rapidly increase in number

• Excellence in PhD dissertations has not been well defined

• Based on the work of an Award jury, a 7-item checklist was developed to review dissertations

• Important features of high quality PhD dissertations include size, breadth of research skills exhibited, coherence of studies, relevance to the field, validity, style, communicative power and ethics, and impact of the work.

Access to dissertations that have been awarded NVMO prizes 2010 through 2016

Teunissen (2010) – http://dare.ubvu.vu.nl/handle/1871/16061

Govaerts (2012) – http://pub.maastrichtuniversity.nl/84114223-248e-46fa-9070-b1791963ca76

Kusurkar (2014) – http://dspace.library.uu.nl/handle/1874/234627

Chen (2016) – http://dspace.library.uu.nl/handle/1874/317368

Glossary

HPE doctoral level scholarship: Demonstrated potential to add to the existing body of knowledge in health professions education through significant educational innovation and research.

Notes on contributors

Olle ten Cate, PhD, is Professor of Medical Education and Director of the Center for Research and Development of Education at the University Medical Center Utrecht, The Netherlands.

Anselme Derese, MD, PhD, is a Professor at Ghent University Faculty of Medicine, Center of Education Development.

Steven Durning, MD, PhD, is a Professor of Medicine and Pathology at the Uniformed Services University (USU).

Patricia O’Sullivan, EdD, is Professor of Medicine, Department of Medicine, University of California, San Francisco.

Acknowledgments

We thank Dr. Marian Govaerts and Dr. Rashmi Kusurkar for assisting in the review processes. Both have once been awarded first prize for their dissertation and have participated in the committee at different phases. We thank the Board of NVMO for inviting us to serve two terms in this committee.

Disclosure statement

None of the authors have a conflict of interest to be declared.