The ethics and epistemology of researching higher degree by research supervision: an encounter with institutional ethics review

ABSTRACT

This article provides a reflexive account of the authors’ experiences of the ethical challenges in conducting a higher degree by research (HDR) supervision project prompted by the ethics review process in a major Australian university. The authors also raise epistemological questions about the specific focus of the study, given the interrelations of ethics and epistemology. The project aimed to examine the conceptualization of ‘poor supervision’ by HDR actors including students and graduates with a view to informing supervision practices. The article seeks to highlight fundamental questions about what is researchable/ unresearchable from ethical as well as epistemological points of view, and who can decide on ethical norms and boundaries in this research space. As HDR education in general and HDR supervision in particular have come under increased scrutiny nationally and globally inviting more research and academic engagement, such an account may be useful in pointing to the complexity of pedagogical, epistemological, and ethical issues and their complex interactions against the processes of institutional ethics review.

KEYWORDS:

• HDR supervision
• ethics and HDR supervision research
• epistemology and supervision research
• ethics review
• effective and ineffective supervision
• HDR education in Australia

Introduction

The higher degree by research (HDR) is the ‘highest form of education’ (Elliott et al. 2020, p. 18), essential for future academics, researchers, and knowledge-leaders across disciplines. It has now emerged as ‘a key constituent of the higher education system: no higher education and research policies can disregard this rapidly expanding sector’ (Halász and Ruano-Borbalan 2022, p. 361). As academic supervision is arguably the most critical aspect of the process and outcome of HDR education, interrogating HDR pedagogy bears utmost significance. However, investigating this education, with its features that are often dependent on close working relationships and inherent power dynamics – 'the high-stakes, intimate tutorial’ (Pare 2011, p. 59) – raises challenges: What are the ethical challenges in researching HDR supervision from the ethics review perspective? What aspects of the phenomenon are researchable and what aspects maybe unethical to research? Who can decide on the ethicality? How do different HDR actors conceptualize effective or ineffective supervision?

Such ethical and epistemological questions are at the centre of the present article, which is based on a research project that aimed to interrogate and demystify what is called ‘poor supervision’. We share our reflections on the project and its aims from ethical and epistemological perspectives. In particular, we reflect upon the ethics review process that significantly reshaped its conceptualization from the original thinking, pointing to the wider relevance of our local narrative. The insights generated and the questions raised are important for all stakeholders in postgraduate research education. The article contributes to our understanding of supervision as pedagogy/research and an object of inquiry, as well as to ethical and epistemological insights into HDR supervision research. It also adds to the narratives of institutional ethics review (e.g. Sikes and Piper 2010) by providing an example of researchers’ lived struggles on the one hand and ethical learning, problem-solving and (un)happy reconciliation on the other (Halse and Honey 2005, Fitzgerald et al. 2006).

The article deviates from the typical structure of empirical research report (e.g. literature review, methods, and findings) as well as the common narrative account of ethics review experiences (e.g. Honan et al. 2013; see also Sikes and Piper 2010). The next three sections provide the background by outlining the landscape of HDR education in Australia followed by discussion of HDR supervision and the university-funded HDR project. We then embark on reflection on our experiences of interacting with the institutional ethics review to raise the ‘researchability’ question in HDR supervision from ethical and epistemological perspectives. In the empirical section of the article that follows, we present selective survey and interview data from the HDR project to revisit the researchability issue. We draw lessons and implications in the conclusion which also highlights the contribution of the article.

The HDR landscape in Australia and beyond

With an enrolment of over 66,000 students (Universities Australia 2020), HDR programmes constitute a significant component of Australian higher education. They are critical for Australian society because of their direct link to education export, human capital development and the construction of a knowledge economy in a competitive world (McGagh et al. 2016). HDR education is also a critical aspect of higher education in Europe, America, Asia, and other parts of the world (see Halász and Ruano-Borbalan 2022) and is probably the oldest and most sought-after form of international education globally. The provision of doctoral qualifications shows institutional research capacity and infrastructure, knowledge leadership and the potential to contribute to society through research knowledge and expertise (Halász and Ruano-Borbalan 2022).

Nationally and internationally, HDR education is increasingly under scrutiny (Pearson and Brew 2002, Manathunga 2005, Halse and Bansel 2012, Buirski 2022, Halász and Ruano-Borbalan 2022). Not surprisingly, HDR supervision, known to be a determinant of the ‘quality’ and outcome of doctoral and master by research education, is a key investigation point. Under the growing influence of neoliberalism in higher education (Fairclough 1993, Manathunga and Bottrell 2019), much of this scrutiny may be motivated by the desire to ensure efficiency and transparency in academic supervision which until recently had been widely perceived as a private space occupied by a single supervisor and their supervisee (Pare 2011, Halse and Bansel 2012, Grealy and Laurie 2019). Unsatisfactory outcomes related to completion rates and the duration of candidature (see, for example, Sinclair 2004) led to this space being monitored and its recognition as a form of pedagogy, particularly in social sciences and humanities fields. Interventions were claimed to assist academic supervisors in the first place, through the implementation of education and training programmes for supervisor development (Pearson and Brew 2002, Manathunga 2005, Miller 2007), even if this was perceived as taking a managerial rather than a pedagogical focus (Manathunga 2005).

In Australia, major reforms to address these issues were recommended by the Australian Council of Learned Academies (ACOLA) Review (McGagh et al. 2016), bringing the ‘quality’ of HDR training to the fore (see also Halász and Ruano-Borbalan 2022). Graduate career development was the key driver of the reforms, justifying public investment in HDR training on the one hand and harnessing the critical benefits of the programme on the other. The career development agenda has shifted the focus of attention from the thesis towards graduate outcomes as a primary target for HDR education. Consequently, universities across Australia have incorporated professional and transferable skills into HDR training alongside research skills development.

HDR supervision

The ‘quality’ question has been the key topic of much of the HDR literature in Australia and globally (Cullen et al. 1994, Pare 2011, McCallin and Nayar 2012, Buirski 2022). Supervision quality is central to international comparability and competitiveness of Australian HDR training. Yet, the ACOLA Review stated as a key finding that despite the general improvement of the supervisory experience, quality is ‘variable between individuals, disciplines, and institutions’ (McGagh et al. 2016, p. xvii), calling for sustained efforts from universities to enhance the quality of HDR supervision. To address this concern, the Review outlined strategies including professionalization of supervision and standardization of supervision approaches; supervisor training; ongoing monitoring and evaluation of performance; and incentivisation of high performance, understood as completions and graduate outcomes.

PhD supervision, ‘the unique mix of teaching, learning and research’ (Wisker 2012, p. 2), is indeed perceived as the ‘heart’ of the process of postgraduate research education. The emphasis on teaching or research as a defining feature of supervision may vary across disciplines, as hard sciences may prioritize the latter while social sciences and humanities the former. Byram and Stoicheva (2020) observed that supervision ‘is an “intimate”, a once-in-a-lifetime experience for the supervisee to which a supervisor brings multiple experiences, and out of this mix is created the unique relationship and working procedure of every supervisee/supervisor combination’ (p. 10). Although ‘[r]ewarding, [and] potentially taken for granted and relatively under-theorised’ (Wisker 2012, p. 2), research supervision is often directly linked to the outcomes and experiences of supervisees in academic, social, emotional, and psychological terms (Elliot et al., 2020; see also Halse and Bansel 2012). For example, feedback on HDR writing has been identified as the most critical and, at the same time, the most contentious component of supervision mediating quality (Pare 2011). The experience of supervision remains the most memorable aspect of postgraduate research education for candidates at the desirable end of their research training journey. Ensuring the quality of HDR supervision is therefore an agenda with the highest priority for every stakeholder in this highest form of intellectual pursuit.

The ‘quality’ agenda has called to bring traditional supervisor-supervisee engagement out of the ‘mysterious garden’ and rather to consider supervision as a teaching and learning activity – a pedagogy (Cullen et al. 1994), although it may take on more of a research character in STEM disciplines, as previously noted. This view of supervision is clearly articulated in the Graduate Research Good Practice Principles: ‘Supervision is appropriately attributed and recognized as a specialized, workload-bearing academic function that has educational and research dimensions’ (Australian Council of Graduate Research n. d.). The growing body of HDR literature, including supervision guides, produced by institutions has emphasized this pedagogical character of HDR training in the past three decades (Cullen et al. 1994, Manathunga 2005, Miller 2007, McCallin and Nayar 2012, Wisker 2012, Guerin et al. 2017, Buirski 2022). This new emphasis has changed the understanding of HDR supervision from an intuitive gift to a learnable skill.

The annual Postgraduate Research Experience Survey (PRES Survey) provides valuable information about the quality and effectiveness of supervision, as experienced by HDR candidates from across universities in Australia and the UK. Supervision is one key topic on which participants’ experiential views and feedback are solicited. The analysis of participant responses in the past few years shows high overall satisfaction ratings by over 85% students (Williams 2019, Pitkin 2020). However, the area that has received the lowest satisfaction rating (by just over 60% participants) over the years is what is called ‘research culture’, ‘a hazy concept’ (Casci and Adams 2020) which is hard to define. These two authors argue that research culture ‘includes the way we evaluate, support and reward quality in research, how we recognise varied contributions to a research activity, and the way we support different career paths’ (p. 1).

While this culture generally refers to issues at the institutional level such as the provision of research support and infrastructure, the quality of supervision may implicate it in a positive or negative way through the provision of feedback, for example (Pare 2011, Starke-Meyerring 2011). Although the overall satisfaction of HDR experience documented by the survey is pleasing, it does not exclude the existence of ‘poor supervision’ or ‘poor quality supervision’ (McGagh et al. 2016). Poor supervision is inimical to quality, and may, even if infrequently occurring, overshadow high performance. Furthermore, it may lead to discontinuations by individuals who are not represented in surveys of current students or graduates. Thus, addressing poor supervision can be strategically important to further enhance HDR training, graduate satisfaction, and outcome.

The ACOLA Review acknowledged poor supervision, but the quality improvement measures recommended do not address it directly. If the HDR literature acknowledges the existence of poor supervision (Cullen et al. 1994, Sinclair 2004, Green 2005, Pare 2011), it is rarely interrogated and demystified. Poor supervision appears to be more real in anecdotes of HDR journeys and experiences often shared in social networks, or on the internet. As academics with considerable involvement in HDR supervision and the management of HDR education in a research-intensive university in Australia, we thought the problem deserved academic and research attention in the interest of all stakeholders including HDR scholars, supervisors, and academic institutions.

The HDR supervision project

Our interest in the research problem (‘poor supervision’) was stimulated by a small-scale seed-funding opportunity that was available from the faculty-level teaching and learning committee in our university. The grant rules clearly articulated a pedagogical focus for prospective applications, which were expected to contribute to the enhancement or innovation in teaching, learning and assessment. In relation to our interest in HDR supervision, two factors shaped the focus of our application. First, although investigating poor supervision from the perspectives of HDR students based on their lived experiences would be highly valuable, we knew that allowing students to share their experiences might lead to discoveries which would go beyond the scope or ability of the research team to handle. Second, from experience we knew that such a project risked not being approved by the ethics committee, which might identify unmanageable ethical issues and challenges. Taking these points into consideration, we proposed a project entitled Enhancing HDR supervision pedagogy: Understanding and acting on ‘poor supervision.’ Although our overarching aim was to understand how HDR candidates defined and understood poor supervision, we did not want to invite students to share their supervision experiences. Instead, we focused on poor supervision at a conceptual level, without purposefully eliciting stories of experiences of supervision students and graduates may have. Three research questions were formulated: (1) How do HDR students and graduates conceptualize poor supervision? (2) What are the key indicators of poor supervision? and (3) What are the implications of the understanding of poor supervision for supervision pedagogy?

In addition to referring to poor supervision as an issue flagged in the ACOLA Review and the HDR literature, we were able to make a case for our proposed research by referring to policy discussion within the university that highlighted poor supervision, calling for ways of addressing it. A practical problem-solving view underlying the conceptualization of the project might have made our application attractive to the committee. The formation of the research team with members from different levels of HDR education at the university might have been another strength of the application. The project application was successful, and we secured the funding. We also partnered with the Graduate School of the university which provided a small amount of additional funding. A PhD graduate who had been supervised by one of the team members and a co-author of this article was recruited to work as Research Assistant. The team, keen to start, applied for ethics approval within the university, following appropriate protocols. This process was the beginning of our learning about the ethics and epistemology of HDR supervision.

Researchability of HDR supervision from an ethical standpoint

We received feedback on our application from the university ethics committee within the stipulated timeframe. However, the comments exposed our naiveté in conceptualizing the project from the ethics point of view. Of the many concerns raised by the committee, several stood out as ethically red-flagged. First, it was commented that the investigation of ‘poor supervision’ would invite hostile testimonies from discontented students and recent graduates (our initial target as research participants) who might voice their grievances about their supervision experiences, and that such complaints were likely to include matters of professional conduct and perceived failings in the way supervisors had performed their role. This concern may have been partly due to the fact that such students would have had limited opportunities to air their dissatisfaction: in the Australian context in particular, ‘few avenues are provided to identify systemic failures in supervision practices at an institutional level’ (Grealy and Laurie 2019, p. 302). Second, it was pointed out that the study was likely to cause reputational damage to advisors, and that there were inadequate measures for their defence and self-protection. Third, the study was construed as collecting students’ complaints about professional conduct of colleagues in the faculty. This ‘invidious situation’ was represented as potentially harming collegiality and causing a direct professional disadvantage to academic supervisors.

The feedback somewhat dampened the enthusiasm of some team members about the project who felt a sense of ethical guilt. They found it difficult to rid themselves of the taint of having proposed a project that potentially violated fellow academics’ reputations and collegiality. Such a violation had been beyond our imagination, as the project had no such intentions. However, to the ethics reviewers, the project was potentially grazing in an ethically questionable territory. We were clear in our minds that the project was about the conceptualization of supervision for which listening to participants’ stories of supervision experiences was not essential, as noted earlier. However, viewing the project as researchers, not as reviewers of ethics applications, we might not have been sufficiently conscious of all its dimensions. Therefore, the feedback was a critical learning point. Specifically, we learned that the way we communicate our intentions in ethics applications may be subject to different interpretations of those intentions. What was needed was showing more humility as researchers acknowledging that our text may be open to multiple and unintended interpretations; and that the clear distinction we saw between a discussion of ‘poor supervision’ as a construct and an invitation to dish the dirt on low performing academics, was less obvious to the ethics committee and therefore, to potential participants.

We submitted a detailed rejoinder to the ethics committee, refuting the potential ethical challenges, taking on board some of the suggestions provided and revising the application. Time was a key issue for this one-year project while the ethics approval process was time-intensive by design, as the committee had fixed dates for meetings. When we received the second-round feedback, we started losing hope about the project.

In addition to some common questions around anonymity, freedom of withdrawal from the study and data security, the committee raised a further number of key issues and provided important suggestions. Below we paraphrase the key points:

• The thrust of the study including the title was ‘prejudicial’; ‘poor supervision’ should be changed into ‘supervision’.

• HDR supervisors should be added to the participant pool.

• The ethics application should include a more thorough discussion of the existing literature on the supervision of HDR candidates.

• The interviews should be conducted by researchers outside the faculty to avoid potential for coercion and bias.

• The application did not acknowledge that the research team members were in positions of power which might result in reputational harm to supervisors.

The requirement of a more thorough discussion of the existing literature was unexpected, as we thought it was an epistemological or discursive question, not an ethical one. Nevertheless, acting on this suggestion was not difficult. Addressing the other issues and concerns presented practical challenges, as they were incompatible with the original funding agreement. For example, the focus on ‘poor supervision’ was the key argument for the project. Therefore, a broader focus on ‘supervision’ would entail such a shift in goals as to require reapproval from the funding authority. Furthermore, while the inclusion of supervisors was sound from an ethical as well as an epistemological viewpoint, it was difficult to act on within the budget of a small, seed-funded project. What was also impractical was the suggestion that the interviews be conducted by researchers from outside the faculty in question, as we were located in that faculty. The final point about the researchers’ position of power was a matter of clarification, which was relatively easy to address.

These concerns of the ethics committee pointed to the sensitivity of the topic of ‘poor supervision’. It appeared that in seeking to research this phenomenon, we as researchers were forcing ourselves into an ethically unsafe space. In our rejoinder, we detailed adjustments relating to those concerns which were addressable in our view, while defending lack of changes which we could not make within the funding agreement and budget constraints.

When we received the third-round feedback from the committee, we were almost certain that we were not making any progress in the ethics application. The committee acknowledged our explanations of poor supervision and the exclusion of supervisors as participants in the research. However, pointing out that these revisions were inadequate, the committee demanded further details on many aspects of the research. The following items paraphrased from the feedback were most critical for the committee:

1. The research merit of the study was still questionable, as the (revised) ethics application did not include research on supervision.

2. The term ‘poor supervision’ should be reviewed, as this might represent an outdated view of supervision.

3. If the researchers were going to investigate the perception/conceptualization of ‘poor supervision’, then they could conduct the research in a different faculty/ university.

4. The project was unclear to the committee. As it was internally funded, it might be an internal Quality Assurance exercise. However, since the project outcomes included publications, it was more than quality assurance.

It appeared that our responses to the previous comments and feedback had had limited impact. Although many questions were raised at the latest round of review, we had the feeling that there was probably some other issue at the heart of the problems, although it remained enigmatic. This can be related to Brown et al.’s (2020) research on the perceived role of the ethics committee and viewing the committee as ‘friends or foes’ by researchers based largely due to the communicational issues and misunderstandings. We concluded that another round of revisions addressing the comments was unlikely to convince the committee. Therefore, we discussed the issues with different colleagues and the best advice that we received was to ask for a meeting with the ethics committee. Accordingly, two members of the research team had a fruitful discussion with a committee representative, following which we received some additional comments and feedback. The key issue was the title of the project, which we were required to modify as a requirement for ethics approval.

Timewise, it was an expensive learning. We had taken almost 11 months to learn that researching ‘poor supervision’ was not ethically justified for the committee because it indicated negativity. The ethics committee was giving us signals about this through their feedback, but we refused to act on the advice since we were under the (wrong) impression that researching ‘poor supervision’ was fine as long as we conducted the research in an ethical way. We also felt that we had an obligation to the funding authority as the agreed project was about understanding this particular kind of supervision. Our ultimate interest was to help improve supervision pedagogy, which would benefit HDR scholars, academics, and educational institutions. However, our experience of the ethical review process taught us that we were following the wrong direction. Therefore, based on the feedback from the ethics committee, the project approved for implementation was: Enhancing HDR supervision pedagogy: Understanding the conceptualization of ‘effective and ineffective supervision’. This was a ‘compromise’ reached in the interest of the project (Halse and Honey 2005) – ‘changes (of whatever kind) which have had to be made in order to be allowed to proceed […] a strategy followed by researchers when they have different views with ethics committee’ (Sikes and Piper 2010, p. 205).

Researchability of HDR supervision from an epistemological standpoint

The time taken by the ethics review process might be seen as a waste, but probably the committee wanted to be confident of the research merit and ethicality of what we had proposed in their own terms. Fortunately, the funding authority was kept informed of the ethics process and the project was extended for another 12 months. We started work on the project with new hope and optimism.

We invited current HDR students and recent graduates from the faculty to participate in a survey, which included four substantive open-ended questions in addition to closed questions about their backgrounds and HDR candidatures. The four questions concerned their conceptualisations and indicators of effective and ineffective supervision and their suggestions for addressing ineffective supervision. Within the limited budget, our target was to reach 50 survey responses from HDR students and graduates. From an epistemological point of view, we were guided by an understanding that this number of responses would be sufficient to capture the diversity of the ways effective and ineffective supervision might be defined and understood. We also wanted to achieve depth in our conceptualization by interviewing about a dozen students and graduates and half a dozen academic supervisors following the survey.

As we took the project out to prospective participants, we were faced with a different kind of challenge. While researching ‘ineffective supervision’ was considered somewhat of a red-flagged territory based on our experience of the ethics review, there was a lot of interest from prospective participants. To our pleasant surprise, following our email invitation, the survey was completed by 160 participants, with the number of current candidates and recent graduates roughly being equal. More impressively, 76 students and graduates expressed their interest in follow-up interviews. Interviewing all interested participants would have been highly desirable, but we were unable to do so with the level of available funding. We ended up interviewing 31 students and graduates by redirecting some funds from two of the team members’ other research and consultancy accounts. The number of interviews with academics remained unchanged from our initial target, as we finished data collection by interviewing seven supervisors.

The key epistemological question that we consider in this article relates to making sense of participants’ conceptualizations of effective and ineffective supervision. Although from our experience and the literature we had some understanding that the space of supervision was amorphous and contested, our project was guided by the assumption (however naïve) that we should be able to develop a reasonably demarcated territory which would assist academic supervisors to adjust their pedagogy and supervision practice. Our understanding of supervision was enhanced by existing work, particularly by the different models and frameworks that had been deployed for educational development of supervisors. For example, Manathunga (2005) tried to shed light on what has been generally understood to be the private space of the one-to-one relationship between the master supervisor and apprentice. Drawing on Brockbank and McGill’s (1998) division of supervision pedagogy, her supervision education programme investigated three learning domains, namely the cognitive (knowing/learning), conative (doing) and affective (feeling) corresponding to educative, administrative, and supportive dimensions respectively. Nonetheless, she acknowledged the limited scope of her focus and achievement, as she could not shed light on some important issues:

I was unable to make a great deal of progress in my attempts to incorporate critical and postmodernist perspectives on the role of power, emotion, irrationality and the body in supervision pedagogy into my programs. Further research is required to investigate effective, sensitive way of exploring this “dirty terrain” … (Manathunga 2005, p. 26)

Some of these issues have been highlighted by Green (2005) using post-structuralist and psycho-analytical lenses and drawing on lived experiences as well as fictionalized accounts of supervision experience. Manathunga (2009) has also underscored the multidimensionality of the supervision space by highlighting cultural and intercultural issues as represented by international and indigenous students.

The undefined scope of supervision was underscored by Miller (2007) who sought to develop an online module on supervision for academic advisors in Australian universities. In problematizing what he called ‘good supervision’, he was led to identify four constructs that point to the hybrid and contested nature of supervision. The constructs included: good pedagogy, good administration, good contribution to knowledge, and good relationships. Thus, he acknowledged that the multidimensional notion of supervision subsumed pedagogical, managerial, epistemological, and interpersonal dimensions. Similarly, Harris’s (2020) CAPES framework refers to five dimensions of supervision namely Communication and Language, Attitudinal Positioning, Partnerships and Relationships, Excellence in Practice, and Skills and Competencies. A comparison of Miller’s and Harris’s frameworks suggests that although there are overlaps between them, they can also be complementary. While Harris provides a more detailed understanding of the various components, it is still possible to incorporate the epistemological dimension from Miller’s work into it.

Elliott et al. (2020) have put forward a legitimate argument that HDR education cannot be fully understood by focusing only on the official aspects of pedagogy, management, and interpersonal relations. What is equally important, as they validly argue, is the notion of ‘hidden curriculum’ (Vallance 1973–74, Martin 1976) in research higher education. It is possible to hypothesize that for every formal/official aspect of supervision, there is a hidden/unofficial aspect that may significantly influence PhD experience in terms of processes and outcomes. For example, even when it is accepted that supervisors as key official actors play the most vital role in shaping the PhD, the role of (unofficial) friends, mentors, colleagues or ‘research buddies’ may not be denied in any PhD journey. Therefore, it can be concluded that no matter how widely we cast our conceptual net, some aspects of the phenomenon of supervision may remain uncaptured. Our understanding of the concept is probably bound to remain an ‘unfinished business’ (Green 2005).

The various dimensions of supervision as identified by the studies reported above are largely based on top-down theoretical understanding. By contrast, we sought to understand the concept based on views and understandings of significant actors including current and past HDR students and supervisors. We were able to interrogate the concept directly and explicitly, which we hoped would enable us to define its potential boundaries. As our participants were experts in conceptualizing research and research-related practices, we hoped that their opinions would be useful for demystifying the notion of supervision. While we were familiar with several frameworks to understand supervision, as discussed above, we believed that an inductive approach would be most suitable for our purpose as our aim was to chart the territory afresh from a conceptual point of view. However, while we may have succeeded in casting our net widely, we were unsure about pulling it back in a way to provide an epistemologically meaningful understanding of the concept of supervision. The ethics review feedback might have played a critical role in the epistemological complexity, as we explain below.

Epistemological challenges

In this section, we present data from the project to point out challenges in conceptualizing supervision even at a broad level. At the stage of conceptualizing the project, we might have been guided by a managerial and problem-solving perspective on supervision. For example, we thought that distinguishing between effective and ineffective supervision would be helpful for pedagogical purposes although this bifurcated view of supervision was a compromise, as the ethics review did not approve the original, singular focus on ‘poor supervision’. However, we came to understand from our data that effectiveness and ineffectiveness were two aspects of the same phenomenon. The absence of clear dividing lines between them was pointed out by a number of interviewees. It was not just the distinction between effectiveness and ineffectiveness; we were also alerted to the epistemological scepticism about knowing what supervision is. As Fiona (pseudonym), one of the academics participating in the study, said:

It's hard to polarize it really. It's a whole, it's a gradient of things that you can or cannot do to be more or less effective. So it's hard to … I hope in your report, you don't say, this is ineffective, this is effective, that's not how it is. I think you have to recognize how things support or what's effective or-or it kind of– [chuckles]. Because it's never black and white. So many elements to it.

Fiona warned us not to take the naïve epistemological view which she might have sensed from our conception of the study. What is effective and what is ineffective were not clearly drawn in black and white terms. She noted that there were many elements to supervision. Megan, another academic, also agreed that there was nothing essentially ‘effective’ and ‘ineffective’ in supervision. As she reasoned:

So it's always hard for me to think in dichotomy, good, bad, effective, ineffective, because I don't think anybody is effective or ineffective. But I think there are practices that for some students are helpful and other students aren't. So for me as a supervisor, I need to learn about my students and to be- for me, to be an effective […] supervisor for that student, I need to understand their needs and also their strengths, and weaknesses, um, because I can't just- I don't treat- You cannot treat everyone the same.

Megan pointed to the relational nature of effectiveness in supervision. In her view, supervisors cannot be labelled effective or ineffective because it does not work like that. She emphasized that supervision refers to practices, and that the same practices may or may not be helpful to all students. She implied that the effectiveness of supervision will depend on the effective identification of the needs, weaknesses (or strengths) or priorities of particular students.

Pamela, another academic, reiterated Megan’s views:

Like the term ‘effective’ is kind of like, uh, you can have whatever strategies you like, but unless they work for the student themselves, like, it doesn't matter. Like it has to be what works for that particular student. And what works for one student may not work for another […]. Well, I guess, um, effectiveness needs to be on a case by case individual basis.

Along the same lines, Sofia, a graduate, believed that supervision was a ‘process’, not a ‘right or wrong thing’:

Effective supervision is a process. There is no necessarily a set rule of all the things that a supervisor needs to do. But having become a professor myself and working with students after my job at [university name]. It would be really helpful, I think, for supervisors to understand when-when they are going to be effective and when they're not going to be as effective with the student. So I'm thinking more of supervision as a process rather than a right or wrong thing.

She also emphasized that there was not a single set of rules that supervisors have to follow to ensure effectiveness: there is no one perfect recipe.

While we sought to determine the scope of supervision in general and its effective/ineffective realization in particular, the outcome showed that supervision can be linked to almost each and everything that can be conceived of about higher degree research education. This is demonstrated by our inductive coding process of the survey and interview data as reported in Tables 1 and 2.

Table 1. Aspects of HDR education with which the conceptualizations of supervision are associated by participants (survey responses).

	Domains associated with supervision	Effective	Ineffective
		Number of participants
1	Academic mentoring, guidance and training	132	99
2	Attitudes, behaviours and styles	111	105
3	Availability and frequency of contact	80	83
4	Feedback	81	77
5	Communication	77	67
6	Management, planning and timeliness	71	51
7	Research progress, performance and outputs	40	39
8	Knowledge, skills, experience and network	40	33
9	Relationship and roles	43	29
10	Personal wellbeing, progress and development	37	23
11	Personal support and mentoring	27	13
12	Needs and expectations	11	9
13	Ethical issues	2	14
14	Culture, language, gender and religion issues	6	8
15	Satisfaction level with aspects of supervision	3	4
16	System, environment and administrative factors	3	3
17	Enrolment and study or work status	1	4
18	Student academic background	2	1
19	Work effectiveness	2	1
20	Funding and financial issues	0	2
21	Personality issues	0	2
22	Workload	0	2
23	No supervisor	0	1

Table 2. Aspects of HDR education with which conceptualizations of supervision are associated by the participants (interview responses).

	Topics associated with to supervision	Effective	Ineffective
		Number of participants
1	Attitudes, behaviours and styles	32	31
2	Relationship and roles	26	27
3	Workload and time	4	26
4	Communication	33	25
5	Health, wellbeing and self-esteem	7	25
6	Research progress, performance and outputs	14	25
7	Academic mentoring, guidance and training	30	21
8	System, policy and administrative factors	7	18
9	Feedback	23	18
10	Culture, language, gender, religion and age issues	5	17
11	Ethical issues	2	16
12	Enrolment and study or work status	5	15
13	Knowledge, skills, and experience	13	13
14	Availability and frequency of contact	9	13
15	Funding and financial issues	1	12
16	Management, planning and timeliness	16	9
17	Personal progress, development and reputation	10	9
18	Milestone panel	2	8
19	Environment	9	8
20	Personality issues	2	8
21	Personal issues	0	7
22	External factors and other people	3	6
23	Needs, expectations and goals	4	6
24	Student academic background and capacity	3	3
25	Personal support and mentoring	12	3

As can be seen from Table 1, the survey data led to identifying 23 sets of factors with which the participants associated effective or ineffective supervision. The range and diversity of factors identified by those with first-hand experience of supervision (as supervisee) may point to an ontology /epistemological reality, which may not be in the best service of pedagogy. Against the stated aims of understanding what effective/ineffective supervision and associated practices look like, any proposition that supervision is a process that can be related to over two dozen sets of things is unlikely to impress those who need to ensure the effectiveness of supervision.

Nevertheless, the data still offer value for pedagogy. Realizing that there is no fixed take on what exactly supervision is and what it entails may lead to pedagogical humility on the part of academic advisors, for example. Supervision is highly contingent and needs constant negotiation with those they work with. Further, while the coding process identified many aspects, the relative weight of factors in relation to effective and ineffective supervision is helpful. This detail allows us to show that although supervision quality is associated with ‘workload’, the contribution of this factor may be minimal compared to others such as guidance, attitudes, communication, and feedback. The table presents data about the relative contribution of the factors based on the number of survey participants who mentioned the factors. This may help to understand more vital and less vital factors that are associated with effective or ineffective supervision.

We repeated the coding process for the interview data, which were contributed by all three categories of participants: HDR students, graduates, and advisors. The categories in both tables are based on open-ended responses from the participants and enabled us to ascertain convergence or divergence between the data collected by the two methods. As can be seen from Table 2, the interview data substantiated all 23 sets of factors identified from the survey data. The interview data also identified attitudes, behaviours, and styles as the most important questions, dropping academic mentoring down to number seven in our hierarchy.

Some qualitative differences in the categories also emerged. For example, the categories of ‘Knowledge, skills, experience and network’ (Table 1) and ‘Knowledge, skills, and experience’ (Table 2) are somewhat different from each other because the interview participants did not mention networking. Similarly, the categories of ‘Needs and expectations’ (Table 1) and ‘Needs, expectations and goals’ (Table 2) are also different, as the interview participants mentioned students/supervisors’ goals in doing research. Finally, we can also see differences between ‘Culture, language, gender and religion issues’ (Table 1) and ‘Culture, language, gender, religion and age issues’ (Table 2).

The divergence between the data in the two tables may suggest that different methods produced different conceptualizations of supervision. However, we interviewed only a subset of the survey participants and the interview also included advisors. The interview questions might have provoked certain responses not captured by the survey. Regardless, the data further complicate the epistemology of research supervision.

Conclusion: ethical and epistemological learning

In conclusion, researching HDR supervision may entail ethical and epistemological challenges (see Aldridge 2020). Our experience in the HDR project highlighted that applying for ethics approvals for research is not a mere administrative exercise, but an essential process contributing to epistemology. Researchers have narrated their experiences of dealing with ethics review, highlighting differences and disagreements (Sikes and Piper 2010, Honan et al. 2013). Given the unequal power relations, some researchers have been reported to ‘compromise’ to reach consensus and reconciliation (see Halse and Honey 2005, Sikes and Piper 2010, Tolich 2016). We have demonstrated how the ethics review process reshaped our research by telling us what kind of supervision we were allowed to research. Certainly, the ethics review process aimed to ensure autonomy, respect and the absence of harm for a particular group of people; at the same time, it also defined and redefined what can and cannot be researched, which is essentially an epistemological question. We were able to predict the consequences of resisting the feedback and suggestions provided by the review authorities – i.e. the project might not have gone ahead at all. However, we could not imagine how the research and its findings would have unfolded had we been allowed to go ahead with our original proposition of researching poor supervision, rather than the compromised and contested variants of effective and ineffective supervision.

As we have reported, effective and ineffective supervision were considered two sides of the same phenomenon. Therefore, researching only effective or ineffective supervision should be sufficient because by default one will show the other side as well. The positive aspect also tells us about the negative, which may be defined as the opposite, or the absence of the positive side. Miller’s (2007) focus on ‘good supervision’ may therefore be an appropriate approach.

It does raise the question, however, that, if ‘ineffective supervision’ was acceptable to the ethics committee, why was ‘poor supervision’ not acceptable? We cannot rule out the suggestion of ‘imagined risks’ (Bell and Wynn 2023) associated by ethics review with certain research problems. Ethics review committees may be more sensitive to risks than researchers shaping not only the review process but also research and its outcomes.

Understandably, the language and representation of research problems matters, as it relates to face-value of our research. An examination of only negative aspects may be perceived that we were engaged in a fault-finding mission, regardless of the validity of the argument; or that we believe that the promotion of effective supervision is only about eliminating bad behaviour, leaving intact the notion that ‘good supervision’ is an unteachable skill. In our view, and our experience, although the question was about ‘ineffective supervision’, some interview participants mentioned the term ‘poor supervision’ in their responses; referring to ‘a poor supervisory team’ and ‘poor supervisory relationship’, allowing us to develop a deeper understanding of the supervisory experience. Effectively, who decided on the conversation boundaries were our participants, and our process of navigating complex pedagogical, epistemological, and ethical issues and their complex interactions helped us meet our aim in this project to understand the areas or domains of activities that can be associated with effective/ineffective supervision, providing a contribution to the conceptualization of supervision.

We acknowledge that this article deliberately focuses on ethical and epistemological issues surrounding HDR supervision research and we have utilized only partial and selective data from the rich dataset to highlight the critical role of ethics and ethics review and its interaction with epistemological questions. The project appears promising from empirical as well as theoretical perspectives and we are interrogating the data in different ways for better understanding of HDR supervision as education, pedagogy, and research mediated by ethics, ethics review and ethical conduct of research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Faculty of Humanities, Arts and Social Sciences, the University of Queensland.