Enabling equality of access in higher education for underrepresented groups: a realist ‘small step’ approach to evaluating widening participation

ABSTRACT

This article presents the evaluation of a small-scale widening participation intervention delivered by a Post-92 university to further education students traditionally underrepresented in higher education. The intervention aimed to provide high quality information, advice and guidance about higher education opportunities and benefits through four workshops, so that participants could increase their confidence and skills and make informed decisions about their future.A realist small steps approach to evaluation was applied to explore the effectiveness of the intervention and why it was successful. Activity based focus groups were employed to address the research questions into how participants acquired and internalised knowledge about higher education and how this led to change or action. The evaluation found that the effectiveness of the intervention was limited because of poor attendance at the workshops. It was also found to be more beneficial to students who were pursuing a higher education pathway, as they highly valued the access to ‘hot’ knowledge the programme provided. However, for some participants, decisions were restricted because of financial constraints. An updated theory of change is presented that includes new and updated enablers that will make the intervention more effective in the future.

KEYWORDS:

• Higher education
• widening participation
• realist evaluation
• critical realism

Introduction

The Office for Students (OfS) has identified that students from low socioeconomic backgrounds, certain ethnic groups, care leavers, carers, people estranged from their families, refugees, mature and disabled students are underrepresented in higher education in England (OfS 2018b). These groups have been identified as facing a range of structural barriers that hinder their progression into higher education, especially high-status institutions. Information and guidance about higher education that enables decision making has been identified as an area where underrepresented groups experience significant disadvantage (BIS 2016). Higher education providers are required to deliver outreach activities that will help increase the access and participation of underrepresented groups within their own institution and across the sector (OfS 2018a).

This article presents the evaluation of the Progress Support Initiative (PSI), an outreach activity delivered by an English Post-92 university to further education students from underrepresented groups. The programme was also open to students who were homeless or at risk of homelessness, or who had recently experienced extenuating circumstances or financial hardship, as the Institution had identified these groups as underrepresented in its student body or as facing access challenges. The initiative aimed to provide high quality information, advice and guidance about higher education opportunities and benefits, so that participants would have the confidence and skills to make informed decisions about their future.

PSI consisted of four 90-minute group mentoring workshops with one session being extended to include a campus tour. Two sessions were delivered on the university campus and two at the participating college. Each session addressed a specific area that the PSI team had identified as challenging for these groups: higher education decision making, identification of transferable and interview skills, Universities and Colleges Admissions Service (UCAS) applications and writing a personal statement, and student life and finance (Figure 1 – Activities). The PSI workshops included a range of activities to engage participants, including presentations, videos, discussion, and individual and group tasks.

Figure 1. PSI theory of change

The workshops were delivered by one widening participation practitioner, and two fully trained student mentors who identified as belonging to one of the underrepresented groups. Two colleges, identified by the regional outreach consortium as needing additional support to help their students into higher education, took part in the initiative. The regional outreach consortium is part of the National Collaborative Outreach Programme that was set up in 2017 with the aim of rapidly reducing ‘the gap in higher education participation between the most and least represented’ and is focused on areas ‘where there is the greatest potential for impact’ (OfS 2019, 4). The consortium has identified and developed relationships with a range of schools and colleges where data and evidence suggest ‘higher education participation by young people is low overall and lower than expected given their GCSE results’ (OfS 2019, 4). The consortium facilitated the initial contact between the colleges and university running the PSI activity. Once the colleges had been approached and had agreed to take part, participants for PSI were recruited from underrepresented groups who were interested in higher education by staff at each of the colleges in September 2018. A total of 26 students participated, 14 from College 1 (C1) and 12 from College 2 (C2). The evaluation took place in May 2019, after all four workshops had taken place.

Harrison and Waller's (2017) realist small steps approach to evaluation was adopted for this project, as it is a pragmatic method that focuses on why an intervention was successful rather than solely on whether it was successful. Their approach is designed specifically for the evaluation of outreach activities and recognises the complex social world in which these widening participation interventions take place (Harrison and Waller 2017).

This evaluation intended to explore the effectiveness of PSI in developing the participants’ knowledge about higher education and how PSI has increased their skills and confidence to make informed decisions about their future. In considering if and why the intervention was ‘successful’, the following questions guided the evaluation: How have participants’ knowledge about higher education developed as a result of PSI? Do PSI participants feel more confident in making decisions about their future? What enabled PSI to be successful?

This article starts by positioning PSI within the existing widening participation literature on information and guidance, and critical realism as a theoretical approach to outreach research and evaluation. Harrison and Waller's (2017) realist small step approach to evaluation, and what makes it different from a traditional realist evaluation, is outlined. The consultation process pursued for the development of PSI’s theory of change and how success is measured is then discussed. Next, the qualitative methodological approach and the use of activity-based focus groups are considered; the findings from each activity are then presented.

The evaluation found that PSI’s effectiveness was limited because of poor attendance at the workshops; it also mainly benefited students who were pursuing higher education as it provided access to informal (‘hot’) knowledge, which was highly valued (Ball and Vincent 1998). PSI did help cement decisions already made about higher education especially for those applying to the university delivering the outreach programme. The decisions of some participants appeared to be restricted because of structural constraints such as financial concerns or location. The results are then discussed in relation to other widening participation research, and the article concludes by presenting a modified theory of change that focuses on the enablers that would make PSI more effective in the future.

Literature review

PSI was predominately developed from practitioner knowledge, lived experience, day to day practice and external policy drivers. While PSI was designed with no explicit connection to existing research, it still aligned with the available literature and addressed a need identified in widening participation research. PSI is a discrete programme, but it also sits within a wider offer of outreach activities delivered by the university that aims to facilitate access into higher education. The delivery of information, advice and guidance is just one small element of the wide path into higher education for students.

One of the major drivers for PSI was the changes in Government policy over the last ten years. Policies have aimed to improve the ‘fairness of opportunity and to stamp out discrimination’ in higher education access by giving students the ‘same choices’ through improving the availability and quality of information about higher education options (BIS 2016, 54). This included information about entry requirements, fees and accommodation costs, course and assessment content, teaching quality, and future career and earning prospects. Recent higher education policies in ‘England have assumed that students make’ choices about their education in an ‘individualised, rational, instrumental and, ultimately, context-free manner’ (Baker 2019, 1). However, this perspective ignores the structural constraints (financial resources, social networks, educational background and social capital) that can restrict or impede the choices students make (Whitty, Hayton, and Tang 2015).

Prospective higher education students from high participating groups have a greater appetite for information about higher education compared to those in traditionally underrepresented groups (Renfrew et al. 2010). Those who ‘consider themselves well informed about [higher education] courses are more likely to be successful in their applications than those who do not’ (Purcell et al. 2008, 161). Those from privileged educational backgrounds also receive more extensive support and guidance as a result of higher levels of social, cultural, and economic capital (Jones 2013).

Widening participation students are less likely to access formal sources of information such as government websites and institutional prospectuses (Moore, Sanders, and Higham 2013). They also ‘require greater help and advice in accessing information and increasing levels of awareness of university options’ (Hunt et al. 2018, 33). However, high quality information can ‘strongly influence’ their ‘intention to pursue post-compulsory education’ (McGuigan, McNally, and Wyness 2016, 484). This information needs to be received at the right educational stage (Sutton Trust 2008) and to be delivered by trained professionals (Borghans et al. 2015) to have a positive impact. These students also want clear and easily accessible information that is appropriate to their circumstances (Moore, Sanders, and Higham 2013) and they highly value information delivered by ‘role models’ or current higher education students (Sanders and Higham 2012).

Research in this area has typically utilised Bourdieu’s (1977, 1986) conceptual lens to ‘understand the role of structure in contributing to unequal [higher education] decisions and choices’ (Baker 2019, 2). There has been a particular focus on capital and how different types, amounts and quality of capital might advantage or disadvantage students when accessing different habitus such as higher education institutions, especially high-status universities (Ball et al. 2002; Bradley and Ingram 2013; Crozier et al. 2008). Drawing on this research and Bourdieu’s notions of capital and habitus, Hayton and Bengry-Howell (2016) have developed an evaluative framework that can be used to show the impact of widening participation intervention. The NERUPI Framework is ‘predicated on a cultural model of widening participation’ and aims to connect metadata, theoretical research and practitioners (Hayton and Bengry-Howell 2016, 46).

However, the use of Bourdieu’s concepts in widening participation research are being challenged by the emergence of social realist epistemologies that are being applied to issues of inequality in higher education (Dyke, Johnston, and Fuller 2012; Case 2015; Kahn 2016; Baker 2019). The ‘critical realist position attempts to avoid’ the ‘structuralist approach’ of Bourdieu, and allows the researcher to ‘explore the relationship between personal, social and cultural resources that emerge, develop and are transmitted over time and across generations’ (Dyke, Johnston, and Fuller 2012, 832). Baker’s (2019) work is of particular interest to this evaluation as she explored how agency is exerted in the face of structural constraints by socio-economically underrepresented further education students making their higher education choices. The research concluded that higher education providers can assist in increasing fairer access through outreach schemes that make enablements which broaden the scope for decisions to be transformed into choices.

This emerging theoretical approach to widening participation has led to calls for social realism to be applied to the evaluation of widening participation activities because they represent complex social interventions (Harrison et al. 2018; Brown 2018). Burke and Lumb (2018) present an evaluative methodology that cautiously uses realist perspectives, while Kahn (2016) and Harrison and Waller (2017) explicitly outline how a realist approach can be applied in the evaluation of widening participation activities.

This approach was chosen because it claims to be designed to ‘engage with the intricate realities of how human choices are made within complex social fields’ (Harrison and Waller 2017, 84). This should allow this evaluation to fully explore how confident PSI participants are in making informed decisions about their future and the interplay between agency, structure and social change.

Methodology

Harrison and Waller (2017) state that their approach sits broadly within the realist worldview and is influenced by Pawson’s (2006, 2013) realist evaluation. They share Pawson’s (2006, 25) belief that ‘contextual constraints’ in which social change programmes operate shape the initiative’s success or failure. Therefore, the fate of an intervention lies in the choices that people make, and these choices are shaped by existing characteristics and relationships that empower or disempower individuals to resist or embrace a programme of change. For Harrison and Waller (2017), higher education choices are constrained by accumulated structural educational inequalities.

The objective of traditional realist evaluations is to discover ‘what works for whom in what circumstances’, rather than merely assessing if something works (Pawson and Tiley 1997, 114). However, Harrison and Waller's (2017, 85) approach differs as their focus is on understanding why an intervention works, not just whether it works. They encourage evaluators to examine the ‘magic box’ of social interventions to establish what processes make the programme successful. They also suggest that widening participation interventions are only effective if they successfully disrupt ‘structural educational inequalities’ (Harrison and Waller 2017, 86).

While they claim not to be prescriptive about particularities of how evaluations should be operationalised, they do suggest five principles of effective evaluation practice. These ‘relate to theories of change, causality, measurement, timescales and disadvantage’ (Harrison and Waller 2017, 85). The small steps approach is also methodologically neutral and encourages the evaluator to use methods appropriate to the needs of the intervention, the participant groups, and the practitioners involved.

Harrison and Waller (2017, 85) state that if an intervention is to cause change, there needs to be ‘clear articulation of the mechanisms’ through which change will occur. They suggest using a theory of change as it can be ‘evaluated in terms of its effectiveness in describing processes and predicting outcomes’ (Harrison and Waller 2017, 85). However, they provide very little guidance on what type of theory of change should be used or how it should be developed.

PSI was designed and delivered without a theory of change being developed; this was undertaken retrospectively before the evaluation took place. Drawing on the guidance from Harries, Hodgson, and Noble (2014), a theory of change was designed in consultation with the PSI team (Figure 1). The theory of change is represented through a logic model as this allows for a substantial amount of detail without being overly complicated.

Originally, the final goal of PSI was to increase the progression and success of underrepresented groups into higher education. However, in alignment with the small steps approach, a final goal was identified that was directly connected to the intervention and measurable within the timeframe of the evaluation without the need for long-term tracking. The final goal or marker of success was identified as ‘PSI Participants feel confident and have the skills to make informed decisions about their future’. Building on the existing research, this outcome assumes that underrepresented groups do not have the confidence, skills, experience, or networks to make informed decisions.

The PSI was delivered with no intermediate outcomes; however, each workshop had four to five learning outcomes. The PSI team and evaluator reflected on these to see how they could be evolved into measurable intermediate outcomes that could be tested and aligned with the final goal. Four intermediate outcomes were articulated, which focused on increasing knowledge, experiences, or changing the behaviour of participants. A small steps evaluation eschews ‘attitudinal measures in favour of those based on knowledge and behaviour’ as this improves the reliability and validity of the evaluation (Harrison and Waller 2017, 86).

The next stage in the development of the theory of change involved the PSI team reflecting on the programme and identifying the ‘enablers’ that allowed the intervention to succeed (Harries, Hodgson, and Noble 2014). They identified three enablers but were unable to express why they were important to the intervention. Understanding why these enablers were important is a key element in assessing the success of the programme. As PSI had already been delivered it seemed unnecessary to consult on the other aspects of the theory of change (activities and inputs), but these stages are represented in the logic model.

Taking into consideration the principles laid out by Harrison and Waller (2017), the purpose of the evaluation, and the operational constraints of the project, a qualitative methodology was chosen. Qualitative methodologies emphasise the ‘context that influences people’s actions or interactions and the meaning that people ascribe to their experience’ (Yilmaz 2013, 313). Focus groups were used as the data collection method as they ‘excel at providing insights into the process rather than outcome’, which is crucial in answering the evaluation questions (Barbour 2007, 30). Using the appropriate heuristic direction, this method provides insight into how people acquire and internalise knowledge, leading to action or change (Kamberelis, Dimitriadis, and Welker 2018).

Focus groups can also shift the power or control from researcher to participants, allowing participants to shape the nature of the discussion (Raby, 2010). The repositioning of the researcher as facilitator can lead to more authentic data, as participants direct the conversation according to what is important to them. Kitzinger (1994) suggests using activities in focus groups as a way to stimulate discussion, break down the power dynamics between participants and provide research artefacts that can be used in analysis.

The focus groups for this evaluation were structured around two activities, concept maps and statement ranking, which were chosen as they would allow the exploration of student knowledge and understanding (Kandiko and Mawer 2013). The activities were also designed to test whether or not PSI had achieved the outcome identified in the theory of change (Figure 1 – Outcomes).

The first activity was used to explore how further education students’ knowledge about higher education was developed, where they obtained that knowledge and how PSI contributed to that process. The concept map method provides a ‘window into students’ minds’, differentiates between ‘information as knowledge and personal understanding’, the ‘perceived importance of concepts’ and the relationship between them (Kandiko and Mawer 2013, 17). The second activity was designed to explore how confident participants felt about processes that involved making decisions about their future. A ranking activity was used as it provides a focused starting point from which participants can share their thoughts and feelings, while still meeting research objectives (Colucci 2007). The statements were developed in consultation with the PSI team and based on PSI content.

The focus group artefacts were examined using thematic analysis, which is best suited for the analysis of ‘evaluation oriented focus groups when the intent is to understand the … impact associated with a programme’ (Massey 2011, 22). The activities were ‘initially analysed visually and holistically, taking emergent structures into account’ (Kandiko and Mawer 2013, 20). This approach allows for a greater level of ‘flexibility to draw out different sorts of connections, relationships or themes’ (Wheeldon and Ahlberg 2017, 6). The activities from each college were also examined for similarities, variance, and common perceptions. The researchers also noted the pathways of the students who created the activity data so that differences could be identified.

Even though the institutions delivering the outreach programme already had a data sharing agreement in place with the colleges, full ethical approval was sought from the evaluator’s institution, as some of the students in the underrepresented groups could be classified as vulnerable individuals. The evaluation was ethically approved, but it was advised that all information sheets and consent forms should be sent at least a week before the field work so participants could fully consider if they wanted to participate. A focus group was run at each college and was open to all 26 participants of PSI. The focus groups were attended by 12 students, four from C1 and eight from C2. All the focus group participants at C1 reported their ethnicity as White, there was one female participant, all were aged between 16  and 18 years old, and all four participants disclosed having some form of learning or physical disability. At C2 two female and four male participants reported their ethnicity as Black, two male participants reported their ethnicity as Asian, they were all aged between 17 and 18 years old, and three male students disclosed a learning disability.

Findings

Dawson and Dawson (2018, 1412) encourage researchers to share methodological challenges and to ‘publish negative or non-significant results as well as positive results’. The ability of the focus group participants to reflect fully on PSI was severely restricted because of limited attendance at the four workshops. Attendance at the PSI workshops decreased significantly throughout the programme, with only four students at C1 and two students at C2 attending the last session and only four students attending all four workshops.

Attendance is a complex issue, with many structural constraints and external factors affecting it. The rescheduling of assessment deadlines at C2 meant students prioritised completing coursework over attending the third PSI workshop delivered at the college. Funding constraints at C2 also meant they stopped providing a minibus service to the university campus for the fourth session of PSI. Consequently, students needed to take two buses for which they had to pay the cost upfront and be reimbursed later. Participants from C1 had a range of complex health and disability needs that had an impact on their general attendance and engagement at college, which therefore affected their attendance at PSI sessions. Whilst each PSI workshop is delivered as a self-contained session, they do supplement each other, developing a student’s understanding and knowledge of higher education. The impact of PSI was contextually constrained by an individual’s choice to attend workshops, but these choices in turn were shaped by existing structures and funding changes (Pawson 2006). The challenges regarding attendance also had an impact on the evaluation’s data collection, as participants could not reflect on all the PSI activities. However, what was also clear is that students pursuing a higher education pathway had acquired knowledge and information about their choices from sources other than PSI so were still able to participate in the focus groups activities that aimed to assess what knowledge students had and where they had obtained it.

Activity 1: concept maps

The first stage of the concept mapping involved participants discussing their current knowledge and understanding about higher education and student life. The facilitators aided this discussion by asking questions or introducing topics from a crib sheet that was based on the content covered in the four PSI workshops (see Figure 1 – Activities). Five concept maps were created, two at C1 and three at C2.

Participants’ knowledge about higher education from both colleges seemed to be predicated on their chosen pathway. The knowledge base of those who had decided on a higher education pathway was very high and detailed, while those pursuing alternative pathways had limited knowledge about higher education. This was particularly evident at C1 where three participants were pursuing alternatives to higher education, such as a level three apprenticeship, working in the family business, or remaining at the college to do further level three studies. Their knowledge about higher education and the decisions that needed to be made to progress to university were shallow, as it was not relevant to their chosen pathway. However, for the one student who was applying to university, they had a wide range of knowledge about finance, bursaries, disabled students’ allowance, and accommodation options. They were also very aware of the course’s academic requirements, content, and the application process.

At C2, five participants were planning on going onto higher education, two were applying for level four apprenticeships, and one was undecided about their future. The seven participants pursuing higher education associated pathways demonstrated high levels of knowledge about all aspects of higher education and student life. They discussed the types of courses and institutions available to them according to their predicted tariff points, and whether or not they could do a work placement with those courses.

The participants at C2 had an extremely high level of awareness of the financial implications and benefits of higher education and spoke extensively about money, fees and debt, with some suggesting they applied to local universities and were choosing to live at home to reduce costs. The apprenticeship students had also selected this route because of the financial benefits of working and free tuition. One participant said they would choose the university that gave them an unconditional offer. All C2 participants also had a good understanding of the application process for higher education as all of them had to write a personal statement regardless of their chosen pathway through unifrog. Unifrog is a complete destination platform which allows students to compare every opportunity and apply successfully for higher education. It also enables students, teachers, and counsellors to work collaboratively on the application process.

Once the focus groups participants’ knowledge about higher education was exhausted, the facilitators moved onto the second stage of the concept map. This involved the facilitators asking probing questions about where participants had gained that knowledge. It was sometimes challenging for participants at both colleges to identify exactly where, when or how some knowledge was acquired. However, they often identified or speculated about multiple sources or occasions where they felt knowledge was or would have been gained. This implies that knowledge acquisition about higher education is cumulative and needs constant reinforcement.

Participants from both colleges identified the colleges’ careers service and college tutors as their primary source of support when making plans and decisions about their future. The second most cited source of information for participants at both colleges was unifrog as it provided all the information needed, in one easily accessible place. All participants also gained information about their chosen pathway through independent research online. Additionally, many of the students had accessed their chosen university's websites to see what additional help, support, or bursaries they could apply for because of their personal circumstances. While many participants had accessed a wide range of sources for information, only two participants had used league tables or comparison sites in their research about courses. Participants at C2 also identified a wide range of informal networks, such as friends, relatives, and social media, where they got information about higher education. These networks seemed very important to these students, and they highly valued the information they received from these sources.

PSI did not necessarily provide students with new information, knowledge or guidance, but it did help to cement existing understanding and provide different perspectives. Participants at C1 found the PSI workshops gave practical tips about the application process, such as how to write a personal statement, which they found beneficial. Participants from both focus groups liked attending the workshops based at the university and the campus tour. For three students who had been unable to attend a university open day, it allowed them to see the accommodation and facilities on offer, which they highly valued. For participants that had applied to the university delivering the outreach activity, PSI reinforced their decisions about which course they wanted to pursue, because during the campus tour, they saw the facilities available for that subject area.

PSI content that addressed broader information about higher education, such as how to apply to Oxbridge, was less valued. However, content that was specific about the courses or bursaries available at the institution which was delivering the outreach programme was highly appreciated. All participants wanted PSI to offer more specific information about the courses they wanted to study and the financial support they would receive.

Participants from C1 and C2 repeatedly referred to ‘informal’ conversations with the PSI lead or student mentors during workshop activities or on the campus tour. Participants appreciated that they could ask the PSI lead specific questions regarding entry requirements for their institutions and about bursaries and disability support. Participants said the mentors shared an example of assignments they were writing and talked about how they balanced work, study, and student life, and the independence they had at university, which participants found very useful and interesting.

Activity 2: statement ranking

Participants were given ten statements about higher education and student life. In pairs, they were asked to discuss the statements and each choose two cards to represent what they felt confident about and two cards that they felt the least confident about. They were then asked to share their choices with the rest of the group. The facilitators explored with the students why they had made the choices they did and what role PSI had in helping them feel confident or not.

Participants considering higher education all chose statements concerning UCAS, the application process, or writing personal statements, as the things about which they felt most confident. They all said they received a lot of support and guidance from college staff to complete applications. They also discussed how useful unifrog was as it sent them reminders of application deadlines, provided exemplar personal statements and was helpful for researching courses and universities. However, the students wanting to progress onto an apprenticeship said they had found the application process confusing, did not feel confident about the information they received and thought PSI should have covered this. Many of the students, regardless of their pathway, felt the least confident about how to perform well in an interview and in identifying transferable skills.

The students who attended the PSI workshop on identifying transferable skills and interview techniques said they found this session very beneficial as it was helpful to have a discussion with the student mentors and hear about their experiences of university and work interviews. Four students said they felt less confident about the financial or disability support they would receive if they chose to come to the institution delivering the outreach activity and suggested that PSI should include this information. However, this was included in the last workshop of PSI, which had very low attendance.

Discussion

This evaluation was aiming to explore the effectiveness of PSI in developing participants’ confidence, skills, and knowledge so they could make informed decisions about their future. It was also seeking to understand why it was successful and what enabled change. For Harrison and Waller (2017, 82) the effectiveness of an intervention is an informal judgement about the ‘amount of change that can be ascribed to a programme’. As this was a small-scale study, it is necessary to discuss the findings in context to existing widening participation research to increase the validity of the evaluation.

Widening participation activities with multiple interventions have been shown to have a positive impact and increase the likelihood of someone progressing to university (Hatt, Baxter, and Tate 2005). PSI’s four workshops were designed to build on underrepresented groups’ knowledge and increase their exposure to higher education over time. However, the inconsistent attendance of participants undermined the effectiveness of the multiple workshops. While participants had been identified as one of the inputs in the PSI theory of change, they were not represented in the enablers. They need to be, going forward, so that adequate strategies can be put in place to encourage attendance, which, in turn, will increase the effectiveness of the programme.

Knowledge about higher education, the application process, financial implications of going to university, and student life seemed to vary according to the pathway students were pursuing. According to Moore, Sanders, and Higham (2013), information delivered through outreach activities has to be applicable and relevant to participants, so it can be applied after an intervention if it is going to increase knowledge, lead to action or change. PSI participants valued information and guidance given through PSI which was relevant and applicable to them. The time spent on the university campus was enjoyed by all, regardless of their pathway, and gave students access to an unfamiliar higher education habitus, which enabled them to ‘anticipate, experience, and reflect’ upon being a higher education student (Hayton and Bengry-Howell2016, 47).

Participants drew on a variety of sources to develop their knowledge. However, there was a preference for ‘hot’ knowledge that was acquired through the ‘grapevine’ and was socially embedded in the students’ ‘networks and localities’ (Ball and Vincent 1998, 377). All PSI participants knew friends and relatives who were at or had been to university and this was an important source of information for them. However, this information source could unintentionally omit the breadth of higher education choices available, especially in relation to allegedly higher status universities as people from underrepresented groups often make decisions based on the belief or perception that some universities are not for them. Students will actively avoid certain universities such as high-status ones, instead seeking out alternatives that they believe or perceive to have the same ‘values or culture’ as themselves (Crozier, Reay, and Clayton 2019, 923). Social media networks, such as Snapchat and Instagram, also seemed essential to how participants at C2 conceptualised student life. However, Turner (2017) found that social media does not necessarily increase a student’s knowledge about university, but it does seem to influence their choice of where to study. The reliance on friends, relatives and social media networks potentially could limit the type and depth of information and guidance students are receiving about all their available options for the future.

PSI participants predominantly relied on college staff for information and guidance about higher education decisions. Shaw (2012) identifies college tutors as a source of ‘hot’ knowledge as the information becomes socially embedded with time. The college staff also acted as ‘enablers’, who had not been identified in the original PSI’s theory of change. The college staff recruited participants on behalf of the institution running the outreach programme and organised the dates and times of the workshops; they, therefore, could play a key role in increasing the attendance of students. Sanders and Higham (2012) found the presence of current students at outreach activities can increase the effectiveness of an intervention. This was found in PSI where the student mentors’ ‘hot’ knowledge about the whole student experience seemed to be highly valued by participants, especially those at C2.

For Harrison and Waller (2017), an effective widening participation intervention should change accumulated structural inequalities, and help students realise their ambitions. Students at C2 appeared to have their choices restricted because of financial consideration or predicted grades that are often inaccurate, especially for students from underrepresented groups (Boliver, Gorard, and Siddiqui 2017). For some participants, the choice of course and the university was determined by its location, and whether or not they could commute from home; this significantly reduces options, which according to Reay (2017) reinforces and exacerbates already existing educational inequalities.

Conclusion

This evaluation set out to answer the following questions: How have participants’ knowledge about higher education developed as a result of PSI? Do PSI participants feel more confident in making decisions about their future? What enabled PSI to be successful? The evidence presented in this article implies that the impact of PSI is limited. However, Harrison and Waller (2017, 85) caution against making ‘strong conclusions about an activity and its causal effects on individuals’, and instead ‘advocate evaluating the success of activities in terms of … intermediate processes’. The low attendance at some of the PSI workshops reduced the effectiveness of the programme. In partnership with college staff, strategies to improve attendance need to be developed to increase the impact of the programme.

PSI enabled some of the participants to feel more confident about their higher education decisions and cement and reinforce knowledge already held, particularly for those who had applied to the university running the outreach activity. PSI also gave the participants access to ‘hot’ knowledge through informal conversations with student mentors, which was highly valued by participants.

The PSI workshop based at the university and the campus tour allowed students to imagine themselves in an academic habitus. Students not planning to progress to higher education seemed to find the programme less relevant to their situation. For some participants, decision making seemed to be shaped by structural constraints such as location of the institution and financial concerns, which might not be experienced by more advantaged students. Some of the challenges PSI faced could be overcome if it were more fully grounded in existing research on applicability and relevance of content (Moore, Sanders, and Higham 2013), and involved students in the content design.

Harrison and Waller (2017) see evaluation as a circular process that should be used to interrogate and hone an intervention. In exploring why PSI was effective, the evaluation highlighted the significant role the enablers played in causing a change in participants. It also provided insight into why existing enablers were important and identified additional ones. The two new enablers are both external to the PSI team and careful consideration will need to be given to how these can be maximised to improve the effectiveness of the programme. The theory of change has been honed to reflect these new insights (Figure 2).

Figure 2. Modified theory of change

The timing of this study meant it was an endpoint evaluation, and further consideration to embedding evaluative practices throughout PSI should be given. Quantitative pre and post measures based on ‘knowledge and behaviours’ could be designed for each workshop. Whilst understanding why something works is important, in order to begin to break down educational inequalities, knowing ‘what works for whom in what circumstances’ is crucial (Pawson and Tiley 1997, 114). This would align the small steps approach with traditional realist evaluation and would allow structural inequalities to be addressed more fully. Future evaluations of PSI should, therefore, capture more detailed demographic information during the field work so the analysis can be done for different underrepresented groups. However, the ethical implications of this should be carefully thought through as PSI has a small number of participants, which would make any conclusion about the impact on different demographic groupings less robust.

A pivotal element to the Harrison and Waller (2017) approach is the theory of change, although the theoretical guidance is not specific about how this should be developed. The theory of change developed for PSI was predominantly based on implementation theory, i.e. it dealt with ‘what was required to translate objectives into ongoing service delivery and programme operation’ (Blamey and Mackenzie 2007, 444). However, for widening participation interventions, programme theory may be more appropriate as it emphasises the ‘causal links between the mechanisms released by an intervention and their anticipated outcomes’ (Blamey and Mackenzie 2007, 445).

The evaluation of widening participation is an emerging field, especially for small-scale institutionally delivered activities such as PSI. The establishment of the Centre for Transforming Access and Student Outcomes in Higher Education (TASO) by the OfS will raise its profile considerably in the future. It is hoped that this evaluation can contribute to that emerging field, and other evaluators and widening participation practitioners will build on its insights to create interventions which bring about equality and social justice in higher education access.

Acknowledgment

The author would like to thank Neil Harrison and Richard Waller for their correspondence about the small-steps approach and for reading a draft of this article. The author would also like to thank Gemma Derrick from the Centre for Higher Education, Research and Evaluation at the University of Lancaster for her insightful and challenging feedback that helped transform the preliminary draft.

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Notes on contributors

Nathaniel Pickering

Nathaniel Pickering is a Senior Lecturer in Research, Evaluation and Student Engagement at Sheffield Hallam University.  His role includes externally funded research within higher education and institutional research and evaluation focused on improving student experiences and outcomes.  Nathaniel's research focuses on access in to higher education, social justice in education, and educational policy discourse.