Capturing life as it is truly lived? Improving diary data in educational research

ABSTRACT

Diary methods have long been used as pedagogic tools in learning, and as part of reflective practice in teacher education, but less often as data collection instruments in educational research. This is in part due to implementation challenges emerging from the time and literacy demands they place on participants. To illustrate the use of diary methods in educational research, we juxtapose two diary studies to reflect on how to use diaries as data collection tools against a backdrop of researcher positionality. In the first example, the teacher-researcher embedded diaries in a curriculum to collect data from language learners in Japan. While learner engagement with the diaries was high, the prescriptive approach led students to tell ‘the teacher what they wanted to read’. In the second example, the researcher used the diaries to collect data on out-of-class learning among language students in Germany. Engagement with the diaries was initially low but improved substantially after daily reminders were sent via mobile phone. Nonetheless, results revealed a possible self-selection bias. Both examples highlight the value in making adaptations to diary methods as the research context necessitates, so that researchers can take into account issues that might distort or produce misleading data.

KEYWORDS:

• Diaries
• journals
• researcher positionality

Introduction

Diary methods in research involve getting participants to record information about themselves, their behaviours, or their thoughts in a written or spoken format, which can then be subjected to various forms of data analysis. The value of diary methods in research is that they are purported to capture ‘life as it is lived’ (Bolger et al. 2003, p. 579) by providing researchers with an insider account (Dörnyei 2007), recorded in a ‘natural setting’ (Krishnamurty 2008, p. 197). In many ways, diaries grant researchers access to domains of knowledge that are difficult to access via other research measures, such as observations or one-off surveys (Day and Thatcher 2009). Diary methods allow participants to become co-researchers by having them collect information about themselves, which may be otherwise difficult to observe or record.

Diaries have had a long tradition in education as pedagogic tools in learning (e.g. Bailey 1990, Parkinson and Howell-Richardson 1990) and as tools for reflective practice and professional development in teacher education (e.g. Jarvis 1992). In education and other sociological research domains, diary methods have been relatively under-utilized as methodological instruments of data collection especially when compared to their wide-spread use in domains such as psychology (Elliott 1997, Rose et al. 2019, Baker, 2021). The infrequent use of diary methods may be, in part, due to implementation challenges that emerge from time and literacy demands they place on participants to engage in record keeping in a reliable and valid manner (Bolger et al. 2003, Hall 2006). Additionally, when compared to clinical approaches in psychology, educational research—especially in classroom settings—is inherently a messy endeavour, where a researcher may have less control over the quality of diary keeping by participants (depending on, for example, the positionality of the researcher and the extent to which the diaries are integrated into the ongoing learning practices). This paper explores the use of diary methods in educational research to reveal ways to overcome these challenges and to improve opportunities for their future use.

In this paper, we begin by outlining key theory underpinning the use of diaries in educational research, before juxtaposing our own experiences in using diary methods in our previous research projects. We use these studies as opportunities to engage in methodological reflection to unpick issues surrounding their design and application. In the first example, we show how diary methods were embedded in a language curriculum to collect data from learners in Japan, for whom one of the researchers was also the teacher. While this approach yielded positive results in terms of learner engagement with the diaries, the prescriptive approach had a risk of diary entries to be disingenuously completed by the participants in order to complete the task. In the second example, an ‘outsider-researcher’ used diaries to collect data on out-of-class second language use among secondary school students in Germany. Learner engagement with the diaries was initially low, although it was improved substantially by sending the students daily reminders via mobile phone. Nevertheless, the researcher discovered that some students in the total sample engaged significantly more actively with the diaries, indicating a self-selection bias. Both examples highlight the value of flexible approaches to diary methods, so that researchers can pre-empt and adapt to issues that might distort or produce misleading data.

Theoretical literature on diary methods

The types of diaries, and issues underpinning their reliability and validity during data collection, are at the core of methodological decision-making. Like most data collection methods, diaries include a broad spectrum of designs that aim to collect quantitative and qualitative data, both cross-sectionally and longitudinally (Hektner et al. 2011). Diary methods can include journal-type instruments that require narrative-style entries from participants in response to broad questions or prompts (e.g. Ma and Oxford 2014). In contrast, log-type diary formats aim to collect data from many participants at numerous data collection points via scalar and short-answer items, which are primed for statistical analysis (Nezlek 2012). Log-type designs are mostly used to track a dynamic construct (e.g. motivation, anxiety, etc.) longitudinally, utilizing data collected at multiple time points to obtain aggregate measures, and to pin-point moments of change (Iida et al. 2012). When researchers decide on a diary design, it is important that they reflect on how they intend to analyse the data, as this will help to inform their choice of the type of diary that best suits their needs.

Other choices of diary types centre on the temporal aspects of when data on the phenomena being investigated are best collected. These decisions may depend on whether the researcher is interested in investigating the phenomena as they unfold over time, or to focus on specific phenomena that only occur at particular points in time (Bolger et al. 2003). Regarding the former, these types of dairies aim to collect data on an ongoing phenomenon at pre-determined time points, such as interval-contingent diaries that collect data at re-occurring time points (e.g. daily, weekly) or signal-contingent diaries at randomized time points (see Wheeler and Reis [1991], for elaboration). For example, a study on school anxiety might ask participants to record their anxiety levels every couple of hours, alongside information on the educational activities they were recently engaged in. In contrast, event-contingent diaries aim to collect data after an event has occurred that is linked to the phenomenon being investigated. For example, the exemplar study on classroom anxiety might ask participants to complete a diary entry only when they have experienced a difficult encounter. A real example of an event-contingent diary in education is Baker’s (2021) study of 13 students, who were asked to record their educational decision-making with regards to their further education. Thus, when designing a diary study, researchers must reflect on the temporal needs of the research to select the most appropriate type of diary to implement.

One of the largest challenges of diary methods centres on the reliability and validity of data collected, as they require a high level of commitment from participants (Bolger et al. 2003). If participants are not invested in a study, they are unlikely to complete their diary entries with requisite frequency or accuracy. That is, a low participant ‘buy-in’ for a study will erode the quality and quantity of data collected. When research focuses on pedagogical practices, diary methods have been attached with good effect to educational tasks and professional development activities to enhance the ‘real-world’ value for participants. For example, Aubrey (2017a, 2017b) made use of his teacher-researcher positionality to incorporate diary keeping as part of a pedagogical activity given to his student-participants, generating 208 journal entries from 42 students in Japan. This study is illustrative of a high participant ‘buy-in’, which was achieved by aligning the researcher's research objectives with the students’ pedagogical goals, and taking advantage of the researcher's unique positionality.

In situations where a researcher has an outsider positionality, it can be more difficult to control the fidelity of diary keeping. Research fidelity refers to assurances that data collection is accurate, consistent, and conforms to the intended research design (Smith et al. 2007). That is, fidelity refers to whether the data are being collected as planned. Ensuring fidelity can be difficult for ‘outsider’ educational researchers, as the diaries are not so easily integrated into ongoing learning tasks and there is no established rapport with the learners which increases participant ‘buy in’. To encourage regular diary keeping, researchers can reduce the time commitments for participants via use of targeted questions for data entry, which could include a mixture of short-answer, multiple choice or scale items. These items also have an advantage for data analysis if a project plans to use statistical analyses, or deductive coding on pre-defined categories and themes (Nezlek 2012). Where longer responses are required, templates to record answers that can fit on a single page can be useful to not overwhelm participants (Parkinson and Howell-Richardson 1990).

Another way to improve participant responses is to reduce the literacy demands of the students. This can be especially important for students who struggle with the language (e.g. are second language speakers, are younger learners, or have a learning disability). Hall (2006) argues that having lower proficiency non-native participants write in their second language may affect the quality and quantity of diary keeping. He suggests, where possible, participants should be permitted to write in their first language. Where participants must complete diaries in their second language, clear instructions and a structured format of questions can help alleviate literacy demands. Allowing students to audio-record their diaries may be another way to reduce the demands of writing. Indeed, digital platforms and the proliferation of mobile technologies have opened new and innovative avenues for diary methods in research, allowing participants to record information in a diary on mobile devices and via social media platforms (Bartlett and Milligan 2015). In a recent paper in this journal, Hewitt (2017) espouses the benefits of oral diaries as research tools, as they cut down on the writing demands of busy participants. As diary methods can be particularly prone to problems of recall bias if time passes between the phenomenon being investigated and the diary record keeping (Carson and Longhini 2002), methods that facilitate diary keeping in tightly-timed situations can be very useful to enhance the accuracy of data produced (Hewitt 2017).

Thus, there are numerous issues researchers must consider when deciding on the most appropriate diary design for their study. These decisions often require a good amount of reflection by the researcher to consider the contextual demands of the research participants and the research site in relation to the requirements of the investigation. In the next section of the paper, we outline two examples of our own experiences of using diaries in research. We reflect on these experiences to unpick some of the challenges and affordances associated with the use of diary methods in educational research.

Exemplar study one: listening journals (Galloway and Rose 2014)

In our first exemplar study, we outline the use of diary methods in classroom-based research at a university in Japan. The study involved the collection of diaries from 108 students studying elective courses in their third and fourth years of an English language degree. At the university, the lead researcher was interested in exploring students’ reactions to varieties of English in the curriculum as an extension of a larger project (Galloway 2011). In the activity, students in four classes—all taught by the lead researcher—took part in an out-of-class listening task each week for ten weeks. The task required students to select a 10-minute audio sample of English speech produced by speakers from a range of linguistic backgrounds, and then to reflect on the task in a handwritten listening journal. In the journal, students recorded the length of the audio, the nationality of the speaker, their reasons for choosing the sample, comments on their observations of language use in terms of vocabulary, grammar and pronunciation, and comments on their reactions to these features. In total 1,092 journal entries were collected and subjected to content analysis. Similar to the above study by Aubrey (2017a), this study demonstrates the powerful effects of a teacher-researcher positionality and the ability to incorporate diaries as part of the curriculum in classroom-based research on participant engagement in diary methods.

Reflections on the problems encountered

The largest issues faced with the implementation of diary methods in this study were at the design stage, where the researcher had to ensure students would fill in the diaries in a consistent and authentic manner. Thus, the core issues that required research reflexivity centred on issues of validity, reliability, and fidelity of diary entries of the participants.

First, a weakness of diary methods in general is that participants do not engage with them in appropriate depth. In this study, the participants were required to complete listening tasks outside of the classroom, limiting the teacher-researcher's control over the study context. The participants were also required to fill in diary entries soon after completing the listening task, adding a further layer of time demands on record keeping—and presenting another noted issue of recall bias in diary methods (Carson and Longhini 2002), discussed further below.

Second, the dual role of the teacher and researcher posed a threat to the authenticity of the entries recorded in the diaries, especially because students are highly accustomed to their writing being scrutinized (Baker 2021). As the listening tasks connected to the content of the curriculum, there was a danger that the students would merely echo the ideas in the curriculum in an effort to ‘tell the teacher what she wanted to read’ (i.e. a social desirability bias in the data). Thus, the diary instrument needed to be designed so that authentic responses were being captured.

Finally, the diaries needed to be designed so that threats of memory recall were minimized for students to ensure fidelity in data collection. It was not common for the students to have their laptops with them at university, and the location of the listening task was flexible with some resources placed in the university library. Students were also encouraged to look for other listening opportunities online or in real life. Thus, the diary needed to be designed so that it could be filled out in a flexible range of contexts, and sometimes ‘on the fly’.

Many of these challenges are typical for methodological decision-making in diary data collection: ensuring participant ‘buy-in’ to enhance engagement, improving validity and authenticity of data collected, and reducing the time demands on participants. These issues needed to be resolved to ensure the diaries were indeed capturing students’ thoughts and experiences as they were truly lived. The unique dual role of the lead teacher-researcher, as well as the unique context of embedding the diaries into a classroom curriculum allowed for several strategies to address these issues.

Strategies to address the problem

To overcome problems of a lack of participant engagement with the diaries, a decision was made from the outset for the listening journals to comprise a compulsory part of the students’ assessed work for the course. This highlighted to the students an importance of completing the diary task each week, and overcame the main weakness of diary methods, which is participants’ lack of engagement with them. As students were to receive a course credit for the diaries, this automatically constituted a genuine ‘buy in’ from participants for the diary entries to be completed. Of course, ethical issues arise with this decision, particularly surrounding the voluntary nature of research participation. As with much classroom-based research, the boundaries between teaching and research became blurred, as the listening journal task held pedagogical value within the curriculum being taught, but also had research value as a potential source of data collection. To overcome this dilemma, while the diaries were compulsory to complete for all students on the course, they were only used for research purposes after voluntary, informed consent was received from each student. This strategy proved very successful in terms of diary participation, with an average of over 10 diary entries per student.

Yet, the compulsory nature of the listening journals introduced new threats to validity of the diary method, with an increased risk that students would view the diaries as an assessment of their knowledge of course content, rather than vehicles to express their genuine attitudes and ideas. To counter this threat, it was decided that the diaries would only be graded on completeness, rather than their content to give students freedom to write genuine entries, even if they were in conflict with the aims of the curriculum. This strategy seemed successful, with numerous themes emerging in the data set which centred on both positive and negative attitudes towards the material students were listening to. Several critical accounts in the diary data provided evidence of honesty in students’ expression of opinions. Nevertheless, the diary data contained numerous positive expressions about the learning experience, and it was difficult to conclude whether these were genuine expressions of beliefs, which was thus a notable limitation of the study.

Finally, to ensure fidelity of data collection, the diaries needed to be designed so that work demands on students were minimized. It was also important that the diaries could be filled out in a flexible range of contexts. The lead researcher consulted the students about different modes that the diaries could take, including computer entry and handwritten formats. The students preferred handwritten formats to avoid a need for immediate access to a computer to complete the diaries. It was also decided that worksheet-style templates would not be used for diary entries. If students did not have these worksheets with them when completing the listening tasks, too much time might pass between the listening activity and diary entry. Thus the diary methods adopted a handwritten approach that required students to record the information within ruled lines in a notebook. If students did not have these notebooks with them during the listening task, they could simply rule lines on a blank sheet of paper, fill out the requisite information, and insert the paper into their notebooks later. While this made the data more troublesome to analyse for the researcher (who had to work with handwritten, as opposed to typed, data), it suited the needs of the students and was thought to enhance the reliability of diary entries overall, by allowing students to create a diary entry in a range of flexible situations.

Reflecting on the data analysis of the study, the diaries may have been further improved through the use of more short answer and multiple-choice/scale items. This would not only have further reduced the literacy demands on the students, but would have facilitated data analysis for the researchers by pre-coding some of the data in defined categories. It would have also created opportunities to quantitively explore the data longitudinally to investigate how attitudes changed throughout the activity.

Discussion of study one: opportunities for educational research

The case of Galloway and Rose (2014) presents key opportunities and challenges for the use of diaries as a data collection method in classroom research. First, if the diaries form a required as part of the curriculum in classroom-based research, there is an opportunity to overcome one of the largest barriers to diary data collection: participant engagement and research buy-in. If a researcher is able to embed the diaries within an established pedagogical activity, student-participants are more likely to value the diaries and engage more deeply in record keeping. The facilitative effect of pedagogically-driven diary research has long been utilized by teacher educators. However, this study has shown the relative ease with which teacher-researchers can use diary methods to collect a large amount of data from student-participants when the diaries serve a dual educational and research purpose. Diaries, therefore, present a unique opportunity to harness the power of the ‘insider’ role of teacher-researchers in classroom research, where they can exert a level of control over data collection which cannot be matched by most researchers of an ‘outsider’ positionality.

The largest challenge associated with the use of diary methods in classroom research is ensuring responses are authentic, valid and reliable accounts of the constructs being measured. This can be particularly problematic if student-participants may be concerned about meeting teacher-researcher expectations of what they record in their diaries. To overcome this challenge, we make the following recommendations:

• If diaries embody a dual role of both a pedagogical task and source of research data, informed consent to use them for the latter is needed. This means all students receive the pedagogical benefits of the task (no one is excluded due to non-consent), but only consenting students become participants in the research.

• If diaries are included as part of assessment in classroom research, they should only be marked on completeness (e.g. incomplete/complete, or pass/fail) so that participants have the freedom to record their genuine beliefs and experiences in them, without fear of their ideas being evaluated for assessment.

• The format of diaries should be designed in consultation with the participants to ensure that they can be completed in a timely manner. To enhance the reliability of diary data, the format should, as far as possible, meet the needs of the students rather than the researcher.

• Diaries should aim to reduce the time and literacy demands on the students via the use of structured questions and inclusion of short answer items or scales, where appropriate.

Although each study poses its own contextually unique challenges, the case of Galloway and Rose (2014) points to the innovative ways in which teacher-researchers can harness their unique positionality to utilize diaries as powerful and ethical sources of research data. The study also highlights potential issues that arise because of this positionality, which may result in inauthentic record-keeping if appropriate assurances are not made.

Exemplar study two: diary of informal second language engagement (Arndt 2019)

In our second exemplar study, we discuss the use of an event-contingent diary as part of a larger research project on informal second language learning (ISLL) among German learners of English. The diary took the form of a recurring online survey that participants accessed on their computers or mobile phones at home, which included questions about engagement with out-of-class language learning activities (e.g. watching TV series, browsing social media, or playing video games). The aim was to collect rich quantitative data for statistical modelling of the links between different aspects of learners’ engagement in informal activities (i.e. behavioural, affective, cognitive, and linguistic dimensions) and their developing proficiency, motivation, and attitudes towards language learning.

A total of 506 secondary school students (ages 15–16) participated in the project across three stages. The first stage consisted of semi-structured focus group interviews in which 47 learners spoke to the researcher about the ways in which they engaged with informal English activities in their daily lives, their reasons for doing so, and how they thought about language learning more generally, both inside and outside the classroom. The findings informed the subsequent development of the diary on engagement in informal learning, as well as questionnaires for measuring motivation and attitudes. The instruments were piloted during the second stage of the study with a group of 105 students, and further validated in the last stage with data from another 354 learners, which was also used to conduct the planned statistical modelling.

Employing an event-contingent diary design, the participants were asked to complete the newly developed Informal Second Language Engagement Questionnaire (ISLE) every time they used English outside of school over the course of one week, except for homework or other school-related activities. The ISLE primarily contained multiple-choice and scale items, yielding quantitative data which were primarily analysed via Structural Equation Modelling and Latent Profile Analysis. It was designed to provide rich, ecologically valid data that is less likely to be affected by memory errors and other recall-related biases than retrospective self-report questionnaires, which have primarily been employed in previous ISLL research (e.g. Arndt et al. [2021], Bolger et al. [2003]).

Reflections on the problems encountered

One of the central challenges in this study was maximizing participant recruitment and engagement with the diary, particularly in the third stage where data from at least 200 students was needed to conduct the planned statistical analyses. In contrast to the first exemplar study in this paper, the position of the researcher in this study was that of an ‘outsider’ trying to recruit volunteers for her study, which meant that a lot of thought went towards making the study as interesting and accessible as possible to schools, teachers, and ultimately students themselves. Similarly, an important consideration was how to motivate the recruited participants to engage with the diary, which required considerable time and effort from them to complete at home, without direct teacher or researcher guidance.

Strategies to address the problem

The design of the diary was heavily informed by the contextual demands and needs of the research participants. This includes the decision to use an electronic instead of a handwritten format and working towards making the content of the diary as comprehensible and meaningful as possible for the participants, so it would be easier for them to complete and more likely to yield rich and valid data.

As part of the focus group interviews conducted during the first stage of the study, the researcher asked the students whether they would prefer to write more open-ended, narrative records of their daily informal language practices, or to complete a more structured diary survey after each activity, and whether they preferred pen-and-paper, email, web or mobile formats. The participants overwhelmingly agreed that they would be most likely to complete a brief electronic questionnaire after engaging in any informal learning activities (i.e. an event-contingent design), since most of these activities already involved using their mobile phones. The focus group interviews also revealed the importance of asking students about the ways in which they talk about the phenomena being researched, in order to avoid research jargon and use terms they were familiar with.

During the pilot data collection and analysis, the primary focus shifted to the participants’ engagement with the diary and how it could be improved. Although they had been asked to complete the diary across one week, 27% of the 105 pilot study participants did not submit any diary entries at all, and 25% engaged with it across only one or two days. 27% submitted entries across three to five days, and 21% across six days or more. Drawing on learner and teacher feedback, the researcher was able to identify several possible reasons for the low engagement, and to plan methodological changes that improved engagement in the subsequent research stage.

First, as noted by Bolger et al. (2003), a major challenge in event-contingent diary designs is that participants may find it difficult to identify each relevant event. In the current study, the students were instructed to ‘make an entry every time they came into contact with English outside of school or homework (e.g. watching a film or TV programme, playing a video game, listening to music, etc.)’. The examples in the instructions did not include the full range of activities elicited in the ISLE questionnaire, which also covered using social media, reading a book or an online text, overhearing a conversation, and speaking or writing in English. To make it easier for future participants to identify relevant events, the researcher decided to modify the instructions for the next data collection period to include a full list of all activities elicited in the diary, and to explain that the students should also record any other English-language activities which the researcher may not have thought to put in the questionnaire. The pilot study also revealed that it was not possible to distinguish between learners who submitted fewer diary entries because they did not engage in any informal learning activities on a particular day and those who forgot about the diary or chose not to engage with it. In the subsequent data collection phase, participants who hadn't engaged in any informal activities on a particular day were therefore asked to access the diary in the evening and simply record ‘I did not use English outside of class today’.

In an effort to increase participation among students who may be prone to forgetting about the diary, the researcher decided to send daily mobile phone reminders to participants in the third stage of the study. Specifically, the students were sent brief messages via WhatsApp,¹ which the participants in the initial focus group interviews had identified as their preferred method for research-related communication. The messages seemed to be effective, as participation rates increased in the third stage of the study: Out of 354 students, 34% now completed the diary on six days or more, 26% across three to five days, and 13% across only one day or two.

Just like in the pilot, however, 27% of all learners did not submit any diary entries at all. This raised the question of whether the diary data may have been affected by self-selection bias, as certain types of learners may have been more likely to complete them than others. The presence of such a bias would, in turn, call into question the validity of the diary data, that is, the extent to which the findings are representative of all students in the target population. Of course, this is an important consideration in any research, but because the diary was only one component of a larger project, which also involved in-class data collection which all participants completed, the researcher had the unique opportunity to directly investigate the presence of self-selection bias in the diary data.

Based on findings from other diary studies (Hektner et al. 2011), it was hypothesized that girls, students with greater English proficiency, and those with more positive attitudes towards informal language learning might have been more willing to engage with the diary. After confirming that the statistical assumptions were met, general linear modelling was used to investigate the extent to which these variables predicted whether a learner participated in the diary portion of the study at all (binomial logistic regression analysis, Table 1), and if so, the number of days across which they submitted diary entries (multiple regression analysis, Table 2). The findings suggested that some self-selection bias was indeed present in the available diary data: As hypothesized, girls and more proficient learners were significantly more likely to engage with the diary, and to submit more entries across longer periods of time. However, learners who held more positive attitudes towards informal learning were more likely to submit diary entries over fewer days (contrary to what was hypothesized).

Table 1. Logistic regression predicting likelihood of participating in the ISLE diary.

	B	SEB	Wald	df	p	Odds ratio	95% CI for Odds ratio
							Lower	Upper
Gender	.74	.28	6.86	1	.009	2.10	1.21	3.66
English proficiency (C-tests)^a	.70	.25	8.07	1	.004	2.01	1.24	3.25
Attitudes towards ISLL^a	−.02	.09	.08	1	.789	.98	.82	1.16

^a These analyses are based on standardized factor scores. Thus, the odds ratios indicate the changes in likelihood of participating in the diary portion of the study per SD increase in the C-test/attitudes scores.

The model had statistically significant predictive power (χ²[3] = 18.40, p<.001), explained 8.6% of the variance in the dependent variable (Nagelkerke R²), and correctly classified 76% of the cases.

Table 2. Multiple regression predicting number of days participating in the ISLE diary.

	B	95% CI for B		SEB	β	t	p
		Lower	Upper
Gender	.88	.33	1.42	.28	.20	3.15	.002
Prior proficiency (C-Test factor score)	.52	.03	1.01	.25	.14	2.09	.038
Attitudes towards ISLL (factor score)	−.19	−.36	−.02	.09	−.15	−2.22	.027

The model had significant predictive power (F[3,235] = 6.88, p<.001) and explained 6.9% of the variance in the dependent variables (adjusted R²).

The observed gender effect mirrors a larger pattern whereby women are usually overrepresented in diary studies, which require participants to be highly engaged and self-directed (Hektner et al. 2011). Regarding proficiency, it seems plausible that lower-level language learners would generally be less motivated to participate in a study about second language acquisition than their more proficient peers. In this study, students with lower proficiency were also found to engage in informal activities significantly less often, which means that they may not have felt that they could contribute any important information in their diary entries. Conversely, participants with more positive attitudes towards ISLL engaged in significantly more out-of-class activities each day. Making a separate diary entry for each activity would have required considerable effort, which may explain why these participants were more likely to stop engaging with the diary after just a few days.

By incorporating the number of days across which participants submitted diary entries into the statistical analyses as a confounding variable, the researcher was later able to confirm that the main quantitative findings of this study were not significantly affected by missing diary data. Nevertheless, this demonstration of a systematic self-selection bias serves as a reminder for researchers to be cautious in the interpretation of their data, and especially when considering the generalisability of their findings to their wider population. That is, when diaries are completed on a voluntary basis, they may be capturing the lives as lived by only a sub-set of the population.

Discussion of study two: opportunities for educational research

Reflecting on the case of Arndt (2019), the study has illustrated several key challenges and opportunities for the use of event-contingent diaries in educational research, especially where the focus is on learner behaviour outside of the language classroom.

Event-contingent diaries offer a way of collecting rich data on individual instances of language use, and the thoughts and feelings that accompany it, which can be difficult to capture using more traditional, retrospective self-report questionnaires (Arndt et al. 2021). The likelihood of recall biases is diminished, and the accuracy of the collected data is enhanced, by asking learners to complete a report soon after the phenomenon being investigated occurs. Since reports are collected ‘in situ’, event-contingent diaries also allow researchers the explore the influence of contextual factors, such as the setting in which learning takes place. When analysed longitudinally, this data can even be used to study dynamic processes in learning.

The largest challenge associated with the use of diary methods to conduct research in an out-of-class setting is motivating enough participants to consistently engage with the diary during their free time, without teacher or researcher guidance. This can be particularly challenging when the researcher's position is that of an ‘outsider’, who must rely on the students’ goodwill and self-discipline for their continuing participation in the study. To overcome this challenge, we make the following recommendations based on our reflection on the second exemplar study:

• Researchers should consult with their participants about both the format and content of the diary. More learners will be willing to engage with a diary if they can complete it quickly, and if it is in a format that is easy for them to keep ‘on hand’. Furthermore, understanding how participants themselves think and speak about the phenomena of interest can help to design questions in a way that is more likely to yield rich and accurate data.

• Instructions for the participants should be clear and comprehensive to make sure that they know how to identify events which are relevant to the study and should be recorded in the diary. This applies particularly to event-contingent designs, but even in interval- or signal-contingent diary studies, it is important for participants to know what they should focus on in their reports.

• It can be difficult to determine whether lack of engagement with the diary reflects a lack of motivation on the part of the participants or a lack of experience with the phenomena that are being studied. We therefore recommend asking participants to also report the absence of any relevant events (e.g. ‘check in’ once per day in event-contingent designs) to aid the researcher in the interpretation of any missing data.

• Similarly, wherever possible, we recommend that researchers remind participants regularly to engage with the diary. This exemplar study has shown mobile phone reminders to be effective, but teacher-researchers could simply give oral reminders to students they meet regularly.

• Finally, we recommend that researchers reflect on the possible presence of a self-selection bias in their research that can occur when certain individuals choose not to take part in, or to consistently engage with, their study. Such patterns should be taken into account during the data analysis and interpretation of the study findings.

Our second exemplar study has highlighted a number of challenges surrounding participant engagement and self-selection bias that can arise in many different research contexts. While it may not always be possible to evaluate the effect of these issues as directly as in the above study, researchers should strive to recognize, pre-empt, and adapt to issues that can potentially limit or distort their findings. The study has also highlighted strategies which ‘outsider researchers’ can employ to connect more closely with their participants and encourage more consistent engagement with their study.

Conclusion: improving diary methods

Although diary methods have not been widely used in educational research, they offer many opportunities to researchers in this field. They are particularly useful for studying learner behaviour and cognition beyond the classroom from the perspectives of the learners themselves and result in very rich, highly ecologically valid datasets which lend themselves to the application of novel analytical approaches. In this reflection paper, we have contrasted two researchers’ experiences of using diary methods to conduct research with second language learners of English. There were two core features that distinguished these studies: The positionality of the researchers (‘insider’ vs. ‘outsider’) and the nature of the learning tasks which the participants were asked to reflect on (an assigned homework task vs. self-directed informal activities). Our reflections have largely focused on the effect that these differences had on the participants’ engagement with the diaries, which is one of the central challenges of this method.

In the first study, the dual role of the teacher-researcher was helpful in facilitating higher student engagement with the diary by making it part of a compulsory homework task. However, this also posed a threat to the validity of the data, raising the question whether the students were giving responses which they thought their teacher would want to hear (i.e. social desirability bias in the data). In the second exemplar study, the researcher had an ‘outsider’ status, which likely contributed to the low initial engagement with the diary instrument, although this was substantially improved after connecting with the participants more closely via daily mobile phone reminders. Other educational research has highlighted further strategies ‘outsider’ researchers have used to ensure participation in diary methods. Baker (2021), for example, details success with diary engagement by using a co-participatory approach, and negotiating the expectations of diary keeping directly with her participants.

Researchers need to take a measured approach to diary methods throughout all stages of the research—from design to implementation to analysis. Based on our own experiences, and our review of the methodological literature, some questions researchers could contemplate for their own diary studies are:

• How can I minimize recall bias to ensure students are able to complete their diaries accurately?

• What are the literacy demands of the diary and how can these be reduced, while maintaining detail in the data?

• What are the time demands of the diary, and how can these be reduced?

• How can I increase participant buy-in for the research project to ensure adequate engagement with the diaries?

• In event-contingent diary designs, how can I differentiate a lack of engagement with diary keeping from a lack of the phenomena being researched?

• How can I minimize social desirability bias in the data collected by diaries?

• When there is a lack of engagement with the diaries, how can I investigate potential self-selection bias in the sample that completes them?

• In classroom research, how can I ensure ethical use of diaries as both a pedagogical task and a research tool?

Careful researcher reflection on diary methods can strengthen methodological rigour, helping researchers to accommodate, pre-empt, and adapt to issues that might produce misleading data and distort research findings. In educational research, in particular, we need to be cognizant of students’ needs within the larger context of the study in which the diary methods are to be applied, and to strike a careful balance between maximizing participant engagement with the diary and collecting consistent, authentic data—that is, capturing life as it is truly lived.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 Parental consent was obtained, and all phone numbers were completely anonymized (i.e. not linked to any other participant data).