“What Do I Say? How Do I Say it?” Twitter as a Knowledge Dissemination Tool for Mental Health Research

Abstract

This study aims to generate evidence-based guidelines for researchers regarding how to effectively disseminate mental health research via Twitter. Three hundred mental health research Tweets posted from September 2018 to September 2019 were sampled from two large Australian organizations. Twenty-seven predictor variables were coded for each Tweet across five thematic categories: messaging; research area; mental health area; external networks; and media features. Regression analyses were conducted to determine associations with engagement outcomes of Favourites, Retweets, and Comments. Less than half (n = 10) of predictor variables passed validity tests. Notably, conclusions could not reliably be drawn on whether a Tweet featured evidence-based information. Tweets were significantly more likely to be Retweeted if they contained a hyperlink or multimedia. Tweets were significantly more likely to receive comments if they focused on a specific population group. These associations remain significant when controlling for organization. These findings indicate that researchers may be able to maximize engagement on Twitter by highlighting the population groups that the research applies to and enriching Tweets with multimedia content. In addition, care should be taken to ensure users can infer which messages are evidence-based. Guidelines and an accompanying resource are proposed.

Mental and substance use disorders are among the leading causes of burden of disease worldwide, affecting over one billion people globally in 2016 (Rehm & Shield, 2019). Despite the prevalence and impact of these disorders, research-to-practice gaps continue to slow the effective implementation of evidence-based practice and policy (Jorm & Malhi, 2013; Proctor et al., 2009; Wainberg et al., 2017; Wiltsey Stirman, Gutner, Langdon, & Graham, 2016). Currently, the average research-to-practice gap for health research is 17 years (Morris, Wooding, & Grant, 2011). Given this slow model of implementation, mental health knowledge dissemination beyond the academic sphere is an increasing point of focus of funding bodies, the public, and researchers (Chambers, Pintello, & Juliano-Bult, 2020; Proctor et al., 2009).

Social media is the leading alternative metric through which medical research is disseminated to the general public (Adams & Loach, 2015). Twitter (www.twitter.com), a microblogging site, is increasingly the social media site of choice for researchers wanting to share scholarly findings (Adams & Loach, 2015). Positive associations between Tweets and article citation rate have been observed across multiple fields of health research, with highly Tweeted articles more likely to be highly cited (Clavier et al., 2020; Eysenbach, 2011; Hayon et al., 2019; Ladeiras-Lopes, Clarke, Vidal-Perez, Alexander, & Luscher, 2020). A review of research dissemination and implementation science resources from 2015 to 16 found that 60% of resources involved social media, with Twitter being the most common (Darnell et al., 2017). Researchers using Twitter can “Tweet” up to 280 characters, including hyperlinks, hashtags and “mentions” (i.e., a Tweet that contains a hyperlink to another account by mentioning their username). The Tweet can be accompanied by enriched media content such as images and videos. Other Twitter users can then “follow” individuals or organizations. Users with higher follower counts have more visible Tweets to the wider Twitter user base.

In April 2023, Twitter was renamed “X,” and shortly after launched a paid subscription option (“Twitter Blue” or “X Premium”) that gives users access to additional features such as the ability to Tweet up to 10,000 characters. As of August 2023 these premium features have been accessed by a small minority (<0.2%) of users, the website name has remained “Twitter,” and nomenclature associated with the website (e.g., “Tweeting” and “Retweeting”) is still widely used across the platform (Binder, 2023a). While the content features of a Tweet (i.e., character limit) have remained very similar since the launch of “X,” the deregulation of content moderation (i.e., what the platform will allow in the messaging of a Tweet) appears to have negatively impacted user engagement. A survey of scientists using Twitter (n = 9,200) released by Nature in August 2023 showed more than half of respondents had reduced their Twitter usage in the past six months, with many reporting this was due to an increase in fake accounts and hate speech on the platform (Vidal Valero, 2023). This loss of scientists also appears to be reflective of Twitter’s broader userbase, with usage statistics shared by Twitter in September 2023 reporting a loss in daily active users ranging between 3.7–11.6% since November 2022 (Binder, 2023b).

Despite the recent loss of Twitter users, leading alternatives such as Threads and Mastodon are currently far less utilized than Twitter. Threads launched in July 2023 with an initial 2.3 million active users, which sharply declined to 576,000 in August 2023 (Clayton, 2023). Mastodon, launched in 2016, experienced a large increase in daily active users following the launch of “X” (from 1.17 million in April 2023–1.71 million as of September 2023), yet this is still a small fraction of Twitter’s 200–250 million daily active users as of September 2023 (Binder, 2023b; Mastodon, 2023). While the changes to Twitter following the launch of “X” have impacted Twitter’s userbase, it currently remains a leading social media platform.

It is estimated that 1 in 5 Australians and Americans use Twitter (Auxier & Anderson, 2021; Yellow, 2020). On average, users will spend 4 hours per week on the platform, and 38% will Tweet at least once during this time (Yellow, 2020). Recent findings indicate Twitter users have higher than average levels of education (with 42% of adult Twitter users holding a Bachelor degree or higher), which may make them a receptive social media audience to health research findings (Pew Research Centre, 2019).

However, due to its popularity and accessibility, Twitter has also been recognized as a source of misinformation. For example, a 2021 systematic review of health misinformation across a number of social media platforms (n = 69 studies) found that misinformation was most common on Twitter (Suarez-Lledo & Alvarez-Galvez, 2021). The dissemination of medical misinformation via Twitter also received increased attention as a result of the COVID-19 pandemic (Kouzy et al., 2020; The Lancet Digital, 2023; Petersen & Gerken, 2021; Scannell et al., 2021; Wang & Lu, 2022). A content analysis of 673 COVID-19 related Tweets sampled in February 2020 found that 1 in 4 (24.8%) contained misinformation (Kouzy et al., 2020), and researchers also noted an increase in preprinted publications that had not yet been peer reviewed being shared via Twitter (Carlson, Harris, & Blekhman, 2020). The growth of medical misinformation on Twitter resulted in the implementation of regulations to identify Tweets containing COVID-19 misinformation (such as the use of written warnings beneath Tweets), however these warnings have now been removed as COVID-19 cases continue to decline (Susarla, 2022). The struggle to contain the spread of medical misinformation during a global pandemic has continued to fuel debate about how to most effectively control medical misinformation on social media platforms, with a number of recent studies highlighting the need for individuals or organizations with expertise to provide reliable sources of information on social media that are engaging, while also delineating their content as evidence-based (Bin Naeem, Bhatti, & Khan, 2020; Chou, Gaysynsky, & Cappella, 2020; Muhammed & Mathew, 2022; Petersen & Gerken, 2021; Walter, Brooks, Saucier, & Suresh, 2020).

Despite this increased need for evidence-based voices on social media, and the potential of Twitter to be a powerful knowledge dissemination tool for health researchers and clinicians, there is a lack of guidance for how to use Twitter to effectively disseminate mental health research specifically.

Existing studies of mental health and Twitter have primarily focused on analyzing public discourse around mental health topics (e.g., people sharing their experiences of living with mental illness, analyzing languages changes relating to mental health in Tweets over time) (Berry et al., 2017; McClellan, Ali, Mutter, Kroutil, & Landwehr, 2017; Stupinski et al., 2022), or Twitter as a tool for symptom detection (e.g., assessing the severity of depression, PTSD or suicidality from Tweet content) (Azam, Agro, Sami, Abro, & Dewani, 2021; Coppersmith, Harman, & Dredze, 2014; O’Dea et al., 2015). These areas of focus are also reflected in studies of Twitter and health more broadly (Sinnenberg et al., 2017), and are distinct from mental health research Tweets, which focus more on sharing developments or findings from mental health research projects and/or project output such as publications.

The neglected questions of “what do I say” and “how do I say it” require addressing to guide researchers using Twitter toward higher content engagement and more impactful knowledge dissemination.

The aims of the current study are twofold;

1. To understand how the written text of mental health research Tweets is associated with user engagement (“What do I say?”)

2. To understand how content features (such as hashtags and external links) of mental health research Tweets are associated with user engagement (“How do I say it?”)

Methods

In the current study, the Tweets of two mental health research organizations were collected, coded, and correlations between content features (e.g., hashtags, subject matter, use of media) and user engagement (e.g., Retweets, Comments) were explored.

Organization Selection

Twitter has no publicly available indexed list that could be used to identify mental health research organizations with the largest number of followers. As such, the authors designed a selection process utilizing the Australian Charities and Not-for-profits Commission (ACNC) charity register. It should be noted that most Australian universities and medical research institutes are included in this list. To be included in the current study, organizations needed to be operating in all Australian states; be focused primarily on mental health and conducting mental health research; and have an active Twitter account.

From this, the two Australian mental health research organizations with the largest Twitter followings were selected (Figure 1). At the time of Tweet selection, the selected organizations Twitter accounts had a total of 38,267 and 8,841 followers, respectively.

Figure 1. Organization selection process.

Tweet Selection

All Tweets from both organizations for the period of September 2018 – September 2019 were extracted using the online tool Followers Analysis (www.followersanalysis.com). This time period was selected to ensure results would be current but limited the impact that national and international emergencies such as the Australian bushfires (beginning November 2019) or the COVID-19 pandemic (beginning March 2020) had on the generalizability of findings. Researchers (EM, LT) reviewed the initial sample of 1,324 Tweets from both organizations, and following a selection process (Figure 2), a final sample of 300 Tweets (178 from Organization 1 and 122 from Organization 2) were selected for content analysis.

Figure 2. Tweet selection process.

Data Collection and Outcome Measures

Engagement

User engagement with selected Tweets was the primary outcome for this study. Based on similar studies, four publicly available engagement metrics for each Tweet were considered when measuring engagement (Kim et al., 2017; Park, Reber, & Chon, 2016). These included number of Favourites (i.e., a user “liking” a Tweet), Video Views, Comments and Retweets (i.e., a user sharing a Tweet). The number of likes and Retweets a Tweet received was extracted from the Followers Analysis tool while number of Video Views and Comments were manually coded by EM, SGB, TG, ZB and CW.

Tweet Content Analysis

Content considered for coding and analysis was limited to the text contained within each Tweet. This encompassed hashtags, mentions, and any hyperlinks included within the text, as well as whether media content accompanied the Tweet or not (but not the content of accompanying media).

An initial codebook to extract key features of Tweets was developed by LT and trialed among a random selection of 30 Tweets independently coded by SGB, TG, ZB, CW, KP, SN, and CW. Coders met to discuss any conflicts and refine the initial codebook. The revised codebook (see Table 1) was then used to code the final sample of Tweets (n = 300).

Table 1. Content codebook for research tweets

	THEMATIC CATEGORY	PREDICTOR VARIABLES (n = 27)
“What do I say?”	MESSAGING	Tweet promotes research results or evidence-based information
		Tweet promotes research program or tool
		Tweet contains announcement
		Tweet features participant recruitment for study
		Tweet includes call to action
		Tweet includes statistic
	RESEARCH AREA	Research area is prevention or early intervention
		Research area is epidemiology
		Research area is treatment
		Research area is translation or implementation
		Research area unclear or NOS
	MENTAL HEALTH AREA	Mental health area equals general or NOS
		Mental health area equals anxiety
		Mental health area equals depression
		Mental health area equals bipolar, schizophrenia or psychosis
		Mental health area equals suicide
		Mental health area equals OCD
		Mental health area equals ADHD or autism
		Mental health area equals substance use
		Mental health area equals trauma
		Mental health area equals general wellbeing
		Tweet focuses on mental health of specific population group
		Tweet features lived experience of mental health
“How do I say it?”	EXTERNAL NETWORKS	Tweets mentions other accounts
		Tweet uses hashtags
	MEDIA FEATURES	Tweet includes media (e.g., image, video)
		Tweet includes hyperlink

NOS = Not otherwise specified

Twenty-seven predictor variables were coded for each Tweet across five thematic categories, which mapped on to the primary aims of the study: Messaging, Research Area, and Mental Health Area analyzed the written text of Tweets (“what do I say?”), whereas External Networks and Media Features analyzed the content features associated with Tweets (“how do I say it?”). Notably, certain content within Tweets was coded across multiple thematic categories. For example, written text that was part of a hashtag (e.g., #depression) was considered part of “what” the user was saying (coded under the thematic categories of Messaging, Research Area and Mental Health Area), as well as “how” the user had chosen to utilize content features to say it (i.e., coded as “Tweet uses hashtags” under the thematic category of External Network).

The authors acknowledged that due to the issue of misinformation on Twitter, further guidance was needed to discern which Tweets were “evidence-based” when coding. For the predictor variable “Tweet promotes research results or evidence-based information,” coders assessed Tweet text only to identify whether the information had come from an evidence-based source (e.g., a peer-reviewed article, conference presentation, or clinical guidelines).

EM and TG initially coded 150 tweets each. Predictor variables in the thematic categories of Messaging, Research Area and Mental Health Area (e.g., “Tweet promotes research program or tool”) were found to be subjective predictor variables with lower levels of initial agreement and as such, underwent inter-rater coding by a second reliability coder (SGB or ZB) to establish validity and reliability.

As inter-rater reliability scores are affected by marginal distributions (Syed & Nelson, 2015), subjective predictor variables underwent a frequency check, and variables present in < 10% or > 90% of the total 300 Tweets across the total primary coder dataset were eliminated. Remaining subjective predictor variables with a Cohen’s Kappa < 0.6 were determined to have poor reliability and were eliminated from further analysis.

Conversely, predictor variables in the thematic categories of External Network and Media Features (e.g., “Tweet mentions other accounts”) were found to be objective predictor variables with a high degree of initial agreement, and as such did not undergo inter-rater reliability tests.

All results were calculated using SPSS (Statistical Package of Social Sciences, Chicago, IL, USA) software version 27.

Regressions for Engagement Outcome Variables

Following coding, remaining predictor variables determined to be valid and reliable were considered for regression, performed by SO. As two of the dependent engagement outcome variables were non-normally distributed count data (i.e., one Retweet, Favourite), Poisson and negative binomial regression models were considered. To test the robustness of significant findings, “organization” was added as a covariate in the models, to account for potential confounds on the dependent engagement outcome variables. Statistics were calculated using R version 4.0.3 (R Core Team, 2021). MASS (v7.3–53; Venables & Ripley, 2002) and foreign (v0.8–80; R Core Team, 2020) packages were used for the Poisson, negative binomial, and binary logistic regression model analyses. Correlations between all predictors, and variance inflation factors for all models were investigated to ensure that multicollinearity assumptions were not violated for regression models.

Results

Primary coder data results (Table 2) showed that for the thematic category of Messaging, Tweets promoting research results or evidence-based information were most frequent (50.0% of Tweets), followed by an announcement (43.0%), most often about collaborations with other research groups or external organizations. For Research Area, Tweets were most likely to be unclear or not otherwise specify the area of research the Tweet related to (48.3%), followed by treatment (22.0%), translation/implementation (18.7%), and prevention/early intervention (17.3%). Nearly two-thirds of Tweets (62.0%) did not specify a mental health condition as their focus, instead referring more generally to mental health research. A majority of Tweets used hashtags (90.0%), media (83.3%), and hyperlinks (78.3%), and just over half of Tweets mentioned another account (56.3%). On average, the organizations used two hashtags (m = 2.2, SD = 1.34) and mentioned one account (m = 1.3, 1.3 SD = 1.27) per Tweet.

Table 2. Predictor variable frequencies and reliability coding results

		Frequency		Reliability coding
THEMATIC CATEGORY	PREDICTOR VARIABLES	(Tweets = 300)		Set 1 (Tweets = 150)			Set 2 (Tweets = 150)			Combined av. (Tweets = 300)
		n	%	Kappa	Std Err	p	Kappa	Std Err	p	Av. Kappa	Av. Std Err
MESSAGING	Tweet promotes research results or evidence-based information	150	50.0%	0.54	0.06	<.001	0.57	0.67	<.001	0.56	0.37
	Tweet promotes research program or tool	82	27.3%	0.61	0.11	<.001	0.31	0.65	<.001	0.46	0.38
	Tweet contains announcement*	129	43.0%	0.65	0.06	<.001	0.57	0.68	<.001	0.61	0.37
	Tweet features participant recruitment for study	26	8.7%	-	-	-	-	-	-	-	-
	Tweet includes call to action*	104	34.7%	0.45	0.08	<.001	0.77	0.54	<.001	0.61	0.31
	Tweet includes statistic	6	2.0%	-	-	-	-	-	-	-	-
RESEARCH AREA	Research area is prevention or early intervention	52	17.3%	0.70	0.07	<.001	0.40	0.96	<.001	0.55	0.52
	Research area is epidemiology	8	2.7%	-	-	-	-	-	-	-	-
	Research area is treatment	66	22.0%	0.56	0.08	<.001	0.34	0.99	<.001	0.45	0.54
	Research area is translation or implementation	56	18.7%	0.19	0.10	.014	0.74	0.68	0.128	0.46	0.39
	Research area unclear or NOS	145	48.3%	0.48	0.07	<.001	0.24	0.76	<.001	0.36	0.41
MENTAL HEALTH AREA	Mental health area equals general or NOS*	186	62.0%	0.70	0.06	<.001	0.67	0.61	<.001	0.68	0.33
	Mental health area equals anxiety	8	2.7%	-	-	-	-	-	-	-	-
	Mental health area equals depression*	31	10.3%	0.93	0.05	<.001	0.96	0.04	<.001	0.95	0.04
	Mental health area equals bipolar, schizophrenia or psychosis	15	5.0%	-	-	-	-	-	-	-	-
	Mental health area equals suicide*	49	16.3%	0.95	0.04	<.001	0.93	0.04	<.001	0.94	0.04
	Mental health area equals OCD	0	0.0%	-	-	-	-	-	-	-	-
	Mental health area equals ADHD or autism	5	1.7%	-	-	-	-	-	-	-	-
	Mental health area equals substance use	10	3.3%	-	-	-	-	-	-	-	-
	Mental health area equals trauma	6	2.0%	-	-	-	-	-	-	-	-
	Mental health area equals general wellbeing	26	8.7%	-	-	-	-	-	-	-	-
	Tweet focuses on mental health of specific population group*	137	45.7%	0.53	0.07	<.001	0.73	0.56	<.001	0.63	0.31
	Tweet features lived experience of mental health	2	0.7%	-	-	-	-	-	-	-	-
EXTERNAL NETWORKS	Tweets mentions other accounts**	169	56.3%
	Tweet uses hashtags**	270	90.0%
MEDIA FEATURES	Tweet includes media**	250	83.3%
	Tweet includes hyperlink**	235	78.3%

NOS = Not Otherwise Specified

*Subjective predictor variables found to be valid and reliable, underwent further regression analysis

**Objective predictor variables not requiring validity or reliability testing, underwent further regression analysis

The most common hashtags used were #mentalhealth, #suicideprevention, #youthmh, and #depression. Mentioned accounts were most likely to be researchers affiliated with the organization such as directors or lead researchers.

Eleven subjective predictor variables were eliminated due to low frequency (present in < 10% of Tweets), with eight of these belonging to the Mental Health Area thematic category. No variables were eliminated for high frequency (>90%).

The remaining 12 subjective predictor variables underwent reliability coding (Table 2).

Average kappa scores between pairs of coders showed variables in the Mental Health Area category had the highest scores; depression and suicide subjective predictor variables both reached significance and had κ scores of > 0.90. Variables in the Research Area thematic category were all found to be unreliable (κ <0.6), and this category was eliminated from further analysis. For the category of Messaging, coders did not reliably agree on whether a Tweet featured research results or evidence-based information (κ = 0.56), or whether the Tweet promoted a research program or tool (κ = 0.46).

Following validity and reliability testing, 10 predictor variables (six subjective predictor variables and four objective predictor variables) remained for regression analyses (Table 3).

Table 3. Regression results for predictor variables found to be valid and reliable

		Retweets			Comments			Favourites
THEMATIC CATEGORY	PREDICTOR VARIABLES	IRR	95% CI	p	OR	95% CI	p	IRR	95% CI	p
MESSAGING	Tweet contains announcement?	0.99	0.79, 1.23	0.891	1.69	0.89, 3.25	0.110	1.19	0.95, 1.49	0.117
	Tweet includes call to action	0.89	0.69, 1.13	0.333	1.65	0.79, 3.48	0.180	0.90	0.70, 1.16	0.413
MENTAL HEALTH AREA	Mental health area equals general or NOS	0.97	0.72, 1.30	0.836	1.87	0.79, 4.83	0.170	1.27	0.95, 1.69	0.086
	Mental health area equals depression	0.92	0.62, 1.39	0.683	1.30	0.37, 4.38	0.670	1.24	0.83, 1.86	0.282
	Mental health area equals suicide*	1.39	0.98, 1.96	0.053	2.75	1.00, 7.87	0.052	1.59	1.13, 2.24	0.007
	Tweet focuses on mental health of specific population group?*	1.12	0.89, 1.41	0.331	2.20	1.10, 4.52	0.028	0.98	0.78, 1.24	0.883
EXTERNAL NETWORKS	Tweets mentions other accounts?	0.94	0.76, 1.15	0.539	1.33	0.70, 2.56	0.390	1.13	0.91, 1.39	0.269
	Tweet uses hashtags?*	0.86	0.60, 1.22	0.400	0.41	0.14, 1.22	0.100	0.63	0.43, 0.89	0.011
MEDIA FEATURES	Tweet includes media?*	1.32	1.00, 1.74	0.045	0.87	0.39, 2.06	0.750	1.19	0.90, 1.55	0.22
	Tweet includes hyperlink?*	2.05	1.53, 2.75	<0.001	0.76	0.34, 1.73	0.500	0.90	0.68, 1.19	0.456

NOS = Not Otherwise Specified, IRR = Incidence Rate Ratio, OR = Odds Ratio, CI = Confidence Interval

* <0.05

All engagement outcome variables had a high level of variability. Videos were only present in eight (2.6%) Tweets, with a mean view count of 1,207 (SD = 734.71). Due to the low frequency of videos and high variation in view count, this engagement outcome variable was eliminated from further analysis. Commenting occurred in 20.3% of Tweets, with a Tweet receiving less than one comment on average (m = 0.49, SD = 1.76). This engagement outcome variable was converted to a “comment/no comment” dichotomous variable for binary logistic regression. Favouriting (i.e., a user “liking” a Tweet) occurred in 96.3% of Tweets, with an average Favourite count of 16 (m = 16.39, SD = 17.97). Retweets occurred in 92.3% of Tweets, with an average Retweet count of 7 (m = 7.19, SD = 7.17). Both Retweets and Favourites were over-dispersed, and negative binomial models had smaller fit indices for both engagement outcome variables. As such, negative binomial models were used for hypothesis testing.

Even after adjusting for the effect of organization (Table 3), Tweets were significantly more likely to receive Retweets if they contained a hyperlink (IRR = 2.05 [CI 1.53–2.75], p = <.001) or included media (IRR = 1.32 [CI 1.00–1.74], p = .045). Tweets were significantly more likely to receive Comments if they focused on the mental health of a specific population group (OR = 2.20 [CI 1.10–4.52], p = .028), and were significantly more likely to receive Favourites if the mental health area was suicide (IRR = 1.59 [CI 1.13–2.24], p = .007). Tweets were significantly less likely to receive Favourites if they used hashtags (IRR = 0.63 [CI 0.43–0.89], p = .011).

Discussion and Conclusion

What Do I Say?

Variables in the “what do I say” thematic categories of Messaging, Mental Health Area, and Research Area were largely found to have no significant impact on engagement. Exceptions were Tweets focusing on the mental health of specific population groups (e.g., youth or emergency workers) being significantly more likely to receive Comments, and Tweets focusing on suicide being significantly more likely to receive Favourites.

These findings aren’t consistent with other content analysis studies focused on health-related Tweets, where results indicate that the written text of a Tweet is associated with engagement across a number of content variables (Ju, Jia, & Cheng, 2023; Nastasi et al., 2018; Scannell et al., 2021; So et al., 2016; Wang & Lu, 2022). Only one study was found that analyzed the association between Tweet content and engagement for mental health Tweets specifically (Ju, Jia, & Cheng, 2023), and no studies of this kind were found that focused exclusively on mental health research Tweets. The content analysis study from Ju, Jia, and Cheng (2023) consisted of 5,361 Tweets from three US mental health organizations posted between 14 November 2018 and –14 November 2019. Researchers analyzed Tweet content through categorical variables similar to our study (e.g., “mentions specific mental illness” and “includes call to action”) (Ju, Jia, & Cheng, 2023).

Analysis of variance tests in Ju, Jia, and Cheng (2023) showed that Tweets mentioning a specific mental illness or mental health in general were more significantly more likely to receive Retweets, Comments and Favourites than Tweets that did not mention mental health. Tweets featuring a call to action were significantly more likely to receive Comments. The top three Retweeted and Favourited topics were stigma, mental health care and suicide/suicide prevention, and the top three Commented topics were stigma, mental health care and statistical overview (Ju, Jia, & Cheng, 2023).

This contrasts with our findings that only Tweets mentioning suicide were significantly more likely to receive Favourites, yet Tweets that mentioned mental health more generally did not receive more Retweets, Comments or Favourites than Tweets that did not mention mental health (e.g., Tweet announced researcher received an award, but it was unclear what for). We also found no consistency in the Messaging or Mental Health Area variables for our top three Retweeted, Favourited or Commented Tweets, and no significant association between Tweets featuring a call to action and engagement variables. The disparity between our findings and Ju, Jia, and Cheng (2023) may indicate that the impact messaging has on engagement may differ between more general mental health Tweets (e.g., advocating against stigma or sharing experiences of care), and mental health research Tweets specifically, where messaging may focus more on factual information or key findings.

Notably, both our study and Ju, Jia, and Cheng (2023) found that Tweets mentioning suicide had high rates of engagement (Ju, Jia, & Cheng, 2023). Contextualizing this finding among broader studies focusing on social media and suicide is difficult, as both our study and Ju, Jia, and Cheng (2023) sampled Tweets from mental health organizations, whereas a majority of existing studies on this topic sample Tweets by hashtags or Tweet text. This method results in Tweet samples dominated by individuals sharing their experiences of suicide (i.e., attempts, ideation or the loss of a loved one), and studies focused on evaluating risk, symptom detection or peer support (Colombo, Burnap, Hodorog, & Scourfield, 2016; Cote et al., 2021; Jeri-Yabar et al., 2019; Rabani, Khan, & Khanday, 2020; Sinyor et al., 2021). The minority of studies evaluating how sharable suicide-related Tweet content is often equate shareability with a “contagion” effect (e.g., the “Blue Whale challenge”) (Khasawneh et al., 2020; Upadhyaya & Kozman, 2022), and this content is not comparable with our Tweet sample.

The increased engagement with suicide-related Tweets in both our study and Ju, Jia, and Cheng (2023) may be indicative of the high prevalence of suicide, which remains among the top four causes of death in Australia and the US for people aged 10–50 (Australian Institute of Health and Welfare, 2021; Centers for Disease Control and Prevention, 2022). Similarly, both our study and Ju, Jia, and Cheng (2023) found Tweets mentioning mental health in general were more common than Tweets mentioning specific mental illnesses, and that highly prevalent mental illnesses such as depression were more represented than less prevalent illnesses such as psychosis or OCD. Further research is needed with larger sample sizes from multiple mental health organizations to make sufficiently powered head-to-head comparisons between what drives engagement for mental health Tweets focused on different topics such as research, advocacy, or specific mental illnesses.

Our finding that researchers coding Tweets could not reliably agree on whether Tweets were evidence-based is representative of the broader issue of misinformation on social media, and how researchers can best delineate their content as evidence-based (Bin Naeem, Bhatti, & Khan, 2020; Chou, Gaysynsky, & Cappella, 2020; Muhammed & Mathew, 2022; Petersen & Gerken, 2021; Walter, Brooks, Saucier, & Suresh, 2020).

Although some Tweets appeared to be “evidence-informed” (i.e., a researcher commenting on their area of expertise), the source of the statement was unclear in the majority of Tweets. This raises further questions about the distinction between content that is easily recognizable as “misinformation” by the general public and researchers or experts (e.g., Tweets claiming COVID-19 vaccines have a high mortality rate) and “poor quality information,” where the level of evidence supporting a Tweet is not recognizable even to those trained to assess it. The inability of Twitter users to clearly communicate the quality of evidence supporting their content may be a constraint of the “microblogging” nature of Twitter, as well as authors not wanting to alienate the general public by citing study design and/or levels of evidence (e.g., NHMRC levels of evidence or the GRADE framework) (Balshem et al., 2011; National Health and Medical Research Council, 2009). A preliminary review of guidance available for researchers or health professionals using Twitter (most of which was not peer-reviewed) did not specify how users can best signal that Tweets are evidence-based within the limited character count available (Centers for Disease Control and Prevention, 2021; John Hopkins Medicine, 2021; Mollett, Moran, & Dunleavy, 2011; Schnitzler, Davies, Ross, & Harris, 2016; Schroeder & Bauman, 2019; Taylor & Francis Group, 2021).

How Do I Say It?

Variables in thematic categories relating to “how do I say it,” such as Media Features and External Networks had more of an impact on engagement. Tweets containing media (such as images, gifs, videos or polls) and hyperlinks had higher rates of engagement. Despite a majority of Tweets in our sample using hashtags (90.0%), the presence of hashtags were correlated with fewer Favourites, and mentioning other user accounts (present in 56.3% of Tweets) was not significantly associated with engagement outcomes.

Our finding that Tweets enriched with multimedia have higher engagement is consistent with existing health and mental health-focused Twitter studies (Chapman & Freeman, 2015; Ju, Jia, & Cheng, 2023; Schlichthorst et al., 2018; Veale et al., 2015; Wadhwa, Latimer, Chatterjee, McCarty, & Fitzgerald, 2017), and best-practice guides outside of peer-reviewed literature for researchers (Mollett, Moran, & Dunleavy, 2011; Taylor & Francis Group, 2021), health professionals (John Hopkins Medicine, 2021; Schroeder & Bauman, 2019) and businesses (Hootsuite Inc, 2021; Twitter, 2021). For example, a recent analysis of 1,032 Tweets from the American Journal of Neuroradiology found that Tweets accompanied by an image were significantly more likely to have an engagement score in the top 25^th percentile (OR = 28.75 [17.654–46.816], p < .001) (Wadhwa, Latimer, Chatterjee, McCarty, & Fitzgerald, 2017). In addition, using multimedia also extends the space available to communicate research findings; images and hyperlinks often contain additional written or visual information that appears beneath the 280 characters of the Tweet to form a more complete message. However, the finding that hashtags were associated with lower levels of engagement contrasts with other health-focused Twitter studies (Elkbuli, Santarone, Boneva, Hai, & McKenney, 2021; Logghe, Pellino, Brady, McCoubrey, & Atallah, 2016; Moorley & Chinn, 2014; Petersen & Gerken, 2021; Wadhwa, Latimer, Chatterjee, McCarty, & Fitzgerald, 2017) and non-peer reviewed guidelines (Hootsuite Inc, 2021; John Hopkins Medicine, 2021; Mollett, Moran, & Dunleavy, 2011; Schroeder & Bauman, 2019; Taylor & Francis Group, 2021; Twitter, 2021) which predominantly encourage the use of hashtags and mentioning accounts to increase user engagement via health events (e.g., conferences or awareness campaigns) (Elkbuli, Santarone, Boneva, Hai, & McKenney, 2021; Schlichthorst et al., 2018), specific health communities (e.g., clinician or patient groups) (Logghe, Pellino, Brady, McCoubrey, & Atallah, 2016; Moorley & Chinn, 2014), or pertinent public health conversations (e.g., COVID-19) (Petersen & Gerken, 2021). Notably, the three top scoring Tweets from our sample (total Retweet, Favourite, Comment count > 100) all contained a hyperlink and an image, yet two of the Tweets contained 2–3 hashtags each. Considering this, as well as the context of this finding in the broader literature, a more purposeful use of hashtags may be appropriate.

To our knowledge, this is the first study to undertake an extensive analysis of which content features of mental health research Tweets are associated with user engagement. From 300 mental health research Tweets of two nationally representative Australian mental health research organizations, preliminary guidelines for mental health researchers have been developed that may maximize engagement while also retaining research integrity (Table 4).

Table 4. Preliminary guidelines for mental health researchers using twitter

What do I say?	How do I say it?
1. Tell the audience what the information is, and where it came from (i.e., your key finding and the evidence-basis is clear) 2. Tell your audience who you’re speaking to (i.e., if your finding relates to a specific population group, make it clear)	Use media that adds to the Tweet, expanding on what the information is and where it came from (e.g., images, videos) Use hyperlinks to verify research information in Tweet Minimize over-use of hashtags (i.e., seek out hashtags that relate to specific events, communities or conversations)

To guide researchers in delineating their Tweets as evidence-based, we also contribute preliminary guidelines that advocate for researchers to consistently answer users’ questions of “what” (is the information), “where” (has it come from) and “who” (is it for) with combinations of text, images and hyperlinks to the source for more information. To facilitate the uptake and implementation of these recommendations, we created a more extensive guidelines resource that features examples of the guidelines in practice using “mock Tweets” (Table 5). These mock Tweets are inspired by Tweets from our dataset, and display common examples of how real Tweets did or did not utilize the recommendations made in the guidelines. In addition, we created examples of how the same Tweet might look after all the recommendations in the guidelines had been implemented. The guidelines are freely available via the Matilda Centre for Research in Mental Health and Substance Use website (https://bit.ly/twitter-guidelines-mh-research).

Table 5. An example of the guidelines in practice using mock tweets

Not using guidelines	Using guidelines
Evidence basis of information is unclear (what is the information, and where did it come from?)	Evidence basis of information is clear (prevalence of trauma in young people, comes from analysis of a clinical trial)
Image used adds nothing to Tweet	Image used adds to Tweet, shows information is also based on expert opinion
Hyperlink to verify research information in Tweet	Hyperlink to verify research information in Tweet
Specific population group that findings relate to is clear (young people)	Specific population group that findings relate to is clear (young people)
Hashtags are not over-used (relate to specific conversations)	Hashtags are not over-used (relate to a specific conversation)

Limitations

We acknowledge the limitations of this study, most notably that the 300 Tweets were sampled from two organizations within the broader field of mental health research. Although attempts were made to control for organization-specific factors during the selection process and analysis, organizational factors that might influence engagement are diverse (e.g., organizational size, resources, and offline community engagement). Although the organizations selected had a broad portfolio of research, their focus on specific mental health areas (e.g., suicide) or population groups (e.g., youth) resulted in other prevalent mental disorders such as anxiety, ADHD, and substance use being under-represented in the sample of Tweets. Further research is needed to determine how these factors impact engagement with Tweets.

In addition, whether mental health research Tweets perform better or worse than Tweets not related to research authored by the same organizations was beyond scope of this paper. Similarly, while the Tweet codebook was extensive, the content of media accompanying Tweets such as images and hyperlinks, and whether the sentiments in Comments and Retweets were supportive or critical of the author’s messaging, were also beyond scope. Further research is needed to determine the sentiments expressed by users who engage with mental health research Tweets.

Whether applying our preliminary guidelines for “what do I say” and “how do I say” is associated with increased engagement or ability to recognize evidence-based content also requires further research. The authors recommend a comparator-control design using similar “before and after” mock Tweets provided in the guidelines. Exploring the impact of the guidelines in practice on Twitter, as well as their translational value to other social media platforms (e.g., Facebook, Instagram or TikTok) is an area for future investigation.

Finally, while social media platforms have rapidly evolved over the past few decades and may undergo further change, the use of microblogging platforms as a tool for disseminating research will likely continue to grow alongside an ever-expanding user base. The content-focused nature of our proposed guidelines (“what” is the information, “where” has it come from, “who” is it for) have potential to be used across this changing social media landscape, serving as valuable principles for researchers looking to disseminate findings to a more general audience while also retaining research integrity.

Conclusion

Our findings produced preliminary guidelines and an accompanying resource for mental health researchers using Twitter to disseminate findings. Researchers may be able to maximize engagement by highlighting the population groups that the research applies to (e.g., youth or emergency workers) and enriching Tweets with multimedia content (e.g., images or hyperlinks). In addition, care should be taken to ensure users can infer which messages are evidence-based, and over-use of hashtags should be avoided.

Acknowledgments

KP received salary support from an Australian Rotary Health Royce Abbey Postdoctoral Fellowship. The authors would like to thank Professor Maree Teesson for her ongoing encouragement and unwavering support for new ideas.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data underlying this article will be shared on reasonable request to the corresponding author.

Additional information

Funding

This work was supported by the PREMISE NHMRC Centre of Research Excellence in Prevention and Early Intervention in Mental Illness and Substance Use [APP1134909].