WordPPR: A Researcher-Driven Computational Keyword Selection Method for Text Data Retrieval from Digital Media

ABSTRACT

Despite the increasing use of digital media data in communication research, a central challenge persists – retrieving data with maximal accuracy and coverage. Our investigation of keyword-based data collection practices in extant communication research reveals a one-step process, whereas our cross-disciplinary literature review suggests an iterative query expansion process guided by human knowledge and computer intelligence. Hence, we introduce the WordPPR method for keyword selection and text data retrieval, which entails four steps: 1) collecting an initial dataset using core/seed keyword(s); 2) constructing a word graph based on the dataset; 3) applying the Personalized PageRank (PPR) algorithm to rank words in proximity to the seed keyword(s) and selecting new keywords that optimize retrieval precision and recall; 4) repeating steps 1–3 to determine if additional data collection is needed. Without requiring corpus-wide sampling/analysis or extensive manual annotation, this method is well suited for data collection from large-scale digital media corpora. Our simulation studies demonstrate its robustness against parameter choice and its improvement upon other methods in suggesting additional keywords. Its application in Twitter data retrieval is also provided. By advancing a more systematic approach to text data retrieval, this study contributes to improving digital media data retrieval practices in communication research and beyond.

Digital media data refer to “data that are collected, extracted, gathered, or scraped from a web-based platform such as a website, social networking site, mobile application, or another virtual space” (Lukito et al., 2023). Such naturally occurring, massive, and nuanced data can help communication researchers gain unprecedented insights into human communication and social interaction, complementing traditional research methods like surveys and experiments. Despite the various sources (e.g., social network sites, discussion forums, and news media sites) and modalities (e.g., texts, visuals, audios, and videos) of digital media data, researchers often follow common analytical steps in practice, which include retrieving data, cleaning and preprocessing data, and applying methods suitable for research objectives (Grimmer & Stewart, 2013). The retrieval of a relevant and comprehensive dataset is a critical first step because it significantly impacts downstream analysis and results (Antoniak & Mimno, 2021; Kim et al., 2016). Yet, this crucial step is often overlooked in digital media research in the field of communication, potentially leading to research limitations (Moreno-Ortiz & García-Gámez, 2023).

Take text data, arguably the primary type of digital media data in communication research. Though many advanced Natural Language Processing (NLP) methods have been frequently applied (e.g., BERT developed by Devlin et al., 2018), relatively little attention has been paid to the data retrieval step. A prevalent practice in communication research is to use a set of keywords as Boolean search strings, informed by domain expertise or prior literature. However, neither guarantees optimal data quality (Antoniak & Mimno, 2021) and problematic keyword selection can lead to biased data collection and compromised research validity (He & Rothschild, 2016; Mahl et al., 2022).

In this article, we synthesize research from social, computer, and information sciences and propose a new method of keyword selection for text data retrieval. Three key insights from related literature form the bedrock of this method: 1) keywords need to be incrementally expanded to capture multifaceted discourses in digital media spaces (e.g., Krovetz & Croft, 1992); 2) the keyword expansion process should harness both human knowledge and computer intelligence (e.g., King et al., 2017; Makki et al., 2018); and 3) word co-occurrence graphs can effectively represent the semantics of text (Mihalcea & Tarau, 2004). Accordingly, we introduce the WordPPR method – an iterative, researcher-driven computational approach to keyword selection for text data retrieval from digital media corpora. This approach incorporates researcher and computer input in an iterative process of (computer) generating, (researcher) evaluating, and (researcher) selecting and expanding keywords. It begins with a set of seed word(s) or phrase(s) identified by the researcher as the starting point, collects data using the seed(s), constructs a word graph based on word co-occurrence, and suggests the most relevant and least ambiguous words or phrases associated with the seed(s) for the researcher to choose from, all of which can be repeated to achieve an optimal level of retrieval precision and recall. Compared to other methods that require processing or sampling an entire corpus or annotating sample data (e.g., Kuzi et al., 2016, Linder, 2017; Rinke et al., 2021), our method is particularly suitable for selecting keywords to collect text data from a massive digital media corpus, as it relies on a local algorithm to incrementally retrieve data.

In what follows, we first highlight the issues with prevailing keyword selection and data retrieval practices in communication research, by presenting a survey of articles from leading communication journals that collect and analyze text data from social media. We then synthesize text data retrieval literature across communication, computer, and information sciences. Next, we introduce the WordPPR method, describe its workflow, and establish its effectiveness using simulations, focusing on its robustness against parameter choice and its performance compared with other methods. Lastly, we demonstrate its application in one social media data retrieval task.

Text data retrieval practices in communication research

Keyword selection is often the basis for text data retrieval, and by extension, measurement construction. Studies across social science disciplines have used keywords for Boolean searches to collect data from social media and other digital media sources to study various topics, such as online public opinion (e.g., Barberá et al., 2019; Choi, 2014; Zhang et al., 2019), digital activism and political protests (e.g., Freelon et al., 2016; Jost et al., 2018; Suk et al., 2023), election or campaign dynamics (e.g., Bovet & Makse, 2019; Jungherr, 2016), and news media discourses (e.g., Barberá et al., 2020; Hart et al., 2020; Stryker et al., 2006). As Jungherr’s (2016) systematic review of Twitter use in election campaign research emphasizes, the keyword or hashtag list for data collection is crucial because it can affect data quality by introducing false negatives (i.e., content that should be included but is not) and false positive (content that should not be included but is).

However, a systematic approach to keyword selection and text data retrieval remains underutilized in communication research (Mahl et al., 2022; Mayr & Weller, 2017). Consider, for instance, the articles published in leading communication journals (Journal of Communication, New Media and Society, Journal of Computer-Mediated Communication, and Political Communication, according to the SCImago Journal Rank indicator within the category of “communication” based on their 5-year journal impact factors). After reviewing each publication in all four journals between 2017 and 2021 (N = 1,179), we identified 75 articles that used keywords to collect social media data and found two general approaches to keyword selection and data retrieval: a one-step approach and a multi-step approach.

The one-step approach hinges on researchers, who select keywords based on either their domain knowledge (e.g., An & Mendiola-Smith, 2019; Borg et al., 2020; Thorsen & Sreedharan, 2019) or other resources (such as existing research, authoritative sources, or unspecified sources, e.g., Scherr et al., 2018; Vicari, 2021). For instance, Stier et al. (2017) use two keywords #ClimateChange and #NetNeutrality to collect tweets, assuming that they capture related policy discourse. The multi-step approach entails separate steps of keyword selection and validation and data collection (Bradshaw et al., 2020; Karsgaard & MacDonald, 2020). For instance, Bradshaw and colleagues (2020) outline four steps for Twitter data retrieval with hashtags. They commence with an initial manual identification of hashtags pertinent to specific political events. This is followed by a subsequent expansion of the dataset, which detects additional salient hashtags through frequency and co-use analyses. A filtration process is then applied, where only the most central hashtags are retained and those generating minor and ephemeral conversations are discarded. Lastly, a focused set of hashtags generating substantial traffic on Twitter is finalized. Our quantitative analysis of the keyword selection and data collection practices across the 75 articles reveals that the overwhelming majority (84%, N = 63) adopt the one-step approach, lacking documentation of the keyword selection process or assessment of the quality of the keywords and retrieved data (see Appendix A for a detailed report).

Researcher-driven, computational, and iterative keyword selection and expansion

The quality of data retrieval can be evaluated by “retrieval precision” and “retrieval recall” (Kim et al., 2016, p. 5; Mahl et al., 2022; Stryker et al., 2006). Retrieval precision refers to the proportion of relevant content in retrieved data; retrieval recall is a measure of the proportion of total relevant content that is retrieved. The one-step, often researcher-centered approach illustrated above, can suffer from poor recall. He and Rothschild (2016) demonstrate that, compared with a baseline set of tweets about the 2012 U.S. Senate elections collected using only candidate names as the keywords, the set of tweets collected via a multi-step keyword expansion approach is 3.2 times larger and produces different conclusions about sentiments and interactions. In a study of Swedish politics, Lorentzen and Nolin (2017) find that the dataset generated by keywords that capture additional hashtags and participants over time has significantly higher quality than the dataset generated by a static, pre-defined set of keywords, in terms of user characteristics, network features, and tweet types. Also, this approach might not be reliable in the sense that different researchers can select different keyword sets for the same retrieval task, resulting in substantially disparate target content and measurement of the construct of interest (King et al., 2017).

Related research in social, computer, and information sciences points to two major directions for maximizing data retrieval quality. First, iterative keyword expansion is critical. Due to lexical ambiguity and variety, expanding a keyword list based on initial seeds can reduce false negatives and increase the completeness of retrieved data (Krovetz & Croft, 1992). This idea is reflected in Parekh and colleagues’ (2018) four-phased social media data collection model that includes initialization, expansion, filtering, and validation. By running real data collection experiments, Kuzi et al. (2016) show that their query expansion technique, which takes an entire corpus, applies the Word2Vec term embeddings and identifies terms similar to the initial query term, can boost retrieval performance significantly compared to results based on the initial term only.

Second, it is crucial to integrate human knowledge and computer intelligence. While keyword selection and data collection practices in communication research tend to be researcher-centered, researchers from computer science, information science, and engineering tend to adopt a computer-centered approach to digital media data retrieval, where little human guidance is involved. A computer-centered approach can efficiently and automatically update relevant keywords and track language evolution, allowing researchers to follow rapidly changing online conversations over time (Liu et al., 2019). This fully automated approach with little human intervention can also be advantageous in terms of minimizing human bias (Liu et al., 2019). In this effort, related work has incorporated natural language processing, deep word embeddings, and clustering algorithms to extract relevant keywords from websites (Lee & Kim, 2018), detect online trolls (Liu et al., 2019), and capture health outbreaks via real-time Twitter information (Edo-Osagie et al., 2020). While efficient, relying on a fully automated model may lack theoretical rigor and increase false positives. Therefore, it is important to combine both computer and human input (Elyashar et al., 2021).

Some methods have incorporated iterative keyword expansion and human-computer input, where manual annotation and machine learning are combined. Makki et al. (2018) present an interactive Twitter data retrieval system that prompts researcher feedback (by asking researchers to label the relevance of collected tweets) while minimizing the number of labeling requests. The ReQ-ReC (ReQuery-ReClassify) framework developed by Li et al. (2014) relies on human-guided active learning and features a double-loop system where an outer loop generates queries and an inner loop takes researcher input to train relevance classifiers, a process that can be iterated multiple times to achieve optimal retrieval results. In a similar vein, Linder’s (2017) method follows the steps of collecting an initial dataset with seed keywords, annotating a random sample, training a classification algorithm to suggest keywords, and repeating the previous steps till satisfaction.

King et al. (2017) “human-led and computer-assisted” method for keyword discovery and selection is predicated on the acute observation that humans can be good at “recognizing whether any given word is an appropriate representation of a given concept” (p. 972). This algorithm uses human input as the starting point such that researchers define “a search set” (e.g., the entire corpus or a dataset obtained based on a set of general keywords) as well as “a reference set” known to be relevant to the construct of interest, which is a subset of the search set (e.g., by coming up with keywords or selecting a subset of the already defined keywords). Then, different classifiers are used to divide the search set data into target vs. non-target data by treating the reference set as labeled data. Lastly, two lists of top-ranked keywords, which are the best features of the target and non-target data, respectively, are generated, and researchers select the keywords based on the two lists.

Similarly, Wang et al. (2016) Double Ranking (DR) method starts with a set of seed keywords developed by researchers to first collect a relevant dataset; it then produces a shortlist of candidate keywords by ranking the words in the dataset. Each word in the shortlist is applied to collect more data, and the percentage of documents in the data containing the seed keywords is computed, producing a final list of keywords for researchers to select from. Both the King et al. (2017) and Wang et al. (2016) methods harness human input by asking researchers to adjudicate the appropriateness of keywords, which we draw heavily upon.

Besides the immediately relevant literature on digital media data retrieval, keyword extraction research, which focuses on producing keywords in a target text dataset for a summary representation, provides a third direction. Such keywords can be identified based on simple statistical measures, such as term frequency (TF) and term frequency-inverse document frequency (TF-IDF) (Qaiser & Ali, 2018). Linguistic knowledge can improve the quality of keyword extraction, such as focusing on noun phrase chunks in text (Hulth, 2003). Pointwise Mutual Information and Information Retrieval (PMI-IR) can effectively quantify the similarity between two words and words with high PMI can be considered as keywords for a text dataset (Turney, 2001). Mihalcea and Tarau’s (2004) TextRank provides a key intuition for our research. TextRank is a graph-based ranking method that treats text units in a document as nodes in the graph and the relations between text units (determined by word co-occurrence within delimited windows) as edges. It ranks the text units by running a graph-ranking algorithm. Abilhoa and De Castro (2014) build on this graph-based approach and develop the Twitter Keyword Graph that takes tweets as input, builds a graph based on word co-occurrence, and ranks words by centrality measures.

Our WordPPR method for keyword selection is built on the three major recommendations from social, computer, and information sciences reviewed above: iteratively expanding keywords to account for lexical variety, integrating both human knowledge and computer intelligence, and representing the order and semantics of text using a word co-occurrence graph. Like many hybrid methods, it consists of the general steps of developing, applying, and validating keywords. The most important feature distinguishing our method from others is its computational efficiency. This method is efficient for and compatible with very large corpora, like the entire data stream from a mainstream social media platform. This is because the WordPPR is a local algorithm that incrementally retrieves documents topically relevant to the seed word(s). Therefore, the computational and memory cost is only impacted by the retrieved documents (which is a relatively small subset of the entire corpus). Some keyword selection methods require processing a complete corpus or a random sample of it for developing and validating keywords (see King et al., 2017; Kuzi et al., 2016). For example, Rinke et al. (2021) expert-informed topic modeling (EITM) performs topic modeling under expert guidance and selects documents with high cumulative probabilities of belonging to relevant topics. However, this method only works when the global dataset is available. It is unfeasible if a researcher needs to build keywords for a massive corpus that is too large to draw a sample from, let alone analyze it. Furthermore, some methods require a non-trivial amount of manual annotation of sample data to determine its relevance (e.g., Makki et al., 2018; Linder, 2017), a process that can be labor-intensive and time-consuming. The human input part of the WordPPR method requires researchers to produce seed keywords and select additional ones from the computer-generated list, which emphasizes human supervision but places a much lighter burden on researchers. In addition, we will demonstrate that the WordPPR method can provide consistent sets of candidate keywords under some basic statistical assumptions and achieve better performance than similar methods.

What is the WordPPR method?

The WordPPR method (Figure 1) consists of four steps: 1) collecting an initial dataset using core/seed keyword(s); 2) constructing a word graph based on the dataset; 3) applying the Personalized PageRank (PPR) algorithm to rank words in proximity to the seed keyword(s) and selecting new keywords that optimize retrieval precision and recall; 4) repeating steps 1–3 to determine if additional data collection is needed.

Figure 1. The workflow of the WordPPR method.

The first step takes human judgment as the initial input, where the researcher determines one or a few core keyword(s) specific to the research topic. This requires researchers to gain a domain-specific understanding of the topic and even to collaborate with domain experts to define the seed keyword(s). Also, the researcher should be restrictive so as to reduce the noise introduced by keywords that are frequently used in unrelated contexts. After selecting the core keyword(s), the researcher can retrieve data (e.g., social media posts) containing the keyword(s) from a database.

The second step centers around constructing a word graph based on the collected data. There are different ways to construct the word graph, but the underlying principle is to treat words (e.g., hashtags, unigrams, bigrams, or trigrams, etc.) as nodes and the relationships between words as edges. After standard preprocessing (e.g., removing stopwords and stemming or lemmatizing words), the edge between two words can be established via co-occurrence in one document (e.g., a social media post) or within a delimited window in the document.

In the third step, the PPR algorithm is applied to the word graph to generate additional keywords closest to the initial keyword(s), as measured by the algorithm. This idea is similar to that in many of the existing query-building systems, like the “word movers’ distance” measure that uses word embeddings to quantify the similarity between new documents and the prototype document (Elyashar et al., 2021). PPR is a variant of Google’s PageRank (Page et al., 1998), which ranks the nodes in a network by their “relational closeness” to the given seed node(s). The PPR vector, $p\in R^n$ (where $n$ is the number of candidate keywords), quantifies such closeness for every node and is defined to be the stationary distribution (i.e., the converging probability distribution of a stochastic process) of the following personalized random walk (a stochastic process). The random walk starts at a given seed node. At any step, the random walker teleports to the seed node with probability $\tau$, or randomly goes to one of the adjacent nodes from the current node with probability $1-\tau$. Here, $\tau$ is called the teleportation constant. If $\tau$ = 0, PPR is equivalent to a random walk on the graph (Pearson, 1905). See Appendix B for a detailed description of PPR. In the section below, we provide a mathematical description of the WordPPR algorithm.

The WordPPR algorithm

The WordPPR algorithm applies PPR to the co-occurrence graph of words, which is undirected and weighted (the directed version can be constructed similarly). Throughout this article, we assume that a given word graph is connected. Selecting the “closest” words to a given word from a word graph is analogous to a targeted sampling of nodes in a graph with a seed node, for which PPR provides a consistent estimate under the stochastic block model (Chen et al., 2020; Karrer & Newman, 2011).

To construct the word graph (Table 1, Step a), we use a set of $m$ documents in the data collected via an initial keyword search. Let $X\in {\left\{0,1\right\}}^{m\times n}$ be the document-term matrix where $X_{iw}=1$ if word $w$ occurs in document $i$ (and 0 otherwise), and $n$ is the number of unique words in all documents. We define the adjacency matrix $A\in R^{n\times n}$ of the word co-occurrence graph as $A=X^{T}X$. So, $A_{wv}$ is the number of documents in which word $w$ and $v$ co-occur. Note that $A$ is usually sparse in practice (containing less than 1% non-zeros in the example below).

Table 1. The WordPPR Algorithm.

Input: document-term matrix $X\in R^{m\times n}$, a set $S$ of $n_0$ seed nodes, teleportation constant $\tau$. Procedure $WordPPR\left(X,S,\pi \right)$: a. Construct the word co-occurrence matrix $A=X^{T}X$.b. Compute the PPR vector $p=PPR\left(A,S,\tau \right)$.▪ Define the graph transition matrix $P\in R^{n\times n}$ with $P_{ij}=A_{ij}/d_i$,where $d_i=\sum _j{A}_{ij}$ is the $i$th row sum of $A$.▪ Define $\pi \in R^n$ with ${\pi }_i=1/n_0$ if $i\in S$, and ${\pi }_i=0$ otherwise.▪ Define the Markov transition matrix $Q=\tau \mathrm{\Pi }+\left(1-\tau \right)P$, where $\mathrm{\Pi }\in R^{n\times n}$ is defined with repeating row of $\pi$.▪ Compute $p$ as the left leading eigenvector of $Q$.c. Compute two vectors $x$ and $y$ defined for any word $w$ as $x_w=\frac{P_w-\tau {\pi }_w}{1-\tau }$ and $y_w=\frac{x_w}{d_w},$ where $d_w=\sum _v{A}_{wv}$ is the degree of word w Output: The frequency and exclusivity measure $x$ and $y$ → a weighted measure FREX.

Next, we calculate the PPR vector, $p\in R^n$, using the initial keywords as the seeds (Table 1, Step b). This step essentially defines the Markov transition matrix that represents PPR on the word graph and calculates the principal left eigenvector of this matrix (Haveliwala, 2002; Jeh & Widom, 2003). When the $A$ is sparse, the principal eigenvector can be calculated efficiently (e.g., using the “Spectra” or “ARPACK” library). To use PPR in WordPPR, we compute a frequency measure and an exclusivity measure for each word in the collected data (Table 1, Step c). Frequency (i.e., simply word frequency) ensures that a given candidate word is capable of including a sufficiently large number of new documents (on top of existing documents obtained using the seed keywords, thus contributing to “retrieval recall”). Exclusivity seeks to locate words that appear mostly in the target topic and infrequently in other topics. This measure ensures that a given candidate word primarily adds useful documents (and not polluting the existing document set, thus contributing to “retrieval precision”). To measure the frequency and exclusivity of words, we define the following two quantities $x,y\in R^n$:

$x_w=\frac{p_w-\tau {\pi }_w}{1-\tau },\mathrm{a}\mathrm{n}\mathrm{d}{y}_w=\frac{x_w}{d_w},$

for any word $w\in \left\{1,2,\dots ,n\right\}$, where $\tau$ is the teleportation constant used for calculating PPR vector $p\in R^n$ (a commonly used default value is 0.25), and $p_w$ is the $w$th element in $p$, and $d_w={\sum }_v^{\hspace{0.17em}}{A}_{wv}$ is the degree of word $w$ (i.e., the total number of words that co-occurred with word $w$). In this definition, $x$ is a consistent estimator for word frequency, and $y$ for word exclusivity (see more details in next section).

The output includes two empirical cumulative distribution functions (ECDFs) applied to the values of $x$ and $y$. To leverage the frequency and exclusivity of words, Airoldi and Jonathan (2016) adopt the harmonic mean of the word’s rank in the distribution of $x$ and $y$ and define a hybrid measure of words:

$FRE{X}_w={\left(\frac{1-a}{ECD{F}_x(x_w)}+\frac{a}{ECD{F}_y(y_w)}\right)}^{-1},$

where parameter $a$ can be interpreted as the weight for exclusivity (which defaults to 0.5). Table 1 summarizes the procedure of WordPPR. Researchers can peruse the suggested top words ranked by the FREX measure and select all the keywords specific to the topic of interest. Here, researchers can inspect a large number of top words to increase the chance of discovering relevant words and in the meantime prioritize the precision of each selected keyword to reduce the noise introduced in the collected dataset.

With the newly selected keywords from step three, the researcher in the fourth step can repeat steps one to three, i.e., retrieving a new dataset, constructing a word graph based on the new dataset, and applying the WordPPR algorithm to generate a new list of keywords for the researcher to choose from. If the researcher cannot find new and meaningful keywords from the list, the data collection process can be terminated, and the first and second rounds of collected data can be combined and cleaned for downstream analysis. Otherwise, the researcher can repeat steps one to three till no additional keywords and data can be added.

Why does the WordPPR method work? Two validations

After explaining what WordPPR is, we turn to the mathematical evidence for why WordPPR works. Chen (2021) demonstrates that if the corpus is generated from the latent Dirichlet allocation (LDA) model (Blei et al., 2003), then ranking words by the $x$ vector output by WordPPR is statistically consistent with ranking words by the frequency measure, and ranking words by the $y$ vector is consistent with ranking words by the exclusivity measure. In what follows, we provide two simulation studies to validate the robustness of the WordPPR method and its comparability with other methods.

Robustness of the WordPPR method

This first simulation study establishes that under the ideal LDA corpus WordPPR can recommend highly accurate keywords for text data retrieval from a given corpus. We performed this simulation in three steps: we first generated text data with LDA (represented by a document-term matrix); then we applied one iteration of the WordPPR algorithm; and finally, we assessed the accuracy of returned keywords in terms of frequency and exclusivity.

Each row of $\varphi \in R^{10\times 500}$ is the word distribution of a topic and is generated from the Dirichlet distribution, ${\varphi }_{t.}\sim$Dirichlet$\left(\beta \right)$ for $t=1,2,\cdots ,10$, where the elements in $\beta \in R^{500}$ are sampled from the exponential distribution, Exponential$\left(10\right)$. We experimented with corpus size (i.e., the number of documents) $m\in \left\{1000,2000,5000,10000,20000\right\}$ and the expected document length $\lambda \in \left\{5,10,20\right\}$. In addition, we studied the effect of teleportation constant with $\tau \in \left\{0.25,0.5,0.75\right\}$. For each combination of parameters, we simulated the document-term matrix for 30 times and evaluated the WordPPR algorithm independently.

To evaluate words ranked by $x$ and $y$ (i.e., the two output vectors of WordPPR), the accuracy in terms of frequency and exclusivity is defined as follows. Let $S\left(x,n_1\right)$ be the set of top $n_1$ words ranked by $x$, then the accuracy in frequency is defined as $Accuracy\left(x,{\varphi }_1,n_1\right)=\left|S\left(x,n_1\right)\cap S\left({\varphi }_1,n_1\right)\right|/n_1.$Similarly, the accuracy in exclusivity for the top $n_2$ words ranked by $y$ is defined as $Accuracy\left(y,\gamma ,n_2\right)=\left|S\left(y,n_2\right)\cap S\left(\gamma ,n_2\right)\right|/n_2$, where ${\gamma }_i=\frac{{\varphi }_{1i}}{{\sum }_j^{\hspace{0.17em}}{\varphi }_{ji}}$ represents the proportion of topic 1 in all the occurrences of each word. In this simulation, we set $n_1$, $n_2=25$.

Figure 2 displays the accuracy of WordPPR’s frequency and exclusivity estimates, for five different document sample sizes (“$m$,” x-axis), three different expected document lengths (“lambda,” colors and shapes), and three different teleportation constants (“tau,” panel columns). The results uniformly show that WordPPR can identify both frequent and exclusive words with high accuracy. Moreover, both the increase in the number of sample documents and the increase in the expected document length improve the accuracy of WordPPR. In addition, the choice of teleportation constant appears to have little effect on the accuracy of WordPPR, except in the top left panel, suggesting that WordPPR is robust against a wide range of parameter settings.

Figure 2. Accuracy of the frequency and exclusivity measures of the WordPPR algorithm for five document sample sizes (“$m$,” x-axis), three document lengths (“lambda,” colors and shapes), and three teleportation constants (“tau,” panel columns). Each point represents the average accuracy of including the top 25 words. The error bars indicate two times the standard deviation of accuracy across 30 repeated experiments.

Comparison with other methods

The second simulation study compares WordPPR with several existing keyword selection techniques and shows that WordPPR can achieve superior performance in terms of producing the first round of additional keywords. The simulation was conducted in three steps: we first generated text data with a mixture of curated LDA and then applied one iteration of WordPPR and other methods, including term frequency (TF), term frequency-inverse document frequency (TFIDF), the non-personalized version of WordPPR (TextRank; Mihalcea & Tarau, 2004), Double Ranking (Entropy, Wang et al., 2016), discriminatory power (DiscPower, King et al., 2017). Finally, we assessed the accuracy of returned keywords in terms of sensitivity and specificity (i.e., receiver operating characteristic, ROC).

First, we sampled 15,000 text documents from the LDA with 10 underlying topics and 1000 unique words, where the first topic was assumed to be the target topic. For each topic $t=1,2,\dots ,10$, we generated the word distribution ${\varphi }_t\in R^{1000}$ from the Dirichlet distribution, ${\varphi }_t\sim$Dirichlet$\left(\beta \right)$, where $\beta \in R^{1000}$ was sampled from the exponential distribution, Exponential$\left(10\right)$. Some existing methods require additional sets of documents. The target set should contain the documents that the researcher sees as relevant to the target topic, for example, by searching the initial keywords in the corpus. The general set should be a random document sample of the corpus. To generate the target set (5,000 documents), we set the topic distribution to be $\alpha =\left(0.901,0.011,\cdots ,0.011\right)$. To generate the general set (10,000 documents), we set the topic distribution to be $\alpha =\left(0.1,0.1,\cdots ,0.1\right)$. The expected document length in both target and general sets was set to 25. For all documents, we also added a very small probability (0.001) of random bit flipping to simulate typos.

For WordPPR, we set the teleportation constant to be $\tau =0.25$ (see above for analysis that shows that WordPPR is robust against choices of $\tau$) and experimented with three weights of the exclusivity parameter $a=0.3,0.5,0.7$ (referred to as WordPPR.3, WordPPR.5, and WordPPR.7 respectively). Although WordPPR only requires a target set of documents, our experiment indicates that including both target and general sets can generate comparably satisfactory results. For TF, TFIDF, and TextRank, we combined target and general sets. For Double Ranking of Wang et al. (2016), since the re-ranking step is not feasible, we skipped it and focused on its computational metric based on entropy. In calculating Entropy, we treated the target set and general set as the “current set” and “random (or reference) set” described in Wang et al. (2016) respectively. For DiscPower, we treated the target set and general set as the “reference set” and “search set” described in King et al. (2017) respectively.

To evaluate keyword rankings, we defined the “ground-truth” set of keywords that a word $w$ must satisfy two criteria to be included: (1) word $w$ is frequent to topic 1 (the target topic) and (2) word $w$ is exclusive to topic 1. Mathematically, we quantify (1) as “${\varphi }_{1w}$ is greater than the 10th percentile of all the elements in ${\varphi }_1\in R^{1000}$.” For (2), define ${\gamma }_1\in R^{1000}$ with ${\gamma }_{1w}={\varphi }_{1w}/{\sum }_{i=1}^{10}{\varphi }_{iw}$. Note that ${\gamma }_1$ is the proportion of each word’s frequency exclusive to topic 1. With that, we quantify (2) as “${\gamma }_{1w}$ is greater than the 10th percentile of all the elements in ${\gamma }_1$.” Combining (1) and (2) resulted in ~50 “ground-truth” keywords, and we used these to calculate the sensitivity and specificity of each tested method. Figure 3 displays the performance (ROC) of all eight methods. Table 2 lists the Area Under the ROC Curve (AUC), which is an aggregate measure of performance. As shown, WordPPR outperforms the other existing methods in ranking keywords, with overall higher sensitivity and specificity. Upon repeating the above simulation 30 times, we consistently observed the same result. The AUC of WordPPR is significantly higher than the others (paired t-test, at the 0.01 significance level). In addition, comparing across different $a$ values of WordPPR, we observed small to minimal variation in AUC (yet all outperforming other methods), suggesting that the performance of WordPPR is relatively stable when varying the weight of exclusivity.

Figure 3. Comparison of different keyword ranking methods (TF, TFIDF, TextRank, entropy, DiscPower, and WordPPR) based on the ROC curve.

Table 2. Comparison of different keyword ranking methods based on the AUC measure (area under the ROC curve).

Method	AUC
TF	0.917
TFIDF	0.900
TextRank	0.917
Entropy	0.920
DiscPower	0.950
WordPPR.3	0.981
WordPPR.5	0.981
WordPPR.7	0.984

How to apply the WordPPR method? An application

To illustrate the workflow and effectiveness of the WordPPR method, we applied it to the collection of #MeToo-related tweets on Twitter. In October 2017, actress Alyssa Milano tweeted to encourage survivors of sexual violence to share their stories using the phrase “me too” in response to the allegations against Harvey Weinstein. Soon, #MeToo was used to call for recognition of the issue and solidarity with victims, which ultimately coalesced into digital connective action (Suk et al., 2021). Beyond the outpouring of stories among sexual violence survivors, the #MeToo movement has kept growing, as activists continue to lead the movement and workplace policy changes have been implemented. Furthermore, #MeToo received waves of attention from politicians and political parties, especially during the Brett Kavanaugh confirmation period and the 2020 US election campaign, when then-Supreme Court nominee Kavanaugh and Presidential candidate Biden were accused of sexual misconduct. In this case study, we define the scope of the data collection to be tweets about the #MeToo movement from October 2017 to December 2020. A longitudinal examination of the movement can demonstrate whether the WordPPR method is suitable to generate a selection of keywords that reflect multiple developments of the movement over several years.

For this application, we relied on the Twitter API product track for academic research, which provided unprecedented access to all publicly available tweets. In the first step, we selected “#metoo” (not case-sensitive) as the keyword for the first round of data retrieval because the movement was propelled by this hashtag. It is a reasonable assumption that the majority of the tweets containing this hashtag are central to the #MeToo conversation. We collected 13,180,909 tweets containing this hashtag from October 1, 2017, to December 31, 2020.

In the second step, we constructed a word graph based on a 10% random sample of the collected tweets. After standard preprocessing (removing stopwords, lemmatizing words, and removing infrequent words), we extracted both hashtags and bigrams in unique tweets and created a word graph where each hashtag or bigram is a node and each edge represents co-occurrence in the same tweet. We included hashtags because they represent key strands of conversations and debates on Twitter, a feature emphasized in Makki and colleagues’ Active Tweet Retrieval Visualization framework (2018). Bigrams were chosen to reduce ambiguity introduced by unigrams. According to Krovetz and Croft (1992), “words are ambiguous and this ambiguity can cause documents to be retrieved that are not relevant” (p.116), and the solution to this ambiguity problem is to use phrases instead of single words 1992.

The PPR algorithm was then applied to the word graph, resulting in a list of keywords ranked by frequency, exclusivity, and FREX (Table 3). We relied on the composite FREX measure and qualitatively evaluated the top 100 hashtags/bigrams with the highest FREX measures. It can be seen that FREX is a more effective measure than mere word frequency because some frequent words are downranked by FREX. For example, relevant keywords like “sexually abuse” (total frequency of 823), “#ibelieveher” (frequency of 864), and “#metoounlessitsbiden” (frequency of 858) are not ranked highly by frequency despite their significant relevance to the research context. The FREX measure, therefore, helps researchers efficiently distinguish keywords specific to the research topic from frequent keywords without contextual relevance (e.g., #news, #women).

Table 3. Top 100 hashtags and bigrams related to #MeToo using the WordPPR algorithm: round one of data collection.

	Word	Total frequency	FREX		Word	Total frequency	FREX
1	#metoo	254253	1	51	#goldenglobes	1034	0.9697161
2	#timesup	22135	0.99955217	52	harassment_assault	1079	0.96942704
3	sexual_harassment	11604	0.99876844	53	#resistance	1526	0.96935846
4	sexual_assault	11546	0.99820627	54	#womensrights	1185	0.96913937
5	#metooindia	3836	0.99742495	55	movement_woman	964	0.96878003
6	sexual_abuse	3637	0.99563334	56	#enoughisenough	1168	0.96801277
7	#news	2269	0.99417824	57	#harassment	1024	0.96773412
8	harvey_weinstein	2504	0.99383599	58	#hollywood	1034	0.96763571
9	sexually_assault	2634	0.9937298	59	#weinstein	1007	0.96633833
10	#women	3402	0.99348076	60	person_year	962	0.96581597
11	#kavanaugh	2992	0.99327285	61	supreme_court	1028	0.96338348
12	#believewomen	2440	0.99249854	62	#domesticviolence	1008	0.96315953
13	#believesurvivors	2527	0.99238145	63	dr_ford	976	0.96308083
14	#blacklivesmatter	3143	0.99212867	64	woman_movement	889	0.9623318
15	#sexualharassment	2410	0.99205011	65	late_daily	822	0.96162592
16	#stopkavanaugh	2137	0.9904827	66	#rapeculture	935	0.96123659
17	share_story	2179	0.99035945	67	#survivor	958	0.96081489
18	sexual_misconduct	2299	0.98969753	68	#womensmarch	944	0.96003131
19	#feminism	2242	0.98880186	69	#kavanaughhearings	922	0.95966496
20	#trump	2890	0.98846809	70	#equality	979	0.9594356
21	social_medium	1982	0.98844479	71	#believeallwomen	891	0.95864155
22	joe_biden	1974	0.98823992	72	bill_cosby	899	0.95819364
23	sexual_predator	2019	0.98823598	73	#metoounlessitsbiden	858	0.95813917
24	#abusefreeindia	3657	0.98742388	74	#ibelieveher	864	0.95742423
25	sexual_violence	2082	0.9872331	75	#ibelievetarareade	877	0.95708765
26	#resist	4172	0.98713163	76	#bluewave	911	0.95690739
27	sexually_harass	1964	0.98711398	77	woman_girl	925	0.9566398
28	year_ago	1878	0.98687342	78	#abuse	897	0.95611033
29	#churchtoo	1775	0.98685472	79	assault_woman	889	0.95551552
30	#sexualassault	1966	0.98577285	80	#democrats	954	0.95526698
31	accuse_sexual	1737	0.98287023	81	woman_forward	873	0.95521994
32	#rape	1796	0.98241164	82	#iftheshoefits	1075	0.95471151
33	#whyididntreport	1601	0.98230331	83	abuse_woman	877	0.95464746
34	#maga	4979	0.98200629	84	woman_woman	871	0.95455617
35	#harveyweinstein	1585	0.9815185	85	year_late	821	0.95399175
36	#metoomovement	1470	0.98138939	86	blasey_ford	932	0.95264183
37	#theresistance	2018	0.98105134	87	#ge	1784	0.95242771
38	tara_reade	1323	0.97914167	88	witch_hunt	774	0.95220317
39	#mentoo	1379	0.97861311	89	nov_pm	960	0.95148879
40	pm_report	2161	0.97814214	90	#sexualabuse	1195	0.9513263
41	woman_speak	1318	0.97781399	91	sexually_abuse	823	0.95036512
42	bill_clinton	1198	0.97585202	92	woman_accuse	842	0.95026842
43	victim_sexual	1307	0.97492163	93	#love	898	0.94994016
44	tarana_burke	1093	0.9748477	94	woman_sexually	827	0.94983316
45	black_woman	1177	0.97456178	95	#oscars	784	0.94980609
46	#feminist	1121	0.97222908	96	#veteran	837	0.9491449
47	due_process	1061	0.97192975	97	#cnn	933	0.9487263
48	#tarareade	1095	0.97143397	98	domestic_violence	807	0.94819124
49	donald_trump	1123	0.9707653	99	#joebiden	808	0.94738005
50	#neveragain	1366	0.9703019	100	violence_woman	811	0.94590482

Bold keywords in black are selected ones.

In step 4, we collected 69,915,400 tweets from the same Twitter endpoint access using the new keywords. Table 4 displays the new top 100 words based on the word graph constructed from a 10% random sample of the second round of data collection. As can be seen in Table 4, the newly suggested words have no significant relevance to the research context to warrant another round of data collection. Therefore, after repeating steps 2 and 3, our data collection stopped, yielding a total of 83,096,309 unique tweets using the following list of keywords: #metoo, #timesup, sexual harassment, sexual assault, sexually assault, sexual abuse, #believewomen, #believesurvivors, #sexualharassment, sexual misconduct, sexual predator, sexual violence, sexually harass, #churchtoo, #sexualassault, #rape, #whyididntreport, #metoomovement, #mentoo, #rapeculture,” #believeallwomen, #metoounlessitsbiden, #ibelieveher, #ibelievetarareade, #sexualabuse, sexually abuse. To assess the amount of noise in the dataset, one author coded 500 tweets randomly drawn from the dataset for relevance to #MeToo and found only 8 out of 500 to be clearly irrelevant. In Appendix C, we also presented keywords suggested by other methods (TF, TFIDF, Entropy, Discriminatory Power, and TextRank) and discussed the effectiveness of the WordPPR method.

Table 4. Top 100 hashtags and bigrams related to #MeToo using the WordPPR algorithm: round two of data collection.

	Word	Total frequency	Frex		Word	Total frequency	Frex
1	#believesurvivors	438	0.99887706	51	#vaw	109	0.97361672
2	#emptythepews	142	0.99816139	52	christine_blasey	1062	0.97258591
3	#tarareade	110	0.99601625	53	#christineblaseyford	112	0.97189253
4	#women	386	0.99571241	54	#scotus	165	0.97089491
5	tara_reade	1052	0.99569727	55	#womensrights	74	0.97011842
6	#fakecases	258	0.99550636	56	sexually_assault	5179	0.96956779
7	#neverbiden	69	0.99529807	57	#hollywood	128	0.96923395
8	#kavanaugh	638	0.9947867	58	#survivor	81	0.96899234
9	#domesticviolence	233	0.9934622	59	#violence	126	0.96881616
10	#ibelievetara	79	0.99345869	60	rape_woman	428	0.96790849
11	#goldenglobes	592	0.99275186	61	#theresistance	139	0.96720809
12	#survivortough	72	0.99233474	62	year_ago	1094	0.96691839
13	#abuse	303	0.99223972	63	#police	106	0.96675247
14	#childabuse	219	0.99213584	64	#legalterrorism	60	0.96621951
15	joe_biden	2194	0.99212689	65	young_woman	551	0.9652397
16	#trump	640	0.9903889	66	#survivors	89	0.9643886
17	#stopkavanaugh	240	0.99019806	67	young_girl	462	0.96423338
18	tara_reade’s	209	0.98999277	68	#ptsd	80	0.96388143
19	#csa	205	0.98988851	69	#whywewearblack	154	0.96370666
20	#kavanaughhearings	220	0.98937536	70	justice_system	313	0.96263768
21	#india	205	0.9882514	71	#gop	202	0.96262217
22	#survivorculture	75	0.98813377	72	#sexual	399	0.96222935
23	#feminism	103	0.98762507	73	hold_accountable	530	0.96143602
24	share_story	693	0.98701231	74	domestic_violence	1824	0.96055446
25	dr_ford	753	0.98700489	75	#voteblue	75	0.9603896
26	#genderbiasedlaws	118	0.98680839	76	wear_black	280	0.96007637
27	rape_culture	476	0.98499003	77	#brettkavanaugh	179	0.95872551
28	#menscommission	140	0.98334079	78	anita_hill	416	0.95782548
29	rape_victim	334	0.98325796	79	#children	115	0.95757029
30	#men	116	0.98282535	80	#grammys	151	0.95733398
31	#ibelievechristineblaseyford	67	0.98179277	81	due_process	346	0.95702246
32	#harveyweinstein	243	0.98172187	82	stand_woman	203	0.9567918
33	#resist	185	0.98151162	83	catholic_church	966	0.9563917
34	sex_abuse	737	0.98076687	84	#donaldtrump	92	0.95613511
35	rape_case	375	0.98070561	85	high_school	807	0.95594708
36	#domesticabuse	85	0.98015939	86	#blacklivesmatter	65	0.95584902
37	#justice	125	0.97986662	87	#democrats	89	0.95519353
38	#biden	80	0.97862591	88	#metooindia	75	0.95493452
39	#joebiden	86	0.97811766	89	year_late	241	0.95415319
40	#murder	150	0.97710098	90	dr_christine	308	0.95352843
41	#trauma	90	0.97699253	91	girl_woman	282	0.95283671
42	#mentalhealth	113	0.97628429	92	sexually_abuse	2094	0.9528064
43	#fathersday	60	0.97532801	93	#equality	61	0.95256364
44	social_medium	598	0.97517045	94	#violenceagainstwomen	64	0.95228858
45	#crime	135	0.97476569	95	#womenempowerment	62	0.95145431
46	blasey_ford	1099	0.97473738	96	#racism	62	0.95122199
47	#sexualviolence	154	0.97465854	97	#law	86	0.95107989
48	#consent	93	0.9742243	98	woman_forward	876	0.94990223
49	#maga	282	0.97407092	99	fake_case	76	0.9497231
50	#harassment	256	0.97406478	100	#news	494	0.9495798

Discussion

With widely available digital media data, significant advancement in computational methods, and the immense potential they hold to enhance communication research, our research community needs to pay more attention to the first step of data retrieval in research that harnesses digital media data and computational methods. In their paper that reviews lexicon selection for measuring bias and stereotypes, Antoniak and Mimno (2021) advocate that “[seeds] and their rationales need to be tested and documented, rather than hidden in code or copied without examination.” This article echoes their call. The same principle applies to the keyword selection process for digital media data collection: keywords used for data collection must be systematically selected, documented, and evaluated.

Since Boolean searches are predominantly applied in our field, we explore the question of how to generate search keywords that can produce a high-quality dataset. After reviewing studies published in leading communication journals over 5 years, we observe a heavy reliance on the one-step keyword selection approach, which can lead to biased and incomplete data. Synthesizing cross-disciplinary research on data retrieval, we distill three key insights: 1) keywords need to be incrementally expanded to capture multifaceted discourses, 2) the keyword expansion process should tap into both human knowledge and computer intelligence, and 3) the word co-occurrence graph can effectively represent the order and semantics of text. These ideas are crystallized in our researcher-driven, computer-assisted, iterative, and graph-based keyword selection method, the WordPPR method.

The WordPPR method requires researchers to first identify core seed words, then collect data with them and feed the data into the WordPPR algorithm, and lastly select additional keywords from the list of keywords suggested by the algorithm. The three steps could be repeated until reaching data saturation. This method is computationally efficient because it does not require analyzing an entire corpus or randomly sampling it. It is particularly advantageous when researchers have to deal with massive data coming from a global social media stream (for example, Twitter’s global stream, which easily goes over hundreds of millions of tweets within 1 day) or data spanning an extensive period. Another advantage of this method is that, unlike other methods, it does not require intensive human labor that is usually in the form of manual annotation of sample data. Instead, human supervision is utilized to select seed word(s) and evaluate additional keywords (as in King et al., 2017; Wang et al., 2016). Therefore, it is an efficient method in terms of both computational cost and human labor. In addition, our two extensive simulation studies demonstrate that under the LDA assumption, the WordPPR method can identify all keywords that define a given topic and achieve better performance than similar methods.

Applying this method in the #MeToo case further demonstrates its effectiveness in identifying keywords and its applicability to data collection from massive corpora. Given that the entire Twitter corpus is too large to analyze or sample, the WordPPR method allows us to zero in on the core data and then incrementally collect additional relevant data efficiently. In our case, the first round of data collection suggests highly relevant and comprehensive keywords for the second round collection, without requiring any additional iteration.

Moreover, our WordPPR method addresses important validity concerns that are central to social science research. For content validity, our method emphasizes an iterative keyword expansion process that harnesses both human knowledge and computational intelligence, aiming to capture the multiple layers inherent in a given discourse from a digital media source. It also emphasizes the role of the researcher’s judgment in the initial seed word selection and subsequent evaluation, which is informed by theoretical guidance and conceptualization. With regard to criterion validity, our method aligns with and improves existing methods. Our simulation analysis indicates that our method generates results that are highly comparable, and in some aspects, superior to those produced by existing methods.

Through the application and validation, we show the importance of the “human-led and computer-assisted” and iterative model for digital data retrieval. In particular, our WordPPR method can be an effective way to reduce researcher bias yet harness researcher knowledge to guide keyword expansion, ultimately increasing the accuracy and completeness of the collected dataset. This workflow can apply to all languages and potentially different sources and types of digital media. Though we only surveyed related research practices in the field of communication and used the findings to illustrate the oversimplified keyword selection approach, this problem might not be limited to communication research. Furthermore, the development of the WordPPR method has drawn research from multiple disciplines, holding the potential to inspire researchers in other disciplines like information and computer sciences. We hope that future research using digital media data can strengthen the data collection process by considering the WordPPR method. Our method code and application code have been made publicly and freely available online.

Given the rapid advancement in the capacities and applications of large language models (LLMs) (Wei et al., 2022), there lies the potential for them to augment the WordPPR method. For example, LLMs could be employed to suggest seed keywords given a meticulously constructed research topic prompt. Furthermore, they might be utilized to extract additional keywords from a representative sample of the initial dataset. It is imperative for future investigations to systematically delve into the integration of LLMs within the realms of keyword selection and data collection.

With digital media data becoming a prevalent and rich source of research data for communication researchers, the importance of applying a systematic way of selecting keywords and retrieving text data cannot be understated. This process should not solely fall on researchers or computers. Instead, an iterative process where human researchers and computer intelligence interact can reduce human bias, augment human capacity, and produce an accurate and complete dataset that maximizes the validity of research findings.

Acknowledgement

We thank our reviewers, the editors, Dr. Karl Rohe, Dr. Nojin Kwak, and Dr. Dhavan Shah for their helpful feedback. We also thank Rui Wang, Dongdong Yang, and Xinxia Dong for assistance with the journal article coding.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The method and application code files as well as the supplementary materials are available at https://osf.io/pcybz/.

Additional information

Notes on contributors

Yini Zhang

Yini Zhang (Ph.D., University of Wisconsin–Madison) is an assistant professor in the Department of Communication at the University at Buffalo, State University of New York. She studies social media, media ecosystem, and political communication, using computational methods.

Fan Chen

Fan Chen (Ph.D., University of Wisconsin–Madison) is a Data Scientist at Google. He studies and develops statistical methods for social media, genomics, and advertisement data. The bulk of this work was completed while he was a Ph.D. student at the University of Wisconsin–Madison.

Jiyoun Suk

Jiyoun Suk (Ph.D., University of Wisconsin-Madison) is an assistant professor in the Department of Communication at the University of Connecticut. She studies the role of networked communication in shaping social trust, activism, and polarization, using computational methods.

Zhiying Yue

Zhiying Yue (Ph.D., University at Buffalo) is a postdoctoral researcher at the Digital Wellness Lab, Boston Children’s Hospital, and Harvard Medical School. Her research interests generally focus on individuals’ social media use and psychological well-being.