WordPPR: A Researcher-Driven Computational Keyword Selection Method for Text Data Retrieval from Digital Media

Figures & data

Figure 1. The workflow of the WordPPR method.

Table 1. The WordPPR Algorithm.

Input: document-term matrix $X\in R^{m\times n}$, a set $S$ of $n_0$ seed nodes, teleportation constant $\tau$. Procedure $WordPPR\left(X,S,\pi \right)$: a. Construct the word co-occurrence matrix $A=X^{T}X$.b. Compute the PPR vector $p=PPR\left(A,S,\tau \right)$.▪ Define the graph transition matrix $P\in R^{n\times n}$ with $P_{ij}=A_{ij}/d_i$,where $d_i=\sum _j{A}_{ij}$ is the $i$th row sum of $A$.▪ Define $\pi \in R^n$ with ${\pi }_i=1/n_0$ if $i\in S$, and ${\pi }_i=0$ otherwise.▪ Define the Markov transition matrix $Q=\tau \mathrm{\Pi }+\left(1-\tau \right)P$, where $\mathrm{\Pi }\in R^{n\times n}$ is defined with repeating row of $\pi$.▪ Compute $p$ as the left leading eigenvector of $Q$.c. Compute two vectors $x$ and $y$ defined for any word $w$ as $x_w=\frac{P_w-\tau {\pi }_w}{1-\tau }$ and $y_w=\frac{x_w}{d_w},$ where $d_w=\sum _v{A}_{wv}$ is the degree of word w Output: The frequency and exclusivity measure $x$ and $y$ → a weighted measure FREX.

Figure 2. Accuracy of the frequency and exclusivity measures of the WordPPR algorithm for five document sample sizes (“$m$,” x-axis), three document lengths (“lambda,” colors and shapes), and three teleportation constants (“tau,” panel columns). Each point represents the average accuracy of including the top 25 words. The error bars indicate two times the standard deviation of accuracy across 30 repeated experiments.

Figure 3. Comparison of different keyword ranking methods (TF, TFIDF, TextRank, entropy, DiscPower, and WordPPR) based on the ROC curve.

Table 2. Comparison of different keyword ranking methods based on the AUC measure (area under the ROC curve).

Method	AUC
TF	0.917
TFIDF	0.900
TextRank	0.917
Entropy	0.920
DiscPower	0.950
WordPPR.3	0.981
WordPPR.5	0.981
WordPPR.7	0.984

Table 3. Top 100 hashtags and bigrams related to #MeToo using the WordPPR algorithm: round one of data collection.

	Word	Total frequency	FREX		Word	Total frequency	FREX
1	#metoo	254253	1	51	#goldenglobes	1034	0.9697161
2	#timesup	22135	0.99955217	52	harassment_assault	1079	0.96942704
3	sexual_harassment	11604	0.99876844	53	#resistance	1526	0.96935846
4	sexual_assault	11546	0.99820627	54	#womensrights	1185	0.96913937
5	#metooindia	3836	0.99742495	55	movement_woman	964	0.96878003
6	sexual_abuse	3637	0.99563334	56	#enoughisenough	1168	0.96801277
7	#news	2269	0.99417824	57	#harassment	1024	0.96773412
8	harvey_weinstein	2504	0.99383599	58	#hollywood	1034	0.96763571
9	sexually_assault	2634	0.9937298	59	#weinstein	1007	0.96633833
10	#women	3402	0.99348076	60	person_year	962	0.96581597
11	#kavanaugh	2992	0.99327285	61	supreme_court	1028	0.96338348
12	#believewomen	2440	0.99249854	62	#domesticviolence	1008	0.96315953
13	#believesurvivors	2527	0.99238145	63	dr_ford	976	0.96308083
14	#blacklivesmatter	3143	0.99212867	64	woman_movement	889	0.9623318
15	#sexualharassment	2410	0.99205011	65	late_daily	822	0.96162592
16	#stopkavanaugh	2137	0.9904827	66	#rapeculture	935	0.96123659
17	share_story	2179	0.99035945	67	#survivor	958	0.96081489
18	sexual_misconduct	2299	0.98969753	68	#womensmarch	944	0.96003131
19	#feminism	2242	0.98880186	69	#kavanaughhearings	922	0.95966496
20	#trump	2890	0.98846809	70	#equality	979	0.9594356
21	social_medium	1982	0.98844479	71	#believeallwomen	891	0.95864155
22	joe_biden	1974	0.98823992	72	bill_cosby	899	0.95819364
23	sexual_predator	2019	0.98823598	73	#metoounlessitsbiden	858	0.95813917
24	#abusefreeindia	3657	0.98742388	74	#ibelieveher	864	0.95742423
25	sexual_violence	2082	0.9872331	75	#ibelievetarareade	877	0.95708765
26	#resist	4172	0.98713163	76	#bluewave	911	0.95690739
27	sexually_harass	1964	0.98711398	77	woman_girl	925	0.9566398
28	year_ago	1878	0.98687342	78	#abuse	897	0.95611033
29	#churchtoo	1775	0.98685472	79	assault_woman	889	0.95551552
30	#sexualassault	1966	0.98577285	80	#democrats	954	0.95526698
31	accuse_sexual	1737	0.98287023	81	woman_forward	873	0.95521994
32	#rape	1796	0.98241164	82	#iftheshoefits	1075	0.95471151
33	#whyididntreport	1601	0.98230331	83	abuse_woman	877	0.95464746
34	#maga	4979	0.98200629	84	woman_woman	871	0.95455617
35	#harveyweinstein	1585	0.9815185	85	year_late	821	0.95399175
36	#metoomovement	1470	0.98138939	86	blasey_ford	932	0.95264183
37	#theresistance	2018	0.98105134	87	#ge	1784	0.95242771
38	tara_reade	1323	0.97914167	88	witch_hunt	774	0.95220317
39	#mentoo	1379	0.97861311	89	nov_pm	960	0.95148879
40	pm_report	2161	0.97814214	90	#sexualabuse	1195	0.9513263
41	woman_speak	1318	0.97781399	91	sexually_abuse	823	0.95036512
42	bill_clinton	1198	0.97585202	92	woman_accuse	842	0.95026842
43	victim_sexual	1307	0.97492163	93	#love	898	0.94994016
44	tarana_burke	1093	0.9748477	94	woman_sexually	827	0.94983316
45	black_woman	1177	0.97456178	95	#oscars	784	0.94980609
46	#feminist	1121	0.97222908	96	#veteran	837	0.9491449
47	due_process	1061	0.97192975	97	#cnn	933	0.9487263
48	#tarareade	1095	0.97143397	98	domestic_violence	807	0.94819124
49	donald_trump	1123	0.9707653	99	#joebiden	808	0.94738005
50	#neveragain	1366	0.9703019	100	violence_woman	811	0.94590482

Bold keywords in black are selected ones.

Table 4. Top 100 hashtags and bigrams related to #MeToo using the WordPPR algorithm: round two of data collection.

	Word	Total frequency	Frex		Word	Total frequency	Frex
1	#believesurvivors	438	0.99887706	51	#vaw	109	0.97361672
2	#emptythepews	142	0.99816139	52	christine_blasey	1062	0.97258591
3	#tarareade	110	0.99601625	53	#christineblaseyford	112	0.97189253
4	#women	386	0.99571241	54	#scotus	165	0.97089491
5	tara_reade	1052	0.99569727	55	#womensrights	74	0.97011842
6	#fakecases	258	0.99550636	56	sexually_assault	5179	0.96956779
7	#neverbiden	69	0.99529807	57	#hollywood	128	0.96923395
8	#kavanaugh	638	0.9947867	58	#survivor	81	0.96899234
9	#domesticviolence	233	0.9934622	59	#violence	126	0.96881616
10	#ibelievetara	79	0.99345869	60	rape_woman	428	0.96790849
11	#goldenglobes	592	0.99275186	61	#theresistance	139	0.96720809
12	#survivortough	72	0.99233474	62	year_ago	1094	0.96691839
13	#abuse	303	0.99223972	63	#police	106	0.96675247
14	#childabuse	219	0.99213584	64	#legalterrorism	60	0.96621951
15	joe_biden	2194	0.99212689	65	young_woman	551	0.9652397
16	#trump	640	0.9903889	66	#survivors	89	0.9643886
17	#stopkavanaugh	240	0.99019806	67	young_girl	462	0.96423338
18	tara_reade’s	209	0.98999277	68	#ptsd	80	0.96388143
19	#csa	205	0.98988851	69	#whywewearblack	154	0.96370666
20	#kavanaughhearings	220	0.98937536	70	justice_system	313	0.96263768
21	#india	205	0.9882514	71	#gop	202	0.96262217
22	#survivorculture	75	0.98813377	72	#sexual	399	0.96222935
23	#feminism	103	0.98762507	73	hold_accountable	530	0.96143602
24	share_story	693	0.98701231	74	domestic_violence	1824	0.96055446
25	dr_ford	753	0.98700489	75	#voteblue	75	0.9603896
26	#genderbiasedlaws	118	0.98680839	76	wear_black	280	0.96007637
27	rape_culture	476	0.98499003	77	#brettkavanaugh	179	0.95872551
28	#menscommission	140	0.98334079	78	anita_hill	416	0.95782548
29	rape_victim	334	0.98325796	79	#children	115	0.95757029
30	#men	116	0.98282535	80	#grammys	151	0.95733398
31	#ibelievechristineblaseyford	67	0.98179277	81	due_process	346	0.95702246
32	#harveyweinstein	243	0.98172187	82	stand_woman	203	0.9567918
33	#resist	185	0.98151162	83	catholic_church	966	0.9563917
34	sex_abuse	737	0.98076687	84	#donaldtrump	92	0.95613511
35	rape_case	375	0.98070561	85	high_school	807	0.95594708
36	#domesticabuse	85	0.98015939	86	#blacklivesmatter	65	0.95584902
37	#justice	125	0.97986662	87	#democrats	89	0.95519353
38	#biden	80	0.97862591	88	#metooindia	75	0.95493452
39	#joebiden	86	0.97811766	89	year_late	241	0.95415319
40	#murder	150	0.97710098	90	dr_christine	308	0.95352843
41	#trauma	90	0.97699253	91	girl_woman	282	0.95283671
42	#mentalhealth	113	0.97628429	92	sexually_abuse	2094	0.9528064
43	#fathersday	60	0.97532801	93	#equality	61	0.95256364
44	social_medium	598	0.97517045	94	#violenceagainstwomen	64	0.95228858
45	#crime	135	0.97476569	95	#womenempowerment	62	0.95145431
46	blasey_ford	1099	0.97473738	96	#racism	62	0.95122199
47	#sexualviolence	154	0.97465854	97	#law	86	0.95107989
48	#consent	93	0.9742243	98	woman_forward	876	0.94990223
49	#maga	282	0.97407092	99	fake_case	76	0.9497231
50	#harassment	256	0.97406478	100	#news	494	0.9495798

Data availability statement

The method and application code files as well as the supplementary materials are available at https://osf.io/pcybz/.