Dictionary-based and machine learning classification approaches: a comparison for tonality and frame detection on Twitter data

Figures & data

Table 1. Coding scheme for the annotation of democracy dimensions.

Title	Coding label	Keyword examples	Description
Competition & fairness of electoral procedures	competition	election, campaign, candidate	Guarantees that elections are competitive, free, open, and fair (precondition for responsiveness). It also refers to the fact that the opposition is free to criticize the government.
Representation	representation	elected representatives, governing coalition, congressmen	Representation of citizens in the formal institutions, which translates in the fact that elected political actors act in citizens’ interest. The notion also refers to the crisis of democracy, but also to the type of representation (e.g. majority vs. proportional).
Self-organized participation of citizens	voice	social movement, strike, riot	Citizen's self-organization into collective actions and reinventing political activism.
Participation of citizens through democratic procedures	institutional participation	direct democracy, participatory budget, e-voting	Citizens’ views are voiced in the political dialogue to be heard by political authorities (e.g. parliament, government, court) through institutional channels (e.g. plebiscite, popular initiative, referendum, petition, e-voting, etc.).
Social fairness & social or political equality	fair	redistribution, social justice, welfare	Social equality is the elimination of social and economic differences (e.g. poverty, income inequality) that would stand in the way of political equality. Political equality ensures that all citizens have equal opportunities to participate politically and to access the law.
Efficiency	efficiency	laundering, embezzlement, corruption	Implies that the democratic system discourages political authorities’ corruption and mismanagement of resources, but also guarantees parliament and government independence (e.g. from lobbies, interest groups, mafia, etc.).
Vertical & horizontal accountability	accountability	transparency, blameworthiness, limited power	Vertical accountability refers to mechanisms through which the people control their representatives. It also implies that elected politicians account, be responsible, answer for their political decisions and give justifications form their policy choices. Horizontal accountability refers to the checks and balances on government power via the courts.
Responsiveness to citizens	responsiveness to citizens	answerability, solution oriented, public opinion	Refers to the capacity to form and implement policies that citizens want (also accounts for democratic technologies such as e-democracy, e-governance, digital-government, etc.).
Responsiveness to other stakeholders	responsiveness to other stakeholders	agreement, convention, freedom of movement	Responsiveness can also be understood in terms of answerability to other governments, supernational bodies (e.g. EU, UN, etc.) or agreement (e.g. Schengen-Dublin).
Freedom & Rule of law	rules	court, freedom of speech, impartiality	Refers to the whole set of rights and liberties available to citizens (e.g. minority rights). The courts treat everyone the same, thus implying equality before the law. Gives information about what citizens think are essential characteristics of democracy (e.g. freedom of expression, freedom of the press, reliability of the press, etc.).
Sovereignism & nationalism	sovereignism	self-determination, nationalization, autonomy	Accounts for the competing views of liberal democracy, such as nationalism and sovereignism (defending a nation’s political autonomy, preserving cultural identity, and shielding the domestic economy), which are often conflated with populism or Euroscepticism.

Table 2. Classification results for predicting the tonality of tweets related to democracy.

Precision	Recall	F-measure	Accuracy	Model	Missing
0.92	0.87	0.89	0.90	BERT	0
0.86	0.89	0.88	0.88	LR	0
0.85	0.78	0.81	0.83	ROCCHIO	0
0.95	0.65	0.77	0.81	Lexicoder	91
0.83	0.74	0.78	0.79	CustomDict	35
0.84	0.61	0.70	0.74	HuLiu	199
0.83	0.58	0.69	0.72	AFINN	177
0.85	0.55	0.67	0.71	LIWC07	790
0.91	0.27	0.42	0.62	NRC	24

Table 3. Classification results for predicting democracy dimensions of tweets related to democracy.

Category	Precision	Recall	F-measure	Accuracy	Model
Accountability	0.66	0.84	0.74	0.89	CustomDict
	0.38	0.56	0.45	0.73	BERT
	0.36	0.30	0.33	0.62	ROCCHIO
	0.23	0.26	0.25	0.59	SVM
Competition	0.69	0.83	0.76	0.90	CustomDict
	0.80	0.63	0.70	0.80	BERT
	0.73	0.50	0.59	0.74	ROCCHIO
	0.63	0.47	0.54	0.72	SVM
Efficiency	0.83	0.81	0.82	0.90	CustomDict
	0.50	0.17	0.25	0.58	SVM
	0.50	0.08	0.14	0.54	ROCCHIO
	–	0.00	–	0.50	BERT
Fair	0.78	0.81	0.80	0.90	CustomDict
	0.20	0.20	0.20	0.59	SVM
	0.33	0.10	0.15	0.55	BERT
	0.00	0.00	–	0.50	ROCCHIO
Institutional participation	0.67	0.81	0.73	0.86	ROCCHIO
	0.56	0.83	0.67	0.84	BERT
	0.63	0.72	0.67	0.81	SVM
	0.95	0.59	0.73	0.79	CustomDict
Representation	0.81	0.77	0.79	0.87	CustomDict
	0.40	0.61	0.49	0.72	ROCCHIO
	0.53	0.48	0.50	0.70	BERT
	0.42	0.45	0.43	0.67	SVM
Responsiveness to citizens	0.61	0.75	0.67	0.84	CustomDict
	0.50	0.32	0.39	0.64	BERT
	0.38	0.32	0.35	0.63	ROCCHIO
	0.29	0.29	0.29	0.60	SVM
Responsiveness to other stakeholders	0.69	0.80	0.74	0.89	CustomDict
	0.36	0.63	0.45	0.80	BERT
	0.40	0.50	0.44	0.74	ROCCHIO
	0.27	0.38	0.32	0.67	SVM
Rules	0.78	0.80	0.79	0.88	CustomDict
	0.67	0.73	0.70	0.83	BERT
	0.59	0.73	0.65	0.82	ROCCHIO
	0.50	0.56	0.53	0.73	SVM
Sovereignism	0.85	0.48	0.61	0.74	CustomDict
	1.00	0.14	0.25	0.57	ROCCHIO
	–	0.00	–	0.50	BERT
	0.00	0.00	–	0.50	SVM
Voice	0.75	0.70	0.72	0.84	CustomDict
	0.85	0.61	0.71	0.80	ROCCHIO
	0.63	0.56	0.59	0.77	SVM
	0.53	0.50	0.51	0.74	BERT

Table 4. Logistic regression on method performance to predict tonality (left) and democracy dimensions (right) as a function text and coding process characteristics.

	Tonality					Democracy dimensions
	CustomDict	Lexicoder	BERT	ROCCHIO	LR	CustomDict	BERT	ROCCHIO	SVM
(Intercept)	1.36 *	2.41 ***	1.31	1.35	0.62	1.32 *	1.16 *	1.47 **	0.93
	(0.65)	(0.71)	(0.87)	(0.73)	(0.81)	(0.63)	(0.54)	(0.55)	(0.54)
Number of words in tweet	−0.00	0.00	0.03	0.01	0.06 *	−0.01	−0.02	−0.02	−0.03
	(0.02)	(0.02)	(0.03)	(0.02)	(0.02)	(0.02)	(0.02)	(0.02)	(0.02)
Tweet contains hashtags (ref: <5)
Hashtag: >5	−0.16	0.51	0.35	−0.38	−0.36	0.57	−0.20	−0.16	−0.25
	(0.41)	(0.48)	(0.65)	(0.43)	(0.50)	(0.44)	(0.35)	(0.35)	(0.35)
Tweet contains mention (ref: no)
Mention: yes	−0.06	0.10	0.84	0.72	−0.07	−0.36	0.04	0.07	0.12
	(0.30)	(0.33)	(0.48)	(0.37)	(0.40)	(0.31)	(0.27)	(0.27)	(0.27)
Original language (ref: German)
French	0.69 *	−0.04	−0.69	−0.52	−0.43	0.75 *	0.16	−0.15	0.39
	(0.35)	(0.37)	(0.52)	(0.39)	(0.45)	(0.34)	(0.30)	(0.30)	(0.30)
Other	0.45	−0.29	−0.86	−0.39	0.15	1.22 **	−0.10	−0.26	0.20
	(0.38)	(0.39)	(0.52)	(0.43)	(0.53)	(0.40)	(0.31)	(0.31)	(0.31)
Annotated sentiment (ref: positive)
Negative	−0.68 *	−2.21 ***	0.60	0.73 *	−0.39	−0.49	−0.09	−0.46	−0.42
	(0.29)	(0.37)	(0.42)	(0.33)	(0.39)	(0.29)	(0.25)	(0.25)	(0.25)
Ambiguity in coding the tweet (ref: no)
Ambiguity: yes	0.33	0.09	0.07	−0.76	0.52	−1.35 **	−1.11 **	−1.28 **	−1.06 *
	(0.47)	(0.50)	(0.68)	(0.45)	(0.78)	(0.42)	(0.42)	(0.43)	(0.44)
Tweet contains a question mark (ref: no)
Interrogation: yes	−0.46	−0.18	−0.97 *	−0.90 *	−0.36	0.65	0.20	0.55	0.45
	(0.34)	(0.37)	(0.48)	(0.39)	(0.48)	(0.53)	(0.40)	(0.42)	(0.40)
AIC	329.02	291.52	193.01	269.39	214.09	320.60	398.44	390.10	400.43
BIC	361.98	324.49	225.98	302.35	247.06	353.56	431.40	423.06	433.40
Log Likelihood	−155.51	−136.76	−87.51	−125.69	−98.05	−151.30	−190.22	−186.05	−191.21
Deviance	311.02	273.52	175.01	251.39	196.09	302.60	380.44	372.10	382.43
Num. obs.	288	288	288	288	288	288	288	288	288

Note: Significativity levels are coded as follows: ‘***’ p< .001, ‘**’ p < .01 ‘*’ p < .05.

Figure 1. Relation between the importance of democracy dimensions for democracy in the ESS (x-axis) and the number of tweets related to each dimension (y-axis).

Table

Search-queries	French	German	Italian	Hashtags
From press review and theoretical inputs
1	democratie	demokratie	democrazia
2	democratique OR democratiques	demokratisch OR demokratisch OR demokratischen	democratico OR democratica
3	anti-democratique OR antidemocratique	undemokratisch OR undemokratische OR undemokratischen	anti-democratico OR antidemocratico OR anti-democratica OR antidemocratica
4			antidemocratica
5	democratie directe	direkte demokratie	democracia diretta
6
7	politique	politik	politico OR politica
8	politicien OR politiciens	politiker OR politikern	politici OR politiche
9	politicienne OR politiciennes	politikerin OR politikerinnen	deputato OR deputata OR deputati OR deputate
10			parlamentari
11	votants	wähler	elettori
12		wahler
13	citoyennete	staatsbürgerschaft	cittadinanza
14		staatsburgerschaft
15	citoyens	staatsbürgern	cittadini
16		staatsburgern
17	citoyen	stimmbürger	cittadino
18		stimmburger
19	peuple	volk	popolo
20			gente
21	elite OR elite	eliten	élite
22			elite
23	elu OR elus OR elue OR elues	volksvertreter OR volksvertretern	rappresentanti
24	conseiller OR conseillers OR conseillère OR conseillères	abgeordnete OR abgeordneten	rappresentante
25	parlementaire OR parlementaires
26	gouvernement	regierung	governo
27	conseil federal	bundesrat	consiglio federale
28	parlement	parlament	parlamento
29	assemblee federale	bundesversammlung	assemblea federale
30	representation AND politique	vertretung
31	constitutionnel OR constitutionnelle	verfassung	costituzionale
32	federal OR federale	federal	federale
33		bundes	federali
34	federales OR federaux	staat OR staatlich
35		staats
36	droits populaires	volksrecht	diritti popolari
37	initiative	initiativ	iniziativa
38			iniziative
39	votation	abstimmung	voto
40	votations	abstimmungen	voti
41	votation populaire	volksabstimmung	voto populare
42	consultation populaire	volksentscheidung
43	scrutin	volksbefragung
44	votations populaires	volksabstimmungen	voti populari
45		abstimmungskampf
46	elections	wahlen	elezioni
47	campagne AND electorale	wahlkampagne	campagna AND elettorale
48
49	bataille AND electorale	wahlkampf	concorrenza AND elettorale
50
51	referendum	referendum	referendum
52
53	referendaires
54
55	décision populaire	stimmvolk	decisione popolare
56		volksentscheid
57	institutionnel OR institutionnelle	institutionel OR institutionnelle	istituzionale OR istituzionali
58	vote OR voter	stimmen	votare
59	contre-projet	gegenvorschlag	controprogetto
60	parti politique OR partis politiques	partei OR parteien	partito OR partiti
61	le parti OR les partis
62	droite AND parti OR partis OR politique	rechte AND partei OR parteien OR politik	destra AND partito OR partiti OR politica
63	gauche AND parti OR partis OR politique	linke AND partei OR parteien OR politik	sinistra AND partito OR partiti OR politica
64	centre AND parti OR partis OR politique	mitte AND partei OR parteien OR politik	centro AND partito OR partiti OR politica
65	extrême OR extreme AND droite OR gauche	rechtsextrem OR linksextrem	estrema AND destra OR sinistra
66	droits politiques	politische rechte OR politischen rechten	diritti politici
67	coalition	koalition OR coalition	coalizione
68	souverainete	soveränität	sovranità OR sovranita
69	souverain	souverän OR souveränen	sovrano
From hashtags review
1				#abst18
2				#abst19
3				#Abstimmungssonntag
4				#AccordCadre
5				#Bundeshaus
6				#Bundesrat
7				#Cantonali19
8				#cantonali2019
9				#chvote
10				#Conseilfédéral
11				#ConsigliodiStato
12				#ConsiglioFederale
13				#contreprojet
14				#controprogetto
15				#deardemocracy
16				#democratie
17				#démocratieparticipative
18				#democrazia
19				#democraziadiretta
20				#Demokratie
21				#DirekteDemokratie
22				#eDemocracy
23				#ef19
24				#ef2019
25				#eGov
26				#eGovernment
27				#elezioni2019
28				#evoting
29				#eVotingMoratorium
30				#facciamolo
31				#gegenvorschlag
32				#granconsiglio
33				#iniziativa
34				#Lobby
35				#Lobbying
36				#nationalratswahlen
37				#NeinzumEURahmenabkommen
38				#NeinzurEU
39				#NoEvoting
40				#nrw19
41				#parlCH
42				#parldigi
43				#participation
44				#politica
45				#politik
46				#politique
47				#Rahmenabkommen
48				#Rahmenvertrag
49				#Rechtsstaat
50				#referendum
51				#sociétécivile
52				#souveraineté
53				#Souveränität
54				#Sovranità
55				#SwissEUrelations
56				#swissregulations
57				#transparence
58				#transparenz
59				#Versicherungslobby
60				#verwaltung
61				#Volksabstimmungen
62				#Volksinitiativen
63				#vorwärts
64				#vot18
65				#vot19
66				#votations
67				#votazioni
68				#wahlCh19
69				#wahlen19
70				#wahlen2019
71				#wahlenCH19