Automated Visual Content Analysis (AVCA) in Communication Research: A Protocol for Large Scale Image Classification with Pre-Trained Computer Vision Models

Figures & data

Figure 1. A research protocol of using computer vision APIs for communication research

Table 1. Metrics for the best-performing model per method: validation set

Method	API(s) used	Precision	Recall	F1-Score
Profit
Expert	All	0.63	0.52	0.45
Supervised ML	All	0.76	0.76	0.76
Custom API training	Clarifai	0.74	0.72	0.72
People
Expert	All	0.68	0.68	0.68
Supervised ML	Clarifai	0.88	0.87	0.87
Custom API training	Clarifai	0.81	0.78	0.78
Planet
Expert	All	0.86	0.85	0.85
Supervised ML	All	0.90	0.90	0.89
Custom API training	Clarifai	0.83	0.84	0.82

Figure 2. Results for profit

Dotted line shows the frequency of the most common category, and it would be expected models performing well would at least surpass this threshold.

Figure 3. Results for people

Figure 4. Results for planet

Table 2. Metrics for the overall best-performing model: test set

Method	API(s) used	Precision	Recall	F1-Score
Profit	All	0.68	0.64	0.63
People	Clarifai	0.81	0.81	0.81
Planet	All	0.82	0.82	0.81

Table 3. Frequency of images in the full sample per sustainability dimension (N = 21,841)

Company	People	Planet	Profit
AXA	41%	16%	59%
Allianz	28%	19%	58%
BASF	36%	23%	52%
BMW	62%	16%	74%
BNPParibas	42%	37%	28%
Bayer	37%	21%	57%
Daimler	32%	21%	62%
Deutsche Telekom	41%	12%	58%
Gazprom	50%	16%	57%
HSBC	33%	20%	55%
ING Group	28%	17%	63%
Nestle	29%	12%	48%
Novartis	54%	15%	41%
Prudential	44%	19%	32%
Roche	43%	21%	48%
Rosneft	44%	18%	62%
Sanofi	43%	9%	64%
Sberbank	14%	3%	86%
Shell	44%	23%	53%
Siemens	6%	7%	27%
Total	40%	23%	56%
UBS	20%	14%	48%
Volkswagen	39%	17%	87%
Zurich	30%	17%	58%
Overall	37%	17%	56%

Table 4. Application contexts for AVCA across communication studies

Type of Image Data	Source	Possible RQs	Sub-field
Party/candidate visual communication	Websites/social media	To what extent do aspects of a politicians’ visual communication affect his/her party’s social media strategy and in how far do similarities/differences predict voter perceptions?	Political communication
Visual employee communication on social media	Social media	To what extent do aspects of visual employee communication predict emotional attachment to the organization?	Organizational communication
Propaganda material	Archives	In how far do historic and contemporary propaganda posters show similarities in terms of image attributes?	Communication History
User-generated visual content	Social media	Which attributes of user-generated visual content facilitate positive word-of-mouth amongst customers?	Marketing
Visuals in corporate reporting	Corporate reports (e.g., annual reports, sustainability reports)	Is an emphasis on environment-related images in sustainability reporting related to improved environmental performance and image?	Corporate communication
News images	Online media	In how far does the visual depiction of organizations in the news affect organizational reputation?	Various