News Credibility and Influence within the Financial Markets

Abstract

How does information credibility, a subjective judgment of investors, affect empirical asset pricing in financial markets? Traditional economic theories are inadequate for interpreting market responses driven by people’s subjective thinking, as these cognitive processes are not encompassed by the concept of utility. We explore these effects by using computational linguistics and deep structured learning algorithms to analyze financial newspapers and social media posts. After controlling for factors related to content and market momentum in our narrative based credibility indicator, we find that news credibility is positively correlated with the returns on assets preferred by experts and negatively correlated with assets preferred by gamblers. Based on this finding, we point out that the efficient-market hypothesis (EMH) is not appropriate in the dominant market of gamblers in the short-term. In the long-term, however, investment motivation does not significantly affect the validity of the hypothesis.

Keywords:

• News
• Media
• Behavioral finance
• Cryptocurrency valuation

JEL CODES:

• G14
• G41

1. Introduction

In an efficient capital market, asset prices should fully absorb all available information quickly and without bias (Malkiel and Fama 1970). Although empirical evidence supports this argument, many studies still question the validity of this insight. For instance, many empirical studies have found ambiguous asset return patterns and thus reject the assumption that the capital market is efficient. Some studies find return reversal, while others find drift patterns when new relevant information is published, as discussed in the survey conducted by Tetlock (2014). One of the key assumptions imposed here is that market participants are strictly rational. For example, the efficient market hypothesis (EMH) assumes that market participants receive and respond rationally and consistently to the information they receive and update their decisions immediately based on this information. Psychologists Glimcher (2011) have criticized economic theories for neglecting people’s goals, hopes or desires; as they are not observable, economists tend to only consider how they constrain people’ preferences in the most limited sense. There are a number of different types of subjective factors, such as emotion (Bollen, Mao, and Zeng 2011), superstition (Hirshleifer, Jian and Zhang, 2018), the weather (Saunders 1993) or even the FIFA World Cup (Kaplanski and Levy 2010) may be regarded as a potential psychological factor affecting the market. From the perspective of emotion, many behavioral finance works (e.g., Tetlock, Saar-Tsechansky, and Macskassy 2008; Uhl 2014; Barberis, Shleifer, and Vishny 2005) indicate the connection between human psychological factors such as sentiment and market performance. However, one important issue often overlooked in previous sentiment analysis in behavioral finance is the perceived credibility of the information. How market participants interpret and respond to information that they do not fully believe remains unclear. Unlike previous sentiment studies primarily consider daily news sentiment (e.g., Tetlock, Saar-Tsechansky, and Macskassy 2008) or investor sentiment (e.g., Brown and Cliff 2005), in this paper we primarily focus on the perception sentiment of newly released information and its impact on the market performance. We define investors’ sentiment on news descriptions as news perception credibility, or simply called news credibility. In the work of measuring the credibility of news, Jin et al. (2014) consider news with different sentiment polarities have no effect to each other. Naturally, when two correlated but different sources of information have similar sentiment polarities, it can be considered that one source considers the other is relatively credible. We are therefore inspired to compare the sentiment of news with the sentiment of investors on social media to construct the news perception credibility indicator.

Yet there remains a lack of knowledge on the effects of news credibility on financial markets. We consider two types of investors participate in the market: experts and gamblers. Experts are all market participants exposed to market assets based on objective information, including financial statements and expert analysis reports. In contrast, we define gamblers as market participants exposed to assets based on subjective factors. We believe that subjective factors have different effects on different types of people. For example, for a perfectly rational economist, the victory of her favorite team should not affect her decision regarding the company stocks that she will buy. (We assume that the firm is not related to the team.) A market is made up of people, so we can reasonably infer that the people who make up the market will have diverse responses to these subjective factors. To distinguish the effects of credibility on experts and gamblers, we focus on two financial markets: the stock market and the cryptocurrency market.

There are many sources of information available to market participants, and the Internet and electronic devices are continuously increasing the speed of information dissemination and reader coverage. Gaziano (1988) argues that people are less likely to pay attention to the press if they do not believe or trust the press. A well-known and reliable source of information is therefore essential for information to be well received and considered “newsworthy” by most market participants. In this analysis, we use two types of markets (the stock market and the cryptocurrency market) to describe the dominant markets of experts and gamblers. Our dataset consists of 349 human-annotated news articles published between 1 March 2018 and 10 October 2018 in the Financial Times. For the experts’ dominant market, we consider the consumer staples sector of S&P 500 firms, as company fundamentals are clearly exposed and relatively easy for market participants to recognize. For gamblers, we consider the Bitcoin cryptocurrency market as the dominant market. Bitcoin was introduced by Nakamoto (2008) as a purely peer-to-peer electronic cryptocurrency that did not require the participation of trusted third parties. Although there are many cryptocurrencies on the market, Bitcoin is one of the most popular, and thus can represent the behavior of most cryptocurrency market participants. Although it is true that some people, especially those who use cryptocurrencies for privacy purposes, use cryptocurrencies for real transactions, Athey et al. (2016) shows that the main function of Bitcoin today is investment. A set of 276,701 comments in StockTwits is used to measure news credibility among market participants. For both markets, we measure abnormal asset returns one day, one week and two weeks after the publication of the news to analyze asset pricing.

This paper has three main objectives. The first is to measure the credibility of information in global financial markets. The second is to identify and test the textual characteristics that can affect information credibility. The third is to broaden our knowledge of asset pricing patterns driven by imperfectly credible information shocks. To answer our research questions, we use a set of news articles about the consumer staples sector of S&P 500 firms and Bitcoin in 2018. We collect all of the news articles relevant to the study of information shocks and annotate them as bullish, neutral or bearish. In addition, we collect pre-annotated comments from a social network to estimate the credibility of these information shocks. To understand what can drive the dynamics of news credibility, we focus on market momentum, news content and macro controls. Finally, we evaluate asset returns after a shock to interpret the effects of imperfectly credible information absorption patterns on a financial market.

This paper extends research on behavioral finance from news sentiment to news perception credibility. This study makes three concrete contributions to the literature. First, we show a clear difference in information credibility between our experts and gamblers. We create a narrative-based credibility indicator and other content-based controls that will help researchers to better understand the effects of narrative events on asset pricing. In this process, an important task is to classify the sentiments of user comments on social media. Unlike most literature that only deals with polar emotions, we also consider the impact of neutral emotions on social networks in financial markets. In order to reduce the noise of missing labels in the neutral annotations that we recognize, we control the weight of this part of neutral emotion by introducing the coefficient λ. Second, we extend our knowledge of gamblers in financial markets by taking into account behavioral factors. We show that news credibility has a positive effect on asset pricing in the dominant market of experts, but this effect is negative in the dominant market of gamblers. One explanation for this result is that when the information is more credible, experts are more likely to rely on the new information they receive, increasing the effect of the news shock. In contrast, information that is not credible strengthens the importance of gamblers’ subjective expectations for asset pricing, thereby causing bubbles. Third, we answer the fundamental question in finance — whether the EMH still appropriate when irrational traders dominant the market. Our finding suggests the answer is yes, but only valid in the long-term of the gamblers’ dominant market.

This paper is organized as follows. The second section reviews the literature that motivates the empirical research in regard to our study. The third section describes the data sources and collection criteria used. The fourth section explains how we measure news credibility in our study. The fifth section presents the main results of the effects of information credibility on market returns. The sixth section explores the robustness of our results, and seventh section concludes the paper.

2. Literature review

We begin our discussion of the role of speculators in asset markets. Economic theory generally considers that speculative activity makes asset markets more efficient (e.g., Grossman and Stiglitz 1980; Grossman 1995) by improving their information and allocation functions. Speculators can be strategic traders who manipulate the stock price for profits by short-selling the stock (Goldstein and Guembel 2008) based on the feedback effect from the stock market to corporate decisions. In contrast, a body of literature (e.g., De Long et al. 1990; Lux 1995; Shleifer and Summers 1990; Shiller 2003) report that speculative activity is driven by noise traders, herding and bubbles can destabilize the market and make asset markets less efficient. Therefore, the overall effect of speculative activity on market efficiency can be positive (through strategic trading) or negative (through the effects of irrational traders). The foremost difference from speculators is that gamblers rudimentarily rely on irrational factors when making investment decisions. If speculators have an indefinite impact on the EMH, then gamblers who recognized as irrational traders might have a more inexplicit effect on the hypothesis. Under the traditional view of the EMH, gamblers who rely on factors unrelated to the fundamental value of the assets will lose money and eventually leave the market (see, e.g., Friedman 1953). However, there is considerable literature (e.g., De Long et al. 1988; Hirshleifer, Subrahmanyam, and Titman 2006) shows that irrational traders can earn more abnormal returns than rational informed investors in some situations, this situation is called the survival of irrational traders. In addition, some speculative assets can be difficult for most traders to assess their fundamental value, thus there may be relatively few rational investors in these markets. Therefore, it is appropriate to assume the EMH should consider the impact of irrational gamblers on asset markets.

A large amount of literature mainly focuses on the impact of news events on the stock market. Traders in the stock market tend to be more rational than asset markets with low fundamental value, such as cryptocurrency markets. Intuitively we can infer that rational and irrational people differently respond to subjective factors. Sentiment analysis is usually a method used by behavioral finance researchers to study the impact of news events and irrational factors on financial markets. (Edmans, Goldstein, and Jiang 2015) study strategic speculators’ incentives to trade on information and indicate the feedback effect rises (decreases) the profitability of buying (selling) on bullish (bearish) news. Antweiler and Frank (2004) suggest that news stories from the Internet help predict market volatility and that the influence of news on stock returns is statistically significant but economically small. Tetlock, Saar-Tsechansky, and Macskassy (2008) use a simple quantitative measure of language in firm-specific news articles and find that market prices consistently underreact to negative words in firm-specific news stories, especially those on fundamental value. Calomiris and Mamaysky (2019) examine how a news article and its context drive risk and returns. They identify the influence of news measures on changes in market performance by country type over time, showing that the content of news articles conveys more information in emerging markets than in developed economies. In general, the previous sentiment studies can be classified into two categories in general. Some studies such as Tetlock, Saar-Tsechansky, and Macskassy (2008) and Uhl (2014) choose daily news sentiment data for analysis and successfully used sentiment indicators to explain the performance of some markets at certain periods of time. On the other hand, Brown and Cliff (2005) represents another analysis approach by looking at investor sentiment rather than news sentiment and show that high investor sentiment leads to overvaluation.

Naturally, it leads to a question of whether it is workable to amalgamate the different sentiment categories for behavioral finance studies. In this paper, we define the news and investors sentiment amalgam as news perception credibility or news credibility for short. If news sentiment is always homogeneous to investors sentiment, then news credibility is redundant. However, as Shang, Brooks, and McCloy (2014) claim market participants treat the sentiment in listed firms’ narratives might deviate from how they value mass media narratives, news credibility is valuable to capture the latent gap between the sources and investors sentiment. One extension of the news and investors sentiment amalgam is the study of news absorption patterns. These studies use the event study approach proposed by Fama et al. (1969) to show how market participants respond to information. For instance, Frank and Sanati (2018) perform a textual analysis of news stories from the Financial Times and find a significant difference in market responses to positive and negative price shocks with regard to new information. Defining a price shock as a price change for a given stock from one day to the next, the authors conclude that the stock market will overreact to good news and underreact to bad news. In addition, the stock market will overreact more to negative no-news shocks than to positive no-news shocks. The authors provide one plausible explanation for these seemingly contradictory results is the attention bias of investors. In other words, investors’ perception of news sentiment rather than news sentiment per se might be more relevant to market performance.

There are controversial definitions of news media credibility in the literature. Abdulla et al. (2002) measure credibility by three dimensions: trustworthiness, currency and bias. Metzger et al. (2003) comprise trustworthiness and expertise, fairness, bias, incompleteness, concern for community, separation of opinion and fact, and accuracy seven dimensions for credibility estimation. Thorson, Vraga, and Ekdale (2010) recognize fairness, completeness, bias, accuracy, trustworthiness, and addressing perceptions of balance in the news six dimensions that appear to be important in news credibility judgments. Some studies on the other hand address a similar issue using the duration of source credibility (Hovland and Weiss 1951), the comparison of credibility between different sources (Abdulla et al. 2002; Cassidy 2007; Lee 1978) or the factors that can influence source credibility (Thorson, Vraga, and Ekdale 2010). The main methodology of these studies is the usage of questionnaires or interviews. However, these multidimensional definitions might only appropriate to study the credibility of media when the samples are relatively small. Newhagen and Nass (1989) claim the credibility of mass media news is to treat news information as a reasonable reflection of the events they depict. Jin et al. (2014) further argue that messages with disparate sentiment polarities have no sway over each other. We are inspired by these views and hence introduce the bi-dimensional news perception credibility that only rely on news sentiment and investors sentiment. Our approach significantly reduces the difficulty to measure the credibility of media, especially in the big data era.

This paper extends research on behavioral finance from news sentiment to news credibility. In addition to contributing to the growing literature on news absorption patterns in the financial market and the correlation between news content and market reaction, this paper introduces the concept of news credibility and studies its role in the news absorption process, which explains previous inconsistent results on news shock return patterns. Our results indicate the usefulness of newspaper and social media narratives for researchers examining traditional and new financial markets from the perspective of behavioral finance.

3. Financial news and social network data

One of the main challenges in our research is to select an appropriate subset to reflect the indispensable characteristics of target markets. For the experts’ dominant market, we select 33 companies from the consumer staples sector of S&P 500 component stocks as our sample with at least one related news that appeared in the newspaper. Our main consideration is that the activities and main products of the companies in this sector are clear to most general investors. Most products in the consumer staples sector, such as food and beverages, are designed to meet the basic physiological desires of consumers. Therefore, companies in the consumer staples sector may be easier to be evaluated by investors, while products in the semiconductor or mining industry may be more complex or unfamiliar. On the other hand, most manufacturers are in monopolistic competition and thus invest plenty of resources in advertisement and product information disclosure. These two reasons lead to the consumer staples sector being highly valued by experts (such as Warren E. Buffett) who predominately rely on objective data to make investment decisions. In addition, these listed companies provide a large amount of substantial data that experts can analyze. The daily news reports (UTC time) of these S&P 500 firms used here are those published by the Financial Times from 1 March 2018 to 10 October 2018. Because our news credibility depends on the daily real-time sentiment of social networks, we must limit the time period of our research to avoid abuse of social network server resources and the expensive cost of data collection, storage and analysis. We therefore choice around 8 months in our study. On the other hand, we hope that Bitcoin is relatively familiar to investors in the selected time period. Simultaneously, there should not be too many factors affecting investors apart from financial news. For example, Bitcoin’s price fluctuations will be much greater than in the previous period in 2020. Tesla’s attitude toward Bitcoin payments and Elon R. Musk’s remarks might be one of the main factors affecting its price. As a result, the period after 2020 may have a certain exogenous influence on our news research. Based on previous considerations, we have decided that our study periods in the stock market and the Bitcoin market will be the same from March to October 2018.

The Financial Times and the Wall Street Journal are both representative and reliable sources of general financial news. Nevertheless, Frank and Sanati (2018) suggest that the Wall Street Journal is more geared toward US readers. In contrast, the Financial Times has a global readership and therefore tends to have higher criteria for selecting and publishing news preferred by global subscribers. In this study, we focus on the opinions of global market participants and therefore choose news articles published in the Financial Times as our main source of information.

To evaluate news credibility in the stock market, we collect 245 stock news articles from the Financial Times website. These news stories are collected if an article appears under the website query topic based on the firm name (for instance, the query topic “Coca-Cola Co.” in the Financial Times indicates the Coca-Cola Company). Using the query topics of articles annotated by this official website prevents us from missing relevant news stories due to common variations in company names.

We select the Bitcoin market to represent the dominant financial market of gamblers. To estimate news credibility in the cryptocurrency market, we collect 104 news articles related to the keyword “Bitcoin” from the Financial Times website. These news stories are selected if an article appears on the website after searching for relevant keywords. Our public narrative data set includes 242,361 unduplicated observations corresponding to the 33 S&P 500 firms of the consumer staples sector and 34,340 comments related to “Bitcoin” in StockTwits. StockTwits is a social network created in 2008 for investors, traders and entrepreneurs to share ideas. In 2013, this social network had 230,000 active users. These data are collected if (i) the articles refer to communities relevant to the markets of interest, as indicated by the hashtags,¹ and (ii) the article is not an advertisement. Compared with news articles in the Financial Times, texts in StockTwits are very unstructured with many casual terms, such as slang, online terms, emoji etc. One of our main reasons for choosing this social network is that it enables users to use “bullish” or “bearish” tags to indicate their market expectations to the public.

Figure 1 illustrates the sentiment variable and comments provided by StockTwits users. These annotations can therefore be used as a proxy for researchers to efficiently study the subjective cognition of market participants on a large scale. In addition, we use textual data from Twitter in our robustness tests. We collect 4,685,417 unique tweets from 29 May 2018 to 10 October 2018 on topics related to “Bitcoin”. More details are provided in the section, Robustness Tests.

Figure 1. Example of StockTwits messages.

Note: This figure illustrates web-page of StockTwits Messages in regard to the “Bitcoin” relevant hashtags.

4. Measuring credibility

As Nagura et al. (2006) indicate that the credibility of online news is related to the judgment of readers, in this section, we propose a method to rate the credibility of financial news from the perspective of investors’ perception of news sentiment. Today, many investors are willing to share their comments about the market with polarized sentiment in social networks. These investor sentiment data are valuable for researchers to measure news credibility by comparing it with market news sentiment. Big data analysis and crawler technology make it possible to analyze investor sentiment based on social networks. Nevertheless, our main empirical challenge is to construct credibility measures for the 276,701 self-annotated comments on the 349 newspaper articles. Each observation with a sentiment label “bullish” indicates that the user is optimistic about the market, “bearish” indicates that the user is pessimistic about the market and an empty label indicates that the user makes no judgment, thus we classify this empty label as “neutral”. Although it is reasonable to speculate that a “bullish” or “bearish” label can correctly indicate the investment trend of these social network users, it is difficult to interpret the cognition of “neutral” label users. On the one hand, a “neutral” label can indicate that these users do not expect the event to have a significant effect on the market. On the other hand, this label can indicate that the news is not relevant to the market, or that users have chosen to omit their expectations concerning the market. The first scenario signals that this news article is de facto neutral for the market, whereas the second scenario reduces the information conveyed by the “neutral” comments.

So far, the most common sentiment analysis approach to textual data has been the binary or polarity classification. This approach only classifies each document as positive or negative. One of the main advantages of the binary classification approach is that it reduces the complexity of the classification problem. However, a disadvantage of this approach is that if most users consider the news neutral for the market, this approach simply ignores this majority of the observations, leading to an overestimation of polarized sentiment following the news shock. To address this problem, we use a parameter λ ∈ [0,1] to control the weight of “neutral” comments among social network users. λ determines the effect of users’ “neutral” expectations on their overall sentiment, a challenge being to choose the appropriate value of λ. There is a tradeoff between a lower λ and a higher λ. If λ is close to 0, our approach will become similar to the binary classification approach as it largely ignores the “neutral” comments of social network users. However, if λ is close to 1, the noise of “neutral” comments will overwhelm the optimistic or pessimistic expectations of other users. In this analysis, we use a lower λ ∈ (0,0.1] after experimenting with different numerical values from 0 to 1 to ensure that the effect of a positive (negative) news shock on the social network will not be overwhelmed by the noise of “neutral” comments.

The rest of this section explains how we develop a continuous sentiment measure for social network users, then discusses the approach we use to identify news shocks from the newspaper. Next, we present the news credibility measures of recognized news shocks based on the sentiment measure from the social network. Finally, we describe how we construct textual measures from relevant news articles to measure subjective factors in terms of news credibility in general financial markets.

4.1. Continuous sentiment in the social network

In the original StockTwits data set, the sentiment of each social network user is a discrete indicator of positive, negative or neutral market expectations. Jin et al. (2014) propose a probabilistic mapping from the binary credibility annotation to measure Microblog credibility. Analogously, we propose several steps to capture the average market expectation among social network users with a continuous sentiment variable through a probabilistic mapping from discrete labels. The goal is to improve the actual information content of the sentiment indicator, allowing us to conduct more reliable studies on the factors that can affect people’s expectations and cognition.

First, for each market-related community i from StockTwits, we group all records according to their publication date t and count the number of each label $N_{\mathit{ist}}.$ s ∈ {0,1,2} is an indicator of the sentiment annotation, with 0 indicating “bearish”, 1 “neutral” and 2 “bullish”. This classification is akin to the methods proposed by Uhl (2014) and Gotthelf and Uhl (2019), which divide news sentiment into three categories: positive, negative, and neutral, but the numerical range in their work is rather between −1 (negative) and 1 (positive). We use a weighted ${\widehat{N}}_{\mathit{ist}}$ instead of $N_{\mathit{ist}}$ to measure the frequency of each sentiment of community i at time t, where ${\widehat{N}}_{\mathit{ist}}$ is given by $${\widehat{N}}_{\mathit{ist}}=\{\begin{array}{c}{N}_{\mathit{ist}}\mathit{\text{if s}}=0\\ \lambda N_{\mathit{ist}}\mathit{\text{if s}}=0\\ N_{\mathit{ist}}\mathit{\text{if s}}=0.\end{array}$$

To compute the overall daily sentiment label for each asset relevant to the community from the social network, ∀i ∈ $Z^{+}$, t ∈ ${\mathbb{R}}^{+},$ we use the following criteria: (1) $$S_{it}≔\arg \underset{s}{\max }\frac{{\widehat{N}}_{\mathit{ist}}}{\sum _s{\widehat{N}}_{\mathit{ist}}},$$(1) (2) $$L_{it}≔\underset{s}{\max }\frac{{\widehat{N}}_{\mathit{ist}}}{\sum _s{\widehat{N}}_{\mathit{ist}}}.$$(2) where $S_{it}$ is the expected overall daily sentiment label and $L_{it}\in \left[0,1\right]$ is the probability that $S_{it}$ accurately measures the overall daily sentiment of social network users at time t in community i.

The second step in measuring sentiment in the social network is to map a discrete $S_{it}$ based on $L_{it}$ to a set $\mathcal{S},$ where $\mathcal{S}≔\{x\in R\mid 0\le x\le 2\}.$ To this end, we use a ${\mathcal{L}}_2$-loss function defined as $\mid y_t-{\widehat{y}}_t{\mid }^2$ to measure the distance between the estimated variable $\widehat{y}$ and the true variable y_. ${\mathcal{L}}_2$-loss is an error measurement instrument widely used in machine learning applications and research (Bühlmann and Yu 2003; Chang, Hsieh, and Lin 2008; Mao et al. 2016; Zhao et al. 2017). ${\mathcal{L}}_2$-loss allows researchers to reduce error sensitivity if the estimated value is close to the observed value and to increase it if the error deviates from the observation. Applying the ${\mathcal{L}}_2$-loss function to estimate $L_{it}$ based on Equation (2)(2) $$L_{it}≔\underset{s}{\max }\frac{{\widehat{N}}_{\mathit{ist}}}{\sum _s{\widehat{N}}_{\mathit{ist}}}.$$(2) at time _t, we obtain the following: (3) $${\mathcal{L}}_2\left(i,t\right)=\mid 1-L_{it}{\mid }^2.$$(3)

If a real annotation exists, the corresponding probability of this label should always be equal to 1. Using $S_{it}$ from Equation (1)(1) $$S_{it}≔\arg \underset{s}{\max }\frac{{\widehat{N}}_{\mathit{ist}}}{\sum _s{\widehat{N}}_{\mathit{ist}}},$$(1) and ${\mathcal{L}}_2\left(i,t\right)$ from Equation (3)(3) $${\mathcal{L}}_2\left(i,t\right)=\mid 1-L_{it}{\mid }^2.$$(3) , we construct the following continuous sentiment measure: (4) $$\zeta \left(i,t\right)\mathrm{ }=\{\begin{array}{c}{N}_{\mathit{ist}}\mathit{\text{if s}}=0\\ \lambda N_{\mathit{ist}}\mathit{\text{if s}}=0\\ N_{\mathit{ist}}\mathit{\text{if s}}=0.\end{array}$$(4) where $\mathrm{\zeta }\left(i,t\right)\in \mathcal{S}$ captures the sentiment dynamics from pessimistic expectations to optimistic expectations among social network users in community i at time t.

Similar to $S_{it}$ in Equation (1)(1) $$S_{it}≔\arg \underset{s}{\max }\frac{{\widehat{N}}_{\mathit{ist}}}{\sum _s{\widehat{N}}_{\mathit{ist}}},$$(1) , we use $S_{i\mathrm{\tau }}^n$ to denote the sentiment of the relevant news article published at time τ. Using $\zeta \left(i,t\right)$ in Equation (4)(4) $$\zeta \left(i,t\right)\mathrm{ }=\{\begin{array}{c}{N}_{\mathit{ist}}\mathit{\text{if s}}=0\\ \lambda N_{\mathit{ist}}\mathit{\text{if s}}=0\\ N_{\mathit{ist}}\mathit{\text{if s}}=0.\end{array}$$(4) , we define narrative-based credibility $\mathrm{\Gamma }\left(i,\mathrm{\tau },t\right)$ in community i at time t with regard to the news shock at time τ as follows: (5) $$\mathrm{\Gamma }\left(i,\mathrm{\tau },t\right)=1-{\left(\frac{S_{i\mathrm{\tau }}^n-\mathrm{\zeta }\left(i,t\right)}{d}\right)}^2,$$(5) where $d≔\left|\sup \mathcal{S}-\inf \mathcal{S}\right|$ is equal to the length of $\left[0,2\right]$ in our study. $\mathrm{\Gamma }\left(i,\mathrm{\tau },t\right)\in \left[0,1\right]$ is a continuous measure of the distance between the direction of the news shock and the sentiment in the social network. By using the ${\mathcal{L}}_2$-loss function of Equation (3)(3) $${\mathcal{L}}_2\left(i,t\right)=\mid 1-L_{it}{\mid }^2.$$(3) , we reduce the sensitivity of the indicator when public sentiment is close to the direction of the news shock, and vice versa.

4.2. Identification of news shocks

A news shock refers to an event published in the news media that can attract people’s attention. We consider this type of information as “newsworthy”. It is a difficult task to identify the type of news that interests each market participant, but we can simplify this task by assuming that each market participant will only be interested in financial news related to the assets in which they wish to invest. It is therefore essential to determine whether a narrative is relevant to the target market. We follow two steps to detect and annotate the relevance of each news article in our data set to ensure that each potential shock can be identified accurately and effectively.

First, we define several classification rules that allow a computer to automatically delete irrelevant news articles. As mentioned earlier, we use the sector name (usually the name of a firm or asset) in the Financial Times to identify and build our news collection. To design the classification rules, we divide the sector name into individual words and remove common stop words like “Co”, “Inc” and “Corp”, which appear in most corporation names. In textual analysis, these procedures are called tokenization and preprocessing. We use the remaining tokens as keywords to match the topics and content of news articles in each sector. The score for a successful match between a token in the topic and a keyword should be 2, and the score for a successful match between a token in the content and a keyword should be 1. As a sector name can generate multiple keywords, we only use the maximum score among all of the keywords. We compare this score with a predefined threshold value θ_k and classify the news as relevant if the maximum score is higher than θ_k. One challenge is to choose the appropriate threshold value. We use θ_k = 3 after considering a tradeoff between the accuracy and efficiency of the classification rules. To avoid the possibility that a small variation in the name of a token related to a keyword can lead to a mismatch with the keyword, we use the “fuzzywuzzy” package in Python to calculate the similarity ratio between each token in a news story and the keyword. For instance, the similarity ratio between “Cola” and “Coca-Cola” is 62 and the ratio between “Estée” and “Estee” is 80. We define a successful match between a token in a news story and a keyword in a sector name if the similarity ratio is greater than a threshold θ_s, and we use θ_s = 60 after experimenting with the transformation of different keyword-related token names.

Our second step is to ask experts with at least a Master of Finance or a degree in other relevant subjects to check the news stories that have been identified as “irrelevant” by our classification rules as some relevant news articles may fail to pass our rules, for example if they include only a few keywords in their title and content. For each discarded news item, we provide the name of the company or financial asset and ask these experts to manually annotate whether the news is relevant or not.

After using our auto-classification rules and the annotations of experts to indicate the relevance of the news stories in our collection, we ask financial experts to interpret and annotate each relevant news article with the sentiment label “bullish”, “neutral” or “bearish”. Following the previous notation, we use $S_{i\mathrm{\tau }}^n\in \{0,1,2\}$ to denote each news shock, where $\mathrm{\tau }\in R^{+}.$ Table 1 summarizes the news shock identification results based on our data.

Table 1. News collection summary with annotations.

	Raw	Relevant	Bullish	Neutral	Bearish
Total documents	349	270	104	59	107
Total words	189,003	149,124	56,979	41,206	50,939

Note: All of the raw news articles are collected from the Financial Times and the relevant news articles are determined by our classification rules and the knowledge of financial experts. We remove punctuation in the title and content of each news article using the string package in Python when calculating the number of words in our data set.

4.3. Computational linguistic approach to the media

Textual analysis is becoming more and more popular among economists, as it provides powerful and efficient tools for quantitatively studying large-scale information incorporated into texts. Shiller (2017) argues that textual analysis will become a more solid field of research in economics in the coming years due to the big data generated by social media and the development of interdisciplinary research involving subjects such as psychology, neuroscience and artificial intelligence. Numbers, symbols, punctuation and topics are direct carriers of information in textual media. To study the credibility of information shocks, it is necessary to control for these plausible linguistic effects on credibility. To this end, in their article, Hansen, McMahon, and Prat (2018) develop “counting measures” and “topic measures” to quantify the reports of the Federal Open Market Committee so that they can study the behavior of policymakers. In this paper, we use counting measures and the latent Dirichlet allocation (LDA) (Blei, Ng, and Jordan 2003) based on topic modeling to quantify news content and determine how the content affects news credibility.

4.3.1. Counting measures

In terms of counting measures, we choose four indicators, i.e. questions, exclamations, numbers and capitals, to capture content-based information of different dimensions from the news narratives. Table 2 summarizes these four counting measures.

Table 2. Summary of counting measures.

Name	Description
Questions	Number of questions asked
Exclamations	Number of exclamation statements made
Numbers	Number of figures presented
Capital	Number of currencies presented

The question measure enables us to estimate the degree of uncertainty in an article. The exclamation measure allows us to roughly measure the intensity of the tone in a news narrative. The number measure can be used as a proxy for numerical language in an article, indicating the quantitative reliability of the source of information. Finally, the capital measure controls the frequency of financial assets presented in the media. All four measures are considered to be content-based characteristics that can affect the credibility of narratives introduced in Equation (5)(5) $$\mathrm{\Gamma }\left(i,\mathrm{\tau },t\right)=1-{\left(\frac{S_{i\mathrm{\tau }}^n-\mathrm{\zeta }\left(i,t\right)}{d}\right)}^2,$$(5) .

4.3.2. Topic measures

Next, we describe how we construct our topic measures of news shocks. In the first step, we analyze the content of a news article based on information hidden in the text. One limitation of counting measures is that this approach omits any information transmitted by text combination. Different news articles can discuss different topics even if they have the same shock direction, affecting market participants to varying degrees. For instance, good news related to agriculture may not affect people as much as good news related to the banking system. We mitigate this concern by using the LDA model to identify underlying topics in a series of news narratives. Topic models are a type of statistical model used to identify topics in a collection of documents, and it has been widely used in the field of information retrieval, textual analysis and natural language processing. As a typical topic model, LDA provides a quantitative and intuitive approach for researchers from different disciplines interested in mining and analyzing information from textual data.

4.3.2.1. LDA topic modeling

The basic concept of the LDA topic model is that each topic is a set whose elements are specific high frequency words tha are more likely to appear in documents discussing that topic. Suppose that there are K topics and D relevant documents in the news collection. The collection can be coded according to a document’s term frequency as vectors ${\mathit{W}}_1,\dots ,{\mathit{W}}_D\in {\mathbb{R}}^N,$ where N is the number of unique elements in a set that includes all words detected in the collection, also called a N size dictionary. To reduce the complexity and noise caused by trivial words, we use the vocabulary indicator in the counting measure stage for all relevant documents to construct the dictionary of size N. To represent each abstract topic, we use vectors ${\mathrm{\beta }}_1,\dots ,{\mathrm{\beta }}_{\mathit{K}}\in {\mathbb{R}}^N.$ Each element in a vector corresponds to a word in the dictionary of size N and the value of each element in a topic is determined by the term frequency of the words in the dictionary. The LDA model assumes that each document includes several topics, which can be denoted by vectors ${\mathrm{\Theta }}_1,\dots ,{\mathrm{\Theta }}_{\mathit{D}}\in {\mathbb{R}}^K.$ Each element of a vector indicates the proportion of the document belonging to this topic, and based on the proportion of each topic, a topic index vector $Z_d\in {\mathbb{R}}^N$ is assigned to each word in the document, where $1 \le d\le D.$

We use a plate notation to visualize the correlations between the variables in the LDA model in Figure 2. The shaded nodes are the observed variables, representing the vectorized training data set in our collection. α and η are parameters to control the predefined conditional probabilities Pr(Θ_d | α) and Pr(β_d | η). According to the graphical representation of the dependencies between random variables in Figure 2, we can obtain the joint distribution of W_d, Z_d and Θ_d: (6) $$\begin{array}{c}\mathrm{P}\mathrm{r}(\mathbf{W}d,\mathbf{\Theta }d,\mathrm{\beta }d,\mathbf{Z}d |\mathrm{ }\mathrm{\alpha },\mathrm{\eta })\mathrm{ }=\mathrm{Pr}\left({\mathrm{\Theta }}_{\mathrm{d}}∣\mathrm{\alpha }\right){\prod }_{\mathrm{k}=1}^{\mathrm{K}}\mathrm{Pr}\left({\mathit{\beta }}_{\mathit{k},\mathit{d}}∣\eta \right)\left({\prod }_{\mathrm{n}=1}^{\mathrm{N}}\mathrm{Pr}\left({\mathbf{Z}}_{\mathbf{n},\mathbf{d}}∣{\mathbf{\Theta }}_{\mathit{d}}\right)\mathrm{Pr}\left({\mathbf{W}}_{\mathbf{n},\mathbf{d}}∣{\mathbf{Z}}_{\mathbf{n},\mathbf{d}},{\mathit{\beta }}_{\mathit{d}}\right)\right)\end{array}$$(6)

Figure 2. Plate notation for the LDA model.

Note: Plate notation is introduced by Buntine (1994) as a type of structured probabilistic models or probabilistic graphical models, the purpose is to present the human designed conditional dependence structure between random variables. For example, an arrow from variable A to B denotes the hypothetical casualty from A to B. If and only if variables A and B are both inside the same plate or rectangular with a subscript number N is this graph used to simplify the notation that independent two self-distributed variables A and B repeat N times under the condition of causality determined by the arrow’s direction. In other words, there are totally N² combinations to represent the casualty from A to B.

Adding the joint distribution equation (6)(6) $$\begin{array}{c}\mathrm{P}\mathrm{r}(\mathbf{W}d,\mathbf{\Theta }d,\mathrm{\beta }d,\mathbf{Z}d |\mathrm{ }\mathrm{\alpha },\mathrm{\eta })\mathrm{ }=\mathrm{Pr}\left({\mathrm{\Theta }}_{\mathrm{d}}∣\mathrm{\alpha }\right){\prod }_{\mathrm{k}=1}^{\mathrm{K}}\mathrm{Pr}\left({\mathit{\beta }}_{\mathit{k},\mathit{d}}∣\eta \right)\left({\prod }_{\mathrm{n}=1}^{\mathrm{N}}\mathrm{Pr}\left({\mathbf{Z}}_{\mathbf{n},\mathbf{d}}∣{\mathbf{\Theta }}_{\mathit{d}}\right)\mathrm{Pr}\left({\mathbf{W}}_{\mathbf{n},\mathbf{d}}∣{\mathbf{Z}}_{\mathbf{n},\mathbf{d}},{\mathit{\beta }}_{\mathit{d}}\right)\right)\end{array}$$(6) on Z_d and β_d, then integrating it on Θ_d gives the following marginal distribution: (7) $$\begin{array}{c}\mathrm{P}\mathrm{r}\left(\mathbf{W}d\mathrm{ }\right|\mathrm{ }\alpha ,\eta )\mathrm{ }=\\ \underset{{\mathbf{\Theta }}_{\mathit{d}}}{\overset{.}{\int }}\mathrm{Pr}\left({\mathbf{\Theta }}_{\mathit{d}}∣\alpha \right){\prod }_{k=1}^K{\sum }_{{\mathrm{\beta }}_{k,d}}\mathrm{Pr}\left({\mathit{\beta }}_{\mathit{k},\mathit{d}}∣\eta \right)\left({\prod }_{n=1}^N{\sum }_{Z_{n,d}}\mathrm{Pr}\left({\mathit{Z}}_{\mathit{n},\mathit{d}}∣{\mathbf{\Theta }}_{\mathit{d}}\right)\mathrm{Pr}\left(W_{n,d}∣Z_{n,d},{\mathrm{\beta }}_d\right)\right)\mathrm{d}{\mathrm{\Theta }}_d\end{array}.$$(7)

By summing the marginal distribution in Equation (7)(7) $$\begin{array}{c}\mathrm{P}\mathrm{r}\left(\mathbf{W}d\mathrm{ }\right|\mathrm{ }\alpha ,\eta )\mathrm{ }=\\ \underset{{\mathbf{\Theta }}_{\mathit{d}}}{\overset{.}{\int }}\mathrm{Pr}\left({\mathbf{\Theta }}_{\mathit{d}}∣\alpha \right){\prod }_{k=1}^K{\sum }_{{\mathrm{\beta }}_{k,d}}\mathrm{Pr}\left({\mathit{\beta }}_{\mathit{k},\mathit{d}}∣\eta \right)\left({\prod }_{n=1}^N{\sum }_{Z_{n,d}}\mathrm{Pr}\left({\mathit{Z}}_{\mathit{n},\mathit{d}}∣{\mathbf{\Theta }}_{\mathit{d}}\right)\mathrm{Pr}\left(W_{n,d}∣Z_{n,d},{\mathrm{\beta }}_d\right)\right)\mathrm{d}{\mathrm{\Theta }}_d\end{array}.$$(7) for each document in the collection, the LDA model transforms it into a log-likelihood function and then executes an unsupervised learning algorithm to measure parameters α∗ and η∗ to maximize the likelihood function as follows: (8) $$\begin{array}{c}{\mathrm{\alpha }}^{\mathrm{*}},{\mathrm{\eta }}^{\mathrm{*}}=\arg \underset{\alpha ,\eta }{\max }L\left(\alpha ,\eta \right)\\ =\arg \underset{\alpha ,\eta }{\max }{\sum }_{d=1}^D\ln \mathrm{Pr}\left({\mathit{W}}_{\mathit{d}}∣\alpha ,\eta \right).\end{array}$$(8)

By identifying the appropriate parameters after training the collection, LDA generates K meaningful word clusters in the data and assigns each document to the most appropriate topic.

4.3.2.2. Topic selection for LDA

One challenge in LDA topic modeling is choosing the appropriate number of topics, K. Typically, a higher K increases the statistical capability of the prediction model, but lowers its interpretability, as each topic may be too specific to be analyzed. In the extreme case where K = D, each document will be assigned one individual topic. Although this model fits the sample perfectly, it is useless for capturing the general patterns in the news articles. Conversely, if K is too small, the latent topics generated may be too vague for topic analysis. To resolve the tradeoff between a high K and a low K, we compare the experimental results of model perplexity in the 20% validation data set with different K values, as shown in Figure 3. We use K = 40 after taking into account the tradeoff between the complexity of the model and the validation perplexity under an appropriate training convergence time.

Figure 3. Topic selection for LDA.

Note: We keep 20% of the original data-set for validation. Perplexity is an indicator in information theory that measures how well a probability model predicts a sample. Usually a lower perplexity denotes a higher likelihood that the probability distribution will fit the sample. Time elapsed measures the convergence time of the model, indicating the complexity of the model.

4.3.2.3. LDA results

We perform LDA topic modeling on the combination of relevant news titles and content in our sample. For each news article, we calculate the probability vectors of 59,783 words after preprocessing, N-gram tokenization and stemming, and identify the most likely topic of each document. Figure 4 shows 38 topics in a heatmap identified by the LDA unsupervised algorithm. We report the top 12 most likely keywords, the darker shades on the heatmap denoting a higher probability that a detected word belongs to one of these topics. Based on the researchers’ understanding of each topic keyword, we use a topic tag to represent the content of each topic. For instance, Topic 3 is “banking”, Topic 18 is “blockchain technology” and Topic 24 is “cryptocurrency market”. Although these topic tags are subjective, LDA provides an intuitive approach for people to understand the central idea of each news narrative.

Figure 4. Topics ranked by level of arousal; terms in topics ranked by probability.

Note: This figure presents the 38 individual distributions over N-size dictionary that LDA unsupervised learning algorithm calculates to indicate news topics. To order the detected topics, we create an arousal index by using the average daily highest VIX value as a baseline to calculate the gap in the number of a topic with a daily highest VIX greater or equal to the baseline versus the number of the topic in the opposite case, and divided by the number of the topic in the relevant news collection. In each row, terms are ordered from left to right according to their likelihoods belong to a single topic of their row.

The LDA algorithm does not automatically order the topics. To quantify the effects of different topics on news credibility, we project the topics generated by LDA into a one-dimensional space of arousal. Arousal is a term in neurophysiology that describes the state of being awoken by stimulation,²enabling us to measure the distance between each topic, which represent the different degrees of stimulus generated by different topics of news shocks. To this end, we define an arousal index using the highest average daily level of the CBOE (Chicago Board Options Exchange) Volatility Index (VIX).

To order the identified topics, we create an arousal index between t₀ and t_n using the highest average daily VIX value $\overline{\mathrm{\sigma }}=\frac{1}{1+n}\sum \left({\mathrm{\sigma }}_{t_0},\dots ,{\mathrm{\sigma }}_{t_n}\right)$ as a baseline. For each date$t_0\le t\le t_n,$ if ${\mathrm{\sigma }}_i\ge \overline{\mathrm{\sigma }},$ vector $T_t^{+}\in {\mathbb{R}}^K$ records the document frequency of each topic. In contrast, vector $T_t^{-}\in {\mathbb{R}}^K$ records the document frequency of each topic if ${\mathrm{\sigma }}_t<\overline{\mathrm{\sigma }}.$ Vector $T≔{\sum }_{t=t_0}^{t_n}\left(T_t^{+}+T_t^{-}\right)$ denotes the total document frequency of each latent topic. Following the notation above, we define the arousal index vector $A\in {\mathbb{R}}^K$ as follows: $$A≔\frac{1}{T}{\sum }_{t=t_0}^{t_n}\left(T_t^{+}-T_t^{-}\right),$$ where each element A_i corresponds to the arousal index of topic i. A higher (lower) value reflects more (less) intense stimulation for market participants during the shock. In our study, t₀ is 1 March 2018 and t_n is 10 October 2018. We sort the latent topics in descending order of A_i from Topic 1 to Topic 38.

The topics with a higher level of arousal (see Figure 5) are sales (2), banking (3) and fund (4), while inactivating topics (see Figure 6) are e-commerce (37), US market (35) and beverage (33). The topics with a trivial relationship to the arousal level include tobacco (11), soft drink (13) and healthcare (15). This result is understandable because news shocks will have a relatively limited effect on consumer behavior toward more dependent products.

Figure 5. Arousal topics in news narratives.

Note: These word clouds present two relatively high arousal level topics in the Financial Times. The size of words is approximately proportional to the probability belongs to the topic computed by the LDA algorithm.

Figure 6. Inactivation topics in news narratives.

Note: The settings of these figures are similar to those presented in Figure 5, except for the relatively low arousal level topics.

4.3.3. Momentum measures

We also consider the role of market momentum in news credibility using abnormal returns (AR) one day before the news shock. The abnormal return is defined as the difference between the actual return and the expected return on an asset. We calculate the expected returns on the stock market using the traditional capital asset pricing model (CAPM) proposed by Sharpe (1964) and Lintner (1975) in Event Study by WRDS. Some studies attempt to provide theoretical asset pricing models for cryptocurrencies. Cong, Li, and Wang (2019) introduce a dynamic pricing model by taking into account platform productivity, endogenous user adoption and user heterogeneity. However, there is a terra incognita in current asset pricing models that make it difficult to satisfactorily interpret expected returns from cryptocurrencies. However, cryptocurrencies, at least Bitcoin, are a type of decentralized currency with a high level of privacy. In other words, the expected volatility of cryptocurrencies should be strongly correlated with the global exchange market. Therefore, we introduce the following approach to measure the expected returns of cryptocurrencies: $$R_{c,t}≔{\mathrm{\alpha }}_c+{\mathrm{\beta }}_c{R}_{m,t}+{\mathrm{\epsilon }}_c\hspace{1em}\hspace{1em}{\mathrm{\epsilon }}_c\sim \mathcal{N}\left(0,{\mathrm{\sigma }}_c^2\right),$$ where $R_{c,t}$ is the expected return of cryptocurrency c at time t, and $R_{m,t}$ is the return of the global exchange market at time t. In our study, we suggest that ${\widehat{R}}_{m,t}=E_{\mathit{SDR},t}/E_{\mathit{SDR},t-1}-1,$ where ESDR represents the value in US dollars of one unit of Special Drawing Rights (SDR).³

5. Results

In this section, we present our findings on the effects of news credibility on market performance. We use t to indicate the time interval since the initial news shock. The linear model we adopt is as follows: (9) $$A{R}_{it}={\alpha }_i+\beta Z_{it}+\mathit{\gamma }{\mathbf{X}}_{\mathbf{t}}+{\epsilon }_i,$$(9) where the dependent variable AR_it indicates the average abnormal returns of the assets observed on market i at time t and Z is people’s expectations regarding information credibility (see details in the section: Momentum Measures). Vector X includes market momentum and other macro controls, such as previous abnormal market returns i, to capture market momentum and the VIX⁴ to control for general market volatility.

In the section, Continuous Sentiment in the Social Network, we present our narrative-based credibility measure for people’s expectations. Nevertheless, news content can affect the result of our measure, so we use the counting measures and topic measures described in the section, Computational Linguistic Approach to the Media, to control for the linguistic effects of news articles on the credibility indicator. Thus, the narrative-based credibility $\mathrm{\Gamma }\left(i,\mathrm{\tau },t\right)$ in Equation (5)(5) $$\mathrm{\Gamma }\left(i,\mathrm{\tau },t\right)=1-{\left(\frac{S_{i\mathrm{\tau }}^n-\mathrm{\zeta }\left(i,t\right)}{d}\right)}^2,$$(5) can be given by (10) $$\mathrm{\Gamma }\left(i,\mathrm{\tau },t\right)={\mathrm{\beta }}_0+{\mathrm{\beta }}_1{Z}_{it}+{\mathrm{\beta }}_{2C_{\mathrm{\tau }}}+{\mathrm{\nu }}_i,$$(10) where vector C_τ includes the counting measures, topic measures, and sentiment dummy variables D(Bearish) and D(Bullish) of news shocks of relevant news articles at time τ. Based on Equations (9)(9) $$A{R}_{it}={\alpha }_i+\beta Z_{it}+\mathit{\gamma }{\mathbf{X}}_{\mathbf{t}}+{\epsilon }_i,$$(9) and (10)(10) $$\mathrm{\Gamma }\left(i,\mathrm{\tau },t\right)={\mathrm{\beta }}_0+{\mathrm{\beta }}_1{Z}_{it}+{\mathrm{\beta }}_{2C_{\mathrm{\tau }}}+{\mathrm{\nu }}_i,$$(10) , we obtain the following reduced form of the effects of news credibility on abnormal market returns: (11) $$\mathrm{A}{\mathrm{R}}_{\mathrm{it}}={\mathrm{\pi }}_0+{\mathrm{\pi }}_1\mathrm{\Gamma }\left(\mathrm{i},\mathrm{\tau },\mathrm{t}\right)+{\mathrm{\pi }}_2{\mathrm{C}}_{\mathrm{\tau }}+{\mathrm{\pi }}_3{\mathrm{X}}_{\mathrm{t}}+{\mathrm{u}}_{\mathrm{i}}.$$(11)

Table 3 presents the results of Equation (11)(11) $$\mathrm{A}{\mathrm{R}}_{\mathrm{it}}={\mathrm{\pi }}_0+{\mathrm{\pi }}_1\mathrm{\Gamma }\left(\mathrm{i},\mathrm{\tau },\mathrm{t}\right)+{\mathrm{\pi }}_2{\mathrm{C}}_{\mathrm{\tau }}+{\mathrm{\pi }}_3{\mathrm{X}}_{\mathrm{t}}+{\mathrm{u}}_{\mathrm{i}}.$$(11) . News credibility has a significant positive effect on short-term stock market returns. This result is in line with our expectations in traditional financial markets that company information is one of the main price drivers, as such information can update people’s knowledge of the fundamental values of companies. However, news credibility has a significant negative effect on the returns of the cryptocurrency market one week after. We interpret this phenomenon as gamblers’ expectations becoming the main source of asset market value when the fundamental value of these assets is trivial or difficult to estimate. Gamblers become less volatile when reliable information is available, allowing assets to return to their fundamental value. However, if the market is flooded with a lot of information that is not credible, this can further stimulate gamblers’ reliance on subjective intuitions rather than reason, thereby creating bubbles in the cryptocurrency market.

Table 3. Effects of news credibility on market performance.

Main Regressors	Stock (1)	Bitcoin (2)	Stock (3)	Bitcoin (4)	Stock (5)		Bitcoin (6)
Credibility	0.243**	−1.326	−0.137	−1.520*	0.126		−0.692
	[0.114]	[0.922]	[0.118]	[0.838]	[0.100]		[0.927]
Topic arousal	−0.272*	−1.853*	−0.070	−2.476***	0.004		−0.081
	[0.155]	[0.937]	[0.161]	[0.935]	[0.141]		[0.943]
Question	−0.181***	−0.145	−0.124*	0.633**	−0.108*		−0.318
	[0.069]	[0.344]	[0.069]	[0.319]	[0.063]		[0.351]
Exclamation	0.068	1.270**	−0.382*	−0.504	−0.290		1.133*
	[0.213]	[0.562]	[0.215]	[0.525]	[0.193]		[0.578]
Numbers	0.160***	−0.293	0.088	−0.080	0.113**		−0.141
	[0.054]	[0.340]	[0.054]	[0.330]	[0.050]		[0.350]
Capitals	−0.037**	−0.114	−0.034**	0.019	0.005		−0.261
	[0.016]	[0.166]	[0.016]	[0.155]	[0.015]		[0.170]
Previous	0.980	11.216*	0.085	24.897***	−6.397***		−20.018***
abnormal return	[1.982]	[6.635]	[1.960]	[5.959]	[1.778]		[6.658]
D(Bearish)	−0.060	0.274	0.030	−0.561	−0.281**		−0.322
	[0.129]	[0.896]	[0.127]	[0.862]	[0.116]		[0.949]
D(Bullish)	−0.060	−0.279	0.029	−1.137	−0.704***		−0.277
	[0.125]	[0.895]	[0.127]	[0.824]	[0.116]		[0.913]
VIX	0.392*	6.770***	−0.066	1.948	−0.215		0.369
	[0.229]	[2.186]	[0.264]	[1.860]	[0.197]		[1.721]
Constant	−1.189*	−17.949***	0.228	−5.149	0.900		0.395
	[0.653]	[6.161]	[0.755]	[5.525]	[0.559]		[5.100]
Period	1 day	1 day	1 week	1 week	2 weeks		2 weeks
Observations	298	111	298	111	298		111

Note: This table reports the results of estimating news shock factors on the average abnormal returns in the stock market and the cryptocurrency market one day, one week, and two weeks after the news shock with macro controls. The neutral threshold λ = 0.02 for measuring the narrative-based credibility. Standard errors in parentheses. Coefficients are labeled based on significance (* p < .1, **p < .05, ***p < .01).

According to the “Yerkes–Dodson law”, subjects’ performance in complex tasks follows a bell-shaped curve when the level of arousal increases. When the level of arousal is relatively low, increased arousal can generate interest and stimulate the performance of subjects. Conversely, when the level of arousal is relatively high, high arousal can become anxiety and therefore decrease performance. Based on our observations, the participants in the stock market and the cryptocurrency market are distributed to the right of the bell-shaped curve, which indicates that when news shock topics are more likely to arouse readers’ emotions, these emotions are more likely to turn into anxiety than interest, reducing people’s expected market returns.

For our counting measures, we observe certain significant effects of the content-based characteristics on the credibility indicator. However, most of them are not dominant characteristics affecting market performance. One predominant characteristic is the abnormal market return the day before a news shock. This past abnormal return is a proxy for market momentum related to people’s expectations. The results show that market momentum has a significant negative effect on the long-term performance of the stock market and the cryptocurrency market, indicating that market performance is absorbed in the long-term. One interesting result is that market momentum has a significant positive effect on the cryptocurrency market in the medium term, but not in the short-term, suggesting that Bitcoin gamblers take at most one week to be affected by price momentum; thus, price momentum rather than news credibility becomes the dominant factor in market performance during this period.

For the news sentiment dummies, we only observe significant negative effects on market performance two weeks after the news shock in the stock market. A reasonable explanation is that the impact of neutral news in the short-run may be underestimated. In the long-run, the value of neutral news may be more obvious for stock market participants, because the stock market has more rational experts than the cryptocurrency market, which affects market performance.

Finally, we see a significant effect of market volatility on both financial markets in the short-term, and that the positive effect of market risk is much stronger among gamblers. However, we do not observe a significant long-term relationship, which means that the market risk indicator only temporarily affects people’s asset return expectations.

6. Robustness tests

6.1. Perturbation analysis

First, we report our robustness test results by tuning the values of the parameters used in the Results section. The first parameter we consider is the threshold value λ, which controls the sensitivity of the news credibility indicator to “neutral” comments in the social network. In our main analysis, we use λ = .02 and make our sentiment analysis model in the social network close to the polarity classifier. In this robustness test, we modify this threshold value to λ = .1 and allow “neutral” comments to have a higher weight in our credibility measure. Table A1 in Appendix A shows that the results of this robustness test correspond to our main results in Table 3. With the exception of slight fluctuations in the estimated coefficients, the significance is consistent with the main estimates.

Second, we modify the threshold value to λ = 0 and only take into account polarized news shocks and comments. We only keep the sentiment dummy variable D(Bearish) in our model and Table A2 in Appendix A shows that the results of this robustness test sustain our main findings with a slight variance of the significance and the estimated coefficients.

6.2. Sample selection

We switch our attention to the selection of our narrative sample in the social network. Our credibility measure depends mainly on comments from StockTwits. To test the possible influence of sample selection bias on our conclusion, our next robustness test measures news credibility using another popular social network platform. To simplify our robustness exercise, we choose Twitter and use the keyword “Bitcoin” to test news credibility in the cryptocurrency market. The Twitter standard Streaming Application Programming Interface (API) allows researchers to randomly request tweets in real time (comments on Twitter) on a specific hashtag at a limited maximum speed. Twitter is one of the largest social networks with a massive number of international daily active users, and thus can reflect the cognition of general market participants. However, a major challenge when using tweets is that Twitter does not provide a self-annotated tag like StockTwits reflecting user expectations in the market. To solve this problem, we use a deep learning approach. Specifically, we choose annotated narratives in StockTwits as training data and then classify each tweet into “bullish”, “neutral” or “bearish”.

Deep learning is a type of machine learning algorithm based on artificial neural networks with multiple hidden layers. The depth corresponds to the number of layers in a neural network, explaining the name “deep learning” when referring to the training process of multi-layer neural networks. Hornik, Stinchcombe, and White (1989) theoretically prove that multi-layer artificial networks with at least one hidden layer can approximate any measurable function to any desired degree of accuracy. Unlike traditional supervised learning algorithms, such as the naïve Bayes classifier or the support vector machine, one advantage of the deep learning algorithm is its powerful ability to learn the relationship between input data and output labels without a strict feature selection process for input. This advantage is very useful for processing highly unstructured social network data, otherwise we have to decide and select specific keywords from our data collection to train our classifier. We discuss in detail the configuration, training and performance of our deep learning model in Appendix B.

We collect 4,685,417 unique tweets from 29 May 2018 to 10 October 2018 related to the “Bitcoin” topic. Table A3 in Appendix A compares the effects of news credibility on the Bitcoin market based on the StockTwits measure and the Twitter measure. Our main results of news credibility in the Bitcoin market are robust, indicating that StockTwits can appropriately represent general users of social networks for researchers to analyze the expectations of gamblers.

7. Conclusions

In empirical asset pricing, external shocks, especially those caused by news, are one of the most important factors affecting asset prices. News credibility is an important but easily overlooked factor in understanding how people perceive and process information in a market. In this study, we propose a statistical learning approach that allows researchers to estimate news credibility using large-scale textual data in newspapers and social media. One challenge in applying our narrative-based credibility indicator is that the linguistic characteristics of news articles can affect the accuracy of the measurements. To solve this problem, we generate a content-based control vector with counting measures and topic measures that can maximize the noise caused by the text structure. For the topic measures, we apply the LDA unsupervised learning model, which allows the machine to automatically detect latent topics in the newspaper. We also develop an arousal index based on psychological theory, which allows us to measure the level of anxiety stimulated by particular news content.

We find that news credibility is positively correlated with stock market returns but negatively correlated with cryptocurrency returns. We explain this result using the idea that when the information is more precise, investors have more resources for valuation and thus amplify news shocks. However, when the information is less credible, gamblers tend to use their own subjective expectations to value assets with low or unclear fundamental values, thereby creating bubbles. Finally, this paper explains how researchers can benefit from machine learning algorithms in textual analysis. In today’s big data era, a large amount of unstructured data are hidden on the Internet, providing reliable data sources for research. Knowledge-based supervised learning, such as deep learning, unsupervised learning based on data patterns, such as LDA, and other machine learning techniques are powerful new tools for researchers in many fields.

Acknowledgements

I am grateful to C. Simon Fan, Hongliang Zhang and Fuhai Hong for helpful comments and suggestions. Thanks also to Joint PhD Workshop with Zhejiang University holding at Lingnan University for their feedback and suggestions.

Additional information

Funding

This work was supported by a grant from National Social Science Fund Youth Project of China (No. 20CJY014): “Study on the Impact of Industrial Structure Upgrading on the Employment Quality of Floating Population and its Countermeasures.”

Notes

1 For instance, the hashtag $BTC.X refers to “Bitcoin BTC/USD”. These hashtags can be used in StockTwits to identify topics in chat rooms

2 Yerkes and Dodson (1908) study the relationship between the level of arousal and subjects’ performance in different tasks and reveal that performance, such as attention and working memory in a difficult task, will follow a bell-shaped curve when the level of arousal continues to increase. This result is also called the “Yerkes–Dodson law” in psychology.

3 The SDR is a potential claim on the freely usable currencies of members of the International Monetary Fund (IMF) and can be exchanged for these currencies. The valuation of the SDR is defined by adding the values in US dollars of a basket of major currencies, based on market exchange rates. From 31 January 2018 to 7 January 2019, the components of the SDR were the US dollar, the euro, the Japanese yen, the British pound and the Chinese renminbi. We use daily observations of the SDR valuation from the IMF database.

4 The VIX is a measure of volatility conveyed by S&P 500 stock index option prices, taking into account the market’s expectation of 30-day volatility.

5 n is the maximum number of tokens in the network. We use n = 15 and ignore the remaining words in each tweet. Therefore, each tweet can be coded as a 15-dimensional row vector and each vector component is a word index in a dictionary.

6 The batch size refers to the number of training samples used in one training iteration. A larger batch size can reduce training times but requires more GPU memory to load the training samples.

Appendix A:

robustness for news credibility

Table A1. Robustness check for news credibility I.

Main Regressors	Stock (1)	Bitcoin (2)		Stock (3)	Bitcoin (4)	Stock (5)	Bitcoin (6)
Credibility	0.266**	−1.486		−0.111	−1.650*	0.102	−0.777
	[0.129]	[0.974]		[0.126]	[0.887]	[0.111]	[0.986]
Topic arousal	−0.267*	−1.856**		−0.066	−2.493***	0.007	−0.075
	[0.155]	[0.935]		[0.161]	[0.935]	[0.141]	[0.943]
Question	−0.184***	−0.124		−0.125*	0.645**	−0.111*	−0.319
	[0.069]	[0.344]		[0.069]	[0.318]	[0.063]	[0.351]
Exclamation	0.063	1.260**		−0.362*	−0.540	−0.287	1.138*
	[0.212]	[0.561]		[0.213]	[0.521]	[0.194]	[0.578]
Numbers	0.158***	−0.294		0.090	−0.096	0.113**	−0.147
	[0.054]	[0.339]		[0.054]	[0.331]	[0.049]	[0.350]
Capitals	−0.038**	−0.112		−0.034**	0.019	0.005	−0.259
	[0.016]	[0.166]		[0.016]	[0.155]	[0.015]	[0.170]
Previous	1.049	11.220*		0.147	24.849***	−6.281***	−19.995***
abnormal return	[1.979]	[6.623]		[1.956]	[5.952]	[1.777]	[6.647]
D(Bearish)	−0.065	0.231		0.044	−0.606	−0.295**	−0.338
	[0.129]	[0.899]		[0.126]	[0.867]	[0.116]	[0.950]
D(Bullish)	−0.062	−0.281		0.026	−1.152	−0.717***	−0.288
	[0.125]	[0.893]		[0.126]	[0.823]	[0.114]	[0.912]
VIX	0.374	6.793***		−0.067	2.013	−0.213	0.406
	[0.228]	[2.183]		[0.264]	[1.859]	[0.196]	[1.727]
Constant	−1.149*	−17.886***		0.214	−5.171	0.920	0.371
	[0.650]		[6.153]	[0.754]	[5.518]	[0.561]	[5.099]
Period	1 day		1 day	1 week	1 week	2 weeks	2 weeks
Observations	299		111	299	111	299	111

Note: This table reports the results of estimating news shock factors on the average abnormal returns in the stock market and the cryptocurrency market one day, one week, and two weeks after the news shock with macro controls. The neutral threshold λ = 0.1 for measuring the narrative-based credibility. Standard errors in parentheses. Coefficients are labeled based on significance (* p < .1, ** p < .05, ***p < .01).

Table A2. Robustness check for news credibility II.

Main Regressors	Stock (1)	Bitcoin (2)	Stock (3)	Bitcoin (4)	Stock (5)	Bitcoin (6)
Credibility	0.239*	−1.270	−0.085	−1.505	0.107	−0.708
	[0.131]	[1.142]	[0.147]	[0.975]	[0.113]	[1.118]
Topic arousal	−0.156	−1.632	−0.096	−2.307**	0.022	−0.208
	[0.168]	[1.209]	[0.191]	[1.115]	[0.156]	[1.222]
Question	−0.212*	−0.236	−0.118	0.583*	−0.127	−0.336
	[0.109]	[0.393]	[0.116]	[0.346]	[0.100]	[0.399]
Exclamation	0.046	0.821	−0.351	−0.566	−0.257	1.251*
	[0.208]	[0.658]	[0.231]	[0.574]	[0.196]	[0.671]
Numbers	0.112	−0.111	0.023	0.089	0.114*	0.034
	[0.069]	[0.408]	[0.076]	[0.381]	[0.066]	[0.425]
Capitals	−0.003	−0.160	−0.003	0.002	0.037**	−0.431*
	[0.019]	[0.226]	[0.021]	[0.204]	[0.018]	[0.231]
Previous	2.019	22.313**	−0.576	15.838*	−3.169	−23.905**
abnormal return	[2.511]	[9.500]	[2.723]	[7.967]	[2.340]	[9.417]
D(Bearish)	−0.033	0.232	0.073	0.109	0.417***	0.139
	[0.115]	[1.055]	[0.130]	[0.935]	[0.098]	[1.073]
VIX	−0.173	6.933**	−0.476	1.458	−0.544**	2.548
	[0.282]	[2.762]	[0.350]	[2.297]	[0.225]	[2.250]
Constant	0.254	−18.300**	1.241	−4.754	0.924	−5.773
	[0.772]	[7.683]	[0.991]	[6.753]	[0.625]	[6.532]
Period	1 day	1 day	1 week	1 week	2 weeks	2 weeks
Observations	214	74	214	74	214	74

Note: This table reports the results of estimating news shock factors on the average abnormal returns in the stock market and the cryptocurrency market one day, one week, and two weeks after the news shock with macro controls. We only consider polarized news shocks and comments and set the neutral threshold λ = 0. Standard errors in parentheses. Coefficients are labeled based on significance (* p < .1, ** p < .05, ***p < .01).

Table A3. Robustness check for news credibility III.

Main Regressors	StockTwits (1)	Twitter (2)	StockTwits (3)	Twitter (4)	StockTwits (5)	Twitter (6)
Credibility	−1.326	−2.382	−1.520*	−2.078	−0.692	−0.540
	[0.922]	[1.437]	[0.838]	[1.284]	[0.927]	[1.377]
Topic arousal	−1.853*	−1.807	−2.476***	−37.743**	−0.081	26.230
	[0.937]	[22.676]	[0.935]	[16.480]	[0.943]	[18.866]
Question	−0.145	−0.251	0.633**	−0.175	−0.318	0.072
	[0.344]	[0.466]	[0.319]	[0.399]	[0.351]	[0.401]
Exclamation	1.270**	1.821**	−0.504	−0.148	1.133*	0.748
	[0.562]	[0.783]	[0.525]	[0.605]	[0.578]	[0.667]
Numbers	−0.293	−0.454	−0.080	−0.437	−0.141	−0.257
	[0.340]	[0.466]	[0.330]	[0.414]	[0.350]	[0.423]
Capitals	−0.114	−0.094	0.019	−0.033	−0.261	−0.057
	[0.166]	[0.247]	[0.155]	[0.198]	[0.170]	[0.189]
Previous	11.216*	32.302*	24.897***	−16.656	−20.018***	−34.448**
abnormal return	[6.635]	[16.521]	[5.959]	[13.282]	[6.658]	[15.261]
D(Bearish)	0.274	−0.222	−0.561	−2.186	−0.322	−1.261
	[0.896]	[1.545]	[0.862]	[1.355]	[0.949]	[1.298]
D(Bullish)	−0.279	−2.186	−1.137	−2.505**	−0.277	−1.731
	[0.895]	[1.535]	[0.824]	[1.213]	[0.913]	[1.308]
VIX	6.770***	−4.377	1.948	−1.692	0.369	3.662
	[2.186]	[5.760]	[1.860]	[3.476]	[1.721]	[2.765]
Constant	−17.949***	12.944	−5.149	7.225	0.395	−7.350
	[6.161]	[15.733]	[5.525]	[9.668]	[5.100]	[7.517]
Period	1 day	1 day	1 week	1 week	2 weeks	2 weeks
Observations	111	56	111	58	111	66

Note: This table reports the results of estimating news shock factors on the average abnormal returns in the stock market and the cryptocurrency market one day, one week, and two weeks after the news shock with macro controls. The neutral threshold λ = 0.02 for measuring the narrative-based credibility. Standard errors in parentheses. Coefficients are labeled based on significance (*p < .1, ** p < .05, ***p < .01).

Appendix B:

deep learning in social networks

To train our Twitter sentiment classifier, we use the Keras library designed by Chollet, et al. (2015) and choose the TensorFlow library as the backend engine in Python. In our model setting, we construct a neural network with one embedding layer to transform 1 times n-word sequences⁵ into 128-dimensional vectors, plus one long short-term memory (LSTM) layer and two dense layers. LSTM is an artificial recurrent neural network that can learn time series and sequential data. This structure is powerful for studying the meaning of sequential tokens without the n-gram preprocessing step for textual data. A dense layer is a regular layer in a neural network implementing the following operation: $$\mathrm{output}=\mathrm{activation}\left(\mathrm{dot}\left(W,\mathrm{input}\right)+b\right),$$ where W is a weight matrix, b is a bias vector, dot(·,·) is a dot product operation between two tensors and activation(·) is an activation function to perform a non-linear transformation. We choose a rectified linear unit (relu) defined by max(0,·) as the activation function. More details on the construction of the network are provided in Figure B1.

Figure B1. Neural network framework.

Note: This framework indicates each layer name in our neural network including the shapes of the input and output tensors. The input layer will receive the encoded 1 × 15 text vector and output to the embedding layer. The embedding layer will project each word index scalar in the input vector to a 128-dimension embedding space. The LSTM and dense layers mainly used to approximate the corresponding mapping between text and labels, and the flatten layer corresponds to reshape the high dimension input tensor to a low dimension tensor for the purpose to be operated in the bottom classification layer.

Training and validation accuracy

One of the challenges of deep learning is to appropriately define the parameters of the model. We experiment with different parameters and ultimately select the 30,000 most frequently used terms, including punctuation and emoji, to code the input narratives. Our collection includes 26,370 annotated training samples and 1,984 testing comments. The size of our training and testing data sets is smaller than the total collection of StockTwits, because we want to assign an equal number of “bullish” and “bearish” comments and reduce the noise of “neutral” comments. To this end, we divide the size of the “neutral” annotated sample by three, as some users may simply ignore or may be unwilling to provide their optimistic or pessimistic market expectations.

The main task of deep learning is to tune W and other parameters in each layer to minimize the distance between the predicted output and the true output using the back-propagation algorithm proposed by Rumelhart et al. (1988). However, the computational complexity of back-propagation in a multi-layer neural network is enormous. For instance, in our relatively shallow neural network, there are 4,137,923 parameters that need to be tuned. To efficiently train our model, we use the NVIDIA TITAN Xp graphics processing unit (GPU) to execute the back-propagation algorithm, as a GPU can perform back-propagation parallel computing more efficiently than a central processing unit (CPU), selecting a batch size of 10,000 data ⁶ to load the data into the GPU.

To validate the parameter setting and training accuracy of our model, we use iterated k-fold validation with shuffling to evaluate the performance of deep learning. This technique is useful when we want to accurately evaluate the overall performance of a classifier when the size of the training samples is relatively small. In practice, we divide the training data set into k = 10 parts. We use each part as a validation data set and train our model with the remaining data. This process will run four times and each time we will shuffle the data rows. Figure B2 presents one of the training and accuracy results, the overall validation accuracy after training is 65.90% (+/ − 2.55%) and the classification accuracy for the remaining 7% of the testing sample is 68.85%.

Figure B2. Deep learning accuracy.

Note: This figure denotes the training and validation accuracy among the iterated K-fold validation with shuffling training process. The horizontal axis indicates the number of a single time the neural network receives the whole training data, called epochs. The vertical axis represents the loss measured by the categorical crossentropy, which is a common loss function in a multi-classification machine learning problem.

By taking into account the noise caused by the “neutral” training data, the performance of our neural network classifier is much better than the baseline accuracy of a three-class classification problem at 33.33%.