Stress measurement using speech: Recent advancements, validation issues, and ethical and privacy considerations

Abstract

Life stress is a well-established risk factor for a variety of mental and physical health problems, including anxiety disorders, depression, chronic pain, heart disease, asthma, autoimmune diseases, and neurodegenerative disorders. The purpose of this article is to describe emerging approaches for assessing stress using speech, which we do by reviewing the methodological advantages of these digital health tools, and the validation, ethical, and privacy issues raised by these technologies. As we describe, it is now possible to assess stress via the speech signal using smartphones and smart speakers that employ software programs and artificial intelligence to analyze several features of speech and speech acoustics, including pitch, jitter, energy, rate, and length and number of pauses. Because these digital devices are ubiquitous, we can now assess individuals’ stress levels in real time in almost any natural environment in which people speak. These technologies thus have great potential for advancing digital health initiatives that involve continuously monitoring changes in psychosocial functioning and disease risk over time. However, speech-based indices of stress have yet to be well-validated against stress biomarkers (e.g., cortisol, cytokines) that predict disease risk. In addition, acquiring speech samples raises the possibility that conversations intended to be private could one day be made public; moreover, obtaining real-time psychosocial risk information prompts ethical questions regarding how these data should be used for medical, commercial, and personal purposes. Although assessing stress using speech thus has enormous potential, there are critical validation, privacy, and ethical issues that must be addressed.

Keywords:

• Life stress
• voice
• speech
• privacy
• ethics
• digital health

Introduction

Hospital rooms, bedrooms, cars, and dorm rooms are locations we still assume are relatively private—places where we believe we can have intimate conversations that will not be transmitted to, or analyzed by, third parties. However, smartphones and smart speakers are rapidly changing all of that. In this article, we discuss the quickly growing capability for these digital devices to assess human stress levels and psychosocial wellbeing, and the critical concerns that are raised by such surveillance. Ultimately, although technologies that can assess stress using speech acoustics hold enormous potential for monitoring and potentially improving human health, serious validation, privacy, and ethical issues exist that must be addressed.

Questions regarding how health data should be acquired and used have circulated for many years. However, the recent increase in availability of low-cost microphones and sensors – and a corresponding increase in interest in digital health – have made these issues a high priority in medical ethics (Rivas & Wac, 2018). By 2020, for example, it is estimated that every individual will own an average of seven internet-connected devices that have the ability to transmit health-related information to distant third parties (Topol, Steinhubl, & Torkamani, 2015). Smartphones and smart speakers are relatively unique in this context, though, as they are already ubiquitous and can non-invasively collect and send large amounts of rich information (i.e., big data) that can be used to indicate real-time disease risk.

If you want to continuously assess a process that greatly affects disease risk, then focusing on stress is an ideal option. This is because stress is implicated in not just a few disorders, but rather is a common risk factor for a variety of different mental and physical health problems, especially when it is chronic (Miller, Cohen, & Ritchey, 2002; Slavich, 2016a, 2016b). For example, greater stress exposure is associated with increased risk for anxiety disorders, posttraumatic stress disorder, depression, chronic pain, coronary heart disease, asthma, respiratory infections, autoimmune diseases, and neurodegenerative disorders, among others (Cohen, Janicki-Deverts, & Miller, 2007; Juster, McEwen, & Lupien, 2010; Slavich & Irwin, 2014). Stress is also associated with accelerated biological aging and premature mortality (Holt-Lunstad, Robles, & Sbarra, 2017; Kelly-Irving et al., 2013; Mayer et al., 2019), making it a critical factor to assess when predicting human health (Epel et al., 2018; Malat, Jacquez, & Slavich, 2017; Toussaint, Shields, Dorn, & Slavich, 2016).

Standard methods for assessing stress

Given these effects of stress on health, numerous approaches have been developed for assessing individuals’ stress levels. The current gold-standard method involves conducting life stress interviews using instruments such as the Life Events and Difficulties Schedule, UCLA Life Stress Interview, and Stress and Adversity Inventory (Monroe & Slavich, in press; Slavich, 2019). In turn, the most commonly used approach involves administering brief self-report questionnaires such as the Perceived Stress Scale (Monroe, 2008). Interview-based measures can be time-consuming and costly, though, and self-report questionnaires often lack item specificity and validity (Cohen, Kessler, Underwood, & Gordon, 1997; Dohrenwend, 2006; Shields & Slavich, 2017). Moreover, both methods are retrospective in nature and subject to (often unmeasured) degrees of cognitive bias and social desirability that can influence the veracity, reliability, and validity of the resulting scores (Monroe, 2008; Monroe & Slavich, 2016).

Stress has also been assessed by measuring biological processes that are upregulated by stress exposure and implicated in disease. Assessing stress in this way has several advantages over and above interview-based and self-report instruments. Two of the most important advantages are that stress-related biomarkers are (a) proximally related to biological disease processes and (b) not subject to self-report biases. The full list of biological indices that are known to increase in response to stress is very long and beyond the scope of the present discussion. As an example, however, these outcomes span cardiovascular, sympathetic, neuroendocrine, and immune outcomes, and include things like heart rate, systolic blood pressure, skin conductance, cortisol, adrenocorticotropic hormone (ACTH), dehydroepiandrosterone (DHEA, and its sulfate ester, DHEA‐S), epinephrine, norepinephrine, α-amylase, and the pro-inflammatory cytokines interleukin (IL)-1β, IL-2, IL-6, and tumor necrosis factor-α (Allen, Kennedy, Cryan, Dinan, & Clarke, 2014; Irwin & Slavich, 2017; Slavich, 2015, in press; Slavich & Auerbach, 2018). These stress signals have become easier to assess over time – for example, smartwatches can now be used to continuously monitor heart rate, skin conductance, and skin temperature – but for the most part, assessing stress-related biomarkers is still relatively invasive, requiring (for example) a blood or saliva sample from the individual (Shirtcliff et al., 2015).

Assessing stress using speech

These limitations of self-report, interview-, and biomarker-based approaches make assessing stress using speech very attractive, especially given that doing so is now relatively inexpensive and non-intrusive. When preparing to speak, an individual must decide which sequence of words will best communicate his or her intended message. Stress can affect these decisions and change the wording, grammar, and timing of speech, which can, in turn, be used as vocal markers of stress (Paulmann, Furnes, Bøkenes, & Cozzolino, 2016; Scherer & Moors, 2019). However, stress induces other changes as well. In order to produce speech, for example, the body modulates the tension of numerous muscles in order to force air through the vocal folds and out the vocal tract to produce sound waves (Titze, 2000). Stress increases both muscle tension and respiration rate, which in turn change the mechanics of speech production and, consequently, the way that speech sounds (Sondhi, Khan, Vijay, & Salhan, 2015; Zhou, Hansen, & Kaiser, 2001).

Current approaches for assessing stress using speech take advantage of these stress-based changes in the quality and pattern of speech acoustics to quantify the amount of a stress a person is currently experiencing. As summarized in Figure 1, this can be achieved by assessing several features, including the fundamental frequency (i.e., pitch), jitter (i.e., changes in pitch over a short period of time), energy in different frequency bands (e.g., Mel Frequency Cepstrum Coefficients; MFCCs), speaking rate, and length and number of pauses made while speaking (Hansen & Patil, 2007). These features are analyzed using machine learning to produce a real-time index of an individual’s stress level (Fernandez & Picard, 2003). The resulting continuous stress signal can, in turn, contribute to quantifying a person’s continuous health risk – something that is not possible with interview-based or self-report instruments.

Figure 1. Assessing stress using speech. (a) When an individual speaks in the presence of (b) an actively recording smartphone or smart speaker, (c) an audio signal is captured. (d) Various features of the audio signal (e.g., pitch, jitter, energy, speaking rate, length, and number of pauses) are then extracted and used as inputs to (e) a machine learning algorithm that yields a stress score. (f) The resulting score can then be integrated into an individual’s clinical chart as an indicator of the person’s potential disease risk.

This last step – namely, analyzing features of speech to produce a stress level output – has been accomplished in many ways. Some models are physiologically based and directly incorporate what is known about speech production and how it changes under stress to estimate an individual’s stress level (e.g., Mendoza & Carballo, 1998; Van Puyvelde, Neyt, McGlone, & Pattyn, 2018). Others models have used only simple acoustic features (e.g., MFCCs) and deep neural networks (e.g., Han, Kyunggeun, & Hong-Goo, 2018; Hansen & Womack, 1996). The interpretability of the physiological models makes them especially attractive to physicians, who often want to know why particular outputs are given. The neural network-based models, in turn, are enticing because they require very little domain knowledge to achieve acceptable accuracies.

Regardless of the particular method used for extracting a stress signal, what is arguably most impressive is how easily this can now be done. Indeed, whereas high-quality speech analysis was once only possible in labs equipped with specialized recording equipment and teams of signal processing experts, researchers can now assess stress in speech inexpensively and without specialized training, using portable devices that can be carried around or placed anywhere in the natural environment. For example, Lu and colleagues recently used Android smartphones to detect instances of stress in multiple environments, including indoors during a job interview and outdoors while participants were interacting with other individuals (Lu et al., 2012). The modeling strategy employed was impressive, accurately detecting the presence of stress in 81% and 76% of cases, respectively, for indoor and outdoor environments, when evaluated against the ground truth of increased skin conductance as assessed using an electrodermal activity sensor that was calibrated for each individual. Opensource programs like openSMILE, in turn, make the collection and analysis of voice data relatively easy for those without a background in signal processing (see Eyben, Weninger, Gross, & Schuller, 2013). In sum, therefore, assessing stress using speech and speech acoustics is now widely possible and relatively inexpensive.

Validation, privacy, and ethical concerns

These technical advancements have transformed our ability to monitor individuals’ stress-related disease risk. Indeed, smartphones and smart speakers like the Amazon Echo and Google Home are now commonplace, with one market analysis suggesting that nearly one million smart speakers will be integrated into hospital rooms by 2021 to help facilitate patient-physician communication (Montany, 2018). Moreover, at least one major university in the United States announced that it placed smart speakers in every dorm room in 2018 to help students communicate with the university and learn about campus-wide events (Montag, 2018), and two other universities were already using the devices in select environments on campus by that time (Brown, 2018).

Given the widespread introduction of these digital devices into previously private settings, the same technology that is empowering our ability to monitor and potentially help individuals under stress is also prompting numerous questions about the validation, privacy, and ethics of this approach to digital health. With respect to validation, the main concern is that the race to promote widespread adoption of this technology is taking precedence over making sure that voice-based approaches for assessing stress are validated against well-established biomarkers of stress exposure and disease risk. The field of digital health, and especially the much broader field of “self-help”, is replete with examples of technologies that have become widely used before being well-validated. One such example is digital “brain training” programs, which acquired more than 50 million users despite possessing little-to-no-evidence that they worked (Simons et al., 2016). Given that we are still in the early days of being able to assess stress using speech, much more carefully conducted validation work is needed to ensure that the stress indices being used have clear clinical utility.

In addition, there are many serious questions about privacy and ethics. With respect to privacy, what if a hospital-based smart speaker discloses HIPAA-protected information to a non-authorized person? Companies that sell smartphones and smart speakers have spent substantial time assuring users that their privacy is not at risk. As summarized in Table 1, however, several recent events have shown that privacy cannot be guaranteed even with huge investments in technology. For example, even Apple, whose leadership speaks the most about privacy and has more than $200 billion in cash on top of enormous technical resources, recently admitted that it had discovered a bug in its FaceTime communication platform that allowed callers to see and hear through the camera of a person they were calling before the person answered the call (Johnson, 2019). Having a device that can listen to you, even if made by a reputable company, thus means not only that your privacy could be one day compromised, but that your stress levels or health status could be potentially revealed without your consent. Similarly, multiple cases have recently been documented in which speakers used for other purposes (e.g., Amazon Echo, which is usually used for shopping or for controlling simple household devices) have been manipulated to listen in on private conversations, save the recordings, and transmit them to a third party (Charlton, 2018). Such hacks appear to be rare at present, but the point is that the technological capability already exists for using these devices for nefarious purposes, which is quite contrary to the goal of improving human health and wellbeing by assessing stress.

Table 1. Examples of recent data breaches with smart speakers.

Year	Device	Description of Incident	Reference
2017	Google Home Mini	Audio was recorded and stored without the wake word being used	Burke, 2017
2018	Amazon Echo	Echo sent a message to an owner’s contacts without the owner knowing	Shaban, 2018
2018	Amazon Echo	Amazon mistakenly sent 1,700 audio recordings to the wrong person	Ivanova, 2018
2018	Amazon Echo	Another Amazon Echo on the same WiFi network as a malicious device could record and transmit the audio that it detected	Charlton, 2018
2019	Apple iPhones, iPads	A FaceTime bug allowed callers to hear the receiver’s audio and see their video feed even if they did not answer the call	Johnson, 2019

Along similar lines, what happens if private conversations captured by stress-assessing smart devices are disclosed (accidentally or on purpose) to a third party? Is an employer or boss allowed to take action if they accidentally overhear something about an employee’s health that may affect their work? If evidence exists that someone is currently under extreme stress, what responsibility does the monitoring party have to act? Do users of the technology have the right to be told, first and privately, that their speech indicates that they are increasingly stressed and may be becoming depressed? Will physicians be more guarded knowing they are also potentially being monitored? After all, their speech can be sampled not just by their own smartphone, but also by their patients’ phones.

In addition, what happens if an advertising company or business uses a voice-based stress assessment technology to take advantage of an individual’s compromised emotional state? Is it ethical to provide stressed individuals with information regarding nearby psychotherapy or anti-depressant medication options? If so, what about fast food options that are known to be strongly preferred when individuals are under stress (Geiker et al., 2018)? In sum, when it is appropriate to use such digital health information for commercial purposes and when is it not?

Finally, what if a smartphone transmits evidence of domestic violence, or if a smart speaker in a dorm room detects self-harm or suicide but a university does not intervene? Is the company that manufactured the technology or that processes the data responsible? What about the company, school, or organization that provided the technology to the user or that has partial or full access to the resulting stress information? All of these scenarios can happen with today’s technology, and the newer smarter sensing approaches will only amplify the accuracy of the information that can be gathered and the scale of the impact it can have – whether for early detection and treatments that may reduce human disease risk, or for accidental or nefarious harm.

Solutions for minimizing risk

To minimize the risks associated with using smartphones and smart speakers to assess human stress levels and psychosocial wellbeing, we must recognize and address the privacy and ethical issues that are raised by these devices with the same vigor that is directed at advancing the technologies themselves. For starters, we believe these challenges can be addressed in part by (a) clearly informing users what the devices are transmitting and assessing, and providing examples of the possible risks involved; (b) enabling users to easily turn the listening function of the devices on and off as they wish; (c) enabling users to also have the audio equivalent of a physical lens cap – a “noise jamming” or other device that ensures that no audio from their speech will be detected in case the “off” button does not work as expected; (d) allowing users to easily control who can access their data and how it is used; (e) permitting users to opt in to having the devices in their surrounding environment; and (f) allowing users to opt out of having their speech logged or analyzed if they must live or work in an environment that listens.

More broadly, we believe it is critical for companies that develop and use these technologies to adopt strict policies to help ensure that users are immediately notified of technological malfunctions and data breaches. In addition, they should have comprehensive plans in place to quickly provide users with adequate identity protection services and compensation after a data breach has occurred. Stories of companies withholding critical information about a recent platform malfunction or data breach are common. When it comes to users’ data, we believe that individuals have a right to immediately know when their information has been inappropriately accessed or used, and that all companies that work with such data should affirm their commitment to putting users’ data privacy and safety first.

Conclusion

In conclusion, stress is a powerful risk factor for poor health that is in dire need of better measurement (Slavich, 2019; Slavich & Shields, 2018), and speech is one process that we can now easily measure to help address this need. To maximize the benefits and minimize the risks associated with monitoring speech, however, we will need to take very seriously the validation, privacy, and ethical issues that are prompted by these technological advancements. We will also need to do a much better job at educating users about these issues and innovating better ways to protect users’ data beyond simply having a device “off” switch.

Looking forward, there are several avenues that could be pursued to make these technologies better and less risky for users. First, as alluded to earlier, research is needed to validate speech-based assessments of stress against stress biomarkers and clinical outcomes. In addition, since much of the original work on assessing stress with speech was conducted in quiet lab settings or with vocal actors, additional research is needed to validate these technologies in a variety of contexts, given that different environments can change an individual’s vocal signature (Giddens, Barron, Byrd-Craven, Clark & Winter, 2013). This will likely require a collaborative effort between private companies and research institutions to consolidate large corpuses of speech data with high-quality stress labels. Second, artificial intelligence techniques have been applied to assess emotional and behavioral states like depression and suicidality using speech (e.g., Cummins et al., 2015), and although the sensitivity and specificity of these assessments have not yet been shown to achieve levels required for medical diagnosis, applying artificial intelligence may well be helpful for enhancing the detection of stress as well.

Third, future methods for assessing stress will undoubtedly benefit from combining voice and facial recognition data to enhance the detection of stress and other emotional processes (Giannakakis et al., 2017), with the addition of other biometric data in the future. Finally, we believe that more crosstalk is sorely needed between developers, privacy experts, and medical ethicists to help ensure that the information gathered by these cutting-edge technologies is handled properly. Digitally driven approaches for assessing stress can ultimately play a key role in the future of digital health. To realize the full potential of this approach while minimizing possible risks, though, balanced attention needs to be paid to the technological, validation, privacy, and ethical issues raised here.

Disclosure statement

No conflicts of interest were reported by the authors.

Additional information

Funding

Preparation of this article was supported by a Society in Science-Branco Weiss Fellowship, NARSAD Young Investigator Grant #23958 from the Brain and Behavior Research Foundation, and National Institutes of Health grant K08 MH103443 to George M. Slavich.