Building trust in remote attestation through transparency – a qualitative user study on observable attestation

ABSTRACT

Internet of Things (IoT) devices have become increasingly important within the smart home domain, making the security of the devices a critical aspect. The majority of IoT devices are black-box systems running closed and pre-installed firmware. This raises concerns about the trustworthiness of these devices, especially considering that some of them are shipped with a microphone or a camera. Remote attestation aims at validating the trustworthiness of these devices by verifying the integrity of the software. However, users cannot validate whether the attestation has actually taken place and has not been manipulated by an attacker, raising the need for HCI research on trust and understandability. We conducted a qualitative study with 35 participants, investigating trust in the attestation process and whether this trust can be improved by additional explanations in the application. We developed an application that allows users to attest a smart speaker using their smartphone over an audio channel to identify the attested device and observe the attestation process. In order to observe the differences between the applications with and without explanations, we performed A/B testing. We discovered that trust increases when additional explanations of the technical process are provided, improving the understanding of the attestation process.

KEYWORDS:

• Usable security
• remote attestation
• user study
• IoT
• trust
• security and privacy
• usability in security and privacy
• human-centered computing
• empirical studies in HCI

1. Introduction

Due to their variety, scalability, and simplicity, Internet of Things (IoT) devices face many security challenges that are difficult to cope with (Alrawi et al. 2019; Hassan 2019; Zhang et al. 2014). For instance, the limited computing power of IoT systems impedes the execution of antivirus software (Zhang et al. 2014). Furthermore, the majority of IoT devices are equipped with closed and pre-installed firmware, which makes them black-box systems. As a consequence, the user has little assurance on whether the system is indeed operating the intended task and not performing malicious tasks (e.g. spying). Incidents such as the well-known Mirai botnet, which targeted consumer devices like routers and IP cameras, have demonstrated the large impact of attacks against vulnerable IoT devices (Kolias et al. 2017). Hence, it comes as no surprise that users have little trust in IoT devices, especially if these devices are equipped with microphones or cameras like smart speakers or smart screens (Zeng, Mare, and Roesner 2017; Zheng et al. 2018).

One prominent research direction to increase trust and security of IoT devices is remote attestation (Ammar, Crispo, and Tsudik 2020; Carpent, Rattanavipanon, and Tsudik 2018; Conti, Dushku, and Mancini 2019; Dushku et al. 2020; Kuang et al. 2019, 2020). Remote attestation is a security service that allows a device, the verifier, to verify the integrity of a remote, untrusted device, the prover. To perform the attestation, the prover inherits a self-measurement routine. Remote attestation is typically implemented using a challenge-response protocol: the verifier sends an attestation request to the prover, which performs a self-measurement and sends the result back to the verifier (Nunes et al. 2019). Remote attestation can assure different security guarantees, varying from firmware integrity to more complex run-time properties such as control-flow (Abera, Asokan, Davi, Ekberg et al. 2016) or data-flow integrity (Abera et al. 2019; Kuang et al. 2020; Nunes, Jakkamsetti, and Tsudik 2021), thereby detecting attacks like persistent malware (Nunes et al. 2019), return-oriented programming attacks (Roemer et al. 2012), and even data-oriented programming techniques that do not alter the control flow (Hu et al. 2016). Remote attestation is an abstract concept. Its security relies on cryptographic protocols and functions, to which users are often oblivious. Existing research shows that users tend to discard or circumvent mechanisms they do not understand or trust (Angela Sasse and Flechais 2005). Users, however, are a critical part of the IoT ecosystem as they are responsible for the devices, and their integrity. Many security mechanisms eventually rely on the user correctly using them, such as virus scanners, firewalls, and passwords (Furnell and Clarke 2012). Today, many security risks in IoT devices already arise due to incautious users that utilise insecure or default passwords, do not apply updates, or configure devices insecurely (OWASP 2018).

However, so far, all remote attestation schemes have been black boxes, which can lead to misuse by users who do not understand and trust the system (backfire-effect Nyhan and Reifler 2010). Bringing the user into the context of remote attestation can increase trust in these security mechanisms. Therefore, HCI research is needed to design new usable security solutions. Recently, the first user-observable remote attestation scheme was presented (Surminski et al. 2023). The authors conducted a user study claiming good usability. However, this still raises the question of whether remote attestation can help to increase trust in IoT devices, e.g. by removing security concerns like the leakage of sensitive information to third parties (Babun et al. 2021; Mare, Roesner, and Kohno 2020). In particular, it has not been investigated how users treat compromised devices and devices that were formerly compromised but have been restored to a benign state. Hence, it remains largely unclear whether users trust the abstract cryptographic concepts of remote attestation schemes, raising the demand for usable security mechanisms.

In this paper, we investigate for the first time whether user-friendly and understandable techniques as well as transparency enhance users' understanding and trust in remote attestation. Examples of user-friendly and understandable techniques are explanations that are not only understood by experts and an easy-to-use design of the application. The concept of trust used in this study entails the users' trust in the correctness and effectiveness of the attestation process, as well as the user's trust, that the attested device runs as intended and no third-party privacy violations are possible. We aim to use transparency as an HCI intervention mechanism to inform the users about the attestation to enable the users to comprehend the process and the promises of remote attestation. Therefore, our main research question is: ‘Does transparency enhance trust in remote attestation?’ We implemented an application of a user-observable remote attestation scheme that has recently been developed (Surminski et al. 2023) into a custom-made demonstrator including a real smart speaker and evaluated it in a user study.

2. Background and related work

In this section, we provide background information on the topics of attestation and trust in IoT devices and present related research. We conclude by pointing out the research gap resulting from the presented related work.

2.1. Remote attestation

Remote attestation allows the verification of the integrity of an untrusted device (Aman et al. 2020). For example, a server can verify whether a client system has an uncompromised version of a software. The main problem in attestation is obtaining a trustworthy measurement of an untrusted device. Even if the system is completely compromised, this measurement must not be altered. To this end, the attested device requires logic to prove its integrity, which is why it is referred to as prover, while the device verifying the reports of the prover is called verifier (Aman et al. 2020). There exist three categories of attestation schemes, depending on their architecture, that is, hardware-based, software-based, and hybrid attestation schemes (Abera, Asokan, Daviet, Koushanfar al. 2016). Hardware-based attestation schemes rely on a trusted computing base inside the untrusted device (Architecure ARM 2009; Costan and Devadas 2016; Kil et al. 2009; Parno, McCune, and Perrig 2010). Software-based attestation schemes rely on precise timing measurements to perform the attestation (Seshadri et al. 2004; Surminski et al. 2021). Hybrid attestation schemes combine hardware and software designs to perform trusted self-measurements (Brasser et al. 2015; Koeberl et al. 2014; Nunes et al. 2019). These attestation protocol schemes were developed for a setting in which a trusted verifier attests one or more prover devices. Typically, remote attestation schemes inherit a challenge-response protocol. The verifier starts the attestation by sending a nonce to the prover to ensure freshness and prevent replay attacks (Nunes et al. 2019).

For IoT devices, there exist many attestation schemes with different advantages. Since IoT systems often consist of many different devices, most existing remote attestation schemes lack the scalability to attest a network of IoT devices. Therefore, the Collective Remote Attestation (CRA) scheme was developed which can perform the attestation of IoT networks (Ambrosin et al. 2020). Kuang et al. (2019) propose a many-to-one remote attestation scheme that eliminates the need for a fixed verifier and thus, reduces the possibility for an attacker to tamper with the attestation process. For the specific threat of roving malware, i.e. malware that relocates itself in the memory, Aman et al. (2020) developed an attestation scheme that can detect this malware. During most remote attestation schemes, the device must not be interrupted. This means that in order to attest a network of devices, all devices must interrupt their normal operation until the attestation is performed. In order to avoid such a network-wide interruption, Dushku et al. (2020) developed an asynchronous attestation scheme allowing parts of the network to operate normally while another part of the network is being attested. Attestation is not only used for IoT devices but inter alia also used for hardware security tokens (HST). It was shown by Pfeffer et al. (2021) that existing authenticity checks have usability issues that cause users to base their trust on ineffective security features such as visual damage of a product. They suggested to use combinations of automation and user interaction to improve the security of attestation procedures.

2.2. Trust in IoT

Trust in IoT devices is essential since they gain more importance in the everyday lives of many people. Smart light bulbs, for example, create easily adjustable ambient light, smart speakers that can be controlled by voice make it easy to adjust the music playing in the background and smart thermostats allow to control the temperature at home from anyplace at anytime. However, IoT devices are notoriously insecure (Alrawi et al. 2019). Manifold prominent examples, such as insecure security cameras (Seralathan et al. 2018), attacks on smart speakers (Yan et al. 2022), and large-scale attacks such as the Mirai botnet (Kolias et al. 2017) have customers worried about the security of home IoT devices (Lafontaine, Sabir, and Das 2021; Nemec Zlatolas, Feher, and Hölbl 2022; Zubiaga, Procter, and Maple 2018). Users are also worried that sensitive or personal information could be leaked to third parties (Asplund and Nadjm-Tehrani 2016; Babun et al. 2021; Mare, Roesner, and Kohno 2020; Park, Kim, and Jeong 2018). Furthermore, IoT devices are designed as black box systems to the user (Zeng, Mare, and Roesner 2017) and are used as is, which is why users do not have control over the software running on the devices. Research showed that security, besides other factors, is an important aspect for people in order to trust an IoT device (AlHogail 2018). The generally positive attitude of the user to use an IoT device (Korneeva, Olinder, and Strielkowski 2021) can then be affected by this trust (Alraja, Farooque, and Khashab 2019) as well as the complexity associated with the usage (van Deursen et al. 2021). Recent work has also shown that many users are unaware of the privacy risk posed by devices that do not have a camera or microphone (Zheng et al. 2018) as well as IoT devices in general (Emami-Naeini et al. 2021; McDermott, Isaacs, and Petrovski 2019; Oser et al. 2020; Remesh et al. 2020) leading to incomplete threat models (Abdi, Ramokapane, and Such 2019; Gerber, Reinheimer, and Volkamer 2018). It was also shown that users often express low privacy concerns (Nemec Zlatolas, Feher, and Hölbl 2022; Williams, Nurse, and Creese 2017) even when using applications which pose security and privacy threats (Park, Kim, and Jeong 2018; Saeidi et al. 2021). This is amplified by the fact that many owners of smart home devices do not fully understand their own data practices (Alshehri et al. 2022; Oser et al. 2020; Williams, Nurse, and Creese 2017). Consequently, while users see privacy and security risks with devices that have built-in cameras or microphones, they often trade them for functionality (Abbott et al. 2022; Remesh et al. 2020; Zeng, Mare, and Roesner 2017) or availability (Asplund and Nadjm-Tehrani 2016; Williams, Nurse, and Creese 2017). This shows the need for solutions that increase the privacy and security of users but minimise the costs of using them (Abbott et al. 2022; Edu, Such, and Suarez-Tangil 2020). Especially, it was shown that these solutions need to establish not only privacy and security but also trust in the devices (Aldowah, Rehman, and Umar 2021).

2.3. Research gap

Trust in an IoT device plays an important role during the adoption process (AlHogail 2018; Menard and Bott 2020). Since security is an integral aspect for people in order to trust an IoT device, mechanisms which increase the security of IoT devices are needed. A key part of security responsibility lies within user awareness as careless behaviour can lead to security breaches (Mahmoud et al. 2015). Consequently, HCI research towards aiding user security awareness and behaviour in dealing with IoT is necessary. While there are mechanisms enhancing the security of consumer IoT devices (Aman et al. 2020; Ambrosin et al. 2020; Dushku et al. 2020; Kuang et al. 2019), there is also a need for mechanisms that are easy for end users of the devices to use (Lafontaine, Sabir, and Das 2021). Here, one way to improve the security of IoT devices is remote attestation. A problem with many traditional remote attestation concepts is that the remote attestation process is completely nontransparent to the user. Thus, the user is neither part of the attestation process, nor can observe the attestation process, and thus often has an incomplete threat and security model (Oser et al. 2020; Zeng, Mare, and Roesner 2017). Previous work showed that it is possible to make remote attestation user-observable (Surminski et al. 2023). However, it has yet to be proven, that this increases the trust in this process. Therefore, we investigate whether users trust the attestation process and explore whether making it more transparent can further improve the trust in remote attestation. Furthermore, we examine another important aspect of remote attestation, concerning the issue of how to deal with a compromised device. Remote attestation by design can only detect a compromise but does not specify how to react to it. In end-user scenarios, it is therefore relevant to determine whether users trust the attestation and whether they trust benign – but formerly compromised – devices.

3. Attestation application

In this section, we present the attestation application used for our study. Based on a remote attestation scheme (Surminski et al. 2023), we built an application that can be used to simulate the attestation process of a smart speaker. The reason for choosing a smart speaker is that these devices have a widespread use and have microphones installed, which poses risks for privacy but enables the attestation via an audio side channel.

3.1. User-observable attestation

SCAtt-man is a remote attestation solution specifically designed for IoT devices (Surminski et al. 2023). To solve the problem of lack of device authentication in software-based attestation and ensure reliable one-hop communication, side channels such as sound and light are used. In SCAtt-man, the missing device authentication is replaced by the user. By using user-observable communication channels, the user can monitor the attestation process and identify the device currently being attested. Figure 1 shows a depiction of this process. To perform the attestation, users can use their smartphones with an attestation app. For practical usage, this functionality could potentially also be integrated into the configuration apps already used for smart speakers, such as the Amazon Alexa app. The app guides the user through the attestation process. First, the user presses the attestation button on the device they want to attest. Then, the app performs the attestation and asks the user to wait. After completing the attestation, the app informs the user about the result. If the attestation fails, the app shows the user how to resolve the problem. A failed attestation can have different reasons: There has been a communication failure, either on the audio communication or the network transmission, or the device has actually been compromised. To rule out transmission problems, the user can repeat the attestation process. After multiple failures, the user can resolve a compromise by resetting the device to factory settings and repeating the attestation process to verify the device's integrity. In an initial user study, it was shown that SCAtt-man has good usability and people trust the attestation (Surminski et al. 2023). However, the prototype only had limited smart speaker functionality as the focus was on the development of the attestation scheme.

Figure 1. The attested device communicates via audio with the smartphone. The user observes this process and can identify the attested device.

3.2. Smart speaker for attestation

To tackle the problem of limited functionality, we re-implemented the SCAtt-man attestation scheme into a state-of-the-art smart speaker. As smart speakers such as Amazon Alexa and Google Home are closed systems, a full implementation of SCAtt-man was not possible as this requires full access to the hardware and software. Therefore, the only solution is to imitate the behaviour of the SCAtt-man attestation scheme.

We developed a smart speaker application for the Amazon Alexa voice assistant that resembles the SCAtt-man attestation process. Amazon Alexa is the most popular smart speaker with a market share of 28.3% (Business Wire 2021). Amazon Alexa offers a wide range of services and keywords and can be used for daily tasks, such as setting timers and alarms, creating shopping lists, telling jokes, or playing music. Many IoT devices allow integration so that Amazon Alexa can be used to control smart homes with voice commands. It can also be used for more complex tasks such as online shopping. Amazon Alexa allows third-party apps and integrations. These apps are called skills and can further extend the functionality of Amazon Alexa. Our implementation consists of three parts: (1) An Alexa skill to implement the attestation behaviour of the smart speaker; (2) The installation of the attestation app on the smartphone running Android 13; (3) A coordination service to allow communication between the app and the skill.

Figure 2 shows the architecture of our setup. For the smart speaker, we used a standard Amazon Echo Dot. We built a smart home setup with a smart lamp and a smart plug. For the attestation, we required a button to start the attestation. For this, we used a Flic button that communicates over Bluetooth LE using a companion app. We used a hardware button to trigger the attestation because we wanted the attestation feature to be easily accessible to the users. Since we investigate user-observable attestation, it would be detrimental to hide the attestation feature inside a menu. A voice command would be an alternative, but in a real-world scenario, this would require every user to read the manual to find this function. We ran the app for the Flic button as well as all other management apps on a dedicated Android smartphone to separate these apps from the user. We designed a custom 3D-printed stand for the Echo Dot and the Flic button to provide the look and feel of a complete, professional device. Figure 4 shows a photo of the final setup. Furthermore, we enhanced the case with a reset button, to perform a reset process during the user study. The Alexa app, like any other Alexa app, was deployed to Amazon AWS. It was not possible to implement actual audio communication between smartphone and smart speaker due to limitations in audio processing in the Alexa API. We, therefore, developed a coordination service that allows coordination between the app and the Alexa skill. This setup allows straightforward usage anywhere as long as Internet connectivity is available. We implemented this service using PHP and an SQL database. This service also has a dedicated web interface to allow the experiment supervisor to influence the attestation result (success/failure) and to activate the extended explanations in the smartphone app, thereby allowing simple control of the complete experiment. We modified the smartphone app so that instead of audio communication it uses our coordination server to start the attestation process. Furthermore, it was extended with explanations. These explanations as well as the result of the attestation can be controlled by the coordination service. The setup for our user study is described in detail in Section 4.1.3.

Figure 2. The architecture of our implementation.

3.3. Application design

We modified the smartphone application (Surminski et al. 2023) to make it work with our simulated attestation process.

3.3.1. Layout

In Figure 3, the different screens of the application for different steps in the attestation process are shown. On the start screen, the user sees the application logo and two buttons. After clicking the explanation button, a pop-up window is shown which explains the attestation process using the following text:

Figure 3. Standard version of the application (top), transparent version (bottom).

Attestation is used to check the security of devices. To do this, you send a device a task that it must solve and return the result. If the result is correct, the device is safe. However, how can you get a reliable reading from a hacked device? After all, if there is a virus on the device, it can influence the measurement and claim that everything is OK. In order to prevent this, the device is sent a mathematically so complex task that the capacities of the device are fully utilised by it. In this way, any additional load caused by malicious code is noticed immediately. That is why the measurement takes a few seconds. However, it is necessary to prevent the device from forwarding the task to another device, on which the solution for the task could be calculated. Therefore, during the scan, the device is automatically disconnected from the Internet and communication is performed over sound. This way you can tell which device is responding and no virus can forward the task.

The other button starts the attestation process. After it is clicked, a screen with instructions on how to start the attestation process on the smart speaker is displayed. The instruction is supported by a picture indicating where the attestation button is located on the smart speaker. After clicking the attestation button on the smart speaker, the simulated audio communication is started. First, the smartphone emits a sequence of tones, then the smart speaker sends a sequence of tones. During this process, the user sees a picture indicating the communication and a text saying that the user should wait until the communication and thus the attestation is done. The progress of the attestation is visualised by a progress bar at the bottom of the display (see third picture in Figure 3). When the attestation process is completed, the user is shown the result of the attestation. In case of a successful attestation, the user observes a green result screen saying that the attestation was successful (see fourth picture in Figure 3). The ‘failed’ screen in red colour informs the user that the attestation may have failed either because the device is compromised or because of a transmission error (see fifth picture in Figure 3). In this case, it is recommended to the user to conduct the attestation again. For this purpose, the user can get back to the start screen by clicking the respective button. If the attestation repeatedly fails, the user should reset the smart speaker by pressing the reset button. A button in the top left corner of the ‘failed’ screen can be pressed to get further information on where the reset button is located (see sixth picture in Figure 3).

Besides the standard version of the application, we also developed a transparent version of the app (cf. Figure 3 bottom row). This application consists of the same screens as the standard version but with additional explanations. At each step, additional explanations are displayed describing what is done in the background to inform the users about the mechanism behind the attestation. Solely the success screen and the screen providing explanations on the location of the reset button were not altered. We developed this version of the application to be able to examine whether additional explanations can improve the understanding of and trust in the attestation process.

4. Qualitative evaluation

In this section, we present the design of our study on user-observable remote attestation and the results from the 35 interviews we conducted.

4.1. Methodology

As mentioned in Section 1, the main research question we aim to address is: ‘Does transparency enhance trust in remote attestation?’ In order to answer this question, it is necessary to understand how trust is generated. We investigate trust in two dimensions: The trust users place in the correctness of the attestation and the trust users place in their devices. Literature indicates that users have less trust when confronted with a black-box system or when they do not understand how their data is processed (Alshehri et al. 2022; Zeng, Mare, and Roesner 2017). This can be translated to: Users are more likely to trust concepts that they can comprehend. Therefore, understanding a concept is key to building trust. As related work shows (Zeng, Mare, and Roesner 2017), software systems are often black-box systems, which makes it hard to understand how a mechanism is working. We aim to make the internal functionality transparent to end users, in order to support their comprehension of security mechanisms. However, literature (Bawden and Robinson 2020) shows, that too much information can cause information overload and confuse the user, which would be detrimental to trust. From this, we derive our first sub-question: ‘How does transparency affect the users’ understanding of remote attestation?'

In this study, we use two approaches to provide the users with transparency: First, we provide users with two different versions of the app: a version providing just the basic functionality, serving as a control group, and a transparent version which contains information on what is happening during each step. Other researchers showed that too much information can lead to information overflow and thus contradict our aim of educating the users about security mechanisms. Therefore, we chose to add small text in every step of the attestation process, explaining what is happening at the moment. Due to space limitations, we only used short text, which should not be overwhelming for users. The investigation of abstract concepts like understanding and trust is prone to biases such as the acquiescence bias (Moss 2016). To combat this, we use a two-fold approach: We construct questionnaire items (see Table A4) to measure the self-reported understanding of the participants and secondly, we interview the participants after the interaction with the demonstrators and ask them to reiterate how they think the attestation works. We then can compare the self-reported understanding with their ability to reiterate the process in the experiment. This way we can limit biases and validate the self-reported understanding to get a more detailed impression of the actual understanding of the participants. Questionnaire items B1 (I understood how the app works) and B2 (I understood how the process behind attestation works) investigate the degree of understanding of the process of remote attestation. We decided to split these items in two, because participants could confuse the usage of the app with the theory behind the process if not asked for both separately. All items are presented with a 6-point Likert scale and a ‘no answer’-option. To answer the first sub-question for the first transparency condition, we compare the derived level of understanding of both groups of the A/B testing.

For the second approach, we embedded an explanation into the app (cf. Section 3.3). On the title screen, users could use the ‘explanation’-button in the top right corner to read the explanation text on how the remote attestation works. This text describes the process behind the attestation in an easily accessible way and was designed for users who would like to learn more about the process. Similar to the first examination, we investigated the effect of reading the explanation text on the understanding of the attestation process by utilising the combined approach of questionnaire items and interview to compare the understanding between the participants who read the explanation text and those who did not.

Since we are investigating whether transparency improves understanding, we also have to investigate, whether understanding leads to more trust in order to answer our main research question. From this we derive our second sub-question: ‘How does the users’ trust change by gaining a deeper understanding of remote attestation?'

To investigate this, we interviewed the participants about their prior experience with smart speakers and how they perceived the attestation process. For this purpose, we utilised the guiding questions from our interview guideline (‘Do you use smart home devices at home?’; ‘How does attestation change your interaction with smart home devices?’; ‘How well were you able to understand how the attestation process works? (Please reiterate in your own words how you think the attestation works.)’). Depending on their reports on prior knowledge, as well as their ability to reiterate the process, we validated whether key features such as audio transmission were understood and whether this understanding increased the participants' trust in the attestation.

Finally, to test the trust in remote attestation we have to consider the case of a compromised device. Therefore, our third sub-question is: ‘Do users trust formerly compromised devices, that have been restored and verified with remote attestation?’.

To answer this, we use two smart speakers in our setup. The user interacts with both, but the first one that is attested should succeed the attestation, while the second one should fail. Then a reset scenario is conducted and the attestation is repeated. In the interview, we investigate with our guiding questions (see Appendix A.1) whether the users would change their behaviour towards the formerly compromised device and which role the attestation process plays in their answer.

4.1.1. Ethical considerations

Since we are gathering personal and personal-relatable information from our participants, we consulted our university's ethics committee and obtained the approval (date of approval and approval number deleted for anonymization). To inform the participants about the aims of the study and the data to be collected, we provided an informed consent form. Following the guidelines of the ethics committee, the form contained information on how the data would be processed, when it would be deleted, and whom to contact to withdraw consent. Personal data collected from the participants were age, gender, education degree, and their IBAN to be able to transfer the compensation of $\text{€}$20. The consent form was mailed to the participants before the evaluation to allow them to read it unaffected by a study situation in which it would be more difficult to reject the terms of the consent form. Before each session, participants were given a hard copy of the consent form, and they were given time to review and sign the document. At the end of the study session, the participants were informed that the attestation process was simulated and no actual attestation was performed. The reason for this is that in our evaluation scenario, one device should be compromised. Since we consider it unethical to use real malware in this evaluation setting, we chose to simulate the attestation results. The participants were informed about this fact at the end of the sessions and asked whether this would change their view on the attestation process. None of the participants stated that it would change their opinion on the attestation process, while some even stated that they expected something like this since they were in a study setting. We concluded each session by asking for the participants' secrecy on the content of the study because it would negatively affect the results of the study if other participants knew how to use the app, which would render the usability evaluation useless. Making transparent why it is important not to disclose the study content to other participants was successful, since we received feedback from multiple participants that they asked other participants beforehand what to expect, but were turned down because of our request.

4.1.2. Participants

We recruited 35 participants for our study from the university environment, by advertising our study on the campus and offering a link on a QR-Code for self-registration for possible time slots. The ages of the participants ranged from 20 to 42 years, with a mean age of 24.6 years, a standard deviation of 4.07, and a median of 23 years. Nine of them identified as female, 26 as male. Although the subject areas of the participants covered a wide variety of topics, this led to a bias in the sample due to the mostly young and educated participants (cf. Appendix A.2, Table A1). However, this participant group is similar to the main user group of smart speakers.¹ Interesting for our study is that the participants had different experiences with smart devices (cf. Appendix A.2, Table A2). Thus, we can examine differences between both groups of users, who on the one hand have experience with smart speakers and on the other hand do not have experience with smart speakers.

Table A1. Demographic information on the participants.

	App version	Did read explanation	Age	Gender	Field of Study
P1	Standard	No	20	male	Electrical engineering
P2	Transparent	Yes	27	male	Electrical engineering
P3	Standard	No	20	female	Mechatronics
P4	Transparent	No	27	male	Mechanical engineering
P5	Standard	No	27	male	no answer
P6	Transparent	No	27	male	Electrical engineering
P7	Standard	Yes	22	male	IT-Security
P8	Transparent	Yes	26	male	Business Informatics; Entrepreneurship and Innovation
P9	Standard	No	23	female	Computer Science
P10	Standard	No	28	male	JBA Philosophy/Political sciences
P11	Transparent	No	23	female	Cognitive Science
P12	Transparent	No	26	female	Civil Engineering
P13	Transparent	Yes	21	male	Political sciences B.A
P14	Standard	No	26	male	Computer Science
P15	Transparent	Yes	24	male	Computer Science
P16	Standard	No	22	male	IT-Security
P17	Transparent	Yes	22	male	Computer Science
P18	Standard	Yes	23	male	Computer Science
P19	Transparent	Yes	33	female	Education in Global Technisation Processes
P20	Standard	Yes	25	male	Computer Science
P21	Transparent	Yes	29	male	Computer Science
P22	Standard	Yes	23	male	Computer Science
P23	Transparent	Yes	25	male	IT-Security
P24	Standard	Yes	22	male	Technology and philosophy
P25	Transparent	No	22	male	Computer Science
P26	Standard	Yes	42	male	JBA History/Computer Science
P27	Transparent	No	24	male	Mechatronics
P28	Standard	No	21	male	Medical technology
P29	Transparent	No	21	female	Biology
P30	Standard	No	26	male	Mechanical Engineering
P31	Transparent	No	23	female	Biomedical engineering
P32	Standard	Yes	22	female	Political science and law
P33	Transparent	No	23	female	Computer Science
P34	Standard	No	24	male	Business Informatics
P35	Transparent	Yes	22	male	Medical technology

Table A2. Prior experience of the participants with smart devices on a scale from 1–6 (Fully Disagree – Fully Agree).

	I have used smart speakers before.	I own a smart speaker or other smart home device.	I trust smart speakers.	I trust my smartphone.
P1	6	6	4	5
P2	1	1	2	3
P3	6	6	6	6
P4	4	6	4	4
P5	6	6	–	4
P6	4	1	4	5
P7	6	6	4	5
P8	1	1	1	3
P9	5	1	2	2
P10	6	6	2	4
P11	5	1	3	4
P12	1	1	4	6
P13	2	1	1	3
P14	3	4	–	6
P15	5	1	1	2
P16	2	1	3	4
P17	6	1	2	4
P18	6	1	1	4
P19	1	1	3	4
P20	6	3	3	4
P21	6	6	4	5
P22	4	1	3	5
P23	2	1	1	5
P24	6	3	1	1
P25	6	6	4	4
P26	1	1	5	5
P27	1	1	1	–
P28	2	1	2	2
P29	6	6	4	5
P30	4	1	4	4
P31	1	4	3	3
P32	6	6	3	3
P33	1	1	3	4
P34	4	6	4	5
P35	6	6	4	4

4.1.3. Materials

For our study, we built a setup to allow the participants to get used to a smart speaker and be able to compare the behaviour of them with compromised and non-compromised smart speakers. The setup consists of the two Amazon Echo Dots with stands described in Section 3.2, a lamp with a smart light bulb and a smart plug connected to a Raspberry Pi to show the function of turning devices on and off via a smart speaker. Thus, a smart home environment should be simulated. A picture of the evaluation setup is shown in Figure 4. Both smart speakers were required for our study, because in our scenario, one of the smart speakers was compromised, i.e. the simulated attestation of it would fail. Thus, we could examine how a user reacts when finding out that one smart speaker is compromised. Additionally, we were able to observe and ask the participants how much trust they have in the compromised smart speaker and how that changes when the smart speaker is reset. After a press on the reset button, the attestation would succeed because the simulated malware would be removed by resetting the device. We provided the participants with a laptop to fill out the questionnaires for our study, which we prepared in a self-hosted instance of LimeSurvey.

Figure 4. Picture showing the evaluation setup.

4.1.4. Procedure

The study was conducted between 01/18/2023 and 01/31/2023. Each session consisted of several parts which were completed by a single participant in about 30 to 50 minutes. In the beginning, we asked participants for demographic information and four questions about their experience with smart speakers (cf. Table A2). All participants were given a short introduction, explaining the scenario and telling them about their tasks. Afterward, the participants were asked to interact with both smart speakers. In order to know what they could do with the devices, a handout with possible actions was provided. The described actions were regular smart speaker commands such as setting a timer or telling a joke. The actions we included to simulate a smart home environment were turning on and off a lamp, changing its colour, and activating a connected device (a Raspberry Pi in our setup) via a smart plug. After the phase of familiarising with the smart speakers, users were asked to perform an attestation. We alternated the versions of the app for each participant, i.e. every participant had either the transparent version or the standard version. First, we assigned the app versions alternately to the time slots for the study, but when some participants did not show up, we alternated the app versions in the order of appearance. In the study scenario, the participant chooses one of the smart speakers to be attested. This first attestation succeeds and the participant is asked to attest the second smart speaker. When attesting this second device, the attestation should fail. On the ‘fail’ screen, the participant is informed about the next steps. The first proposed advice is to retry the attestation in case of a transmission error. When the attestation fails repeatedly, the participant should reset the device using the button on the back of the device. After resetting the device, the participant is asked to attest the device again. This time, the attestation succeeds showing the successful reset of the smart speaker. After testing the attestation app, the participants answered the UEQ-S (Schrepp, Hinderks, and Thomaschewski 2017) giving us information about the user experience when using the app. Additionally, the questionnaire contains questions about the understanding of the process (see Appendix A.3.2). We continued with a semi-structured interview for which we prepared some guiding questions to get more information about the users' perceptions. The study material can be found in Appendix A.3.

4.2. Results

Since we used a smartphone application to evaluate the attestation mechanism, we wanted to ensure that the usability of our application was high enough to not influence our results. For this reason, we used the UEQ-S questionnaire. The results (cf. Figure 5) show that our application has a good to excellent usability and thus does not influence our evaluation negatively.

Figure 5. Results of the UEQ-S questionnaire.

4.2.1. Measuring understanding based on the app version

In order to examine the effects of the app version on the understanding of the attestation process, we analysed the Think-Aloud study. There, we found evidence that many participants with the nontransparent application asked for more explanations on what to do. Participant P1 asked, for example: ‘Are there more explanations?’ Similarly, P3 told us that we should implement an explanation in the app. P5, P13, and P28 asked for more information on ‘Why should I do the attestation process?’, which shows that the meaning behind attestation was not understood. Some participants requested to be able to follow individual process steps better. For example, P9 wished ‘to see what the application tests at the moment’ and said that ‘there were no explanations what I have done or how the attestation checks that the system is secure’. Similar comments were made by P16, who asked for information ‘that explain what is happening and why’ and stated that ‘I could see that something is happening but not what exactly happens’ and P28 remarked ‘What is happening in the background and what do the tones mean?’. Confusion was also expressed by P10, who stated: ‘Normally I am no fan of tutorials but in this case, a more obvious explanation would be great’. P25 admitted ‘I only have a vague guess’ about what is happening during the attestation process and wished for ‘a more transparent overview of the methodology of the attestation’. P30 said ‘I have not understood what the application does’ and wished for ‘small pieces of information on every screen’, which describes our transparent version. P34 described the missing information as a reason for lack of trust in the process, as he is ‘not as convinced as he would be when having a technical explanation’. P14 even concluded that the process was ‘not at all’ comprehensible.

On the other hand, participants who received the transparent version of the application but did not read the explanatory text also asked for more information. Here, P4 stated: ‘I don't know what I did or what the background behind this process is.’ and P29 said ‘I think I did not understand well how this works’. P12 reflected on the fact that the explanation button was not initially seen or used, and thus the attestation process could not be comprehended. More generally, P27 wished to have more information on why attestation should be used and explained that he knew what to do because of the texts provided by the transparent version, but could understand the process ‘only a little bit’. The type of information requested can be categorised into more general information on the attestation process, and technical details for further information on the topic. One participant (P1) asked for both. Six of the participants with the standard version and four of the transparent version asked for general information on the process. Nine participants with the standard version and eight with the transparent version asked for further technical details.

We measured the understanding of the process in two different ways: The self-rating of the participants using questionnaire item B2 (I understood how the process behind attestation works) and the question in the interview to describe the process in their own words. Regarding questionnaire item B2, participants with the transparent version answered with a total score of 71 which is a mean score of 3.94 (4 = ‘Slightly Agree’), whereas the participants with the non-transparent version answered with a score of 59, resulting in a mean score of 3.47 (3 = ‘Slightly Disagree’). When the participants in the discussion were asked how well they could comprehend the process of the attestation, there was a slight difference between the participants with the transparent and the nontransparent version: We found discrepancies between the self-rating and the observed understanding of some participants: P4, P10, P14 and P29 claimed to understand the process in the questionnaire, but could not explain it in the interview. On the other hand, P19 underestimated her understanding in the questionnaire. While those who had little to no understanding at all were evenly distributed among the versions (6 transparent; 8 non-transparent), those who were able to freely reiterate how the process works, which was considered as high understanding by us, were more likely to be in the transparent version (8 transparent; 2 non-transparent). Participants with medium understanding were considered as those who had a general idea but could not explain the process in detail (4 transparent; 7 non-transparent).

4.2.2. Measuring understanding based on the explanation text

For the examination of the influence of the explanation text on the understanding of the attestation process, we analysed the interviews and compared the results to the self-reported understanding in the questionnaire. One remarkable finding is that some users who did not read the explanation wished for an explanation. P4 said that it would be great if he could read somewhere about how the process works. And P5 objected to the missing ‘?’-button for an explanation of functions. Similarly, P16 wished for an ‘i-button which explains what is happening and why’. P33 liked that the app ‘is very simple’, but would like ‘to have a possibility to get more information’.

The positive influence of the explanations on the understanding of the attestation process was confirmed by several participants. For example, P11 read the explanation and reflected after conducting the attestation that before reading the explanation, he was not sure whether the attestation would work. This was confirmed by P21 who said ‘You could see that something is happening but not what exactly happens; with the explanations, it became clear what happens’. P15 said that the process could be well comprehended and ‘I have a plan [of what is happening]’. P17 elaborated that ‘there was an explanation button’ and that the process was ‘explained well’. In addition, the suitability even for technology non-experts was noted. In this regard, P17 elaborated that the process ‘seems to be plausible, also for people who have not a high affinity for technology’ and it is ‘quite foolproof’. P18 explained that the process could be comprehended well and that ‘the idea could be understood’. P19 said that ‘the explanation was very good, short but detailed and not overloaded’ and ‘I could follow it [the explanation] as a student who does not study computer science’. P32 stated ‘I think the explanation is good. I am not so familiar with the devices. I study politics. I think it is well explained for people who know almost nothing [about IT-Security].’ P22 who only read the explanation after conducting the attestation (using the nontransparent application) said ‘I would have understood the process better if I read the explanation first’. P23 confirmed ‘it was nice that there was an explanation; without that, I would have pushed a button without knowing what was happening’. Thus the process could be ‘100% comprehended’. P35 said ‘I understood how it works; the explanation was fairly clear’. Several other participants (P15, P17, P18, P35) concluded that because of the explanations, the process as well as the general idea behind the process could be well comprehended.

Similar to the analysis of the app version's impact, we analysed those who asked for more information in the app. Out of ten participants who requested general information on the process, only one read the explanation, while nine did not. Among the 17 who wanted further technical details to look for more information on the topic eleven read the explanation and six did not. Regarding questionnaire item B2 (I understood how the process behind attestation works), participants who read the explanation answered with a score of 74 which is a mean score of 4.63 (5 = ‘Agree’), whereas the participants with the nontransparent version answered with a score of 56 resulting in a mean score of 2.95 (3 = ‘Slightly Disagree’). However, when the participants were asked in the discussion how well they could comprehend the process of the attestation, there was a clear difference between the participants who read the explanation and those who did not. In total, 16 participants read the explanation and 19 did not. 13 of those who had little to no understanding at all did not read the explanation, while only one of the participants who read the explanation could not comprehend how the attestation process worked. Furthermore, six of those who did not read the explanation and five of those who read it had a medium understanding of the process. Most significantly, ten of those who read the explanation were able to freely reiterate how the process worked, while none of those who did not read the explanation were able to do so.

4.2.3. The connection of trust and understanding

Please note, that the evaluation of the perceived trust and the understanding is indirectly affected by the two different app versions and whether the explanation was read since these conditions affected the understanding. In the following, we will not distinguish between the versions since these effects are already covered above.

The participants who had specific expectations based on prior knowledge are the first to show that understanding affects trust. For example, P18 and P25 knew that training data is important to train language models but were skeptical of smart speakers because of privacy violation incidents that were made public. This indicates that the level of trust is influenced by prior knowledge and thus can also be affected by understanding obtained during the use of a security feature. During the interviews, we found that there are different types of trust which were influenced by the understanding of the process: On the one hand, we observed trust in the process and the belief that this technical measure can indeed detect malicious modifications of the smart device. On the other hand, we detected that understanding of the process resulted in disappointment because the participants lost their naive and illusionary trust in the device: Before learning how attestation works, they believed that this would also prevent the manufacturer from collecting data via the smart speaker. Learning that attestation 'only' prevents malicious third parties from (mis-)using the device reduced their illusionary trust which originated from their former perception of what protection means and who was perceived to be the antagonist. In total, 17 participants of our study stated that their handling of smart speakers would not change because of the attestation. The main reason for this answer is that the users' problem is not that the smart speaker could be hacked but that the personal and spoken information is sent to the manufacturer of the devices (P7, P9, P22, P34). Since attestation can only detect that a device is compromised but not whether the information is sent to the manufacturer or not, attestation does not solve this basic skepticism. These participants were slightly disappointed when they learned that the attestation process only guarantees that no malicious third party has access to the device and that attestation cannot limit the amount of data that is transmitted to the manufacturer. Therefore, transparency does increase trust in the attestation process but weakens the illusionary trust in misconceived security and privacy. This is also one of the reasons why participants indicated that they do not share some personal information with the smart speaker (P8, P15).

General trust issues with companies were brought up by several participants: P12 said: ‘I lose the control over my data and do not know what the companies actually do with it’. The struggle to minimise the amount of data that is given away is described by P28: ‘I try to minimise tracking in my life. I make the three extra clicks to decline tracking in cookie banners and get angry with my friends when they click on ‘accept all’. I studied computer science and am aware of the risks associated with data collection. However, sometimes you get ads for something you were talking about a few days earlier and then you know: somewhere data was collected’. The fear that companies may monetise the collected data was shared by P5, who stated the concern that people can be manipulated to buy products. On the other hand, P1 had little concerns about giving away data: ‘I know that companies get my data but I am interested in the technology. Furthermore, I am young and have nothing to hide’. Another factor for indifference towards data collection despite lacking trust in companies was stated by multiple participants: Resignation. After experiencing that their data already was disclosed to companies, many only could 'trust' the companies that their data was not misused.

Tracking happens everywhere: via smartphone, browsers, or social networks. This could provide benefits to save time but then you get a suspicious offer for things you may need and then it gets creepy. It all depends on what is actually done with our data by the companies (P30).

‘I am aware that companies gather my data. It is the same issue with my smartphone’ (P33). This attitude was shared by P11. P29 continued: ‘Google has my data anyway via my phone. You can't do anything against it’. Similarly, P27 stated:

They know all of us already. As soon as you start using the service of one company you lose all inhibitors. In my social environment, I frequently hear: ‘Google already has my data thus I do not care anymore about my data’.

A similar statement was made by P9. However, even those who were not concerned about the data collected by companies showed signs of increased trust in smart speakers after learning about attestation: For example, P17 does not think that Amazon cares about what he talks about in his living room, but still finds the concept of a device listening to his voice a bit creepy: ‘You do not know who else is listening’. Therefore, attestation is useful to increase the trust of users who think similarly.

When investigating the trust of the participants towards the security provided by attestation, it became clear that the mental models of the users played a major role. These depended on how the participants imagined the motives and possibilities of different actors. P3 and P7 thought that hackers were no threat to them because they were not interesting enough to be targeted. P21 shared this belief, but still wanted to take action to prepare: ‘I do not believe that I get personally hacked, but I do want to secure my data’. P35 distinguishes between different devices to form a threat model: 'Regarding privacy invasion I am more afraid of my smartphone than a smart speaker. The smart speaker is only in one room, but my phone is always by my side'. And finally, many still were convinced that data collection by companies is a bigger threat than malware and hacker attacks: ‘I am not sure whether a hacker attack is the biggest problem. I am more afraid that a company misuses my data. […] Big companies gathering my data is more scary than hackers’ (P35). P24 underlined the perceived threat by data collection: ‘I think a bigger threat than a private hacker is the power amassed by big companies. This could pose a threat to society’. The evaluation of different actors and their capabilities was summarised by P21:

I am not concerned that secret services are getting data from smart speakers. I got a smartphone after all. I am not afraid of governmental surveillance, but I do not want companies to use my data for marketing; My biggest fear is not a specific ‘hacker organisation’ but security breaches in general.

These differences in the perceived power of different actors are similar to the misplaced illusionary trust in attestation. Users have specific mental models and these influence their threat models and the level of trust they place in someone or something. Therefore, prior knowledge is a major factor in evaluating threats and trust. This can be achieved by transparency during the interaction with a tool or by prior experience as P34 illustrates:

I turn a blind eye to the influence of big companies in my everyday life. I once requested my data from Google and they had data from my childhood. Until now nothing bad happened to me, but it depends on how much you trust the company.

4.2.4. Trust in formerly compromised devices

Regarding our third hypothesis, we examined (1) whether device handling would change due to the possibility of attestation and (2) whether the interaction with the device hacked in our study would change after successful attestation following a reset (see guiding questions (iv) and (vi) in Appendix A.1). Regarding the first question, six participants stated that the attestation increased their trust in the smart speaker (P5, P20, P21, P23, P25, P28). Although they described an improved trust in the device, some users stated that the handling of the smart speaker still would be the same (P20, P21, P25). Three users reported that the attestation leads to an increased sense of security (P6, P16, P35). The greatest influence of attestation was on five participants, who stated that the possibility of attestation would be a reason to purchase the device, although they do not currently own a smart speaker because they do not trust them (P11, P12, P18, P19, P28).

For the second question, 17 participants responded that a formerly compromised device would not be handled any differently than a non-compromised device after a reset and subsequent successful attestation. For participant P4, the reset with a successful attestation would increase the trust in the device. Seven participants stated that they would be more skeptical about the formerly compromised device and would do the attestation more frequently. Four participants would ask themselves how the smart speaker could be compromised (P5, P6, P22, P33). For six participants the compromised device would lose their trust and would not be used anymore. Two of those would also lose trust in similar devices, even if they had not been compromised (P1, P7). In conclusion, attestation influences the trust and sense of security positively and can also be a reason to buy a device. One participant even stated that he would pay extra for a device with a built-in attestation function (P4): In case of a device costing $\text{€}$50, he would be willing to pay $\text{€}$10–15 extra for the attestation function. Otherwise, a fundamental skepticism towards smart speakers and their manufacturers cannot be changed even by attestation. However, most of the participants trust the attestation when a formerly compromised device is attested successfully after a reset.

5. Discussion and conclusion

In this section, we reflect on the results to test our hypotheses and answer our research questions.

5.1. Findings and implications

First, we examined the difference in understanding of the attestation process between users of the transparent version of the app and those using the nontransparent version. During the interviews, we observed that participants with the nontransparent version of the app asked more frequently for more information on the process of the attestation. However, this might be an effect of the sample size, the design of the versions itself, or the expectations of the participants. The latter can be derived from the quotes of P5 and P16 who asked for a ‘?’- or ‘i’-button for more information. This implies that users do not expect transparency in each step but a central source of information. In the interviews, we could observe that participants who had the transparent version of the app had a slightly better understanding of the process than those who received the nontransparent version. Therefore, we formulate the hypothesis that the transparency difference in the app version does not have a relevant influence on the understanding of the attestation process.

Second, we analysed the difference in understanding of the attestation process between users who read the explanation text and those who did not. We could show that the majority of those who wanted general information on the process did not read the explanation, while the majority of those who read it requested further technical details to dive deeper into the topic. Additionally, we could confirm that all participants who were able to reiterate how the process works in the interview had read the explanation. Overall, we formulate the hypothesis that users understand the process of attestation better when reading the explanation text in the app.

Therefore, we show that making security mechanisms transparent for end users supports their understanding of the overall mechanism. This combats the problem of users not understanding how their data is processed, as stated by Alshehri et al. (2022) and the practice of designing smart speakers as black-box devices (Zeng, Mare, and Roesner 2017). Therefore, we can answer our first research question (‘How does transparency affect the users’ understanding of remote attestation?') with: Increased transparency led to an increase in understanding in our sample.

When investigating how understanding and trust are related, we found that the mental models of the participants played a major role. Those who had experience with security issues or heard of privacy violations were more skeptical than others. Furthermore, many participants thought that they were not interesting enough to be targeted by hackers. However, in the context of IoT, this is not the correct threat model. Users are more likely to fall victim to broadly deployed ransomware or botnets than being targeted as individuals by hacker groups. Therefore, educating these users is highly important to achieve actual security. By utilising transparency the understanding of users can be enhanced, which proved to be essential for building trust in the security mechanism and to reduce false mental models such as misplaced trust. This illusionary trust was placed in the attribution process due to false mental models and by attributing properties to the system due to lacking understanding.

Therefore, we can answer the second research question (‘How does the users’ trust change by gaining a deeper understanding of remote attestation?') with: Understanding affects the mental models of the users and enables them to estimate the correct level of protection. Furthermore, false attributions are revoked and real trust can be built.

Our third sub-goal was the investigation of whether participants would trust a formerly compromised device after it was reset and successfully attested. During the interviews we received mixed feedback: On the one hand, resetting and attesting a corrupted device increased the trust of some participants. On the other hand, some participants indicated that they would become suspicious of all similar devices if they discovered a corrupted device in their vicinity. Most participants agreed that the possibility of attestation does not affect how they interact with IoT devices in general. Some indicated that their trust is strengthened, but they would not share more information with the smart speaker than they would without attestation. Even if the device was compromised and restored, the behaviour towards the device would not change. However, participants stated that they would be alarmed and conduct the attestation process more often after the discovery of a corrupted device. Some stated that they would test the device once a day for at least some weeks after regaining trust in the device. Others stated that they would stop using the smart speaker if it was compromised again after the reset. This aligns with findings from literature pointing out that security is an important factor for users to trust IoT devices (AlHogail 2018). With these findings, we can answer our third research question (‘Do users trust formerly compromised devices, that have been restored and verified with remote attestation?’) with: Yes, most users do trust formerly compromised devices if they have been restored and verified via remote attestation.

Overall we could investigate that transparency indeed increased the trust in remote attestation. By giving the users the possibility to comprehend the process, they were able to evaluate the result of the attestation themselves and could take action as they deemed necessary. The fact that most participants of our study would use a formerly compromised device after reset and successful attestation shows the level of trust towards the attestation.

5.2. Limitations and future work

Our study is subject to several limitations. First, our sample consists of relatively young and well-educated participants. Although this resembles the target group of smart speakers closely, it still makes our study not representative of a broader target group. Second, our methodology is a qualitative study. While qualitative studies allow for a more in-depth investigation of a topic, they are not representative on a large scale. By reaching saturation during the interviews due to the lack of new input from the participants, we can claim that most perspectives are represented. However, even though we interviewed participants of different nationalities, a sample from a different country may bring up other results since risk cultures differ based on the background of people. For example, Reuter et al. (2019) showed that Germany had the strongest privacy concerns of all investigated European countries when interacting with social media. Therefore, we encourage other researchers to use our study design as a basis to conduct comparative studies or even quantitative studies on this topic in future work to broaden the perspective of the research community on this topic. Another limitation is the suspicion of some participants that the smart speakers were not really compromised. This indicates a different perception of the situation in this study compared to the real world which could result in a different behaviour in the real world. Future work could also develop other methods of transparency to evaluate their effect on the trust of users towards attestation. Another interesting approach would be testing the limits of transparency. In our study, we found that transparency enhances the understanding of the users and thus the trust in the attestation process. The results of the usability scale and the fact that the participants could reiterate more or less how the process worked led to the conclusion that our setup provided a useful amount of information. However, including too much information could lead to information overload and thus reduce the level of understanding. Testing the limits of the amount of information which is helpful and the point when it becomes detrimental could greatly enhance the research on information overload and decision fatigue.

5.3. Conclusion

Finally, we can summarise our findings to answer our main research question: Does transparency enhance trust in remote attestation? In our study, we confirmed that transparency as an HCI intervention does increase the understanding of processes such as remote attestation. We were also able to confirm that understanding the process of remote attestation leads to increased trust in the overall process. One example of this is the audio communication between the devices. Most participants thought it would just be feedback for them to show that the attestation is in process. However, by realising that audio communication is useful to avoid remote attackers, users did trust the attestation process more. This trust even allows them to rebuild trust in formerly compromised devices which were restored and verified by remote attestation. Therefore, we conclude that trust is based on prior knowledge as well as mental models. Transparency affects both by increasing the understanding and thus the knowledge about a security mechanism, and on the other hand it reduces illusionary trust which originates from mental models and false attributions. Thus, transparency helps build real trust in the actual functionality of security and privacy features. Therefore, transparency indeed supports users to calibrate their trust to the trustworthiness of the system and we can answer our main research question in the affirmative, and conclude that transparency enhances trust in remote attestation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Grant 236615297 (SFB 1119 CROSSING); and this research work has been funded by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE and Technische Universität Darmstadt.

Notes

1 https://speakergy.com/smart-speakers-statistics/.

Appendix

A.1. Interview guideline

A.1.1. Outline

(1)	Introduction (5 minutes): First, the evaluation begins with the introduction of the topic and the informing of the test subject. For this purpose, the Informed Consent (mailed beforehand to the participant) is handed out and discussed in detail. When a brief introduction to the topic is given, care should be taken not to anticipate too much, as this could bias the results. We do not want to ‘put words in the mouths’ of the participants. It is important to clarify that the purpose is to test a particular tool for its practicality and that the purpose is not to evaluate the participants themselves. Finally, the consent of the participant and their signature on the Informed Consent are obtained.
(2)	First questionnaire (5 minutes): The participant is asked to fill in the questionnaires that have already been opened on a laptop computer. Care should be taken to ensure that the interviewers gives the test person a little more space and does not give the impression that they are permanently looking over their shoulder. (a) Demographics (b) Prior experience with smart speakers
(3)	Hands-on experiment (10–15 minutes): The experimenter instructs the participant to please say each thought and idea out loud. The so-called think-aloud method helps with the evaluation, as it is important to collect initial impressions of the product and to record how the participant reacts to the device. In this way, problems and pitfalls can be identified that would otherwise not be explicitly noted. (a) warm-up with the smart home setting: Since not every participant has experience with smart speakers, the first task is to familiarise with the smart home setup which was prepared. The participants can freely interact with both smart speakers. For inspiration a sheet with example commands is given. (b) Step 1: Successful attestation of the first smart speaker: The participant is instructed with Please explore the app and conduct the process for both speakers. The attestation process for demonstrator 1 runs smoothly. The participant performs the required steps and receives the message that the attestation was successful. Important: The interviewer has to note whether the explanation button was used. (c) Step 2: Failing attestation of the second smart speaker: The attestation for demonstrator 2 fails because the device is corrupted. The participant receives a message stating that the attestation has failed and that the process should be repeated. (d) Step 3: Repeat the attestation process for the second smart speaker: The attestation for demonstrator 2 is repeated. It fails again. The participant is prompted to ‘reset’ the device in order to restore the delivery status. This is simulated by pressing the key on the device. (e) Step 4: Restore the second speaker: The participant presses the reset button and the instructor resets the simulation. (f) Step 5: Successful attestation of the second smart speaker: The attestation for demonstrator 2 is repeated and can now be performed successfully.
(4)	Second questionnaire (5 minutes) (a) UEQ-S questionnaire for usability (b) Understanding of the app and the process
(5)	Discussion and Interview (10 minutes) (a) Guiding questions: (i) Which feature would you like to add? (ii) What did you particularly like? (iii) Do you use smart home devices at home? (iv) How does attestation change your interaction with smart home devices? (v) How well were you able to understand how the attestation process works? (Please reiterate in your own words how you think the attestation works.) (vi) You reset the device that failed the attestation. How does this affect your interaction with this device in the future? (b) Reveal of the simulated attestation: The ethical guidelines require us to inform the participants about the simulated attestation, because this counts as ‘deceiving’ the participant. Therefore, we have to clarify the attestation was simulation and ask for the approval of the participants (Due to the regulations of the ethics board the participants could drop out of the study if they perceive the ‘deception’ as inappropriate). (c) Asking for discretion: At the end we ask the participants not to reveal the content of the study to others who are yet to participate in the study. This would negatively influence the results of our study.

A.2. Participants

A.3. Questionnaires

All questionnaires were originally in German and translated to English for this paper.

A.3.1. Smart speaker experience

We used the questions shown in Table A3 to survey the experience of the participants with smart devices.

Table A3. Questions on smart speaker experience.

Item	Question	Scale
A1	I have already used smart speakers.	6-Point-Likert-Scale (Fully Disagree – Fully Agree)
A2	I own a smart speaker or other smart home devices.	6-Point-Likert-Scale (Fully Disagree – Fully Agree)
A3	I trust smart speakers.	6-Point-Likert-Scale (Fully Disagree – Fully Agree)
A4	I trust my smartphone.	6-Point-Likert-Scale (Fully Disagree – Fully Agree)

A.3.2. Items of the questionnaire to answer sub-research question 1 and 2

Items were translated from German to English for this paper.

Table A4. Table showing items of the questionnaire after the experiment to investigate self-reported understanding.

Item	Question	Scale
B1	I understood how the app works.	6-Point-Likert-Scale (Fully Disagree – Fully Agree)
B2	I understood how the process behind attestation works.	6-Point-Likert-Scale (Fully Disagree – Fully Agree)

A.4. Hardware

• 2 Smart Speakers with custom printed stands and Flic button

• IoT devices to create a smart home setting

• Smartphone with attestation app

• Laptop for the interviewer to control the versions and the results of the attestation

• Laptop for the participants to answer the questionnaires (hosted via own LimeSurvey instance)