Interruptibility Estimation Based on Head Motion and PC Operation

Abstract

Frequent and uncontrolled interruptions by information systems that do not reflect the user’s state can result in fragmented working times and decreased intellectual productivity. To avoid adverse interruptions, interruptibility estimation methods based on PC operation information have been proposed. However, workers who use PCs to accomplish their primary tasks occasionally engage in paperwork. Occasional paperwork activities, which are not reflected in the PC’s operation information, can cause estimation errors. This study focuses on using the position of the head, posture, temporal motion, and continuity of the head position and posture while a worker is at his or her desk as indices to reflect engagement in the task at hand. Based on an analysis of the relationship between the head-related parameters and interruptibility, an interruptibility estimation algorithm is proposed using four head-related indices that reflect interruptibility during PC and non-PC work. Experiments indicate that estimation accuracy improves as a result of incorporating these indices in the algorithm.

1. INTRODUCTION

The popularization of the Internet and information systems has made online communication and data access easier. Various systems have been developed to facilitate communication and information sharing among workers at satellite offices and home offices. However, the spread of online communication systems has also increased the likelihood of interruptions by information systems. Frequent and uncontrolled interruptions distract the user and fragment working time (Lantz, 1998; Renaud, Ramsay, & Hair, 2006). Previous studies have suggested that switching tasks causes suspension and resumption of the problem state and memory processes related to the previous task (Altman & Trafton, 2002). Consequently, interruptions that do not reflect the user’s state may result in decreased intellectual productivity (Mark, Gonzalez, & Harris, 2005).

Numerous studies have attempted user state estimations with various techniques. These techniques include PC operation-based methods, which typically rely on counting keystrokes or mouse clicks (Honda et al., 1997; Minakuchi, Takeuchi, Kuramoto, Shibuya, & Tsujino, 2004), and sensor-based methods, which involve sensors that detect conversations or events (Danninger & Stiefelhagen, 2008; Forgaty et al., 2005; Milewski, 2006). However, the features utilized in the previous studies do not necessarily reflect intellectual activities that do not have observable outputs. Studies on human multitasking (Borst, Taatgen, & Van Rijn, 2010; Monk, Boehm-Davis, & Trafton, 2004) have suggested that resumption lags (RLs) when interruptions occur at task breakpoints are significantly shorter than the lags for interruptions at other times, even during an intellectual task. Therefore, focusing on task breakpoints appears to be a potential way to control the timing of information system interruptions.

Other studies have proposed that focused application switching (AS) could be an alternative to task breakpoints during PC work (Czerwinski, Horvitz, & Wilhite, 2004; Tanaka, Matsumura, & Fujita, 2010). A method based on AS was proposed in order to estimate user interruptibility; it represents the subjective degree to which a user can be interrupted, and the feasibility of applying this method to a real work environment was demonstrated (Tanaka et al., 2010; Tanaka, Fukasawa, Takeuchi, Nonaka, & Fujita, 2012). However, workers who primarily perform their jobs on PCs would occasionally do paperwork as well, and conversations among workers occur frequently. These activities are not reflected in the interruptibility estimated by the PC operation-based methods. Therefore, these non-PC activities can cause estimation errors and lead to excessively high interruptibility estimations of an uninterruptible state.

Kapoor and Picard (2005) suggested that several factors such as internal state, user engagement, posture, and task progression affect interruptibility during work. In particular, it has been suggested that the posture of the head reflects changes in the body’s position, gaze target transitions, and changes in concentration regardless of the nature of the task (McDuff, Karlson, Kapoor, Roseway, & Czerwinski, 2012). Therefore, head-related information is expected to represent the internal state, which differs from the physical activity, of workers. In such a case, incorporating indices of head motion might improve the estimation accuracy of the PC-operation-based interruptibility estimation, regardless of work target.

To establish an interruptibility estimation method applicable to general desk work, this article discusses the relationship between interruptibility and head motion features and proposes an interruptibility estimation algorithm. The effect of head motion on the accuracy of the interruptibility estimation is experimentally investigated.

2. RELATED WORK

2.1. Cognitive Cost of Interruption and User Interruptibility

The cognitive cost of interruption has been studied in human multitasking research. The studies suggested that switching tasks requires suspension and resumption of memory processes associated with the suspended task and causes RL (Altman & Trafton, 2002). The relationship between leading and subsequent tasks was also reported to affect the length of RL. For example, if the two tasks are strongly related, part of the memory process is shared, which shortens RL (Monk et al., 2004; Salvucci, 2010; Salvucci & Bogunovich, 2010). In other words, RL increases if a worker is interrupted by an unrelated task. It has been pointed out that the interruptions that do not consider the user’s state or task state, increase RL, decrease intellectual productivity, and increase fatigue and mental stress (Mark et al., 2005). Therefore, an interruption timing control taking into consideration the user’ state would be desirable.

Various techniques have been studied as ways of making user state estimations. A number of studies have focused estimating the busyness of users from PC operation activities such as keystrokes and mouse clicks (Honda et al., 1997; Minakuchi et al., 2004). These activity-based methods seem to be able to estimate busyness adequately during tasks that have observable physical activities. However, physical features do not necessarily reflect intellectual activities. On the other hand, attempts have been made to estimate busyness using electromyograms, heart rate, and body acceleration (Chen, Hart, & Vertegaal, 2007). However, physiological signals are easily affected by many factors other than the target activity. Moreover, this approach requires sensors to be put on the user’s body, which is often impractical. Another approach involves installing various sensors such as microphones and cameras in the work place (Danninger & Stiefelhagen, 2008; Forgaty et al., 2005). However, the related studies intended to estimate the type of user activity in terms of physical statuses such as having a visitor or using a computer. These studies depend on the assumption that interruptibility is basically determined by the type of task, including personal and social activities. The problem with this assumption is that temporal changes in interruptibility are not principally reflected when the worker continues a single task. Thus, these approaches do not seem to provide an effective way to control the timing of interruptions.

The relationship between interruptibility and task breakpoints has also been studied. It was experimentally revealed that RL for interruptions that occur at intentional task breakpoints is significantly shorter because the preceding task was already suspended (Borst et al., 2010). For example, immediately after a user saves a file to complete the current task is an appropriate time for an interruption. In contrast, interrupting a worker while he or she is continuously typing a document is inappropriate (Iqbal & Bailey, 2005). These studies indicate that user interruptibility instantaneously increases at task breakpoints even when the user is engaged in an intellectual activity. Therefore, for workers using information systems, automatic real-time breakpoint estimation appears to be a potential solution for controlling the timing of interruption.

To estimate the interruptibility of a breakpoint, the levels of task breaks have been categorized on the basis of task-structure analysis (Iqbal & Bailey, 2006). However, in an actual office environment, workers engage in various tasks. Task-structure analysis of all task combinations appears to be impractical. Therefore, an automatic estimation method, which is robust to a variety of task types, aims, and application software, would be desirable. Another study (Tanaka et al., 2010) focused on using AS as an alternative to task breakpoints during PC work. An experimental analysis of the relationship between interruptibility and AS demonstrated that interruptions at AS moments are significantly more acceptable than those during continuous work. A user interruptibility estimation method based on AS allowed detecting the timing of interruptions so that would have less cognitive impact. However, the frequency of AS depends on the task type. Thus, this method might not offer sufficient opportunities to present needed information. To detect more presentation opportunities, an interruptibility estimation algorithm involving “not-application-switching” (NAS) was proposed (Tanaka et al., 2012). This algorithm has the ability to estimate the interruptibility of research and development engineers, clerical staff, and managers performing PC work. However, the problem associated with occasional non-PC activities such as paperwork occurs in real office-work scenarios. In such cases, PC operation activities would be detected intermittently, and they wouldn’t be differentiated from a state of stagnation. This means non-PC work results in false high-interruptibility estimations. Thus, attributes of workers other than PC operation should be incorporated in the estimation.

2.2. Relationship Between Head Motion and Interruptibility

Measuring and quantifying non-PC work is difficult. Quantifications have been attempted using various devices, for example, a motion sensor attached to a pen (Minakuchi et al., 2004). However, it is not practical to deploy sufficiently versatile sensors for a wide variety of non-PC work in a real office environment. Previous studies have suggested that, in addition to work activities, user engagement and task progress affect interruptibility during work (Mark, Gudith, & Klocke, 2008; Matsuda, Kuramoto, Shibuya, & Tsujino, 2005). In particular, head motion is expected to be closely associated with the worker’s attitude toward the task and changes of the work target (Kapoor & Picard, 2005; McDuff et al., 2012; Sumi, Tanaka, & Matsuyama, 2004). Head motion is expected to be a significant factor in an interruptibility estimation because the head moves when the eyes change their focus and when the body moves. For example, when a worker concentrates on a task, his or her head move forward to concentrate on what he or she is gazing at. The position and motion of the head may thus represent the internal state of the worker such as concentration.

Therefore, we have focused on head motion while people are engaged in desk work and experimentally analyzed the relationship between interruptibility and head motion by using a three-dimensional motion capture system to record participants performing paper-based tests and jigsaw puzzle tasks (Kimura, Tanaka, & Fujita, 2011). The results revealed that interruptibility is lower when the subject assumes a head-forward posture and it increases when the head turns up regardless of the task type (Abe, Tanaka, & Fujita, 2012). However, so far, we have not investigated the estimation algorithm itself. Furthermore, the use of a 3D motion capture system limits the practicality of the method in terms of its size and cost. Therefore, in this study, we attempted to improve the interruptibility estimation accuracy by detecting the head motion using a camera and face recognition software and by incorporating indices of head motion in the interruptibility estimation algorithm.

3. METHOD: COLLECTING DATA FOR ANALYZING HEAD MOTION AND INTERRUPTIBILITY

We conducted a set of interruption experiments to collect data for analyzing the relationship between head motion and interruptibility.

3.1. Participants

Thirteen university students (two women) and two faculty members from the Department of Computer Science participated in the experiments. They were experienced computer users and skilled in office software operation and programming. All participants had some skill in touch typing. The participants were 22 to 36 years of age (M = 24.73, SD = 4.667) and were unpaid volunteers. Thirteen participants were right-handed, and two participants were left-handed. No participant had any medical impairment that would have affected his or her ability to operate a computer, such as extreme shortsightedness.

3.2. Environment

The experiment was performed in the environment shown in Figure 1. The experiments were conducted in the participants’ normal working environments. The participants used their own desktop PCs at their own desk. In particular, they used single PC monitor, which was placed at the center of the desk. To recognize their head motions, a camera, having a resolution of 640 × 480 pixels, was placed in front of the monitor. All participants used Microsoft Windows OS and installed the experimental system, which recorded data on their PC. The applications they used depended on their work. However, all participants used word processor, spreadsheet, and presentation applications. The experimental scenes were recorded by a video camera to confirm the tasks that had been performed. Figure 2 is a photograph of the experimental environment.

FIG. 1. Experimental environment.

FIG. 2. Photograph of experimental scenario.

The experimental system recorded the head motion, PC operations, and subjective interruptibility scores. Figure 3 illustrates its configuration. Resident software recorded the PC operations. At the same time, a camera and face-tracking software (Seeing Machines, faceAPI) recorded the motion of the head. During the experiment, the system occasionally interrupted the participants and required them to evaluate their level of interruptibility. The recorded PC operations, head motions, and subjective interruptibility assessments are described in detail in the following subsections.

FIG. 3. Configuration of experimental system.

PC operation and head motion

The recorded PC operations were mouse clicks, mouse wheel usage, active window name (.exe name), process ID, window messages (quit and clipboard), and the number of open windows. The sample interval was set to 500 ms, which is similar to the sample interval used in a previous study (Tanaka et al., 2012).

The XYZ coordinates were assigned to right and left, up and down, and back and forth head motions, respectively. Pitch, yaw, and roll angles represented rotations around the XYZ axes. The sample interval of the head motion was set to 500 ms.

Subjective interruptibility evaluation

The system presented a dialog window with a chime that interrupted the participant on the basis of the interruption rules. The interrupted participant evaluated their interruptibility at the time and inputted an interruptibility score into the dialog window. The participants evaluated interruptibility on a 5-point scale from 1 (absolutely uninterruptible) to 5 (absolutely interruptible). Table 1 shows the interruption rules. The conditions and thresholds were experimentally determined on the basis of previous findings that indicated the interruptibility correlated with head position, movement along the z axis, and pitch angle (Abe et al., 2012; Kimura et al., 2011).

TABLE 1 Interruption Rules of Experimental System

Conditions	Detail
Head-forward posture	Z position stayed forward from reference posture for 10 s
Facing up posture	Pitch angle exceeded the angle while looking at the top of the PC monitor
Forward translation	Forward Z movement in 2 s exceeded 200 mm
Backward translation	Backward Z movement in 2 s exceeded 200 mm
Pitch rotation	Absolute value of pitch rotation in 2 s exceeded 25 degrees
No movement	No interruption for 6 min

3.3. Procedure

The participants were initially introduced to the experiment by their reading a written document and being given an explanation by the experimenter. They installed the experimental system software on their PC and engaged in their usual work. During the experiment, they were forbidden from leaving their desk and chatting with their colleagues for a long time. They were instructed to not change the positions of the PC monitor, camera, or their chair. Furthermore, they were instructed to not hide their faces intentionally from the camera. During the experiments, the participants freely performed PC and non-PC work according to their own demands. In addition, we asked them to report their activity outline for the experimental period at work breakpoints during the experiment. We recorded each participant for 5 hr a day. The experiments were conducted in the early afternoon, after which the participants had nothing urgent on their schedules.

To reduce the influence of individual variations, we defined the reference posture as sitting straight in the chair, facing the desk, with both hands on the keyboard. Moreover, we calibrated the zeroes of the Z position and the pitch angle using the Z position of the reference posture and pitch angle when the user looked at the top of the monitor.

We instructed the participants to imagine that each interruption was a request for a 5-min conversation and intuitively evaluate their interruptibility at the time. Furthermore, we set the minimal interruption interval to 2 min and instructed the participants to disregard the frequency of interruptions in their evaluation.

TABLE 2 Summary of Results for Analysis Data

Head-Related Conditions	Interruptibility Score	Frequency	Total Frequency	Average Interruptibility
Head-forward	1	50	169	2.4
	2	50
	3	32
	4	21
	5	16
Facing up	1	8	105	3.8
	2	15
	3	8
	4	31
	5	43
Forward	1	3	102	3.7
transition	2	15
	3	18
	4	35
	5	31
Backward	1	6	74	3.3
transition	2	18
	3	14
	4	20
	5	16
Pitch rotation	1	11	62	2.9
	2	14
	3	16
	4	15
	5	6
No movement	1	67	273	2.8
	2	66
	3	37
	4	55
	5	48
Total		785	785	3.0

3.4. Overview of Recorded Data

Seventy-five hr worth of recorded data were obtained from the 15 participants, and these were divided into three kinds of situation on the basis of the video records and the self-reports: that is, PC work, non-PC work including some PC work, and other activity including leaving the desk. The durations for the three situations were 54, 15, and 6 hr, respectively. The observed PC activity included entering data, programming, and writing reports or papers. The non-PC work included reading and writing paper documents and constructing devices. Most of the non-PC work consisted of reading and writing paper documents. Such non-PC tasks are frequently associated with occasional PC activity and 10 participants performed them. The other non-PC tasks included leaving the desk, eating lunch, chatting with others, and nondesk work, such as playing games on their mobile phones. We targeted 69 hr worth of data items including PC and non-PC tasks.

In the postexperiment interview, all participants indicated that their interruptibility varied depending on the interruption timing. Most of the participants were aware that they are more interruptible when their head moves away from the working target. Such head movements tend to occur at task breakpoints. Whether the frequency of interruptions would affect the interruptibility score was a concern. However, no participants indicated that the effect of the interruption frequency was problematic. All the participants indicated that they could perform tasks as usual.

TABLE 3 Summary of Correlative Tendencies

	Correlation With Interruptibility
Participant No.	Z-Position	Pitch Angle	Z-Movement
1	Negative	Positive	Negative
2	Negative	Positive	Negative
3	Positive	Positive	Negative
4	Negative	Positive	Positive
5	Negative	Positive	Negative
6	Negative	Positive	Negative
7	Negative	Positive	No correlation
8	Negative	Positive	Negative
9	Negative	Positive	Negative
10	Negative	Positive	Negative
11	Negative	Positive	Positive
12	Negative	Positive	Negative
13	Negative	Positive	Negative
14	Negative	Positive	Negative
15	Negative	Positive	Negative

Table 2 summarizes the frequency and average interruptibility of each interruption condition and interruptibility score. Compared with the “No movement” condition, interruptibility for the “Head-forward” condition appears to be lower, whereas those of the “Facing up,” “Forward transition,” and “Backward transition” conditions appear higher. However, a repeated measures two-way analysis of variance showed a significant interaction between the participant and interruption condition, F(70, 699) = 1.596, p < .01. After that, we examined the simple main effect of each factor. The simple main effect of the participant was significant in the “Head forward,” F(14, 699) = 3.076, p < .01, and “No movement,” F(14, 699) = 4.753, p < .01, interruption conditions . This suggests that the effect of head posture and movement on interruptibility varies with individuals. Moreover, the simple main effect of the interruption condition was significant in six participants, F(5, 699) = 3.201 to 8.781, p < .01, and marginal in three participants, F(5, 699) = 1.919 to 2.926, p < .1. Although individual variation in effect on interruptibility is suggested, the head postures and movements appear to be feasible indices for making interruptibility estimations. Furthermore, because two participants obviously changed their sitting posture during the experiment, we tried to exclude their data and applied repeated measures two-way analysis of variance again. As a result, the interaction between participant and the interruption condition became not significant, and the main effect of the interruption condition became significant, F(5, 628) = 12.692, p < .01.

To discuss the generality of the tendencies of the participants’ head motions, we examined the effects of the back and forth position, pitch angle, and back and forth movement on the interruptibility of each participant. Table 3 summarizes the correlation tendencies in terms of the sign of the regression coefficient. The head forward position was more uninterruptible in 14 out of 15 participants. All participants tended to evaluate interruptions when at an upward posture as more interruptible. The backward movement was more interruptible for 12 participants. The two or three tendencies were observed in 14 out of 15 participants. This implies the existence of a common relationship between head motion and interruptibility. Therefore, in the following sections, the data set is analyzed without dividing it up according to the individual participants. Section 4.1 presents a detailed analysis in relation to the selection of estimation indices.

4. REFLECTION OF HEAD MOTION ON INTERRUPTIBILITY ESTIMATION

This section discusses the selection of head-related indices and describes an improved interruptibility estimation algorithm that reflects head motion.

4.1. Selection of Head-Related Indices

In the previous section, it was shown that the several Z and pitch parameters are significantly related with interruptibility. In general, a static head position and posture reflect the spatial relationship between the head and work target. Therefore, it is hypothesized that they reflect the degree of user engagement in the task at hand. The occurrence of translational and rotational movements such as moving away from the work target may represent a temporal reduction in concentration. Furthermore, the continuity of a specific posture or movement may represent continuity of concentration. Therefore, we examined the feasibility of three groups of parameters as indices for the interruptibility estimation: head position and posture, translational and rotational movement, and continuity of posture and motion.

Head position and posture and interruptibility

We analyzed the relationship between interruptibility and the Z head position, as well as the pitch angle. The Z head position and pitch angle were individually calibrated for each user using the reference posture described in the previous section. To discuss the feasibility of the estimation indices, we attempted to classify the calculated values into two groups using nonhierarchical cluster analysis.

Figure 4a shows the results of the classification based on the Z position. The Z position data were divided into two clusters. The average interruptibility of one cluster, which had an average Z position of 74 mm ahead of the reference posture, was 2.6. The average interruptibility of the other cluster, which has an average Z position of 148 mm behind the reference posture, was 3.5. The difference between the interruptibilities of the two clusters was significant, F(1, 616) = 58.8, p < .01. This suggests that interruptibility is low while a worker assumes a head-forward posture and that the back and forth position of the head can be an interruptibility estimation index.

FIG. 4. Cluster analysis results.

Figure 4b shows the results of a similar classification based on upward and downward postures. In one cluster with an average pitch angle of 24.0° below the top of the monitor, the average interruptibility was 2.9. In another cluster, with an average pitch angle of 0.2° above the top of the monitor, the average interruptibility was 3.4. The difference between the interruptibility of the clusters was significant, F(1, 438) = 11.1, p < .01. The results suggest that interruptibility increases while a worker assumes a head-upward posture. Therefore, we selected this parameter as an estimation index.

Transitional and rotational movement

The relationship between interruptibility and the back-and-forth moving distance, as well as upward and downward rotation, were analyzed. The distance was calculated as the difference between the current Z position and the one 1.5 s before. The calculation time was determined on the basis of the results of the correlation analysis between the interruptibility and distance by changing the time difference from 0.5 s to 3.0 s. Similarly, the pitch rotation angle was calculated by subtracting the current pitch angle from the one 1.5 s before. The nonhierarchical cluster analysis was applied to both data sets.

Figure 4c shows the results of the classification based on the back-and-forth movement. The average interruptibility of one cluster, the average distance of which was 201 mm, was 3.6. The average interruptibility of the other cluster, which had an average distance of 27.6 mm, was 2.9. A significant difference in interruptibility between the clusters was observed, F(1, 447) = 25.1, p < .01. The cluster analysis was applied on the basis of the pitch rotation. The average pitch rotation angles were 35.5° downward and 29.0° upward, and the average interruptibilities were 2.8 and 2.9. However, no significant difference was observed in this case.

The results demonstrate that interruptibility increases temporarily when the head moves backward. Therefore, we selected the 1.5-s Z moving distance as the estimation index. In contrast, the pitch rotation angle did not significantly reflect interruptibility. A potential cause is the variation of the situation when the pitch rotation occurs. Pitch rotation occurs when a worker looks not only above the monitor for a short break but also back to the monitor after looking at the keyboard. Thus, the pitch rotation was not applied in the estimation.

Continuity

To discuss the effect of the continuity of posture or movement, we analyzed the relationship between interruptibility and the continuity of head position, posture, translation, and rotation for several durations. To analyze the relationship between interruptibility and the continuity of a head-forward posture, we calculated the rate of head-forward posture duration in the previous 1 min. The duration, which provides the highest correlation, was selected from ones ranging from 30 s to 5 min. Figure 4d shows the results of the nonhierarchical cluster analysis. The average rates of head-forward posture durations of the two clusters were 0.7 and 0.8, and the average interruptibilities were 4.0 and 1.5. There was a significant difference between the two clusters, F(1, 614) = 2065.1, p < .01. Similarly, we analyzed the relationship between interruptibility and other parameters for several durations, such as the integral of pitch rotation and the frequencies of head upward rotation, backward translation, and switching between forward and backward translation. However, these results did not suggest any significant correlations. These results are discussed in section 6.2.

The results suggest that a higher rate of the head-forward posture duration significantly reflects a less interruptible state. In contrast to the results for positions and movements, it appears that the classification affects the interruptibility score more than the rate of head-forward posture duration. We conducted Bonferroni multiple comparisons of the rate of the head-forward posture duration for each interruptibility score. The results revealed that the average rate for the first interruptibility state was significantly higher than the other interruptibility states. In other words, longer head-forward postures were primarily observed while the worker is not interruptible. The difference in interruptibility scores between the two clusters divided at a rate of 0.8 was significant, F(1, 474) = 13.68, p < .01. Therefore, the rate of the head-forward posture duration in the previous 1 min was selected as an estimation index. Table 4 summarizes the results.

TABLE 4 Summary of Head Motion Analysis

	Position/	Translation/
Head Motion	Posture	Rotation	Continuity
Back and forth	p < .01	p < .01	p < .01
Pitch	p < .01	×	×

4.2. Interruptibility Estimation Algorithm Reflecting Head-Related Indices

As described in section 2.1, there are a number of interruptibility estimation methods based on PC operations. One approach is based on PC operation activity, whereas another approach is based on AS as a task breakpoint. While a worker engages in non-PC work, AS does not occur. Therefore, we attempted to extend the interruptibility estimation algorithm so that it would work when AS is not detected (NAS; Tanaka et al., 2010).

Overview of base interruptibility estimation algorithm

The base interruptibility estimation algorithm uses the four PC-operation-related indices shown in Table 5. Because the indices might not have a linear relationship to interruptibility, each estimation index is binarized. Index 1 represents the detection of the feature. The difficulty in the base algorithm is that it strongly depends on the amount of PC operation activity because three of the four indices are activity-related features. If the worker engages in non-PC work, the algorithm falsely estimates interruptibility as high because of the decrease of the operation activity. Therefore, the four head-related indices are expected to reduce the estimation error.

TABLE 5 PC Operation-Related Indices

Feature
ID	Definition	Effect on Interruptibility
A	Keystroke detection in past 20 s	High
B	More than 30% time rate of PC operation for past 2 min	High
C	Both keystroke and mouse operation detection in past 2 min	High
D	Detection of transition from Explorer in past 5 min	High

Improved estimation algorithm

As described in section 2.2, the previous studies suggested the feasibility that head motion is closely associated with the worker’s internal state, which is different from physical operations. Therefore, we examined the effect of the head-related indices on accuracy by incorporating them in the PC operation-based estimation algorithm. The selected four head-related indices are shown in Table 6. Because the indices might not have a linear relationship with interruptibility, and the statistical significance was accounted for in the nonhierarchical cluster analysis in section 4.1, all head-related indices were binarized, similar to the PC-operation-related indices. The threshold for each index was defined on the basis of the averages of the divided clusters.

TABLE 6 Head-Related Indices

Feature
ID	Definition	Effect on Interruptibility
E	Forward position relative to reference posture	Low
F	More than 80% time rate of head-forward posture for past 1 min	Low
G	Backward motion larger than 150 mm within past 10 s	High
H	Facing up over the PC monitor within past 10 s	High

Even if all the indices are statistically significant, they might have different influences on interruptibility. Therefore, we applied multiple linear regression analysis using forward selection with interruptibility as a dependent variable. Equation 1 represents the improved estimation algorithm using the four head-related indices. Variables A to H represent the indices shown in Tables 5 and 6. The coefficients for the indices were determined according to the results of the multiple linear regression analysis. The estimated value is normalized between 0 and 1.

Similar to the previous studies, the improved algorithm using Equation 2 estimates interruptibility in three levels, that is, high, neither, and low (see Iqbal & Bailey, 2006; Minakuchi et al., 2004; Tanaka et al., 2012). The thresholds in Equation 2 were experimentally determined. In the following sections, the interruptibilities (high, neither, and low) correspond to the experimentally obtained five-level subjective scores (5 and 4, 3, and 2 and 1).

(1)

(2)

4.3. Trial Estimation With Analyze Data Set

Before the evaluation experiment, we applied the proposed algorithm to the analyze data set as a trial. We examined 779 interruptibility scores. The scores were obtained for NAS states after more than 5 min had elapsed in the experiment. The works during the experiment were manually identified on the basis of the recorded data and reports by the participants. The work categorized as PC work included cases in which the participant used the keyboard or mouse or gazed at the PC’s monitor continuously for more than 5 min. The other cases were identified as non-PC work, and they possibly included a mixture of PC and non-PC work. Figure 5 shows the estimation results of the previous and proposed methods. Separate results are shown for PC work (Figure 5a) and non-PC work (Figure 5b). Each graph represents the rate of the frequency at which the actual interruptibility score was within the estimated interruptibility-level range. Thus, this rate of the frequency for the high interruptibility represents the precision of the high-interruptibility estimation. In contrast, the rate of the frequency at which the actual low interruptibility score was within the estimated high interruptibility level represents the error rate with high distraction risk in which an uninterruptible state is mistaken as interruptible.

FIG. 5. Estimation results for analysis data set: (a) PC work; (b) non-PC work.

The precisions of the previous method for low and high interruptibility levels during PC work were 60% and 68%, respectively. The error rate of high distraction risk was 20%. On the other hand, the precisions for low and high interruptibility levels during non-PC work were, respectively, 43% and 54%, more than 15% lower than for PC work. Furthermore, the error rate of high distraction risk increased to 35%. We concluded that the non-PC activity significantly impaired the accuracy of the PC-operations-only method, which is similar to the conclusion of the previous study. In contrast, the proposed method improved the precision for low and high interruptibility levels during PC and non-PC work. In particular, the precision for high interruptibility during non-PC work improved to 79%, and the error rate of high distraction risk fell to 10%. In addition, the F-measures improved for both PC and non-PC work. As the recall for the low interruptibility estimation greatly increased in both work states, the results suggest that the head-related indices contributed to the detection of the uninterruptible state.

5. EVALUATION OF PROPOSED METHOD

Another set of experiments was conducted to examine the improvement of the estimation accuracy of the proposed method.

5.1. Method and Data

Using the experimental system described in section 3, we performed a set of interruption experiments under the same conditions as those in the analysis, including the limitation on the conversation. Four university students and two faculty members participated in the experiment for 5 hr per day. Thirty hr of records were collected. Table 7 shows a summary of the experimental results. The estimation targets were 332 interruptibility scores obtained in NAS states after more than 5 min had elapsed in the experiment. The tendency of the obtained data was approximately the same as that of the analyze data set, except for a lower interruptibility in relation to forward movement of the head. Forward head movement was detected when a worker assumed not only a forward-leaning posture but also a natural posture after leaning on the backrest. In contrast, backward movements of the head had a robust tendency. These results encouraged us to incorporate backward movements in the estimation.

TABLE 7 Summary of Results for Evaluation Data

Head-Related Conditions	Interruptibility Score	Frequency	Total Frequency	Average Interruptibility
Head-forward	1	32	102	2.4
	2	29
	3	16
	4	14
	5	11
Facing up	1	2	31	3.5
	2	4
	3	7
	4	13
	5	5
Forward	1	0	16	2.7
transition	2	9
	3	3
	4	4
	5	0
Backward	1	1	66	3.8
transition	2	11
	3	8
	4	27
	5	19
Pitch rotation	1	0	9	3.8
	2	2
	3	1
	4	3
	5	3
No movement	1	33	110	2.6
	2	30
	3	12
	4	20
	5	15
Total		334	334	2.9

5.2. Estimation Results

Figure 6 shows the estimation results for the evaluation data set of the previous and proposed methods. Separate results are shown for PC (Figure 6a) and non-PC (Figure 6b) work. The previous method’s precision for high interruptibility was 45% during non-PC work, and this value was similar degree to the case of the trial estimation. However, there was no reduction in precision of low interruptibility during non-PC work. In comparison with the analyze data set, the evaluation data set may have contained more low interruptibility activity with a significant amount of PC operations.

FIG. 6. Estimation results for evaluation data set: (a) PC work; (b) non-PC work.

Compared with the previous method, the proposed method improved the precision of high interruptibility during PC work by 5% and during non-PC work by 16%. The reductions of high-distraction-risk error rates were 8% for PC work and 19% for non-PC work. These results suggest that the head-related indices improve the accuracy of interruptibility estimation. In particular, the fewer high-distraction-risk errors will allow information systems to inform workers with less risk of distraction. Furthermore, the estimation algorithm improved the F-measures, precisions, and recalls on two data sets. The results indicate that reflecting the attitudes of workers using head motion indices in addition to ones for PC activity improves the accuracy of the interruptibility estimation.

6. DISCUSSION

6.1. Interruptibility Measure Incorporating Head Motion

The experimental results suggest that incorporating head motion indices reduces the estimation error caused by occasional non-PC work, especially in an office environment (Tanaka et al., 2012). However, head motion essentially reflects changes in the target of one’s gaze, not the task activity itself. Furthermore, head motion indices improve the estimation accuracy of someone working on a PC without engaging in any other sort of activity. However, activities such as using the keyboard and mouse are already reflected in the estimation. Therefore, it is suggested that head-related indices reflect factors other than task-related activities.

Cognitive science studies have suggested that interruptions incur a cognitive cost. This cost is associated with cognitive memory usage required to suspend and resume the original task (Altman & Trafton, 2002). Therefore, the subjective interruptibility of a worker is affected by the cognitive cost of the interruption. For instance, the cognitive cost of an interruption is lower and subjective interruptibility is higher at task breakpoints (Iqbal & Bailey, 2005, 2006).

On the other hand, Mark et al. (2008) reported that participants who dealt with a task in earnest made an effort to compensate for losses in time caused by interruptions. Their report suggests that interruptibility is affected by the cognitive cost and the attitude of the worker. Head motion represents a physical relationship between the worker and the target task. Our indices reflect continuous gazing on the target and temporary suspension of work on the target. Therefore, it is speculated that head motion reflected attitude factors, such as concentration and task engagement.

In general, a worker’s attitude in an office environment is affected by diverse factors, such as the urgency and importance of their work and physical conditions (Matsuda et al., 2005; Tanaka et al., 2014). If head motion reflects the worker’s attitude, which is a consequence of those factors, it would be useful for improving the estimation accuracy in a wide variety of tasks and situations.

6.2. Feasibility of Additional Head-Related Indices

The analysis to select the head-related indices examined but did not incorporate many candidates in the index for basically the same reason. For example, we analyzed the relationship between interruptibility and pitch rotation angle for several durations, as described in section 4.2. Figure 7 shows the relationship between interruptibility and the 1-min integral of the absolute value of pitch rotation. As shown, pitch rotation integrals during lowest interruptibility (1) and highest interruptibility (5) periods have approximately equal values. This coincidence can be interpreted as follows. Concentrated gazing at a task target decreased head pitch motion; however, the relaxing state, which is opposite in terms of interruptibility, also decreased head motion. Thus, the pitch rotation integral was not used because a simple threshold has trouble distinguishing low and high interruptibilities. Similar undesirable coincidences of values for different interruptibility levels occurred for other head motion features, such as yaw rotation and x-axis translational movement. To leverage these nonmonotonic features, we will have to investigate methods other than a simple threshold.

FIG. 7. Relationship between interruptibility and 1-min integral of absolute value of pitch rotation.

Long-term postural changes caused by fatigue and other factors should be considered when applying a simple threshold. In the experiment, one participant—Participant 4 in Table 3—changed his sitting position significantly in the middle of the experiment. It appears that the tendency of the back-and-forth position and movement thereof were affected by the change in position. Here, dynamic recalibration of the threshold might reduce the effect of changes in the sitting position.

In this study, we attempted to improve the interruptibility estimation method based on PC-operation indices by incorporating additional head-related indices. The results of a multiple regression analysis led us to introduce four head-related indices in an extended estimation algorithm. However, the head-related and PC-operation indices have different origins. PC activity more strongly reflects the intensity of a task, whereas head motions are expected to reflect the worker’s attitude toward the task. Therefore, as a primitive trial, we introduced several logical multiples of the indices to the multiregression analysis for the design of the estimation algorithm. The results suggest that index combinations such as typing with a head-forward posture have stronger correlations than any single index. To make further improvements, we need to investigate the semantic relationships among head-related, PC operation, and other indices.

Our previous experiments in a real office work environment revealed several causes of estimation error (Tanaka et al., 2012). In particular, occasional non-PC work was a major cause of estimation error. However, half of the total errors were caused by conversations. Telephone conversations and discussions with other workers occasionally occur in office environments. Workers usually do not use their PCs when they are in conversation. Thus, the PC operation-based method falsely estimates interruptibility during a conversation as being high. However, interruptions are not acceptable, because conversation is a task with high cognitive loads. The period between the end of a conversation and resumption of a task is also considered as a breakpoint between different types of work. Therefore, interruptibility is expected to be high shortly after a conversation ends, which is similar to the case of AS. To broaden the range of applicable scenarios, social factors must be reflected by detecting conversations and other interactions.

6.3. Limitations

Research on interruptibility estimations is aimed at real office environments and improving intellectual productivity. Our study targeted office work requiring intellectual activity. Furthermore, we assumed that the main tasks would be on PCs connected to the Internet, which is the main source of interruptions. Therefore, simple tasks that do not require intellectual activity or PC operations are beyond the scope of this study. Moreover, at present, the proposed method is for an environment in which a worker sits in front of a single working target. The estimation method needs to be extended so that it can deal with multiple-PC or multiple-desk environments. Furthermore, this study had only 20 participants. Empirical studies with a larger number of participants and in wider variety of environments and job types should be conducted.

Previous studies have suggested that multitasking can be understood at various levels, such as activity, action, and operation. PC operation-related indices, which correspond to “operation” levels, were effective for making automatic estimations because of their objectivity and ease of detection. However, they do not directly relate to the objective or motivations of the worker. Head motion implied the feasibility to reflect worker attitude. This remains a speculation, and further investigations will have to be conducted to confirm it.

7. CONCLUSION

We studied four head-related indices significantly related to interruptibility and incorporated them in an interruptibility estimation algorithm. Experiments demonstrated the effectiveness of using head motion, especially in cases that involved occasional work that wasn’t on a PC. In the future, we intend to incorporate indices of the conversational state into the estimation method and examine the semantic relationships among head-related, PC operation, and other indices.

FUNDING

This work was partly supported by the Japan Society for the Promotion of Science (KAKENHI); the Ministry of Education, Culture, Sports, Science and Technology; Japan Fund for Smart Space Technology toward Sustainable Society; and the National Institute of Information and Communications Technology, commissioned research.

Additional information

Funding

This work was partly supported by the Japan Society for the Promotion of Science (KAKENHI); the Ministry of Education, Culture, Sports, Science and Technology; Japan Fund for Smart Space Technology toward Sustainable Society; and the National Institute of Information and Communications Technology, commissioned research.

Notes on contributors

Takahiro Tanaka

Takahiro Tanaka, Ph.D., is an Assistant Professor at the Department of Computer and Information Sciences, Tokyo University of Agriculture and Technology. His research interests include intelligent agents, user state estimation, human-agent interaction, and human-computer interaction.

Ryosuke Abe

Ryosuke Abe graduated from the Department of Computer and Information Sciences at Tokyo University of Agriculture and Technology in 2012 and received a Master of Engineering degree. His research interest is user state estimation from head motion during work.

Kazuaki Aoki

Kazuaki Aoki, Ph.D., is an Assistant Professor at the Department of Computer and Information Sciences, Tokyo University of Agriculture and Technology. His research interests include pattern recognition, machine learning, user state estimation, and human interfaces.

Kinya Fujita

Kinya Fujita, Ph.D., is a Professor at the Department of Computer and Information Sciences, Tokyo University of Agriculture and Technology. His research interests are smart human–computer interfaces, communication in virtual space, and haptic virtual reality.