2,181
Views
7
CrossRef citations to date
0
Altmetric
Research Articles

Straying Off Course: The Negative Impact of Mind Wandering on Fine Motor Movements

ORCID Icon & ORCID Icon
Pages 186-202 | Received 14 Jan 2021, Accepted 27 May 2021, Published online: 04 Aug 2021

Abstract

The goal of this study was to examine how various degrees of perceptual decoupling during mind wandering affect fine motor control. We hypothesized that while under normal circumstances attention ensures an optimal control strategy that leads to accurate motor performance, during mind wandering the process becomes disrupted. In this study, we conducted a computer-based experiment with a tracking task. During mind wandering, motor movements were more erratic and less variable, indicative of reduced attentiveness to the continuous demands of the external task. Importantly, the deeper the reported mind wandering, the less accurate and less variable were the mouse movements, suggesting that perceptual decoupling may take place in a graded rather than in an all or nothing manner. Greater movement intermittency was associated with higher tracking accuracy, suggesting that more corrective movements toward a moving target were functional to task performance. Moreover, greater variance in velocity was negatively correlated with tracking accuracy. These findings suggest that periods of inattention to the task have a negative impact on fine motor movement control by making behavior unpredictable, providing support for the idea that there is a decoupling of sensory-motor processes during mind wandering.

Introduction

Both gross and more fine-grained movements are necessary for most activities in daily life. Gross motor skills concern the coordination and movement of larger body parts, such as arms and legs (Khalaj & Amri, Citation2014), contributing to static and dynamic balance (Pitcher et al., 2003; Valentini et al., Citation2014), while fine motor skills refer to smaller movements of hands, wrists, and fingers, requiring close eye-hand coordination and contributing to manual dexterity (Pitcher et al., 2003; Suggate et al., Citation2018; Valentini et al., Citation2014). Fine motor skills are arguably most used nowadays in online interfaces, from school age children who learn and play with their tablets to working age adults who spend the majority of their days in front of a computer. This ever-growing interaction between humans and computer interfaces across all ages highlights the increasing importance of investigating the dynamics of fine motor movements in online tasks (e.g., Hocherman et al., Citation2004; Mathew et al., Citation2020).

Various factors influence – and often impair – motor control. These may be physical (e.g., breaking a limb), neurological (e.g., Parkinson’s disease), or age related. During childhood, motor abilities evolve from being purely involuntary reflexes to being intentional actions upon the environment. As people age, they also experience a deterioration in their motor control abilities (Lee et al., Citation2013; Michimata et al., Citation2008; Stirling et al., Citation2013), which is exacerbated by cognitive decline (Kluger et al., Citation1997). However, the most important factor which influences motor control is attention, as it regulates movements in individuals across all ages.

Past research demonstrates that individuals with attention disorders often have impairments in both fine and gross motor abilities (Kaiser et al., Citation2015; Pitcher et al., 2003). Particularly, with regards to fine motor skills, various studies have reported that children with ADHD have less precise, less stable, more corrective and jerkier movements during tracking tasks (for a review, see Kaiser et al., Citation2015). Similarly, Tucha and Lange (Citation2001) found the handwriting of boys with ADHD to be less legible. In addition to the motor control impairments associated with attention deficit disorders, it has been found that focusing attention on another cognitive task while performing a movement leads to impairments in motor learning (Song, Citation2019; Taylor & Thoroughman, Citation2008). Moreover, there appears to be an advantage in motor performance when focusing externally on the task rather than focusing internally on body motions and mechanics (Lohse et al., Citation2014; Wulf, Citation2013; Wulf et al., Citation1998).

In the current study, we examine fluctuations in attention caused by mind wandering, a state in which attention decouples from external perceptual input. Mind wandering has been found to occur between 30 to 50% of the time across a wide variety of tasks, both in and outside of laboratory settings (Smallwood & Schooler, Citation2015). A commonly used method to investigate mind wandering involves directly asking individuals about the content of their thoughts (whether they are focused on the external task at hand or mind wandering) in real time. For the most part, individuals tend to be able to reliably identify and report the contents of their thoughts, as indicated by high correlations between subjective self reports of mind wandering and objective behavioral and neurophysiological measures (Smallwood & Schooler, Citation2015). As attention decouples from perception, there is often a decrease in alertness and sensory-motor processing of the external environment (Kam & Handy, Citation2013; Smallwood & Andrews-Hanna, Citation2013). This is demonstrated, for example, by systematic differences in oculomotor movements during reading prior to reports of mind wandering. For example, fixations during mindless reading have been found to be longer and less sensitive to lexical and linguistic features than during normal reading (e.g., Foulsham et al., Citation2013; Reichle et al., Citation2010). This observation is in line with studies reporting changes in both accuracy and reaction times during button presses or mouse clicks in vigilance tasks. For these tasks, reduced accuracy either because of failing to respond or failing to inhibit an incorrect response (e.g., Smallwood et al., Citation2004), as well as generally more variable reaction times (e.g., McVay & Kane, Citation2009; van Vugt & Broers, Citation2016), have been reported during mind wandering states.

According to Baddeley’s model of working memory (Baddeley, Citation2003; Baddeley & Hitch, Citation1974), the visuo-spatial sketchpad is responsible for storing information of a visual or spatial nature for a short period of time. Meanwhile, visuo-spatial working memory resources would be associated with visuomotor control processes (Repov & Baddeley, Citation2006). In support of this claim, Spiegel et al. (Citation2013) demonstrate that motor movement planning and execution impacted performance in working memory tasks. Conversely, Bathurst and Kee (Citation1994) found that having participants perform a verbal task while finger tapping significantly changed tapping rates. According to the attentional-resource hypothesis of mind wandering (Smallwood & Schooler, Citation2006), both task-related attention and mind wandering compete for limited executive resources. As attentional resources are decoupled from perception and redirected to internal trains of thought during mind wandering (Kam & Handy, Citation2013), it is possible that motor control of movements also becomes impaired.

An exploration of changes in motor movements allows for a more fine grained measurement of the temporally dynamic processes which take place during mind wandering as we monitor and adjust of our movements in response to external demands. For example, during a visuomotor tracking task, Kam et al. (Citation2012) found increased tracking errors during periods of mind wandering relative to periods of focused attention. More recently, we found that movements during a reaching task toward a target became initially slower and overall more complex during periods of mind wandering (Dias da Silva & Postma, Citation2020). These types of tasks generate a rich set of continuous measures collected across time as individuals follow the path of a target on a screen with either a mouse, joystick, or similar tracking device. There is some evidence indicating that as attention decouples from perception during mind wandering in a graded fashion, sensory and motor processes may become attenuated to different degrees, ranging from becoming partially attenuated, only reducing stimulus processing to an extent, to becoming completely attenuated (Kam & Handy, Citation2013; Schad et al., Citation2012). It is therefore likely that the impact of mind wandering on fine motor measures varies with the degree of perceptual decoupling.

In sum, it seems that as attention directs itself inwards, a decoupling of sensory and motor systems from the environment becomes manifest in wavering behaviors, ranging from oculomotor movements, to finger movements during button presses or clicks, to more dynamic movements during reaching and tracking tasks. In what follows, we first provide an overview of findings regarding mind wandering during different task conditions. We subsequently describe the methods used to measure motor movement execution, in particular the movement of the computer mouse during online tasks.

Mind Wandering under Different Task Conditions

As we adjust our movements in response to demands of the external environment, our expected responses vary according to the type of task being performed. Given that mind wandering can occur during tasks with varying levels of complexity, the consequences of mind wandering vary as attention decouples from the external environment depending on the activity being performed. As very easy tasks demand minimal resources, the mind can wander without hurting performance of the primary task. However, mind wandering tends to be more detrimental to performance on tasks that require more resources (Randall et al., Citation2014, Citation2019; Smallwood, Citation2013). For example, during a relatively predictable task such as driving on an empty highway, the driver can both drive a car and think about the next vacation destination. Driving with no GPS on a busy road, on the other hand, may require the involvement of working memory to retrieve the directions to one’s destination and an episode of mind wandering would likely be highly disruptive to the driver’s performance (Lin et al., Citation2016). However, a variety of factors, including task difficulty (Seli et al., Citation2018), previous knowledge, level of expertise (Simonsohn, Citation2015; Smallwood & Andrews-Hanna, Citation2013), time on task (Thomson et al., Citation2014), working memory capacity (Randall et al., Citation2019; Smallwood, Citation2013), drowsiness (Stawarczyk et al., Citation2020), and fatigue (Walker & Trick, Citation2018) – to name a few – can affect this relationship.

Mind wandering has been investigated in tasks with varying levels of complexity. Vigilance tasks (e.g., go/no-go tasks, SART, or oddball), choice reaction time tasks, and low-load visual tasks, are characterized as having low complexity (Randall et al., Citation2014) and typically require relatively few attentional resources. On the other hand, tasks involving reading comprehension, assessments of complex cognitive abilities (e.g., working memory) and visual search tasks with a high-load or with competing stimuli are characterized as having high complexity (Randall et al., Citation2014) and typically require more attentional resources. In general, mind wandering is more likely to take place under lower tasks demands, as they require fewer attentional resources in order to be completed, and as task demands increase, mind wandering decreases (McVay & Kane, Citation2010; Smallwood & Schooler, Citation2006). However, under too high task demands which require resources beyond an individuals’ capacity, mind wandering rates increase again (Randall et al., Citation2014).

Next to complexity, another factor that has been shown to influence mind wandering is time on task. Various studies have found that mind wandering increases with time on task (McVay & Kane, Citation2009, 2012; Smallwood et al., Citation2006) for generally unfamiliar and repetitive assignments. Previous studies indicate that typical sustained-attention tasks are both experienced as boring and monotonous, as indicated by reduced heart and respiration rates (Pattyn et al., Citation2008), and effortful, as indicated by subjective reports of task workload and stress (Grier et al., Citation2003). One possibility is that initially, participants must get used to the unfamiliar, novel task. As time goes by, practice leads to a habituation to the task and potentially, automation of task performance. This allows for executive resources to be freed up for mind wandering (Risko et al., Citation2012). Mind wandering then potentially increases because of understimulation, leading to low levels of arousal as a consequence of the task’s monotony (Brosowsky et al., Citation2021).

Alternatively, as sustaining attention toward a monotonous task is effortful, this leads to a depletion in executive control as the task progresses, which in turn, leads to increases in mind wandering (McVay & Kane, Citation2010). According to the first account, performance decrements over time are a result of executive resources devoted to mind wandering rather than to the task, while according to the second account, performance decrements are a result of cognitive depletion. According to both of these accounts, increasing task demands should lead to greater performance decrements over time (Thomson et al., Citation2015).

In terms of motor movements, some tasks require relatively rudimentary levels of motor control (pressing a button), while others require slightly higher (reaching for a stationary target), or even more fine-grained levels of motor control (tracking a moving target). In visuomotor tracking tasks, movements are guided by sustained visual attention to a moving target in a fine-grained manner. When tracking a target, movements tend to be intermittent (step and hold-like), with discrete positional corrections toward a target (Craik, Citation1947; Russell & Sternad, Citation2001; Weir et al., 1989). However, the amount of intermittency is subject to both individual variability and task variability (Mathew et al., Citation2020; Russell & Sternad, Citation2001). More complex fine motor tasks, for example, are associated with greater intermittency and reduced smoothness of movements (Weir et al., 1989). When tracking nonpredictable pseudo-random targets, movements tend to be more intermittent and less smooth, as “participants must react to unexpected changes with a delay” (Russell & Sternad, Citation2001, p. 330). In contrast, when tracking repetitive targets such as sinusoidal waves, participants’ responses are smoother (Miall, Citation2003; Russell & Sternad, Citation2001). In addition, practice or removal of online visual feedback are associated with a reduction of movement intermittency and increased smoothness of movements (Russell & Sternad, Citation2001). An exploration of the effects of mind wandering during different kinds of motor tasks may provide insight into how dynamic measures associated with task complexity change according to an individual’s locus of attention.

Visuomotor tracking tasks are particularly suitable for investigating the moment-to-moment dynamics of wavering behavior associated with mind wandering during guided movements. The underlying dynamics of motor control present in these tasks are relevant for activities we perform in our daily lives, such as interacting with online interfaces, as well as other activities requiring eye-hand coordination such as driving, writing, or cooking. Considering the ubiquity of mind wandering in our daily lives (Killingsworth & Gilbert, Citation2010; Smallwood & Schooler, Citation2015), and the fact that complexity in terms of motor control has been largely overlooked in the literature, there is a need to investigate further the effect of mind wandering on movements under such tasks in laboratory settings.

During tasks assessing continuous movement performance, it seems that increased behavioral variability is associated with an external focus of attention. According to the constrained action hypothesis (McNevin et al., Citation2003), an external focus of attention on the effects of a movement allows for automatic control associated with improved motor performance. Meanwhile an internal focus of attention on movement patterns themselves engages executive resources and disrupts these automatic control processes, resulting in poorer motor performance. An abundance of research suggests that more variable movements are associated with an external focus of attention irrespective of how well-learned a task is (Lohse et al., Citation2014; Wulf, Citation2013). Although mind wandering and an internal focus of attention toward movement dynamics cannot be equated, both require executive resources and seemingly disrupt automatic control processes engaged during motor performance. Variability in movement, therefore, should also decrease during episodes of mind wandering as opposed to maintaining an external focus of attention.

In addition to reduced movement variability, previous research has found that smoother movements – reflected as less intermittent or jerky movements) are also associated with an external focus of attention (Kal et al., 2013). As such, internally oriented attention during periods of mind wandering should conversely be associated with less smooth, more intermittent movements.

In sum, mind wandering is generally more likely to take place under low demand tasks and less likely as task demands increase (Smallwood & Schooler, Citation2006). Moreover, mind wandering is also likely to increase during monotonous activities, either because of low attentional demands required or because of cognitive depletion resulting from effortful processing required for continuous performance. With time on task increasing, participants become habituated to these tasks, and mind wandering becomes more likely as a result of low levels of arousal. In tasks assessing continuous movements, mind wandering seems to be related to poorer tracking accuracy and more complex reaching, indicative of changes in arousal. Greater intermittent and less variable motor movements, previously found to be related to an internal focus of attention, could also potentially be indicative of mind wandering.

Predicting Mind Wandering from Movements

In recent years, a considerable amount of attention has been paid to automatic detection of mind wandering in educational and other contexts (Bosch, Citation2016; Hutt et al., Citation2016; Jin et al., 2019; Zhang & Kumada, Citation2017). In particular, a variety of bodily behaviors ranging from eye movements to head movements and posture have been used as features for predictive modeling of off-task states. Indeed, process tracing methods such as mouse tracking have demonstrated to be a promising avenue of research in the tracking of off-task behavior in tasks with varying characteristics, under both laboratory and naturalistic settings. For example, in a visuomotor tracing task, participants were required to track the sinusoidal path of a ball on a screen with a joystick (Kam et al., Citation2012). During episodes of mind wandering, participants made greater tracking errors compared to reports of being focused on the task at hand. In a more perceptually and cognitively demanding task, we found that different mouse movement features are predictive of mind wandering during performance of an Operation Span Task (Dias da Silva & Postma, Citation2020). More complex movements (more x- and y-flips) likely indicated changes in arousal prior to reports of mind wandering. In addition, we found that time to reach maximum deviation was longer, indicating that participants took longer to commit to a response prior to mind wandering probes.

In the study of Cetintas et al. (Citation2010), mouse tracking features have been used to detect students’ off-task behavior during interaction with an intelligent tutoring system. Three features were extracted from students’ mouse movements. The first indicated the largest time interval in which the mouse was not used to solve a problem. The second and the third features included both the average x- and average y- movement every 200 milliseconds, respectively. Both the amount of time the mouse was idle as well as the amount of the total distance traveled by the mouse (computed from the x- and y- movements) were important predictors of off-task behavior during interaction with an intelligent tutoring system. In comparison to models using only time and performance features, models using mouse movement features improved predictive accuracy from approximately 60% to 90%.

In user engagement research, mouse movements have been employed to track user attention to online web pages. Arapakis and Leiva (Citation2016) engineered a wide set of mouse cursor features for data collected from users exploring information in proxy search engine result pages. By feeding these features to a random forest classifier, the authors were able to predict user attention with an accuracy of 76% and an AUC (balanced measure of accuracy taking into account both the true positive and the true negative rate) of 86%. As feature engineering can be extremely time consuming and requires domain expertise, in another study, Arapakis and Leiva (Citation2020) instead used raw mouse-cursor data from user interactions with search engine result pages for both convolutional and recurrent neural networks to predict user attention, achieving up to 70% in F1-scores and in AUC scores.

Based on the findings reported above, it can thus be concluded that measures from continuous hand movements contain valuable information about the locus of attention (external vs. internal). These measures have been used to infer off-task states and mind wandering states, both self-reported or coded by an observer, or inferred from user responses. In general, more tracking errors, more complex movements, initially slower movements, more idle time, and overall mouse cursor distance seem to be important features for predicting inattentive states.

Current Study

The goal of the present research is to examine differences between dynamic measures from tracking fine motor movements prior to states of mind wandering and focused attention. Measuring movements in a task in which participants must track a moving target on a screen with a mouse allows us to explore how movements are adjusted in real time using ongoing sensory information under different different degrees of perceptual decoupling during mind wandering. Such movement patterns provide insight into the underlying dynamics of eye-hand coordination which we use on a daily-basis, for example, when interacting with a computer, writing or driving a car. In the current visuomotor tracking task, attention must be focused both on the semi-predictable movements of the target itself and on the cursor controlled by the subject. Whenever the target becomes red (the so-called ‘oddball’), participants are instructed to click on it as quickly as possible. Performing this task thus requires both continuous monitoring of the visual target, as well as selection of the appropriate motor response in terms of the position and timing of the mouse in response to the stimulus (i.e., keeping the mouse inside the target). As such, attention must be focused both on the target and on the mouse cursor. We expect that whenever participants report to have been mind wandering:

  • H1) The responses to the oddball will be less accurate (McVay & Kane, Citation2012) and slower (Dias da Silva & Postma, Citation2020).

  • H2) Tracking movements will be less accurate in terms of the position of the mouse within the ball (the distance between the mouse and the ball will be smaller, Kam et al., Citation2012).

  • H3a) Given that the task chosen for the study is monotonous and relatively long, we expect that self-reported mind wandering rates will increase over time (Farley et al., Citation2013; Risko et al., Citation2013; Thomson et al., Citation2014) and that H3b) tracking performance will decrease (Randall et al., Citation2014), as a result of reduced vigilance and habituated movements to monotonous, repetitive stimuli (Peiris et al., Citation2006; Thomson et al., Citation2014).

  • H4) We expect that movements will be more intermittent whenever participants report to be mind wandering, in that less smooth, more corrective movements toward the ball are necessary for accurate tracking per trial (Kal et al., 2013; Mathew et al., Citation2020).

  • H5) As movement variance has been found to be indicative of an internal focus of attention (Lohse et al., Citation2014; Wulf, Citation2013), we expect the variability of cursor speed to be lower during self-reported mind wandering states.

  • H6) Greater degree of disengagement from the task during self-reported mind wandering episodes will predict poorer tracking accuracy, more intermittent movements, and less variable velocities.

If mind wandering affects tracking performance, and retrospective self-reports of mind-wandering and focused attention is valid, then it should be possible to identify episodes of mind wandering from signatures in the motor tracking performance. To this end, we applied machine learning methods to model the tracking data.

Methods

Participants and Procedure

This study was approved by the university’s Institutional Review Board (REDC #2019/98). In total, 45 participants between 18 and 33 years of age (M = 22.53, SD = 3.36, N = 43), 28 female, participated in the experiment in return for course credit. Two participants were excluded from the analysis because they fell asleep during the study and did not follow the instructions. In order to participate in the experiment, participants had to be right-handed and have 20/20 vision or corrected vision. Upon arriving at the experimental session, participants signed a consent form and answered questions about their demographics1 and mind wandering experiences in daily life (reported in Dias da Silva et al., Citation2020). Next, after a training session in which participants familiarized themselves with the task and with the mouse cursor, they began the visuomotor tracking oddball task (), during which they were interrupted with thought probes assessing whether they were focused on the task or mind wandering. Participants were instructed to track the path of a black ball that “bounced” along a white computer screen as closely as possible, while also responding to rare targets (1 target to 9 non-targets), also denoted as ‘oddballs’. That is, whenever the ball turned red, participants had to click on it as quickly as possible. The task consisted of 5 blocks, each lasting approximately 10 minutes. There were 60 targets per block (300 targets total). Pseudorandomly throughout the task, participants were asked to answer the following question: “Were you focused on the task just before this question?”. They could have answered either ‘yes’, indicating their were focused on the task, or ‘no,’ indicating they were mind wandering. If they answered ‘no’, they were presented with further questions, assessing the quality of their thoughts. More specifically, we asked participants to indicate on a scale from 1 to 10, to what degree 1) their attention was disengaged from their surroundings [perceptual decoupling], 2) they imagined being somewhere else [mental navigation], and 3) the content of their thoughts varied [content variation]2. During the experiment, EEG data was collected (analysis not included in this report). The experiment lasted about 90 minutes (30 minutes for EEG preparation and filling out questionnaires and 1 hour for performing the visuomotor task).

Figure 1. Illustration of the trial sequence in the visuomotor task. Participants tracked the path of a black ball (with a 50 pixels radius, moving with a speed of 5 pixels per 16 milliseconds) that “bounced” around a 1366 cm cm wide by 768 cm long screen. Whenever the ball reached either the top, bottom, left or right corners of the screen, this was marked as a trial, and it would bounce back toward the opposite direction. Neither the ball nor participants’ movements left a trail. In 1 out of 10 trials, the ball would turn red (e.g., bottom middle panel) and remain still for one second. During this time, participants had to click on the red “oddball” as quickly and accurately as possible.

Figure 1. Illustration of the trial sequence in the visuomotor task. Participants tracked the path of a black ball (with a 50 pixels radius, moving with a speed of 5 pixels per 16 milliseconds) that “bounced” around a 1366 cm cm wide by 768 cm long screen. Whenever the ball reached either the top, bottom, left or right corners of the screen, this was marked as a trial, and it would bounce back toward the opposite direction. Neither the ball nor participants’ movements left a trail. In 1 out of 10 trials, the ball would turn red (e.g., bottom middle panel) and remain still for one second. During this time, participants had to click on the red “oddball” as quickly and accurately as possible.

Instrumentation

The experiment ran on a 22-inch Dell desktop, with a 1366 by 768 resolution. Participants sat approximately 70 centimeters in front of the screen. Stimulus material was presented with a display refresh rate of 60 Hz on a white background. Reaction time and correct target detection for each participant were recorded. A Dell USB 3 Button Scrollwheel wired Optical Mouse was used to record cursor coordinates during the task. Mouse coordinates were sampled at 60 Hz.

Labeling Data: Mind Wandering Responses

Participants reported to be focused on the task 60% of the time (SD=19%) and mind wandering 40% of the time (SD=19%). The interval between probes was on average 39.58 trials (SD = 22.81), i.e., from the moment when the ball left one side of the screen and reached the other side, see ), lasting in total M=33.18 seconds (SD=18.07). Based on the participants’ answers to each probe question, we labeled trials preceding the probe as either ‘Mind Wandering’ or ‘Focused’.

Mouse Tracking Measures

Trajectories were measured per trial from the moment the ball left one side of the screen (whether it be left, right, top, or bottom) to the moment it reached another side of the screen. We determined the average tracking accuracy, trajectory intermittency, and variance in velocity per trial. In addition, we extracted 17 mouse features3 with the R package Mousetrap (Kieslich & Henninger, Citation2017) from the x- and y-coordinates recorded for each participant per trial. All trajectories were aligned to a common starting position and were remapped onto one side of the screen, before computing these additional measures (see ). Further descriptions of features can be found under Results.

TABLE 2. Unstandardized means and standard deviations of mouse tracking features for each class – Focused attention and Mind wandering (N = 43).

Machine Learning

We trained 5 different algorithms – Naive Bayes (NB), Linear Discriminant Analysis (LDA), K-Nearest Neighbors (KNN), Treebag (TB), and Random Forest (RF) to predict mind wandering from 17 mouse tracking features (features from ) along with tracking accuracy per trial. To minimize the chance of overfitting, eighty percent of the participant data were in the training/validation set, and 20% were held out for testing. We used a leave-one-participant-out cross-validation procedure to train our classifiers in a participant-independent manner (Dias da Silva & Postma, Citation2020; Pham & Wang, Citation2015; Yatani & Truong, Citation2012), such that each model was trained on N − 1 participants, and one participant was held out for validation, for N folds, where N is the number of participants in the training/validation set. Trained models were then tested on the hold-out set and were compared to a corresponding chance baseline computed by randomly sampling classes from the test set.

Results

Oddball Detection

Overall, participants clicked on the oddball correctly on 92% of trials (SD=.08). Errors were determined as either not clicking within the red ball (target) or not responding. Probes were presented pseudo-randomly during the task – for a total of 75 probes (15 per block) – after either two, four, or six oddballs. Oddballs preceding probe questions were labeled as either ‘Mind Wandering’ or ‘Focused’ based on the participants’ answers to each probe question. As responses to the oddball stimulus were negatively skewed, we performed a paired Wilcoxon’s singed-ranks test in order to examine whether the difference in accuracy between oddball trials in which participants were mind wandering and trials in which they were focused was significant. In support of H1, results indicated that responses during mind wandering (Mdn =.94) were significantly less accurate than responses during focused attention (Mdn=.96,Z=3.42,p<.001). Moreover, reaction times for correct responses during focused attention (M = 622.05, SD = 82.04) were faster than reaction times during mind wandering (M = 652.21, SD = 87.49). Reaction times during mind wandering and focus trials were normally distributed. Paired t-tests indicated that this difference was significant (t=6.15, df = 42, p <.001).

Tracking Performance

In line with a previous study by Kam et al. (Citation2012), the Root Mean Squared Error (RMSE), was calculated as, RMSE=1nΣi=1n(xiTiσi)2 where xi = the participant’s mouse position across time, Ti = target position4 across time, and ni = the number of samples for the participant’s trajectory array. In support of H2, after conducting a Wilcoxon signed-rank test due to non-normal distribution of the variable in question, we found that the RMSE was larger during states of mind wandering (Mdn = 124.79) than during states of focus (Mdn = 114.45), Z=3.92, p < 0.001. Conversely, an independent samples t-test indicated that the average tracking accuracy (normally distributed), calculated as the percentage of points in which the mouse was within the ball was higher whenever participants reported having been focused (M =.58, SD =.19) than when participants reported having been mind wandering (M=.54, SD =.21, t(42)=4.12, p <.001). See for a sample snippet of a participant’s movements while tracking the trajectory of the ball.

Figure 2. Sample trajectory of one participant across 3 trials. The straight, faded lines represent the trajectory of the ball, while the uneven lines represents the trajectory of the participant.

Figure 2. Sample trajectory of one participant across 3 trials. The straight, faded lines represent the trajectory of the ball, while the uneven lines represents the trajectory of the participant.

Figure 3. Mind wandering rates (Mean + SD) across blocks.

Figure 3. Mind wandering rates (Mean + SD) across blocks.

Mind Wandering Over Time

Partially in line with H3a, in which mind wandering rates were expected to increase over time, we found that participants reported more mind wandering after Block 1 (see ); however, mind wandering rates remained approximately stable from Blocks 2 through 5 (40-44%), indicating that there is a ceiling on the amount of time participants engage in mind wandering thought within the present task (see ). We conducted a Repeated Measures Analysis of Variance (ANOVA) that treated Block (1, 2, 3, 4, 5) as a within subject factor. Results indicated that the effect of Block on the proportion of mind wandering responses to the thought probes was significant, F(2.64,110.88)=3.67, p =.018, np2=.08. Post-hoc analyses with Bonferonni corrections for multiple comparisons indicate that only mean differences between Block 1 and 2 (p=.016), and between Blocks 1 and 3 (p =.025) were significant.

TABLE 3. Performance metrics for predicting mind wandering for multiple models on data using 17 mouse tracking features () and tracking accuracy per trial.

Tracking Performance Across Time

As this task required a considerable amount of motor control for tracking the ball at a constant speed during the span of about one hour, we were interested in whether the tracking accuracy declined across the 5 different blocks. We conducted a Repeated Measures Analysis of Variance (ANOVA) that treated Block (1, 2, 3, 4, 5) as a within subject factor. Results indicated that the effect of Block (as a proxy for time on task) on tracking accuracy was only marginally significant, F(2.49,104.76)=2.81, p =.053, np2=.06. Post-hoc comparisons with Bonferroni corrections indicate that mean differences between Blocks 1 and 2 (p =.043), 2 and 3 (p =.047), and between 2 and 4 (p =.021) were significant. From , we observe that tracking accuracy initially increases slightly from the first to the second block, but then declines. H3b) was therefore not supported. We see that participants’ tracking of the ball was more accurate whenever they were focused on the task than when they were mind wandering (see ).

Figure 4. Tracking accuracy per block and per Mental State (FA – Focused Attention, MW – Mind Wandering, N = 37).

Figure 4. Tracking accuracy per block and per Mental State (FA – Focused Attention, MW – Mind Wandering, N = 37).

In order to control for the effect of mind wandering on tracking performance across time, we also conducted a Repeated Measures Analysis of Variance with both Block and Mental State (mind wandering v. focused attention) as within-subject factors (N=37)5. We found that there was indeed a significant main effect of mind wandering on tracking performance, F(1,36)=13.30, p =.001, np2=.27. However, the main effect of Block was not significant, F(2.77,99.59)=1.87, p =.144, np2=.05, nor was there an interaction effect between mental state and block F(3.38,121.59)=1.42, p =.237, np2=.04. Therefore, when controlling for mind wandering, the effect of time on tracking accuracy was no longer significant.

Intermittency and Variability of Movements

In order to estimate the amount of intermittency in tracking movements between attentional states (Mathew et al., Citation2020), we investigated how often mouse movements alternated between acceleration and deceleration phases (see ). Contrary to H4, by means of a Wilcoxon signed-rank test, we found no significant differences between the intermittency of movements under mind wandering (Mdn=6.21), and focus conditions (Mdn=6.14,Z=.17,p=.866). In addition to changes in acceleration, we computed the standard deviation of velocity per trial (Mathew et al., Citation2020). After conducting Wilcoxon’s signed-rank tests, we found that, in line with H5, variability of velocity per trial was lower during mind wandering (Mdn=.49) than during focused attention (Mdn=.51,Z=2.40,p=.016).

Figure 5. Sample movement intermittency of one trajectory in terms of acceleration (pixels/ms2). The dashed line represents the acceleration of the moving ball – which at a constant velocity, was 0.

Figure 5. Sample movement intermittency of one trajectory in terms of acceleration (pixels/ms2). The dashed line represents the acceleration of the moving ball – which at a constant velocity, was 0.

For mind wandering intervals, we also assessed to what degree attention was disengaged from the current task (perceptual decoupling), to what degree participants imagined being somewhere else (mental navigation) and to what degree the content of their thoughts varied (content variation). We thus conducted three separate linear mixed effect models6 using the package lme4 (Bates et al., Citation2015) with degree of decoupling, mental navigation, and content variation as fixed effects, and participants as random intercepts. As dependent variables, we used tracking accuracy, intermittency, and velocity variability, respectively. P-values were obtained from likelihood ratio tests of the full model with degree of decoupling, mental navigation and content variation as fixed effects against the model with only participants as random intercepts. We found that the model for predicting tracking accuracy was significant (χ2(3)=41.15, p <.001), such that only the decoupling significantly contributed to lowering tracking accuracy (see ). However, the model for predicting movement intermittency was not significant (χ2(3)=5.40, p <.145). Finally, the model predicting velocity variability was significant (χ2(3)=23.08, p <.001), such that only the effect of decoupling was significant in lowering velocity variability (see ) . In sum, we provide support to H6) by observing that higher degrees of decoupling, but not of mental navigation or content variation, significantly contributed to reduced tracking accuracy and variation in velocity whenever participants reported that they were mind wandering.

TABLE 1. Perceptual decoupling, mental navigation and content variation as predictors for tracking accuracy, intermittency, and velocity variability.

Since tracking accuracy can be seen as a proxy for attentive performance on the task, we also investigated the relationship between tracking accuracy, intermittency and variability in velocity per participant. We found that intermittency of movements, measured as changes in acceleration and deceleration phases per trajectory, was highly positively correlated with tracking accuracy (N = 43, r =.87, p <.001). Meanwhile we found variability in velocity (N = 43, r=.49,p=.001) to be negatively correlated with tracking accuracy7. Participants who performed more accurately on the task also had more intermittent movements, with less variable velocities.

Exploratory Analyses: Gender and Age

Recent research has reported a male advantage in motor control during various tasks, and particularly in visuomotor tracking (see Mathew et al., Citation2020). As such, we were interested in exploring whether this held true in our study and whether gender played a role in the relationship between mind wandering and tracking. As equal variances could not be assumed due to unequal Male (N = 15) and Female (N = 28) sample sizes, and because variables were not normally distributed, we report results for Mann-Whitney U tests. Confirming recent findings (Mathew et al., Citation2020), we found than men had higher tracking accuracy (Mdn =.69) than women (M =.53, Z=2.32,p=.020). Moreover, men’s movements were more intermittent (Mdn = 6.72) than women’s (Mdn=5.92,Z=2.83,p=.005). Men’s velocity profiles (Mdn =.66) were slower than women’s (Mdn =.70, Z = 2.63, p =.009; and, interestingly, women’s velocities were more variable (Mdn=.58) than men’s (M =.42, Z = 4.03, p <.001).

In order to explore whether there were any age differences with regards to motor control in our sample of university students, we performed Spearman correlations between age, tracking accuracy, mean velocity, standard deviation of velocity, and mind wandering frequency. We found that age was negatively correlated with accuracy (N = 43, r=.32, p =.038), intermittency (N = 43, r=.33, p =.033), and with mean velocity (N = 43, r=.37, p =.014).

Additional Features

In addition to measures of error, intermittency and variance, we report the means and standard deviations of various different mouse-tracking features (Kieslich & Henninger, Citation2017) used in a previous study to investigate mind wandering (Dias da Silva et al., Citation2020). Features were aggregated per participant and per mental state – mind wandering vs. focused (see ) – before computing means and standard deviations. Specifically, x- and y-pos min and x- and y-pos max refer to the minimum and maximum x- and y-positions reached by the mouse, respectively per trial. The x- and y-pos flips refer to the number of directional changes along x- and y-axis, respectively. The x- and y-pos reversals refer to the number of crossings of the x- and y-axis, respectively. Idle time refers to the amount of time the participant remained without moving the mouse per trial. Total distance refers to the total amount of distance in pixels covered by the mouse per trial. Vel max, acc max, and acc min refer to the peak velocity, peak acceleration, and minimum acceleration, respectively, per trial.Vel max time, acc max time, and acc min time refer to the point in time at which peak velocity, peak acceleration, and minimum acceleration were reached.

Machine Learning Results

We trained 5 different algorithms to predict mind wandering from 17 mouse tracking features (features from ) along with tracking accuracy per trial. A leave-one participant-out cross-validation procedure was performed on the training data over 35 folds with 2550 samples – where data from one participant (75 samples) was held out over each fold. Tuning parameters were set at the caret package’s default (Kuhn, Citation2008), such that for LDA (Linear Discriminant Analysis) and TB (Treebag), there were no parameters to be tuned. For NB (Naive Bayes), LDA (Linear Discriminant Analysis), and RF (Random Forest), highest accuracy was used to select the optimal model. For NB, the final values used for the optimal model were laplace = 0, usekernel = TRUE and adjust = 1. For KNN, the final value used for the model was k = 7. For RF, the number of trees to grow was 500 and the optimal parameter mtry used for the model was mtry = 2. After training the models on 80% of the participants, we tested them in the remaining 20%. Results can be found in . We report the classification performances in terms of the F-1 score (balanced average of precision and recall), followed by Sensitivity (true positive rate, i.e., recall), Specificity (true negative rate), and Overall Accuracy of all classes. As can be seen, Accuracy was slightly above chance for all models. Although F-1 and Sensitivity scores were above chance for NB, Specificity scores were not. KNN performed above a random chance baseline for all performance metrics. LDA, TB and RF had above chance Specificity scores, but not F-1 or Sensitivity scores.

Discussion

The primary aim of this study was to investigate differences between fine-grained movement dynamics during states of mind wandering and focused attention during a visuomotor tracking task. We obtained particular insights into the dynamics of eye-hand coordination that are relevant both for interactions with computers but also for movements in day-to-day life. Our findings provide support for an attentional resources view of mind wandering (Smallwood & Schooler, Citation2006), in which resources necessary for visuomotor performance (Repov & Baddeley, Citation2006) are instead directed inwards toward mind wandering, and sensory and perceptual processes become decoupled from the task at hand (Kam et al., Citation2012; Kam & Handy, Citation2013).

Confirming our expectations in H1, we found that whenever participants were mind wandering, they both missed the oddball more frequently and had slower response times to the oddball. In line with previous work (Arapakis & Leiva, Citation2016, Citation2020; Cetintas et al., Citation2010; Dias da Silva & Postma, Citation2020; Kam et al., Citation2012) and confirming H2, we found that movements change in relation to one’s attentional state. Notably, our findings provide further support for a decoupling of sensory and motor processes as the mind wanders. More specifically, and confirming Kam et al. (Citation2012)’s findings, we found an increase in tracking error, and conversely, reduced tracking accuracy, during periods of mind wandering compared to periods of focused attention during the visuomotor tracking task. Ongoing monitoring of the visual target as well as selection of the appropriate motor movement (in terms of position and timing of the mouse) in response to the stimulus are necessary for accurate task performance. During mind wandering, attention no longer directs the the motor system to control the task outcome. Responses that miss the stimulus altogether are related to poorer tracking scores, reflective of reduced attention to and poorer monitoring of the target.

Based on previous studies (Brosowsky et al., Citation2020; Thomson et al., Citation2014), we expected that mind wandering rates would increase (H3a) and tracking performance would decrease over time (H3b). However, we found that mind wandering rates increased only from the first to the second block, remaining stable from the second block onwards, indicating that there were ceiling effects for the amount of time spent mind wandering during this task. In terms of tracking performance over time, we observed that tracking accuracy initially increases from the first to the second block, but then declines. However, when controlling for mind wandering, the differences in tracking accuracy across time were no longer significant. Previous studies assessing fine motor control have shown improvements in visuomotor tracking performance across training sessions (Boyd & Linsdell, Citation2009; Boyd & Winstein, Citation2004). However, task duration, instructions, and goals likely play a role in generating these learning effects. In these studies, the task was repeated across three to five different days, with each session lasting between 25-45 minutes each. Moreover, there was more diversity in the task patterns that needed to be tracked, contributing to less task monotony (i.e., participants had to track random and repeated sine-cosine waveform with a joystick on a screen).

Although the current task was challenging in terms of its fine motor demands, it could be that it is more similar to the typical sustained attention and vigilance tasks used to study mind wandering than to the task used by Boyd and Winstein (Citation2004) and Boyd and Linsdell (Citation2009). As such, the task’s monotony allowed little room for learning, as there was nothing to learn beyond the monotonous tracking of the ball bouncing on the screen and clicking on it during rare occasions when it became red (Brosowsky et al., Citation2021). Alternatively, it may be that learning would take place if this task was performed across various days as in Boyd and Winstein (Citation2004) and Boyd and Linsdell (Citation2009). In addition, due to the task’s length (and the fact that participants were aware that they would be performing such a repetitive task for about an hour), it is likely that participants quickly became less motivated to focus on the task, and engaged in mind wandering instead (Brosowsky et al., Citation2020; Thomson et al., Citation2014), which in turn, led to no improvements in tracking performance. Recently, Brosowsky et al. (Citation2020) found that increased levels of motivation mitigate declines in performance caused by mind wandering as time on task increases. Thereby, in future studies, it would be relevant to assess levels of motivation while performing the task.

In line with our predictions in H5, we did find differences – although small – in the standard deviation of velocity per trial, such that the variation in velocity during mind wandering was lower than during focused attention. This falls in line with previous research that has found reduced variation in movement under conditions in which participants focused their attention internally on one’s body motions rather than externally toward the task (Lohse et al., Citation2014). It has been suggested that variability in movement velocity is actually functional to task performance and a consequence of coordination toward the task. Meanwhile a reduction in variability may reflect a loosening of processes that help maintain attentional focus and motor control during performance of the task at hand (Wulf, Citation2013). At a first glance, our findings seem to be at odds with this view, as reduced variability in velocity was also associated with increased tracking accuracy. However, previous tasks reporting greater variability of movement to be associated with improved task performance did not involve a moment-to-moment measure of movement accuracy. Instead, they assess the relationship between a series of movements which lead to a temporally distant outcome (i.e., expert hammer swings, shooting a basketball, dart throwing, volleyball serves, etc., see Lohse et al., Citation2014; Wulf, Citation2013).

In order to qualify the type of thoughts and quantify the depth of mind wandering episodes, we also asked participants to what extent their thoughts were disengaged from the current environment (perceptual decoupling), to what extent they imagined being somewhere else (mental navigation), and to what extent the contents of their thoughts varied (content variation). Confirming H6, we found that higher degrees of perceptual decoupling, significantly contributed to reduced tracking accuracy and variation in velocity. This falls in line with previous research reporting that deep levels of mind wandering, reflective of high degrees of decoupling, tend to lead to a greater attenuation of external stimuli processing (Kam & Handy, Citation2013; Schad et al., Citation2012).

Contrary to our predictions in H4, we found no differences between mind wandering and focused attention in the intermittency of movements. However, intermittency was actually highly strongly related to tracking accuracy, a proxy for focused attention on the task, indicating that more corrective movements were also functional to task performance. More complex tasks in terms of motor control are associated with more intermittent movements (Russell & Sternad, Citation2001; Weir et al., 1989) being necessary for task performance. Thus, as a proxy for mind wandering, poorer tracking accuracy was related to less intermittent movements, reflecting poorer adjustment to the complex task demands.

Moreover, even though we found mind wandering states to be associated with slightly lower movement variabilityFootnote8, we found that poorer tracking accuracy was associated with higher movement variability. If we assume that tracking accuracy could be a proxy for focused attention on the task, then poor tracking accuracy could also be a behavioral index of mind wandering. There seems to be some discrepancy between participants’ binary reports of mind wandering (yes vs. no) and poor tracking accuracy as a behavioral index of mind wandering. While individuals are generally able to accurately report whether or not they were mind wandering, they are unable to pinpoint the exact onset of their thoughts. Therefore, we did not know when exactly, a mind wandering episode began. In line with previous research (see Jin et al., 2019; Smallwood et al., Citation2007; Stewart et al., Citation2017), we analyzed trials up to approximately 30 seconds prior to reports of mind wandering (with varying degrees of decoupling, mental navigation, and content variation) or focused attention. However, it is likely that both in trials where participants responded that they were mind wandering and in trials in which they responded that they were focused, there was a mix of attentional states. Therefore, subjective reports to the probe experiences were a more general, but less precise measure of mind wandering than moment-to-moment tracking accuracy. Nevertheless, we can conclude that overall poorer tracking accuracy reflect conditions in which participants were primarily mind wandering, but movement intermittency and variability likely reflect more precise gradations in the level of mind wandering throughout the task.

Predicting Mind Wandering from Mouse Movements

Finally, we also investigated whether we could predict self-reported mind wandering states from mouse movements. In addition to differences in tracking accuracy and movement variability prior to mind wandering reports, there also seemed to be differences in other variables we explored, namely, in those reflecting movement complexity (i.e., x-position flips), maximum velocity, acceleration, and total distance traveled per trial during the tracking task. However, this feature set did not distinguish between mind wandering and focused attention in our machine learning models very well.

In a previous study (Dias da Silva & Postma, Citation2020), we used a similar mouse tracking feature set to distinguish between three attentional states (focused attention, task-related interferences, and task-unrelated thoughts) during a complex working memory forced-choice reaching task. Whereas task-unrelated thought refers to thoughts that are irrelevant to the current task, task-related interference refers to thoughts involving a preoccupation with performance of the current task (i.e., “How long will this task take?”, “I wonder how well I am doing.”, “What is the purpose of this task?”). Although distinct from one another, both task-unrelated thoughts and task-related interferences involve a decoupling of attention from the external task toward internal thoughts and feelings. Our predictive models were better at distinguishing task-unrelated thought and focused attention, but worse at predicting task-related interferences (often below or just at chance level). We found that both features pertaining to trajectory complexity (x- and y-position flips) features (i.e.: time to reach maximum deviation and maximum velocity) pertaining to the time it took to make a commitment to a response were most important for predicting task-unrelated thought. Participants took longer to commit to a response and made more x- and y-position flips, indicating more complex trajectories, whenever they were having task-unrelated thoughts.

Beyond the most evident difference in terms of reaching and tracking goals in the study reported in Dias da Silva and Postma (Citation2020) and in the current study, the tasks differed in terms of their complexity, in terms of the types of probes presented, and in terms of the period of time assessed prior to each probe as a mind wandering episode. Notably,the tracking task was more more complex in terms of motor control necessary as participants continuously monitored the position of their mouse in relation to a target on the screen in a fine-grained manner for approximately one hour. Sustaining attention over extended periods is challenging, especially when the task is monotonous, but requires vigilance to rare targets (Thomson et al., Citation2015). The reaching task, however, was more complex in terms of the attentional and cognitive load necessary to complete the task (keeping items in working memory for later recall while at the same time solving compound math equations and responding with reaching movements).

Moreover, during the tracking task, participants did not receive feedback concerning their performance nor were there any rewards for accurate performance. During the reaching task, participants did receive feedback concerning their performance on the task, which potentially contributed to interfering thoughts, leading participants to think about their performance on the task, and consequently more pronounced differences in temporal aspects of reaching movements toward a target response. There are, however, both advantages and disadvantages to providing feedback during a task. On the one hand, feedback can encourage more accurate performance during a task, however, it can also be a source of interference, leading to a bias in the types of mind wandering thoughts which arise. As such, it is valid to investigate motor movements under both tasks which do provide feedback and under tasks that do not. Further research could provide insight into the effect of feedback on modeling predictions between mind wandering and attentive states in both tracking and reaching tasks.

As this reaching task and the current visuomotor tracking task reported in this paper were not matched in terms of their designs, it is not possible to make direct quantitative comparisons between both. However, it does does seem that variables reflective of movement complexity (i.e., x- and y-position flips) are indicative of states of mind wandering across these different types of motor tasks.

Conclusion

Our findings demonstrate that mouse movement features have valuable information for distinguishing between states of mind wandering and focused attention. In particular, during a visuomotor tracking task, mind wandering was associated with poorer tracking performance and less variable movements, indicative of less attentiveness toward the task. These findings were exacerbated by degree of perceptual decoupling, such that whenever participants reported higher levels of disengagement from the task during mind wandering, their movements were less accurate and less variable. This suggests that perceptual decoupling is not a binary state, but that attention decouples from perception in a graded fashion. Interestingly, poorer tracking accuracy, a potential proxy for mind wandering, was associated with less intermittent movements and with more variable velocities, possibly revealing a loosening of control processes in response to a moving target.

During performance of fine motor tasks, external and internal stimuli compete for attentional resources. The occurrence of mind wandering thoughts may lead to unpredictable movements, which under particular circumstances may lead to mistakes, and sometimes fatal outcomes. Therefore, continued investigation of fine motor behavior associated with mind wandering, taking into consideration both task characteristics, individual differences and depth of mind wandering episodes, can further our understanding of the coupling between attention, perception and action and minimize the unwanted effects of deviations in fine motor control whenever perceptual decoupling takes place.

Acknowledgment

We would like to acknowledge Prof. Dr. Óscar Gonçalves, who was involved in the conceptualization of the study and Diogo Branco, who was involved in the technical setup of the study.

Data Availability Statement

The data that support the findings of this study are openly available in https://doi.org/10.34894/VZSY1L.

Notes

1 Participants were asked the questions 1) “What is your Gender”, and could respond either ‘Male’ or ‘Female’, and 2) “What is your age.”

2 These questions were inspired by items in the Mind Wandering Inventory (Gonçalves et al., Citation2020).

3 Note that in the forced-choice reaching task we include 1 additional feature assessing the time it took for participants to start moving their mouse towards the target per trial and 9 additional features of spatial attraction towards (competing) targets, which are irrelevant for the purposes of this tracking task.

4 The target position was determined as the coordinates of the center of the ball at each time point.

5 As some participants did not report either mind wandering or being focused in some blocks, they were excluded.

6 Note that we took into account only the trials in which participants responded that they were not focused on the task.

7 We performed Spearman correlations on data aggregated per participant, as the data was non-normally distributed.

8 When observing only the trials in which participants reported mind wandering, higher degrees of perceptual decoupling seem to be driving this reduction in movement variability.

References