Evaluations of Commercial Sleep Technologies for Objective Monitoring During Routine Sleeping Conditions

Abstract

Purpose

The commercial market is saturated with technologies that claim to collect proficient, free-living sleep measurements despite a severe lack of independent third-party evaluations. Therefore, the present study evaluated the accuracy of various commercial sleep technologies during in-home sleeping conditions.

Materials and Methods

Data collection spanned 98 separate nights of ad libitum sleep from five healthy adults. Prior to bedtime, participants utilized nine popular sleep devices while concurrently wearing a previously validated electroencephalography (EEG)-based device. Data collected from the commercial devices were extracted for later comparison against EEG to determine degrees of accuracy. Sleep and wake summary outcomes as well as sleep staging metrics were evaluated, where available, for each device.

Results

Total sleep time (TST), total wake time (TWT), and sleep efficiency (SE) were measured with greater accuracy (lower percent errors) and limited bias by Fitbit Ionic [mean absolute percent error, bias (95% confidence interval); TST: 9.90%, 0.25 (−0.11, 0.61); TWT: 25.64%, −0.17 (−0.28, −0.06); SE: 3.49%, 0.65 (−0.82, 2.12)] and Oura smart ring [TST: 7.39%, 0.19 (0.04, 0.35); TWT: 36.29%, −0.18 (−0.31, −0.04); SE: 5.42%, 1.66 (0.17, 3.15)], whereas all other devices demonstrated a propensity to over or underestimate at least one if not all of the aforementioned sleep metrics. No commercial sleep technology appeared to accurately quantify sleep stages.

Conclusion

Generally speaking, commercial sleep technologies displayed lower error and bias values when quantifying sleep/wake states as compared to sleep staging durations. Still, these findings revealed that there is a remarkably high degree of variability in the accuracy of commercial sleep technologies, which further emphasizes that continuous evaluations of newly developed sleep technologies are vital. End-users may then be able to determine more accurately which sleep device is most suited for their desired application(s).

Keywords:

• wearables
• consumer sleep technologies
• sleep duration
• sleep efficiency
• sleep staging

Abbreviations

APE, absolute percent error; ACSM, American College of Sports Medicine; bpm, beats per minute; CI, confidence interval; EEG, electroencephalography; EMG, electromyography; EOG, electrooculography; eg, exempli gratia; Q1, first quartile; FDA, Food and Drug Administration; Gen, generation; HR, heart rate; cm, height in centimeters; IQR, interquartile range; LED, light emitting diode; LOA, limits of agreement; Max, maximum; MAPE, mean absolute percent error; MdAPE, median absolute percent error; Min, minimum; NTC, negative temperature coefficient; n, number of trials; PPG, photoplethysmography; PSG, polysomnography; REM, rapid eye movement; SE, sleep efficiency; app, smartphone application; SD, standard deviation; Q3, third quartile; TST, total sleep time; kg, weight in kilograms; y, years of age.

Acknowledgments

The authors would like to express gratitude to the Rockefeller Neuroscience Institute at West Virginia University for support in data collection.

Disclosure

Financial competing interests: none. Non-financial competing interests: none. The authors report no conflicts of interest for this work.

Additional information

Funding

This study was funded internally by the West Virginia University, Rockefeller Neuroscience Institute.