Sleep Accuracy: Garmin loses, Fitbit wins in scientific study

Affiliate Disclosure: All links pay commission

Reading Time: 2 minutes

Sleep Accuracy: Garmin loses, Fitbit wins in Scientific Study

9 commercial devices went head-to-head in this scientific sleep study that is claimed to be internally funded by West Virginia University, Rockefeller Neuroscience Institute.

The study comprised 5 people over an extended period of 98 data points (nights) with various scientific controls that looked at basic metrics: Total sleep time (TST), total wake time (TWT), and sleep efficiency (SE).In a nutshell, Fitbit Ionic showed the least variation with Garmin Vivosmart 4 Band and Apple Watch 3 (SleepWatch app) having the highest, or worst, variation. Even the Whoop Strap 2.0 and Polar A370 fared better than the Garmin.

Source: Nature & Science of Sleep Journal

Here are some charts from the report’s findings followed by some of my thoughts and key take-out points from the results.

Here are my thoughts

I don’t believe all the results

At least I don’t believe they are representative results for the brands on test. Even if I did buy them, I would like to have seen the researchers use significantly more people, significantly more data points and use newer devices which would give the manufacturers a fairer shot. Also omitting EMFIT QS was somewhat strange as, in my opinion, it is the best consumer-grade sleep-tracking device.

Interestingly the researchers found that “Consistent trends across all devices were observed in their failure to determine the amount of time its user was awake rather than sleeping“. At least that seems to tie in with what many of you often say.

Surprisingly when it came to sleep stage analysis the limited number of data points showed that

  • Measurements of light sleep obtained from WHOOP were much less biased and illustrated strong consistency between trials
  • the “Ability to estimate deep sleep was remarkably poor for all devices” and that
  • they all overestimated REM sleep time“.

As we’ve heard many times before, the sleep devices we use simply do not accurately measure sleep stages.

It just goes to show that you can’t always take any single view or a single technology as being correct. Clearly, technologies work differently on different people as I am at pains to say whenever I show my workout-related oHR stats. That said, the researchers swapped devices between subjects and do state that, “None of the remaining subjects had significantly different error distributions from each other.” Luckily readers here tend to be an intelligent bunch who read around and get a variety of different views before making up their own minds.  I would be interested to know, below, how much weight you would place in this study compared to your own personal experiences.

 

Reader-Powered Content

Follow the5krunner
This content is not sponsored. It's mostly me behind the labour of love which is this site, meaning it's entirely reader-powered content ❤️ I'd really appreciate it if you'd follow, subscribe or consider getting me an e-Coffee ❤️ Alternatively please buy the reviewed product via the5krunner.com (which costs you no extra) and, for that, I receive a small commission. Thank you!
 
Affiliate Disclosure: All links pay commission
Support this site with purchases at these partners - should click to a local choice in your country
Subscribe
Notify of
guest
21 Comments
oldest
newest most voted
Inline Feedbacks
View all comments
Stromz

Having used Polar, Coros, Suunto, and now Whoop – the Whoop is far and away the most accurate for me in regards to properly capturing time asleep and wake up time. Thing even auto identifies the odd nap I do take.

Xave

Do you wear Whoop all the time?

Stromz

Yeah pretty much 24/7. It does a pretty decent job of tracking workout HR if worn on the bicep or upper arm.

The recovery insights are what are keeping me pretty well locked in to the platform. Once I have meaningful sample size to find some of my own conclusions, I may dump it.

Nick K

Agree with the first commenter. WHOOP analysis rocks. The most consistent detection of sleep, very believable detection of REM and deep phases, etc. I’d say newer Sleep+ Stages from Polar is fairly close too when it works: 15-20% of days the device doesn’t get sleep start correctly missing it by up to 1.5 hours sometimes.

Ōura is close too but too many false positives interpreting restless/REM sleep as wake time. Fitbit was far worse in that regard.

Garmin… depends. The new on-device analysis available to F6, etc. seems to be a huge step forward. The old analysis, available on VA4 and all, is a bad joke.

gtom

I consider this a good feedback to anyone putting too much focus on “advanced metrics” from consumer grade devices. All devices were within 7 – 22 of Mean Absolute Percent Error for sleep duration and 3 – 12% MAPE in sleep efficiency. All good for basic metrics from those devices.

Compare this to MAPE in “advanced metrics” like sleep stages.

Light sleep MAPE 13 – 40. That’s still good.
Deep sleep 46 – 133. A joke across the board.
REM 62 – 79. LOL?

ehaul

If you don’t buy it and we’re intelligent, don’t use a headline like that to get our attention. Marked as waste of my time.

Chris

It’s a pity that old devices have been compared. I’d love to check out the latest Polar Precision Prime, the FirstBeat algorithm on Garmin watches (e.g. Fenix 6), the Apple Watch 6 and the Honor Band 5, for example. The sleep analysis of the latter is very accurate, although it is evaluated in the Huawei cloud.

Benedikt

with only 5 people, I can do that study at work more accurate.
It is not a scientific study, its only a study done by scientists.

Benedikt

Maybe it was a side project or a „Semesterarbeit“, some sort of annual homework for a student with no budget.

Michael

Not a big study, but I’ve switched from Fitbit Charge 3 to Vivosmart 4 and I agree Garmin is as horrible at sleep tracking, as it is better at run tracking. The worst cases are several nights Garmin thought I had 8+ hours of deep sleep. But almost everyday it overestimates sleep time, failing to detect when I wake up.

John

Garmin VA3 user here. I find the total sleep time is reasonably accurate, but it fails to register times I wake in the night if I don’t move a whole lot. It registers a total amount of deep sleep but looking at the movement chart it looks like it’s badly off. That said, the days I wake not feeling well rested are usually the days it says I didn’t get much deep sleep.

Tracking sleep isn’t hugely important to me, it’s just interesting to see my heart rate slowly drop overnight while I sleep.

Mihail Bolog

You hate they did not say Apple fared best, are’ t you?

tfk, the5krunner

me?
no. it is what it is.

antinomy
  1. n=5 is still better than n=1
  2. the amount of time it takes to conduct a study, write it up and have it peer reviewed means that it’s unlikely to ever be based on the most recent devices
antinomy

I think you’ve missed my point. Your doubts about their findings are based on what is, presumably, a set of single-tester subjective reviews of the wearable devices and brands discussed. Nothing wrong with this of course, but glasshouse, stones etc.

I imagine getting funding to do a large-scale test using every device currently offered by Apple, Garmin, Fitbit, Polar etc. would be prohibitively expensive. (And who would want to fund it anyway, other than the brands themselves?) In the meantime this seems a perfectly reasonable contribution to the field…

Justin

I’ve used every of the major devices and their top-end models (Polar, Garmin, Coros). Every last one reports generally the same thing about me:

I get a lot of light sleep, some REM, and very little deep sleep. In the case of Polar and Coros, both get my time frame of sleep within minutes of actually falling asleep and waking. Garmin was the worst at this by gauging when I fell asleep by hours and with the same result for waking (nearly an hour after I was awake).

Coros for me, while barebones, seems to be the best at getting awake time accurate, as well as getting my RHR near perfect night to night. I just wish they’d record in 12hr time in the app (a unity of time across the platform would be nice COROS!!).

Stromz

Rant mode enabled 🙂

Completely off topic but they need freaking waypoints. I’ve been hammering customer service about this. Vertix would be a killer actual “Outdoor” watch if it had any sort of waypoint functionality…

Pretty much agreed w your assessment though lol