FTC: Affiliate Disclosure: All links pay commission
Sleep Accuracy: Garmin loses, Fitbit wins in Scientific Study
9 commercial devices went head-to-head in this scientific sleep study that is claimed to be internally funded by West Virginia University, Rockefeller Neuroscience Institute.
The study comprised 5 people over an extended period of 98 data points (nights) with various scientific controls that looked at basic metrics: Total sleep time (TST), total wake time (TWT), and sleep efficiency (SE).In a nutshell, Fitbit Ionic showed the least variation with Garmin Vivosmart 4 Band and Apple Watch 3 (SleepWatch app) having the highest, or worst, variation. Even the Whoop Strap 2.0 and Polar A370 fared better than the Garmin.
Source: Nature & Science of Sleep Journal
Here are some charts from the report’s findings followed by some of my thoughts and key take-out points from the results.
Here are my thoughts
I don’t believe all the results
At least I don’t believe they are representative results for the brands on test. Even if I did buy them, I would like to have seen the researchers use significantly more people, significantly more data points and use newer devices which would give the manufacturers a fairer shot. Also omitting EMFIT QS was somewhat strange as, in my opinion, it is the best consumer-grade sleep-tracking device.
Interestingly the researchers found that “Consistent trends across all devices were observed in their failure to determine the amount of time its user was awake rather than sleeping“. At least that seems to tie in with what many of you often say.
Surprisingly when it came to sleep stage analysis the limited number of data points showed that
- “Measurements of light sleep obtained from WHOOP were much less biased and illustrated strong consistency between trials“
- the “Ability to estimate deep sleep was remarkably poor for all devices” and that
- “they all overestimated REM sleep time“.
As we’ve heard many times before, the sleep devices we use simply do not accurately measure sleep stages.
It just goes to show that you can’t always take any single view or a single technology as being correct. Clearly, technologies work differently on different people as I am at pains to say whenever I show my workout-related oHR stats. That said, the researchers swapped devices between subjects and do state that, “None of the remaining subjects had significantly different error distributions from each other.” Luckily readers here tend to be an intelligent bunch who read around and get a variety of different views before making up their own minds. I would be interested to know, below, how much weight you would place in this study compared to your own personal experiences.
Reader-Powered Content
This content is not sponsored. It’s mostly me behind the labour of love which is this site and I appreciate everyone who follows, subscribes or Buys Me A Coffee ❤️ Alternatively please buy the reviewed product from my partners at their regular price. Thank you! This really is reader-powered content.
FTC: Affiliate Disclosure: All links pay commission. As an Amazon Associate, I earn from qualifying purchases.
Having used Polar, Coros, Suunto, and now Whoop – the Whoop is far and away the most accurate for me in regards to properly capturing time asleep and wake up time. Thing even auto identifies the odd nap I do take.
Do you wear Whoop all the time?
Yeah pretty much 24/7. It does a pretty decent job of tracking workout HR if worn on the bicep or upper arm.
The recovery insights are what are keeping me pretty well locked in to the platform. Once I have meaningful sample size to find some of my own conclusions, I may dump it.
Agree with the first commenter. WHOOP analysis rocks. The most consistent detection of sleep, very believable detection of REM and deep phases, etc. I’d say newer Sleep+ Stages from Polar is fairly close too when it works: 15-20% of days the device doesn’t get sleep start correctly missing it by up to 1.5 hours sometimes.
Ōura is close too but too many false positives interpreting restless/REM sleep as wake time. Fitbit was far worse in that regard.
Garmin… depends. The new on-device analysis available to F6, etc. seems to be a huge step forward. The old analysis, available on VA4 and all, is a bad joke.
I consider this a good feedback to anyone putting too much focus on “advanced metrics” from consumer grade devices. All devices were within 7 – 22 of Mean Absolute Percent Error for sleep duration and 3 – 12% MAPE in sleep efficiency. All good for basic metrics from those devices.
Compare this to MAPE in “advanced metrics” like sleep stages.
Light sleep MAPE 13 – 40. That’s still good.
Deep sleep 46 – 133. A joke across the board.
REM 62 – 79. LOL?
If you don’t buy it and we’re intelligent, don’t use a headline like that to get our attention. Marked as waste of my time.
It’s a pity that old devices have been compared. I’d love to check out the latest Polar Precision Prime, the FirstBeat algorithm on Garmin watches (e.g. Fenix 6), the Apple Watch 6 and the Honor Band 5, for example. The sleep analysis of the latter is very accurate, although it is evaluated in the Huawei cloud.
with only 5 people, I can do that study at work more accurate.
It is not a scientific study, its only a study done by scientists.
ha! that’s one of my favourite lines too.
I agree.
but why did they do it like this? strange. If it were sponsored by a company I could understand the motivation. But it appears not to be sponsored.
Maybe it was a side project or a „Semesterarbeit“, some sort of annual homework for a student with no budget.
and using the gadgets the group of friends already had.
Yes, I did think that in one of my more sceptical moments
Not a big study, but I’ve switched from Fitbit Charge 3 to Vivosmart 4 and I agree Garmin is as horrible at sleep tracking, as it is better at run tracking. The worst cases are several nights Garmin thought I had 8+ hours of deep sleep. But almost everyday it overestimates sleep time, failing to detect when I wake up.
Garmin VA3 user here. I find the total sleep time is reasonably accurate, but it fails to register times I wake in the night if I don’t move a whole lot. It registers a total amount of deep sleep but looking at the movement chart it looks like it’s badly off. That said, the days I wake not feeling well rested are usually the days it says I didn’t get much deep sleep.
Tracking sleep isn’t hugely important to me, it’s just interesting to see my heart rate slowly drop overnight while I sleep.
You hate they did not say Apple fared best, are’ t you?
me?
no. it is what it is.
true
running 4k is better than sitting on the couch. You wouldn’t win a 5k race tho.
I think you’ve missed my point. Your doubts about their findings are based on what is, presumably, a set of single-tester subjective reviews of the wearable devices and brands discussed. Nothing wrong with this of course, but glasshouse, stones etc.
I imagine getting funding to do a large-scale test using every device currently offered by Apple, Garmin, Fitbit, Polar etc. would be prohibitively expensive. (And who would want to fund it anyway, other than the brands themselves?) In the meantime this seems a perfectly reasonable contribution to the field…
no, not really. i get your point.
98 nights seems reasonable
5 subjects doesn’t seem reasonable for ‘science’.
brand choice is reasonable but using all that old tech isn’t great.
20 subjects with newer tech would have made me happy. I probably would have glossed over 10 subjects not being enough but still been annoyed by the older tech….sure include some old tech but new tech too.
It does add SOMETHING to the understanding but i’m just not quite sure what.
what it does add is an inference that Fitbit is better than Garmin, which i doubt. Hey, I must be wrong as the science above says i’m wrong.
In reality, I’ve personally done WAY more than 5×98 nights of looking at sleep data and I never try to dress that up as science (as it isn’t)
thank you for making your points nicely, apologies if I’m coming across as a bit grumpy today.
I’ve used every of the major devices and their top-end models (Polar, Garmin, Coros). Every last one reports generally the same thing about me:
I get a lot of light sleep, some REM, and very little deep sleep. In the case of Polar and Coros, both get my time frame of sleep within minutes of actually falling asleep and waking. Garmin was the worst at this by gauging when I fell asleep by hours and with the same result for waking (nearly an hour after I was awake).
Coros for me, while barebones, seems to be the best at getting awake time accurate, as well as getting my RHR near perfect night to night. I just wish they’d record in 12hr time in the app (a unity of time across the platform would be nice COROS!!).
Rant mode enabled 🙂
Completely off topic but they need freaking waypoints. I’ve been hammering customer service about this. Vertix would be a killer actual “Outdoor” watch if it had any sort of waypoint functionality…
Pretty much agreed w your assessment though lol
yes i concur with your opinion. the right question here is : does the sleep monitors and measures how many times i woke up for pee time. interesting if i placed the smartwatch on the table and continue to watch netflix in the wee hour, it assumed that i was sleeping and the accuracy of the monitoring is in question.
i am looking for a reliable accurate smart watch tracker for sleep monitoring purpose. would you be able to recommend one?
if conducting a scientific experiment then putting the device on the side cabinet would not be in the methodology.
so you have to accept that user error is going to cause some rubbish results. don’t do it!
to determine sleep/wake times should be possible by most of the well-known brands. garmin, apple…
the most practical and is EMFIT or you could go down the ring route and just wear it at night. those are the best bets.
you can get surprisingly good results from a smartphone under your pillow
sadly i can’t recommend a smartwatch tracker if you want really accurate results. i can recommend many if you just want indicative results. personally i’d get the apple watch 6 and then there are MANY apps that you can try until you find one that you think is correct. (whereas if you get a fitbit or garmin you are effectively guided toward the one option…the manufacturer option)