new Gesture Controls Can Be Used In Sports Apps
I have been using the next generation of Apple Watch software and there is a unique new feature that WILL become more widespread in sports watches even though it is not intended as a sports feature per se…even higher-level athletes using Garmins would find this useful.
There is a video of it in action further above.
In a nutshell:
You can use hand gestures instead of taps, presses and swipes to control your watch.
For controlling fitness usage that will mean adding laps/segments and scrolling screens.
How it works
Gesture Controls are configurable. In the example I describe here, I used a fist clench gesture which the Apple Watch converted into an action to page through screens. Other gestures and watch actions can be used.
Q: Does it work well?
A: Yes! Gesture recognition is great but gestures are limited in scope right now and the actions you can assign to them have limitations.
Background
For many years, there have been several methods available to control your watch whilst exercising. How well each method works varies depending on personal preference at your level of exertion. Let’s take a specific example of how we can currently scroll through screens on our watches/apps.
- Buttons – Perhaps the best and most widely used method. You know what pressing a button is like! You also know that 4 or 5 buttons are needed for most sports watches to be treated as truly credible…at least for now.
- Touch – taps & swipes on a watch screen or heart rate monitor have been used in the past to trigger various actions. Usually, a swipe on your touchscreen will switch between the screen of your sports profile/app and many years ago Polar had the Heart Touch feature working with an H7 and V800.
- Audio Control – This works by voice recognition and as such only smartwatches have voice recognition through Wear OS, watchOS or similar, advanced firmware.
Some of these methods require you to bring your arms together or to interact with a specific part of a touchscreen and that can be non-trivial if you are exerting yourself and using a touchscreen swipe. For a cyclist and perhaps an XC runner, these interactions can add a small amount of danger to the interaction!
Are buttons perfect? A: No. Sometimes you might have to contort your right hand to press the bottom on your left wrist and then you might find that your laggy Garmin has made you do an extra press and you have to page back.
How Gesture Control Works
You’ll need watchOS 8 and iOS 15.
Currently, 4 gestures are recognised by the Apple Watch 6/SE and these are fist clench, double fist clench, pinch and double pinch. Once you have followed the instructions from the Apple Watch to recognise your gestures, you assign each gesture to an action on the watch.
Open the Watch app on your iPhone and go to Accessibility> Assistive Touch>On.
These screens take you through the steps and you can see on the second screen where the double pinch is assigned the backward action (usually back to the previous screen)
Great, you’re done…kinda.
You can now use the Apple Watch and control it to open apps and navigate the screens and options within the apps. It’s fairly powerful but its usefulness depends on which gestures you assign to which actions. The normal actions during workouts would be some combination of a basic screen tap, left/right screen swipes, hardware button press and a crown press/rotate.
But now your problems start.
Every app you use is subtly different and each app will interpret actions differently. For example, the Apple Workout app requires a double tap to take a lap, whereas the iSmoothRun app requires a simultaneous press of the two hardware buttons. The Stryd Apple Watch app scrolls up and down rather than left to right and lets the side BUTTON (not the crown) take a lap.
So now you can see why I chose the example of switching between screens during your workout. Perhaps taking a lap would be more usefully done with a hand gesture but you can’t because none of the apps I looked at could be configured to allow a lap to be taken with the actions currently supported by gesture control.
This will eventually be resolved with new gestures recognised by Apple like finger click or pinching other fingers together. Then the hand gesture actions themselves need to expand to cover double screen taps and various button combinations. Finally, app developers need to modify how their apps work with the available gesture actions.
What I got to work
My goal was to start an app as normal (fingers) and then use one gesture to take laps and another to scroll screens
iSmoothRun would take a screen tap as a way to scroll screens as would the Apple Workout app as shown in this video.
I couldn’t get LAP functionality to work at all with any gestures, although it might be possible on other apps.
Take Out
Apple intends gestures to be used as Accessibility Features. Which is great.
However, it also shows me that these features can work really well in some sports scenarios. So well, that they WILL be used by watch and app developers in the near future starting on the Apple Watch 7 / watchOS 8 in September and indeed Samsung introduced some gestures to Galaxy Watch 3 in 2020.
The use of gesture control will allow sports watch designs with fewer than 4/5 buttons. That could be a boost to the sporting credibility of the traditional 2-buttoned smartwatch.
I got the screen scrolling to work well but couldn’t quite get my apps to take a lap. However, this feature is only in Beta right now so we are at the early stages and its scope will grow over time.
What I would like to see
- More gestures recognised – such as pinching a different finger
- More watch actions are supported – such as a watch shake or screen double-tap.
- Enable app developers to create a gesture profile that only works within their app rather than at a system level.
Interesting!
Looks a bit too erratic compared to good old reliable buttons, but I’m old-school Polar, they always work, reliably, quickly, perfectly. Obviously 5 buttons is the preferred format for that though (up, down, select, back, and “other menus/options)… having used Garmin, Polar, and now Coros for awhile, nothing beats Polar’s button-ui still, and I don’t see gestures really doing much to offset the loss of buttons down to 2 or 3. My opinion of course.
One thing I do like about my v800 (not sure if any other model ever had it) was their user-selectable heart-touch, AND their “tap” functions. So it became a 5-button with 2 extra “triggers” watch that didn’t require 2 hands (you could “tap” on anything to trigger a tap (switch data screens, turn on light, take a lap, whatever you selected for that profile), and the heart touch… well, to be honest, I set mine up to show the current time, handy when cycling, swimming, etc, rather than taking up data view space or requiring “switching screens via buttons” to get to the time display. They certainly could have added gesture functionality since they already had cadence detection, and “daily activity” motion sensing built-in.
Other than maybe water activities, honestly, it seems that “voice” would make more sense over gestures for the “next UI” jump. “Screen up” “screen down”. “Take a lap” “Start Workout”… THAT would be useful (thinking of “Gestures”, if cycling (rowing, etc)… you really can’t do those, because you are holding the workout equipment, without taking your hand off those items, so it’s really not better (if you have a button UI, you take off the “non-watch” hand, reach over, and press the button(s) reliably, rather than taking off the “watch-hand” and hopefully gesturing successfully while bumping over the road… yeah, the more I ponder, the less useful it seems, I know when running, I’m often “gripping/relaxing” to loosen up my arms, as well as lots of shaking them out now and then, no telling what that might ultimately trigger.
Sorry, random ponderings “new features” get the gray matter churning.
it’s much smoother to use than in the video, i had to contort my arm to get the video and it didn’t quite recognise my clenches as well as it would normally. the pinch is a more subtle gesture
v800.yes. the v/v2 has a tap the screen option too
i think voice is too unreliable right now. i use it a fair amount with google and apple.
if the gesture is combined with an activation gesture there would be no (few) accidental triggers. but then the activation gesture would defeat the speed at which i think it would need to work to be practical. if it can recognise a thumb and fourth finger pinch then i guess that wouldn’t be done accidentally.
actually you mention swimming, it would be a great way to add a lap when swimming. when i stop to press lap in ows i sink a bit!