Uploaded November 2023 | Updated September 2026, 2 hours ago
Demo Video: youtu.be/AHh2vYQVb_8
Abstract:
The ability to track a user’s arm pose could be valuable in a wide range of applications, including fitness, rehabilitation, augmented reality input, life logging, and context-aware assistants. Unfortunately, this capability is not readily available to consumers. Systems either require cameras, which carry privacy issues, or utilize multiple worn IMUs or markers. In this work, we describe how an off-the-shelf smartphone and smartwatch can work together to accurately estimate arm pose. Moving beyond prior work, we take advantage of more recent ultra-wideband (UWB) functionality on these devices to capture absolute distance between the two devices. This measurement is the perfect complement to inertial data, which is relative and suffers from drift. We quantify the performance of our software-only approach using off-the-shelf devices, showing it can estimate the wrist and elbow joints without the user having to provide training data.
Citation:
Nathan DeVrio*, Vimal Mollyn*, and Chris Harrison. 2023. SmartPoser: Arm Pose Estimation with a Smartphone and Smartwatch Using UWB and IMU Data. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 79, 1–11. doi.org/10.1145/3586183.3606821
*Equal contribution
Demo Video: youtu.be/AHh2vYQVb_8
Abstract:
The ability to track a user’s arm pose could be valuable in a wide range of applications, including fitness, rehabilitation, augmented reality input, life logging, and context-aware assistants. Unfortunately, this capability is not readily available to consumers. Systems either require cameras, which carry privacy issues, or utilize multiple worn IMUs or markers. In this work, we describe how an off-the-shelf smartphone and smartwatch can work together to accurately estimate arm pose. Moving beyond prior work, we take advantage of more recent ultra-wideband (UWB) functionality on these devices to capture absolute distance between the two devices. This measurement is the perfect complement to inertial data, which is relative and suffers from drift. We quantify the performance of our software-only approach using off-the-shelf devices, showing it can estimate the wrist and elbow joints without the user having to provide training data.
Citation:
Nathan DeVrio*, Vimal Mollyn*, and Chris Harrison. 2023. SmartPoser: Arm Pose Estimation with a Smartphone and Smartwatch Using UWB and IMU Data. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 79, 1–11. doi.org/10.1145/3586183.3606821
*Equal contribution
![TouchTools [CMU/Qeexo]](https://i.ytimg.com/vi/N8s8NJf34fM/mqdefault.jpg "TouchTools [CMU/Qeexo]
Gestures on todays touch devices are simplistic, solely relying on the number of fingers (1-finger pan, 2-finger zoom, 3-finger swipe, etc). We propose instead that gesture design be inspired by the manipulation of real world artifacts specifically tools. The result is a set of rich gestures for touch interaction that leverages user familiarity and fluency with real world objects.")
