Uploaded October 2018 | Updated September 2026, 2 hours ago
Learn more: gierad.com/projects/ubicoustics
Despite sound being a rich source of information, computing devices with microphones do not leverage audio to glean useful insights about their physical and social context. In this project, we present a novel, real-time, sound-based activity recognition system. We start by taking an existing, state-of-the-art sound labeling model, which we then tune to classes of interest by drawing data from professional sound effect libraries traditionally used in the entertainment industry. These well-labeled and high-quality sounds are the perfect atomic unit for data augmentation, including amplitude, reverb, and mixing, allowing us to exponentially grow our tuning data in realistic ways. We quantify the performance of our approach across a range of environments and device categories and show that microphone-equipped computing devices already have the requisite capability to unlock real-time activity recognition comparable to human accuracy.
Learn more: gierad.com/projects/ubicoustics
Despite sound being a rich source of information, computing devices with microphones do not leverage audio to glean useful insights about their physical and social context. In this project, we present a novel, real-time, sound-based activity recognition system. We start by taking an existing, state-of-the-art sound labeling model, which we then tune to classes of interest by drawing data from professional sound effect libraries traditionally used in the entertainment industry. These well-labeled and high-quality sounds are the perfect atomic unit for data augmentation, including amplitude, reverb, and mixing, allowing us to exponentially grow our tuning data in realistic ways. We quantify the performance of our approach across a range of environments and device categories and show that microphone-equipped computing devices already have the requisite capability to unlock real-time activity recognition comparable to human accuracy.
![TouchTools [CMU/Qeexo]](https://i.ytimg.com/vi/N8s8NJf34fM/mqdefault.jpg "TouchTools [CMU/Qeexo]
Gestures on todays touch devices are simplistic, solely relying on the number of fingers (1-finger pan, 2-finger zoom, 3-finger swipe, etc). We propose instead that gesture design be inspired by the manipulation of real world artifacts specifically tools. The result is a set of rich gestures for touch interaction that leverages user familiarity and fluency with real world objects.")
