KIMLAB (Kinetic Intelligent Machine LAB) | [2023 Weekly KIMLAB] Specifying Target Objects in Teleoperation Using Speech and Natural Eye Gaze @KIMLAB2020 | Uploaded December 2023 | Updated October 2024, 9 minutes ago.
In this study, we propose a new intent detection framework to teleoperate robotic arms based on human speech and natural eye gaze. Our framework applies instance segmentation on the robot's camera image and predicts the human's intended object through matching eye-gaze data, instance masks, instance classes, and transcribed words.
***This work will be presented at the IEEE-RAS Humanoids 2023 in Austin.
2023.ieee-humanoids.org
Yu-Chen (Johnny) Chang, Nitish Gandi, Kazuki Shin, Ye-Ji Mun, Katherine Driggs-Campbell, Joohyung Kim, ”Specifying Target Objects in Robot Teleoperation Using Speech and Natural Eye Gaze,” IEEE-RAS International Conference on Humanoid Robots, December 2023
In this study, we propose a new intent detection framework to teleoperate robotic arms based on human speech and natural eye gaze. Our framework applies instance segmentation on the robot's camera image and predicts the human's intended object through matching eye-gaze data, instance masks, instance classes, and transcribed words.
***This work will be presented at the IEEE-RAS Humanoids 2023 in Austin.
2023.ieee-humanoids.org
Yu-Chen (Johnny) Chang, Nitish Gandi, Kazuki Shin, Ye-Ji Mun, Katherine Driggs-Campbell, Joohyung Kim, ”Specifying Target Objects in Robot Teleoperation Using Speech and Natural Eye Gaze,” IEEE-RAS International Conference on Humanoid Robots, December 2023
![[2023 Weekly KIMLAB] Specifying Target Objects in Teleoperation Using Speech and Natural Eye Gaze](https://i.ytimg.com/vi/FWkA-QXKoic/hqdefault.jpg "[2023 Weekly KIMLAB] Specifying Target Objects in Teleoperation Using Speech and Natural Eye Gaze
In this study, we propose a new intent detection framework to teleoperate robotic arms based on human speech and natural eye gaze. Our framework applies instance segmentation on the robots camera image and predicts the humans intended object through matching eye-gaze data, instance masks, instance classes, and transcribed words.
***This work will be presented at the IEEE-RAS Humanoids 2023 in Austin.
https://2023.ieee-humanoids.org/
Yu-Chen (Johnny) Chang, Nitish Gandi, Kazuki Shin, Ye-Ji Mun, Katherine Driggs-Campbell, Joohyung Kim, ”Specifying Target Objects in Robot Teleoperation Using Speech and Natural Eye Gaze,” IEEE-RAS International Conference on Humanoid Robots, December 2023")
![[2023 Weekly KIMLAB] Exploring the Capabilities of a General-Purpose Robotic Arm in Chess Gameplay](https://i.ytimg.com/vi/GsLxrXSdOqA/hqdefault.jpg "[2023 Weekly KIMLAB] Exploring the Capabilities of a General-Purpose Robotic Arm in Chess Gameplay
As the field of human-robot collaboration continues to grow and autonomous general-purpose service robots become more prevalent, robots need to obtain situational awareness and handle tasks with a limited field of view and workspace. Addressing these challenges, KIMLAB and Prof. Yong Jae Lee at the University of Wisconsin-Madison utilize the game of chess as a testbed, employing a general-purpose robotic arm.
To achieve successful robotic manipulation in playing board games like chess, the robot must follow game rules and prioritize human safety, all while operating in a limited workspace with field-of-view constraints. A pipeline is proposed that considers the rules of the chess game, the constraints of the robot, and environmental conditions to generate valid moves and countermoves. To validate the algorithm, experiments were conducted with the robotic platform, and the parameters of the system were tuned. The proposed algorithm achieved competitive performance, with an average robot turn success rate of 91.94%. This research demonstrates the potential of a general-purpose robot with a single camera to engage safely in competitive tasks with humans, even with these constraints.
***This work will be presented at the IEEE-RAS Humanoids 2023 in Austin.
https://2023.ieee-humanoids.org/
Kazuki Shin, Sankalp Yamsani, Roman Mineyev, Hongyu Chen, Nitish Gandi, Yong Jae Lee, Joohyung Kim, ”Exploring the Capabilities of a General-Purpose Robotic Arm in Chess Gameplay,” IEEE-RAS International Conference on Humanoid Robots, December 2023")
![[2023 Weekly KIMLAB] Interactive Robotic Backpack with PAPRAS](https://i.ytimg.com/vi/NsQyspbpdVE/hqdefault.jpg "[2023 Weekly KIMLAB] Interactive Robotic Backpack with PAPRAS
The Weekly KIMLABs second video unveils PAPRAS:Backpack.
While movies and animations often showcase characters and superheroes with additional limbs, bringing such concepts into the real world involves a multitude of considerations. In this video, we delve into the development of a skeleton estimator utilizing a multi-IMU system and an interactive motion controller designed for our robotic backpack.
***KIMLAB is gearing up to unleash the PAPRAS: Backpack at the Demo Expo during IROS 2023!!!
The Demo Expo will be rocking from 3:30-5:00 pm on Wednesday, October 4, 2023, in the Exhibition Hall. Be sure to track us down and witness the live demo at IROS2023.
Chaerim Moon, Sean Taylor, Kevin G Gim, Sankalp Yamsani, Kazuki Shin, Kyungseo Park, Joohyung Kim, Robotic Backpack System with Pluggable Supernumerary Limbs, Demonstration Sessions, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023), 2023
The backpack hardware was created in 2021, and there are additional video clips showcasing this system.
https://youtu.be/byzNdOhueeQ
https://youtu.be/5O2HC9aiIYU
https://youtu.be/uq5AF7WDP1Y")
![[KIMLAB x PSYONIC] Turned to the Dark Side (Star Wars General Grievous theme)](https://i.ytimg.com/vi/OAsdMvVJb6U/hqdefault.jpg "[KIMLAB x PSYONIC] Turned to the Dark Side (Star Wars General Grievous theme)
Hello There,
KIMLABs PAPRAS:Orthrus sharpened its Star Wars lightsaber skills thanks to PSYONICs Ability Hands!!!
-PSYONIC Ability Hand
https://www.psyonic.io/ability-hand")
![[2023 Weekly KIMLAB] MOMO: Mobile Object Manipulation Operator](https://i.ytimg.com/vi/OvcFSEICq9w/hqdefault.jpg "[2023 Weekly KIMLAB] MOMO: Mobile Object Manipulation Operator
Server robots have become a common sight in restaurants. The next step is equipping them with robotic arms for enhanced mobile manipulation. In this direction, KIMLAB is partnering with HDHyundai Robotics on MOMO(Mobile Object Manipulation Operator). MOMO features the HDHyundai Robotics B1 as its mobile base, incorporating the adaptable PAPRAS. There are two mounts on each side of its torso and an extra neck mount, and each side arm gains one more DOF through a prismatic joint. The systems primary goals include autonomously clearing floor obstructions and delivering items to humans without human intervention. For the design of MOMO, substantial consideration was given to the modularity and adaptability of the system. This allows the mobile manipulator to be optimized for diverse tasks and scenarios.
***KIMLAB will bring MOMO to IROS 2023!!!
The Demo Expo will be rocking from 3:30-5:00 pm on Wednesday, October 4, 2023, in the Exhibition Hall. Be sure to track us down and witness the live demo at IROS2023.
Sankalp Yamsani, Kevin Gim, Tyler Smithline, Richard Qiu, Chaerim Moon, Sungmin Kang, Roman Mineyev, Kyungseo Park, Yoon-Koo Kang, Seulbi An, Sunghwan Ahn, Joohyung Kim, MOMO: Mobile Object Manipulation Operator, Demonstration Sessions, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023), 2023
HD HYUNDAI ROBOTICS S1 (Based on B1)
https://service.hyundai-robotics.com/en/product_descriptions/hyundai_s1")
![[2023 Weekly KIMLAB] Development of a 3-DOF Interactive Modular Robot with Human-like Head Motions](https://i.ytimg.com/vi/TlIII8uEOL0/hqdefault.jpg "[2023 Weekly KIMLAB] Development of a 3-DOF Interactive Modular Robot with Human-like Head Motions
In the realm of robotics, the simplicity of Wall-Es head belies its remarkable capabilities: it mirrors human-like movements and possesses essential perception abilities, facilitated by an integrated camera. This design stands as the epitome of robotic head engineering. At KIMLAB, we have embraced this iconic design, transforming it into a modular robot head compatible with the PAPRAS docking mount. Utilizing the embedded IMU, our robot maintains its orientation even when the base is in motion. Our head module replicates several human head gestures, enriching interaction capabilities. Moreover, we have incorporated facial detection and tracking features, ensuring seamless Human-Robot Interaction for various applications.
Chaerim Moon, Sankalp Yamsani, Joohyung Kim, “Development of a 3-DOF Interactive Modular Robot with Human-like Head Motions,” IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), August 2023
Paper link: https://ieeexplore.ieee.org/document/10309462")
