Adaptive physical Human-Robot Interaction using Dynamical Systems [Clearpath Ridgeback] LASA 2026-09-19 | Adaptive physical Human-Robot Interaction using Dynamical Systems [Clearpath Ridgeback]
Action Contextualization: Adaptive Task Planning and Action Tuning using Large Language Models LASA 2024-10-29 | Large Language Models (LLMs) present a promising frontier in robotic task planning by leveraging extensive human knowledge. Nevertheless, the current literature often overlooks the critical aspects of robots' adaptability and error correction. This work aims to overcome this limitation by enabling robots to modify their motions and select the most suitable task plans based on the context.We introduce a novel framework to achieve action contextualization, aimed at tailoring robot actions to the context of specific tasks, thereby enhancing adaptability through applying LLM-derived contextual insights.Our framework integrates motion metrics that evaluate robot performances for each motion to resolve redundancy in planning.Moreover, it supports online feedback between the robot and the LLM, enabling immediate modifications to the task plans and corrections of errors.An overall success rate of 81.25% has been achieved through extensive experimental validation.Finally, when integrated with dynamical system (DS)-based robot controllers, the robotic arm-hand system demonstrates its proficiency in autonomously executing LLM-generated motion plans for sequential table-clearing tasks, rectifying errors without human intervention, and showcasing robustness against external disturbances.Our proposed framework also features the potential to be integrated with modular control approaches, significantly enhancing robots' adaptability and autonomy in performing sequential tasks in the real world.
Tube Acceleration: Robust Dexterous Throwing against Release Uncertainty (T-RO 2024) LASA 2024-04-12 | This video is the multimedia material of the publication "Tube acceleration: robust dexterous throwing against release uncertainty", Yang Liu and Aude Billard, IEEE Transaction on Robotics (T-RO), 2024. Open access: ieeexplore.ieee.org/document/10494917To throw generic objects that differ in mass, shape, material, and deformability, instead of case-by-case adaptation, what about synthesizing robust release motion to compensate for our ignorance of object properties?Tube Acceleration is a novel solution concept that drives the robot's end-effector to stay in the family of admissible object flying trajectories, ensuring robustness against release uncertainty. Through insightful observations, Tube Acceleration can be found via convex optimization within 50ms with bounded error. This facilitated robust throwing motion achieves a high accuracy and success rate at throwing a variety of complex deformable objects with dexterous throwing configurations. For planar throwing, the success rate exceeds 97%.
SAHR Project Presentation LASA 2024-03-05 | Meet the SAHR project and what was done in it. This is an insight in robotics research meant to bring a better understanding to a broad audience.
Avoidance of Concave Obstacles through Rotation of Nonlinear Dynamics LASA 2023-12-10 | The implementation of this video can be found under:github.com/hubernikus/nonlinear_obstacle_avoidanceThe open-access publication can be found under:arxiv.org/abs/2306.16160Controlling complex tasks in robotic systems, such as circular motion for cleaning or following curvy lines, can be dealt with using nonlinear vector fields. In this paper, we introduce a novel approach called rotational obstacle avoidance method (ROAM) for adapting the initial dynamics when the workspace is partially occluded by obstacles. ROAM presents a closed-form solution that effectively avoids star-shaped obstacles in spaces of arbitrary dimensions by rotating the initial dynamics towards the tangent space. The algorithm enables navigation within obstacle hulls and can be customized to actively move away from surfaces, while guaranteeing the presence of only a single saddle point on the boundary of each obstacle. We introduce a sequence of mappings to extend the approach for general nonlinear dynamics. Moreover, ROAM extends its capabilities to handle multi-obstacle environments and provides the ability to constrain dynamics within a safe tube. By utilizing weighted vector-tree summation, we successfully navigate around general concave obstacles represented as a tree-of-stars. Through experimental evaluation, ROAM demonstrates superior performance in terms of minimizing occurrences of local minima and maintaining similarity to the initial dynamics, outperforming existing approaches in multi-obstacle simulations. The proposed method is highly reactive, owing to its simplicity, and can be applied effectively in dynamic environments. This was demonstrated during the collision-free navigation of a 7 degree-of-freedom robot arm around dynamic obstacles.
Dynamical System Modulation For Obstacle Avoidance of Decentralized Swarms LASA 2023-09-27 | Efficient and safe multi-agent swarm coordinationin environments where humans operate, such as warehouses,assistive living rooms, or automated hospitals, is crucial foradopting automation. In this paper, we augment the obstacleavoidance algorithm based on dynamical system modulationfor a swarm of heterogeneous holonomic mobile agents. Asmooth prioritization is proposed to change the reactivity of theswarm towards the specific agents. Further, a soft decoupling ofthe initial agent’s kinematics is used to design an independentrotation control to ensure the agent reaches the desired positionand orientation simultaneously. This decoupling allowed theintroduction of a novel heuristic, the virtual drag. It minimizesthe disturbance influence an agent has when moving through itssurroundings. Additionally, the safety module adapts the velocitycommands from the dynamical system modulation to avoidcolliding trajectories between agents. The evaluation was per-formed in simulated assisted living and hospital environments.The prioritization successfully increased the minimum distancerelative to a moving agent. The safety module is observedto create collision-free dynamics where alternative methodsfail. Additionally, the repulsive nature of the safety moduleaugments the convergence rate, thus making the proposedmethod better applicable to dense real-world scenarios.The paper was published as:Douce, L. N., Menichelli, A., Huber, L., Bolotnikova, A., Paez-Granados, D., Ijspeert, A., & Billard, A. (2023, October). Agent Prioritization and Virtual Drag Minimization in Dynamical System Modulation For Obstacle Avoidance of Decentralized Swarms. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.Link to the paper:https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/632370/Swarm_DS_Furniture-IROS_2023.pdf?sequence=1Link to the source code to recreate simulations:github.com/epfl-lasa/autonomous_furniture
Bimanual dynamic grabbing and tossing of objects onto a moving target LASA 2023-09-16 | This video shows the experimental validation of a control framework that enables a bimanual robotic system to grab and toss objects onto a moving target. It shows the dual-arm coordination required by such a dynamic task and the fast adaptation abilities when the moving tossing target is subject to perturbations.
Decentralized Collision-free Autonomous Furniture LASA 2023-08-08 | This is a video using the algorithms described in following paper:ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9981583The source code repository can be found on:github.com/epfl-lasa/autonomous_furnitureIn order to facilitate and assist the indoor mobility of people with special needs, the classically static objects in the environment, such as furniture, can be rendered mobile. The need for efficient and safe autonomous coordination of a mobile furniture swarm arises. We present a closed-form approach for mobile furniture obstacle avoidance and navigation withinan indoor environment. The approach shows that each mobile furniture agent, defined by a polygonal surface, does not collide with any static or mobile obstacle (e.g., a person is moving around). All controllable mobile furniture converges towards a defined goal position and orientation. We showcase the application of this algorithm in simulation on mobile furniturefor smart environments. Results demonstrate that the proposed method can coordinate a swarm of mobile furniture to get out of the way of a mobile agent representing a person with limited mobility passing through the room while avoiding obstacles and converging towards a predefined target pose.
Online Active and Dynamic Object Shape Exploration with a Multi-fingered Robotic Hand LASA 2023-06-12 | The sense of touch can provide a robot with a wealth of information about the contact region when interacting with an unknown environment. Nevertheless, utilizing touch information to plan exploration paths and adjust robot posture to improve task efficiency remains challenging. This paper presents a novel approach for the online tactile surface exploration of unknown objects with a multi-degree of freedom robotic hand. We propose an exploration strategy that actively maximizes the entropy of the acquired data while dynamically balancing the exploration’s global knowledge and local complexity. We demonstrate that our method can efficiently control a multi-fingered robotic hand to explore objects of arbitrary shapes (e.g., with a handle, hole, or sharp edges). To facilitate efficient multi-contact exploration with a robotic hand, we offer an optimization-based planning algorithm that adapts the hand pose to the local surface geometry online and increases the kinematic configuration of each finger during exploration. Ultimately, we compared our approach to state of the art in a simulated environment. Experimental results indicate that our proposed methods can guide a multi-finger robotic hand to explore efficiently and smoothly, thereby reconstructing the unknown geometry of a variety of everyday objects, with significant improvements in data efficiency and finger compliance when compared to state-of-the-art approaches.
Reactive Collision-Free Motion Generation in Joint Space via Dynamical Systems and MPC (MPPI) LASA 2023-06-08 | Supplementary video for the corresponding paper submission.
Self-Correcting Quadratic Programming-Based Robot Control LASA 2023-04-17 | IEEE Transactions on Systems, Man, and Cybernetics: SystemsDigital Object Identifier 10.1109/TSMC.2023.3262954In this video, we introduce our Self-Correcting Quadratic Programming-Based controller, which enables robotic systems to undertake complex motions and interactions, even in situations where the underlying model of the environment and robot dynamics is unmodeled. Our approach augments traditional QP-based controllers with a learned residual inverse dynamics model and an adaptive control law, allowing for online adjustment to account for uncertainties and unforeseen disturbances. We extensively evaluate our method in several simulated robotic scenarios and validate our approach in physical robotic systems with unknown environmental models. Check out our paper for more information!ieeexplore.ieee.org/document/10102575open-access: http://infoscience.epfl.ch/record/301975?&ln=en
Motion Planning and Inertia Based Control Experiments - Impact Aware Manipulation LASA 2023-02-09 | ...
Demonstration of the robotic foot platform and the anastomosis 4-handed task in laparoscopy surgery LASA 2023-01-24 | ...
EMG-driven shared human-robot compliant control for in-hand object manipulation in hand prostheses LASA 2023-01-12 | Journal of Neural Engineeringiopscience.iop.org/article/10.1088/1741-2552/aca35fAuthors:Farshad Khadivar, Vincent Mendez, Carolina Correia, Iason Batzianoulis, Aude Billard, and Silvestro MiceraAbstract:Objective. The limited functionality of hand prostheses remains one of the main reasons behind the lack of its wide adoption by amputees. Indeed, while commercial prostheses can perform a reasonable number of grasps, they are often inadequate for manipulating the object once in hand. This lack of dexterity drastically restricts the utility of prosthetic hands. We aim at investigating a novel shared control strategy that combines autonomous control of forces exerted by a robotic hand with electromyographic (EMG) decoding to perform robust in-hand object manipulation. Approach. We conduct a three-day long longitudinal study with eight healthy subjects controlling a 16-degrees-of-freedom robotic hand to insert objects in boxes of various orientations. EMG decoding from forearm muscles enables subjects to move, proportionally and simultaneously, the fingers of the robotic hand. The desired object rotation is inferred using two EMG electrodes placed on the shoulder that record the activity of muscles responsible for elevation and depression. During the object interaction phase, the autonomous controller stabilizes and rotates the object to achieve the desired pose. In this study, we compare an incremental and a proportional shoulder-decoding method in combination with two state machine interfaces offering different levels of assistance. Main results. Results indicate that robotic assistance reduces the number of failures by 41% and, when combined with an incremental shoulder EMG decoding, leads to faster task completion time (median = 16.9 s), compared to other control conditions. Training to use the assistive device is fast. After one session of practice, all subjects managed to achieve tasks with 50% less failures. Significance. Shared control approaches that give some authority to an autonomous controller on-board the prosthesis are an alternative to control schemes relying on EMG decoding alone. This may improve the dexterity and versatility of robotic prosthetic hands for people with trans-radial amputation. By delegating control of forces to the prosthesis’ on-board control, one speeds up reaction time and improves the precision of force control. Such a shared control mechanism may enable amputees to perform fine insertion tasks solely using their prosthetic hands. This may restore some of the functionality of the disabled arm.
Fast Obstacle AvoidanceBased on Real-Time Sensing LASA 2022-12-31 | The open-access paper can be found here:ieeexplore.ieee.org/document/9999335The software used to program the robot is open-source:github.com/hubernikus/fast_obstacle_avoidanceHumans excel at navigating and moving through dynamic and complex spaces, such as crowded streets. For robots to do the same, it is crucial that they are endowed with highly reactive obstacle avoidance which is adept at partial and poor sensing. We address the issue of enabling obstacleavoidance based on sparse and asynchronous perception. The proposed control scheme combines a high-level input command provided by either a planner or a human operator with fast reactive obstacle avoidance (FOA). The sampling-based sensor data can be combined with an analytical reconstruction of the obstacles for real-time collision avoidance. Thus, we can ensure that the agent does not become stuck when a feasible path exists between obstacles. Our algorithm was evaluated experimentallyon static laser data from cluttered, indoor office environments. Additionally, it was used in shared-control mode in a dynamic and complex outdoor environment in the center of Lausanne. The proposed control scheme successfully avoided collisions in both scenarios. During the experiments, the controller took 1 millisecond to evaluate over 30000 data points.
Neural Joint Space Implicit Signed Distance Functions for Reactive Robot Manipulator Control LASA 2022-12-09 | Video accompanying paperieeexplore.ieee.org/document/9976191M. Koptev, N. Figueroa and A. Billard, "Neural Joint Space Implicit Signed Distance Functions for Reactive Robot Manipulator Control," in IEEE Robotics and Automation Letters, doi: 10.1109/LRA.2022.3227860.
Dual-arm control for coordinated fast grabbing and tossing of an object LASA 2022-06-21 | This video shows an application case of Dynamical Systems (DS)-based motion generation for dual-arm robotic arms to grab and release an object in one swipe: swift pick-and-toss. Unlike traditional pick-and-place operations that grab objects with quasi-zero contact velocity, in this application, the robots are allowed to grab and release objects with non-zero contact velocities. This yields a faster and more energy-efficient grab and release process than the traditional pick-and-place operation.
Avoiding Dense and Dynamic Obstacles in Enclosed Spaces: Application to Moving in Crowds LASA 2022-05-03 | The open-access paper can be found at:ieeexplore.ieee.org/abstract/document/9765824Find also the open source implementation:github.com/epfl-lasa/dynamic_obstacle_avoidanceThis video presents a closed-form approach to constraining a flow within a given volume and around objects. The flow is guaranteed to converge and to stop at a single fixed point. The obstacle avoidance problem is inverted to enforce that the flow remains enclosed within a volume defined by a polygonal surface. We formally guarantee that such a flow will never contact the boundaries of the enclosing volume or obstacles. It asymptotically converges towards an attractor. We further create smooth motion fields around obstacles with edges (e.g., tables). Both obstacles and enclosures may be time-varying, i.e., moving, expanding, and shrinking. The technique enables a robot to navigate within enclosed corridors while avoiding static and moving obstacles. It was applied on an autonomous robot (QOLO) in a static complex indoor environment and tested in simulations with dense crowds. The final proof of concept was performed in an outdoor environment in Lausanne. The QOLO robot successfully traversed a marketplace in the center of town in the presence of a diverse crowd with a non-uniform motion pattern.Reference to the paper / video with:Huber, Lukas, Jean-Jacques Slotine, and Aude Billard. "Avoiding Dense and Dynamic Obstacles in Enclosed Spaces: Application to Moving in Crowds." IEEE Transactions on Robotics (2022).
Safety Assessment for Personal Mobility Devices (PMD) and Service Robots LASA 2022-03-29 | Published work: Paez-Granados, D., Billard, A. Crash test-based assessment of injury risks for adults and children when colliding with personal mobility devices and service robots. Nature Scientific Reports 12, 5285 (2022). doi.org/10.1038/s41598-022-09349-9 Abstract: Autonomous mobility devices such as transport, cleaning, and delivery robots, hold a massive economic and social benefit. However, their deployment should not endanger bystanders, particularly vulnerable populations such as children and older adults who are inherently smaller and fragile. This study compared the risks faced by different pedestrian categories and determined risks through crash testing involving a service robot hitting an adult and a child dummy. Results of collisions at 3.1 m/s (11.1 km/h/6.9 mph) showed risks of serious head (14%), neck (20%), and chest (50%) injuries in children, and tibia fracture (33%) in adults. Furthermore, secondary impact analysis resulted in both populations at risk of severe head injuries, namely, from falling to the ground. Our data and simulations show mitigation strategies for reducing impact injury risks below 5% by either lowering the differential speed at impact below 1.5 m/s (5.4 km/h/3.3 mph) or through the usage of absorbent materials. The results presented herein may influence the design of controllers, sensing awareness, and assessment methods for robots and small vehicles standardization, as well as, policymaking and regulations for the speed, design, and usage of these devices in populated areas.
Contact-Initiated Shared Control for Foot Interfaces for 4 Handed Manipulation (2/7) LASA 2022-03-03 | Multimedia extension part of the paper published in the International Journal of Robotics Research.In this video we show the performance of the control strategy with force assistance but no coordination assistance. Amanhoud W, Hernandez Sanchez J, Bouri M, Billard A. Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces. The International Journal of Robotics Research. 2021;40(8-9):986-1014. doi:10.1177/02783649211017642 Can be found originally in doi.org/10.1177/02783649211017642.
Contact-Initiated Shared Control for Foot Interfaces for 4 Handed Manipulation (3/7) LASA 2022-03-03 | Multimedia Extension part of the paper published in the International Journal of Robotics Research.In this video show the performance of the control strategy with full assistance (force and coordination)Amanhoud W, Hernandez Sanchez J, Bouri M, Billard A. Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces. The International Journal of Robotics Research. 2021;40(8-9):986-1014. doi:10.1177/02783649211017642 Can be found originally in doi.org/10.1177/02783649211017642.
Contact-Initiated Shared Control for Foot Interfaces for 4 Handed Manipulation (4/7) LASA 2022-03-03 | Multimedia Extension part of the paper published in the International Journal of Robotics Research.In this video we show the performance of the control strategy with coordination assistance and no force assistance.Amanhoud W, Hernandez Sanchez J, Bouri M, Billard A. Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces. The International Journal of Robotics Research. 2021;40(8-9):986-1014. doi:10.1177/02783649211017642 Can be found originally in doi.org/10.1177/02783649211017642.
Contact-Initiated Shared Control for Foot Interfaces for 4 Handed Manipulation (5/7) LASA 2022-03-03 | Multimedia Extension part of the paper published in the International Journal of Robotics Research.In this video we explain the mapping between the foot platforms and the robotic arms, for the coordinated and uncoordinated control strategies.Amanhoud W, Hernandez Sanchez J, Bouri M, Billard A. Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces. The International Journal of Robotics Research. 2021;40(8-9):986-1014. doi:10.1177/02783649211017642 Can be found originally in doi.org/10.1177/02783649211017642.
Contact-Initiated Shared Control for Foot Interfaces for 4 Handed Manipulation (6/7) LASA 2022-03-03 | Multimedia Extension part of the paper published in the International Journal of Robotics Research.In this video we highlight some remarks related to human motor control during the four handed manipulation. Amanhoud W, Hernandez Sanchez J, Bouri M, Billard A. Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces. The International Journal of Robotics Research. 2021;40(8-9):986-1014. doi:10.1177/02783649211017642 Can be found originally in doi.org/10.1177/02783649211017642.
Contact-Initiated Shared Control for Foot Interfaces for 4 Handed Manipulation (7/7) LASA 2022-03-03 | Multimedia Extension part of the paper published in the International Journal of Robotics Research.In this video we highlight the importance of force assistance for the bipedal sub-task of the four-handed task. Amanhoud W, Hernandez Sanchez J, Bouri M, Billard A. Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces. The International Journal of Robotics Research. 2021;40(8-9):986-1014. doi:10.1177/02783649211017642 Can be found originally in doi.org/10.1177/02783649211017642.
Efficient Configuration Exploration in Inverse Dynamics Acquisition of Robotic Manipulators LASA 2022-03-01 | You can find the paper in the following link:ieeexplore.ieee.org/abstract/document/9561587/media#media
Credit Assignment Safety Learning from Human Demonstrations LASA 2021-11-01 | Published in AAAI AI-HRI 2021 Fall Symposium. Access arXiv paper at: arxiv.org/pdf/2110.04633.pdfAuthors: Ahalya Prabhakar and Aude Billard
Multimodal Sensory Learning for Real-time, Adaptive Manipulation LASA 2021-11-01 | Published in AAAI AI-HRI 2021 Fall Symposium. Access the arXiv paper at: arxiv.org/pdf/2110.04634.pdfAuthors: Ahalya Prabhakar, Stanislas Furrer, Lorenzo Panchetti, Maxence Perret and Aude Billard
Contact-Initiated Shared Control for Foot Interfaces for 4 Handed Manipulation (1/7) LASA 2021-09-11 | Multimedia Extension part of the paper published in the The International Journal of Robotics Research.In this video, we explain the four handed task and show the performance of the control strategy without assistance (baseline)Amanhoud W, Hernandez Sanchez J, Bouri M, Billard A. Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces. The International Journal of Robotics Research. 2021;40(8-9):986-1014. doi:10.1177/02783649211017642 Can be found originally in doi.org/10.1177/02783649211017642.
Unfreezing Social Navigation: DS based Compliance for Contact Control in Robot Navigation LASA 2021-07-20 | Presentation at Robotics Science and Systems (RSS-2021) - Workshop on Social Robot NavigationPaper: http://infoscience.epfl.ch/record/287442?&ln=enAbstract—Soon human-robot interactions in pedestrian areaswill be beyond the novelty effect with the deployment of delivery robots, autonomous personal mobility vehicles, and surveillance robots. Proxemics and other social rules guide these interactions, nonetheless, contactless navigation might be yielded infeasible by pedestrian density in certain areas or by adversarial pedestrians. In such scenarios, freezing the robot might go against bystanders safety and task completion might only be feasible under controlled contact interactions. We present a force-limited and obstacle avoidance integrated controller through a time-invariant dynamical system in a closed-loop force controller that let the robot react instantaneously and drive around pedestrians. Mitigating the risk of collision is done by modulating the velocity commands upon detecting a pedestrian and absorbing part of the contact force through active compliant control when the robot bumps inadvertently against the pedestrian.Work by; Diego Paez-Granados, Vaibhav Gupta, Aude BillardThis work was part of the EU CrowdBot Project.Contact: Diego Paez
Reactive Navigation for Service Robots: Collision Avoidance and Compliant Control LASA 2021-04-28 | This short clip shows the standing mobility vehicle "Qolo" with a reactive navigation control system for assisting a driver to navigate crowded environments through a safe interaction of obstacle avoidance and compliant control.Work presented as DEMO for EU H2020 CrowdBot ProjectWork by: Diego Paez-Granados David Gonon Vaibhav Gupta Aude BillardRedirecting Driver Support Controller: doi.org/10.1109/LRA.2021.3068660Method for Passive Dynamical Systems:doi.org/10.1109/icra40945.2020.9197509Hands-free Navigation System:doi.org/10.1109/IROS45743.2020.9340875Qolo robot:doi.org/10.1109/IROS.2018.8594199
Gaze tracking and object identification LASA 2021-02-01 | Development of real-time gaze-tracking method and object detection for identification of the object of interest. Each of eyes is recorded from one of the two cameras placed below the eye and pupils are identified using traditional image processing (conversion to grayscale, morphological transformation and contour extraction). The pupils coordinates are associated with the gaze coordinates in the world-camera by training two Support Vector Regressors, one for each of the coordinates (x,y). A state-of-art object detector is employed to identify the objects in the world-camera. The object that the gazed is fixated to is identified based on the properties of the bounding box and the gaze coordinatesResearcher:Iason BatzianoulisRelated code:- github.com/yias/eurekaRes/tree/master/gazeAcknowledgements:This research is supported by the Swiss National Science Foundation through the National Centre of Competence in Research Robotics and the Hasler Foundation.
Decoding of grasp types from muscle activity of individuals with transradial amputation LASA 2021-02-01 | Development of a decoder that addresses the noise that derives from the stochastic nature of electromyographic (EMG) signals and increase the efficiency of the pattern-recognition system of myoprostheses. First, we separate the reach-to-grasp motion into three phases, with respect to the kinematics of the amputated arm. We model the dynamic muscle contractions with Gaussian distributions over the different phases and the overall motion. A multivariate analysis of variance (MANOVA) on the muscle activity reveal significant differences among the motion phases. The introduction of phase-segmentation in real-time control of hand-myoprosthesis enables the detection of grasp intention early in the reaching motion and improves the overall classification accuracy.This method offers a more natural and intuitive control of prosthetic devices, as it will enable controlling grasp closure in synergy with the reaching motion. This work contributes to the decrease of delays between the user’s intention and the device response and improves the coordination of the device with the motion of the arm.Reference paper:Batzianoulis, I., Krausz, N., Simon, A. et al. Decoding the grasping intention from electromyography during reaching motions. J NeuroEngineering Rehabil 15, 57 (2018). doi.org/10.1186/s12984-018-0396-5Acknowledgements:This research is supported by Swiss National Science Foundation through the National Centre of Competence in Research Robotics and the United States’ National Institutes of Health
Robotic in-hand object-manipulation: Combining muscle-activity decoding with autonomous controller LASA 2021-01-24 | Using tactile sensors on the fingertips of a robot hand, we estimate the desired contact forces to the object for guaranteeing grasp stability. In addition, we generate trajectories of the fingertips by a dynamical system to perform a rotation of the object. We combine this autonomous controller with an electromyographic decoder for receiving high-level commands from a user. The user commands the fingers to open or close around the object and rotate it. Researchers:Iason Batzianoulis, Farshad Khadivar, Vincent Alexandre MendezAcknowledgements:This research is supported by Swiss National Science Foundation through the National Centre of Competence in Research Robotics and the Hasler Foundation.
Robot arm trajectory generation: Obstacle avoidance with computer vision LASA 2021-01-12 | Exploiting the benefits of state-of-art object detection, we identify and localize the obstacles in the workspace of a robot arm. Given the obstacles’ position, we model the obstacles as ellipsoids. Based on the location, size and shape of the ellipsoids, we modulate a linear dynamical system for generating the robot trajectories that avoid the obstacles when moving towards a target. Whilst the passive controller of the robot arm facilitates a safe interaction with humans, the generation of the robot trajectories from dynamical system enables a rapid reconfiguration of the trajectories after perturbations. Researcher:Iason BatzianoulisRelated code:- Object detection: github.com/yias/eurekaRes/tree/master/gaze- Robot arm motion generator: github.com/yias/robot_arm_motion- Kuka LWR control interface: github.com/epfl-lasa/kuka-lwr-ros- Mask R-CNN: github.com/matterport/Mask_RCNNRelated work:- Mask R-CNN, arxiv.org/abs/1703.06870- A Dynamical System Approach to Realtime Obstacle Avoidance, https://infoscience.epfl.ch/record/174759?ln=en- Passive Interaction Control with Dynamical Systems, https://infoscience.epfl.ch/record/221276?ln=enAcknowledgements:This research is supported by the Swiss National Science Foundation through the National Centre of Competence in Research Robotics and the Hasler Foundation.
Learning dynamical systems with bifurcations LASA 2020-12-14 | doi.org/10.1016/j.robot.2020.103700Highlights:•Learning bifurcation parameters to encode a dynamical system with discrete and periodic dynamics.•Exploiting Hopf bifurcations to smoothly switch across periodic and non-periodic phases.•Using diffeomorphism to acquire nonlinear limit cycles from demonstration.#bifurcation #nonlinear #limit_cycle #supervised #learning #robotics #LfD #demonstration #diffeomorphism #gmm #pca #lasa #kuka #humanoid
Foot Control of Surgical Laparoscopic Gripper via 5 DoF Haptic Robotic Platform LASA 2020-11-09 | Foot haptic interface of 5 degrees of freedom to control a surgical robotic manipulator. In this demo we focus on controlling a laparoscopic forceps to grasp objects with the foot, and developed a shared control strategy to improve, completeness, efficiency and reduce fatigue.#robotics #hri #haptics #epfl #lasa #rehassist #sti #foot #5dof #surgery #gripper #laparoscopy #hasler #future #technology #science #shared #control #surgical training #simulation
Reactive Controller for a Convex Non-holonomic Robot to Travel in Crowds LASA 2020-10-22 | This video documents a method for Obstacle Avoidance in Crowds.Full paper in Open-Access: doi.org/10.1109/LRA.2021.3068660 Work done by David Gonon, Diego Paez-Granados, and Aude Billard at LASA, EPFL, Switzerland. We thank all our co-workers appearing in the video for their help.This work's source code (C++ w/ ROS) is publicly available at: github.com/epfl-lasa/rds
Force Adaptation in Contact Tasks with Dynamical Systems LASA 2020-06-30 | Presentation video at ICRA 2020
Adaptive Co-Manipulation (Master-Slave Strategy) LASA 2020-03-09 | Complementary video for ICRA 2020 paper "A Dynamical System Approach for Adaptive Grasping, Navigation and Co-Manipulation with Humanoid Robots"Authors: Nadia Figueroa, Salman Faraji, Mikhail Koptev and Aude BillardWebsite: epfl-lasa.github.io/iCub-Assistant
Shared human–robot proportional control of a dexterous myoelectric prosthesis LASA 2019-12-10 | Presenting our research on shared control algorithm for prosthetic hands. The novelty of the approach lies in it's ability to improve grasp around the object using information from pressure sensors integrated with the hand. The algorithm allows the subject to control robotic hand's fingers through muscle signals (EMG) decoding and in addition improves the grasp on the object whenever a contact is detected. Please find the accompanying paper published in Nature Machine Intelligence in the link here: nature.com/articles/s42256-019-0093-5
Four Handed Laparoscopic Robotic-Surgery with Foot Haptic Interfaces LASA 2019-10-30 | In conventional minimally invasive surgery, the surgeon requires the support of a trained assistant. However communication (speech & gestures) is often subjective, which could pose an efficiency problem. Our research aims at endowing the surgeon with the capacity to complement the work of both hands with two robotic arms that are controlled via haptic foot interfaces of five degrees of freedom. One foot controls an endoscope (camera) while the other foot controls a complementary retractor ( in this demo resembled with a hook). The robots are teleoperated using impedance control modulated via linear dynamical systems.
Four-Arm Manipulation via Feet Robotic Interfaces LASA 2019-10-30 | Presented at AAAI Fall Symposium Series. AAAI 2019. arxiv.org/abs/1909.04993
DS-based Adaptive Grasping, Navigation and Co-Manipulation with Humanoid Robots LASA 2019-09-22 | Supplementary video for the following paper submitted to ICRA 2020: "A Dynamical System Approach for Adaptive Grasping, Navigation and Co-Manipulation with Humanoid Robots".Abstract: In this paper, we present an integrated framework that provides compliant control of an iCub humanoid robot and adaptive reaching, grasping, navigating and co-manipulating. We use state-dependent dynamical systems (DS) to (i) coordinate and drive the robot's hands an object in position and orientation to grasp an object, and (ii) drive the robot's base while walking/navigating. The use of DS as motion generators allows us to adapt smoothly as the object moves and to re-plan on-line motion of arms and body to reach the object's new location. The desired trajectory generated by the DS are used in combination with a whole-body compliant control strategy that absorbs perturbations while walking and offers compliant behaviors for grasping and manipulation tasks. The desired dynamics for arm and body can be learned from demonstrations. By integrating these components we achieve unprecedented adaptive behaviors for whole body manipulation. We showcase this in simulations and real-world experiments where iCub robots (i) walk-to-grasp objects, (ii) follow a human (or another iCub) through interaction and (iii) learn to navigate or co-manipulate an object from human guided demonstrations; whilst being robust to changing targets and perturbations.Authors: Nadia Figueroa, Salman Faraji, Mikhail Koptev and Aude BillardWebpage: epfl-lasa.github.io/iCub-Assistant
Locally Active Globally Stable Dynamical Systems: Experiments on iCub for Navigation/Co-Manipulation LASA 2019-07-03 | Supplementary video for the following paper submitted to IJRR:"Locally Active Globally Stable Dynamical Systems: Theory, Learning and Experiments"Abstract: State-dependent Dynamical Systems (DS) offer adaptivity, reactivity and robustness to perturbations in motion planning and physical human-robot interaction tasks. Learning DS-based motion representations from non-linear reference trajectories is an active research area in robotics. Most approaches focus on learning DS that can (i) accurately mimic the demonstrated motion, while (ii) ensuring convergence to the target; i.e. they are globally asymptotically (or exponentially) stable. When the objective of a task is to reach a target while tracking a reference trajectory, if perturbations are present a compliant robot guided with a DS will ultimately reach the target, albeit failing to track the reference trajectory. In this work, we propose a novel DS formulation referred to as the locally active globally stable DS (LAGS-DS). The LAGS-DS provides both global convergence and stiffness-like symmetric attraction behaviors around a reference-trajectory in regions of the state-space where trajectory tracking is important. This allows for a unified approach towards motion and impedance encoding in a single motion model without parameterizing the controller of the robot; i.e. stiffness is embedded in the motion model. To learn LAGS-DS from demonstrations we propose a learning strategy based on Bayesian non-parametric Gaussian mixture models, Gaussian processes and a sequence of constrained optimization problems that ensure estimation of stable DS parameters via Lyapunov theory. This novel DS and learning scheme is extensively validated on writing tasks with a KUKA LWR manipulator, as well as navigation and co-manipulation scenarios with iCub humanoid robots.Contact: Nadia Figueroa [nbfigueroa.github.io]Funding: Cogimon [https://cogimon.eu/]
Locally Active Globally Stable Dynamical Systems: Experiments on KUKA LWR for Writing Tasks LASA 2019-07-02 | Supplementary video for the following paper submitted to IJRR:"Locally Active Globally Stable Dynamical Systems: Theory, Learning and Experiments"Abstract: State-dependent Dynamical Systems (DS) offer adaptivity, reactivity and robustness to perturbations in motion planning and physical human-robot interaction tasks. Learning DS-based motion representations from non-linear reference trajectories is an active research area in robotics. Most approaches focus on learning DS that can (i) accurately mimic the demonstrated motion, while (ii) ensuring convergence to the target; i.e. they are globally asymptotically (or exponentially) stable. When the objective of a task is to reach a target while tracking a reference trajectory, if perturbations are present a compliant robot guided with a DS will ultimately reach the target, albeit failing to track the reference trajectory. In this work, we propose a novel DS formulation referred to as the locally active globally stable DS (LAGS-DS). The LAGS-DS provides both global convergence and stiffness-like symmetric attraction behaviors around a reference-trajectory in regions of the state-space where trajectory tracking is important. This allows for a unified approach towards motion and impedance encoding in a single motion model without parameterizing the controller of the robot; i.e. stiffness is embedded in the motion model. To learn LAGS-DS from demonstrations we propose a learning strategy based on Bayesian non-parametric Gaussian mixture models, Gaussian processes and a sequence of constrained optimization problems that ensure estimation of stable DS parameters via Lyapunov theory. This novel DS and learning scheme is extensively validated on writing tasks with a KUKA LWR manipulator, as well as navigation and co-manipulation scenarios with iCub humanoid robots.Contact: Nadia Figueroa [nbfigueroa.github.io]Funding: Cogimon [https://cogimon.eu/]
Supplementary Video for Geometric Invariant Segmentation and Action Discovery LASA 2019-06-06 | This video shows unstructured and unconstrained demonstrations of 4 complex sequential tasks used in [1] to evaluate the geometric invariant segmentation and action discovery algorithm.Tasks evaluated: 1) Wiping a door fender, 2) Wiping a Rim Cover, 3) Dough Rolling and 4) Zucchini Peeling[1] Figueroa N. and Billard A., "Geometric Invariance of Covariance Matrices for Unsupervised Clustering, Segmentation and Action Discovery in Robotic Applications". Submitted to Journal of Machine Learning Research (JMLR).
![Adaptive physical Human-Robot Interaction using Dynamical Systems [Clearpath Ridgeback]](https://i.ytimg.com/vi/7BjHhV-BkwE/hqdefault.jpg)