Learning Highly Dynamic Skills Transition for Quadruped Traversing Through Constrained Space Arc Lab 2025-09-25 | Robot performs lifelike, explosive agility through constrained obstaclesPaper link: advanced.onlinelibrary.wiley.com/doi/full/10.1002/adrr.202500032Although legged animals are capable of performing explosive motions while traversing confined spaces, replicating this behaviorin quadrupedal robots has been a longstanding challenge. Here, we propose a hierarchical reinforcement learning pipeline thatempowers the robots to perform aggressive locomotion through constrained obstacles—a narrow gate. The imitation learning technique is used to train the low-level policy, which mimics the behaviors of real animals and forms a set of diverse skills. The high-level controller, having an awareness of the capability of low-level skills and acquiring the gate information via vision-based detection, determines the suitable maneuvers with collision-free trajectories to traverse it dynamically. Notably, we also verify that this framework can be extended to other highly dynamic tasks. This is one of the first works that perform autonomous and agile aerial gate traversal tasks on ground-walking robots, extending the lifelike agility of legged robots to match that of their biological counterparts.
MGDP: Mastering a Generalized Depth Perception Model for Quadruped Locomotion Arc Lab 2026-03-27 | Title: MGDP: Mastering a Generalized Depth Perception Model for Quadruped LocomotionAbstract:Perception-based Deep Reinforcement Learning (DRL) controllers demonstrate impressive performance on challenging terrains. However, existing controllers still face core limitations, struggling to achieve both terrain generality and platform transferability, and are constrained by high computational overhead and sensitivity to sensor noise. To address these challenges fundamentally, we propose a generalized control framework: Mastering a Generalized Contrastive Perception Model (MGDP). We leverage NVIDIA Warp to enable efficient parallel computation of depth images, thereby mitigating the inherent high computational cost. MGDP extracts low-dimensional terrain feature representations from multi-modal inputs (depth images and height maps) and integrates an explicit depth map denoising mechanism. This process not only facilitates effective decoupling of perception from dynamics but also significantly reduces the memory. Furthermore, we design terrain-adaptive reward functions that modulate penalty strengths according to terrain characteristics, enabling the policy to acquire complex locomotion skills (e.g., climbing, jumping, crawling, squeezing) in a single training stage without relying on distillation. Experimental results demonstrate that MGDP not only endows the policy with superior cross-terrain generalization capability but also enables fast and efficient fine-tuning across diverse quadruped robot morphologies via its pre-trained, dynamics-decoupled perception model. This vigorously advances the development of unified, efficient, and generalized frameworks for quadrupedal locomotion control.
MILD: Tractable Terrain Modeling for Learning Improved Bipedal Locomotion on Deformable Surfaces Arc Lab 2026-01-26 | Paper link: ieeexplore.ieee.org/document/11302803Enabling robots to walk on yielding terrain is vital for applications ranging from disaster response to planetary exploration. While bipedal robots hold immense potential, their locomotion on deformable surfaces remains limited as current simulators fail to capture the spatiotemporal heterogeneity of such yielding substrates. We present MILD, featuring a physics-grounded discrete-element contact solver that accurately simulates spatially varying foot-terrain interactions. Complementing this model, we train a terrain-aware locomotion controller via deep reinforcement learning with latent modulation and proprioceptive estimation.Quantitative comparisons against state-of-the-art methods show our approach generates more diverse and realistic contact scenarios during training, resulting in controllers that exhibit natural adaptation on real deformable surfaces. Through hardware experiments, we demonstrate the system's capability for online terrain identification and adaptation across a wide range of surface stiffness.
[RA-L25] Breaking the Static Assumption: A Dynamic-Aware LIO Framework Arc Lab 2025-12-15 | Breaking the Static Assumption: A Dynamic-Aware LIO Framework Via Spatio-Temporal Normal AnalysisIEEE page：ieeexplore.ieee.org/document/11207655Arxiv：arxiv.org/abs/2510.22313v1Github：github.com/thisparticle/btsa
TORM: Transparent Objects Reconstruction and Manipulation with Multi-View Segmentation Arc Lab 2025-12-01 | TORM: Transparent Objects Reconstruction and Manipulation with Multi-View SegmentationAccepted by RA-L 2025IEEE page: doi.org/10.1109/LRA.2025.3632680
Lasso Gripper: A String Shooting-retracting Mechanism for Shape-Adaptive Grasping Arc Lab 2025-11-06 | Lasso Gripper: A String Shooting-retracting Mechanism for Shape-Adaptive GraspingAccepted by ROBIO 2025arxiv page: arxiv.org/abs/2506.14163
MARG: MAstering Risky Gap Terrains for Legged Robots with Elevation Mapping Arc Lab 2025-09-24 | Title: MARG: MAstering Risky Gap Terrains for Legged Robots with Elevation MappingAbstract:Deep Reinforcement Learning (DRL) controllers for quadrupedal locomotion have demonstrated impressive performance on challenging terrains, allowing robots to execute complex skills such as climbing, running, and jumping. However, existing blind locomotion controllers often struggle to ensure safety and efficient traversal through risky gap terrains, which are typically highly complex, requiring robots to perceive terrain information and select appropriate footholds during locomotion accurately. Meanwhile, existing perception-based controllers still present several practical limitations, including a complex multi-sensor deployment system and expensive computing resource requirements. This paper proposes a DRL controller named MAstering Risky Gap Terrains (MARG), which integrates terrain maps and proprioception to dynamically adjust the action and enhance the robot's stability in these tasks. During the training phase, our controller accelerates policy optimization by selectively incorporating privileged information (e.g., center of mass, friction coefficients) that are available in simulation but unmeasurable directly in real-world deployments due to sensor limitations. We also designed three foot-related rewards to encourage the robot to explore safe footholds. More importantly, a terrain map generation (TMG) model is proposed to reduce the drift existing in mapping and provide accurate terrain maps using only one LiDAR, providing a foundation for zero-shot transfer of the learned policy. The experimental results indicate that MARG maintains stability in various risky terrain tasks.
Learning Aggressive Animal Locomotion Skills for Quadrupedal Robots Solely from Monocular Videos Arc Lab 2025-08-11 | Title: Learning Aggressive Animal Locomotion Skills for Quadrupedal Robots Solely from Monocular VideosJournal: npj RoboticsAbstract: The quest for agile quadrupedal robots is limited by handcrafted reward design in reinforcement learning. While animal motion capture provides 3D references, its cost prohibits scaling. Video learning provides an efficient alternative yet suffers from 2D limitations and joint tracking failures during explosive motions. We address this with a novel video-based framework. First, robust 2D pose estimation constructs a skeleton graph model, enabling Kalman filter-based joint position fusion. Next, a spatial-temporal graph convolution network aggregates spatial pose features via graph convolutions and temporal dynamics through dilated convolutions, recovering 3D joint trajectories. These trajectories are mapped to the robot’s joint space to formulate generative imitation learning. Real-robot deployment demonstrates successful learning of complex motions: gallop (high-speed), tripod (fault-tolerant), bipedal (quadrupedally challenging), and backflip. The proposed framework significantly advances robotic locomotion capabilities.
Learning Autonomous and Safe Quadruped Traversal of Complex Terrains Using Multi-Layer Elevation Map Arc Lab 2025-07-29 | Title: Learning Autonomous and Safe Quadruped Traversal of Complex Terrains Using Multi-Layer Elevation MapsAbstract:Legged robots hold great promise for agile and flexible mobility across diverse and unstructured terrains, inspired by the remarkable adaptability of bipeds and quadrupeds in nature. However, achieving robust autonomous locomotion in cluttered and complex environments remains a significant challenge. In this work, we present a hierarchical control framework for quadrupedal robots that enables safe and autonomous traversal of cluttered terrains. Central to our approach is a novel multi-layer elevation map representation, which is generalized enough to capture a wide range of terrains. To further improve policy generalization and maneuverability, we incorporate terrain augmentation, knowledge distillation, and carefully designed reward functions. Extensive simulation experiments demonstrate that each component contributes to improved policy generalization, and that our terrain representation is more efficient and informative than existing alternatives. By training a terrain compressor in simulation, we successfully deploy our system on a low-cost quadrupedal robot in real-world environments, showcasing the practicality and robustness of our approach.
Learning stable bipedal locomotion skills for quadrupedal robots on challenging terrains Arc Lab 2025-07-10 | Reinforcement learning has made remarkable strides in advancing quadrupedal locomotion. However, achieving bipedal locomotion for quadrupedal robots remains extremely challenging due to less contact with the surface. Additionally, during the transition from quadrupedal to bipedal locomotion, the body axis shifts from horizontal to vertical, and the center-of-mass rises suddenly. Here, we present TumblerNet, a deep reinforcement learning controller that enables robust bipedal locomotion for quadrupedal robots. Our proposed framework features an estimator that estimates the center-of-mass and center-of-pressure vector and rewards based on this vector, which allows the learning controller to monitor and maintain the balance of the robot during bipedal locomotion. As such, the proposed framework, although only trained on flat ground in simulation, can be directly deployed in a real robot on various terrains without additional training. The proposed framework exhibits exceptional robustness against various challenging terrains (uneven and soft terrains) and external disturbances, with automatic fall recovery.
OmniNet: Omnidirectional Jumping Neural Network with Height-awareness for Quadrupedal Robots Arc Lab 2025-06-11 | Title: OmniNet: Omnidirectional Jumping Neural Network with Height-awareness for Quadrupedal RobotsArticle Link: recently received by IEEE Robotics and Automation letters (RA-L). Paper link coming soon.In the robotics community, it has been a longstanding challenge for quadrupeds to achieve highly explosive movements similar to their biological counterparts. In this work, we introduce a novel training framework that achieves height-aware and omnidirectional jumping for quadrupedal robots. To facilitate the precise tracking of the user-specified jumping height, our pipeline concurrently trains an estimator that infers the robot and its end-effector states in an online fashion. Besides, a novel reward is involved by solving the analytical inverse kinematics with pre-defined end-effector positions. Guided by this term, the robot is empowered to regulate its gestures during the aerial phase. In the comparative studies, we verify that this controller can not only achieve the longest relative forward jump distance, but also exhibit the most comprehensive jumping capabilities among all the existing jumping controllers.Keywords: Legged Robots, Reinforcement Learning, Machine Learning for Robot Control
FR-Net: Learning Robust Quadrupedal Fall Recovery on Challenging Terrains through Mass-Contact Predi Arc Lab 2025-05-07 | Title: FR-Net: Learning Robust Quadrupedal Fall Recovery on Challenging Terrains through Mass-Contact PredictionArticle Link: doi.org/10.1109/LRA.2025.3569117Recovering from falls on complex terrains is a significant challenge for quadrupedal robots. This paper introduces FR-Net, a recovery framework based on mass-contact prediction, enabling quadrupedal robots to recover quickly from arbitrary fall poses in diverse environments. The main contributions of this work are threefold: 1. A Mass-Contact Predictor Network is proposed to estimate the robot's state using limited sensory inputs, generating robust recovery strategies across various terrains. 2. A safety-driven reward function is designed to ensure safe recovery motions, even on steep terrains, avoiding dangerous rolling behaviors. 3. The cross-platform generalization capability of FR-Net is validated through simulations and real-world experiments, demonstrating outstanding performance on the Go2 robot and in multiple challenging scenarios.Keywords: Reinforcement Learning, Quadrupedal Robots, Failure Detection and Recovery
DEIO: Learning-based Event-Inertial Odometry Arc Lab 2025-04-22 | Github: github.com/arclab-hku/DEIOWebsite: kwanwaipang.github.io/DEIOThe first deep event-inertial odometry framework, which combines learning-based event data association with traditional nonlinear graph-based optimization. Even though training on synthetic data, it still outperforms over 20 state-of-the-art methods across 10 challenging real-world event benchmarks.
Spatial-Temporal Optimization of Foot Trajectories for Controlled Football Kicking in Bipedal Robots Arc Lab 2025-04-14 | Humanoid robot soccer poses several challenges, particularly in maintaining system stability during aggressive kicking motions while achieving precise ball trajectory control.Current solutions, whether traditional position-based control methods or reinforcement learning (RL) approaches, exhibit significant limitations. Model predictive control (MPC) is a prevalent approach for ordinary quadruped and biped robots. While MPC has demonstrated advantages in dynamic motion control for legged robots, existing studies often oversimplify the leg swing progress, relying merely on simple trajectory interpolation methods. This severely constrains the foot's environmental interaction capability, which is particularly detrimental for tasks such as ball kicking.This study innovatively adapts the spatial-temporal trajectory planning method, which has been successful in drone applications, to bipedal robotic systems. The proposed approach autonomously generates foot trajectories that satisfy constraints on target kicking position, velocity, and acceleration while simultaneously optimizing swing phase duration. Experimental results demonstrate that the optimized trajectories closely mimic human kicking behavior, featuring a backswing motion. Simulation experiments confirm the algorithm's efficiency, with trajectory planning times under 1 ms, and its reliability, achieving nearly 100 % task completion accuracy when the soccer goal is within the range of -90° to 90°.
Stable Imitation of Multi-Gait and Bipedal Motions for Quadrupedal Robots over Uneven Terrains Arc Lab 2025-03-11 | Imitation learning has proven effective for quadrupedal robot control by demonstrating capabilities such as precise motion tracking and executing complex maneuvers like backflips and rapid trotting while achieving faster convergence compared to traditional reinforcement learning. Despite these advancements, challenges remain, existing methods struggle with stability, generalization, and real-world adaptation, particularly in the task of dealing with disturbances and varying terrains. To overcome these obstacles, we propose a novel framework that integrates a Long Short Term Memory (LSTM) data regularization module, model-based stability reward functions, and an interpolation-based dataset augmentation technique. This comprehensive approach refines motion data, enhances stability, and improves the robot's ability to handle disturbances. Our method supports a variety of data inputs from different robots, simulators, and controllers, and enables the robot to perform diverse motions, including tripod walking, trotting, pacing, bounding, pronking, and even bipedal walking, while maintaining stability over a variety of uneven terrains. By implementing variable frequency imitation learning, we significantly improve the generalization of learned behaviors across different gaits and speeds. By involving the dynamics modeled reward, we enhance the robot's adaptability and robustness in dynamic environments. These contributions mark an important step towards deploying quadrupedal robot imitation learning policies in more unpredictable and challenging real-world scenarios.
Flying in Highly Dynamic Environments with End-to-end Learning Approach Arc Lab 2025-02-06 | Obstacle avoidance for unmanned aerial vehicles like quadrotors is a popular research topic. Most existing research focuses only on static environments, and obstacle avoidance in environments with multiple dynamic obstacles remains challenging. This paper proposes a novel deep-reinforcement learning-based approach for the quadrotors to navigate through highly dynamic environments. We propose a lidar data encoder to extract obstacle information from the massive point cloud data from the lidar. Multi frames of historical scans will be compressed into a 2-dimension obstacle map while maintaining the obstacle features required. An end-to-end deep neural network is trained to extract the kinematics of dynamic and static obstacles from the obstacle map, and it will generate acceleration commands to the quadrotor to control it to avoid these obstacles. Our approach contains perception and navigating functions in a single neural network, which can change from a navigating state into a hovering state without mode switching. We also present simulations and real-world experiments to show the effectiveness of our approach while navigating in highly dynamic cluttered environments.
A dexterous and compliant aerial continuum manipulator for cluttered and constrained environments Arc Lab 2025-01-27 | Nature Communications paperPaper link: nature.com/articles/s41467-024-55157-2PDF link: rdcu.be/d7B8CThis paper proposes a highly dexterous and compliant aerial continuum manipulator (Aerial Elephant Trunk). We have proposed the design, designed the shape estimation method, developed a feedback controller, and proposed a whole-body motion planning module such that the UAV and the continuum manipulator could carry out tasks as a whole.AET can perform various challenging aerial manipulation tasks, including but not limited to:1) grasping object of various sizes and shapes;2) traversing constrained pipelines with various shapes;3) aerial writing/painting;4) performing manipulation in various complex environments.#robot #drone #uav #airplane #robotics #artificialintelligence #technology #learning #deeplearning @UAVfutures @fpvdrones @meninododronefpv @Thedroneracingleague @RobotFutureAI
AET performs manipulation in complex environments with obstacles. Arc Lab 2025-01-24 | This video shows that AET is capable of performing manipulation in various complex environments with obstacles.#artificialintelligence #robot #drone #airplane #fpv #ai @Thedroneracingleague @meninododronefpv @UAVfutures @fpvdrones @RobotFutureAI @Incrediblestories-1 @DR1Racing @DronesFPV
AET pipeline traversal Arc Lab 2025-01-24 | This video shows that AET is capable of traversing constrained pipelines with various shapes.#robot #drone #fpv #airplane #artificialintelligence #artificialintelligence @Thedroneracingleague @UAVfutures @RobotFutureAI @meninododronefpv @Incrediblestories-1
AET aerial grasping Arc Lab 2025-01-24 | This video shows that AET can grasp various objects with different sizes and shapes.#artificialintelligence #robot #drone #airplane #fpv #ai @Thedroneracingleague @meninododronefpv @UAVfutures @RobotFutureAI @Incrediblestories-1 @fpvdrones
AET aerial shape deformation Arc Lab 2025-01-24 | Paper: Peng, R., Wang, Y., Lu, M. et al. A dexterous and compliant aerial continuum manipulator for cluttered and constrained environments. Nat Commun 16, 889 (2025).Paper link: rdcu.be/d7czr AET is a dexterous aerial continuum manipulator that can achieve various aerial manipulation tasks in complex environments.AET can: 1) grasp various objects with different sizes and shapes;2) pass through constrained pipelines with various shapes;3) perform manipulator in various complex environments with obstacles.This video shows the aerial shape deformation of AET.#drone #robot #robotics
DAPTP: Distributed Awareness Planner using Time Potential for Dynamic Target Search Arc Lab 2024-09-12 | ...
FAPP: Fast and Adaptive Perception and Planning for UAVs in Dynamic Cluttered Environments Arc Lab 2024-05-29 | Fast and adaptive perception and planning for UAVs in dynamic cluttered environments.Perception; motion planning; UAVs; dynamic obstacle avoidance; object segmentation; object detection and tracking; point cloud
Multi-Agent Search and Rescue for Unknown Located Dynamic Targets in Completely Unknown Environments Arc Lab 2024-03-29 | Multi-Agent Search and Rescue (MASAR) tasks, challenged by unknown environments and the unpredictable movements of unknown dynamic targets, suffer from inefficiencies in traditional map coverage techniques which require repeated sweeps. Addressing this, our study introduces a novel MASAR framework based on Multi-Agent Reinforcement Learning (MARL), featuring innovative elements like state, reward, and network structure design, alongside a Heterogeneous Curriculum Training algorithm and a hybrid decision mechanism. These components collectively enhance performance in dynamic environments, improve model generalization, and mitigate issues like sparse rewards and policy bias. In grid map simulations, our approach, HMA-SAR (Heterogeneous Multi-Agent Search and Rescue Framework), demonstrated consistent superiority over the traditional frontier-based method and other MARL algorithms, in metrics such as success rate, steps count, and the number of targets fetched. The practical applicability of our approach was further validated through experiments in Gazebo and real-world scenarios. Additionally, scalability tests in grid maps revealed substantial improvements in success rates and task completion times with increased agent deployment
MorAL: Learning Morphologically Adaptive Locomotion Controllerfor Quadruped on Challenging Terrain Arc Lab 2024-03-02 | Accepted by IEEE Robotics and Automation Letters (RA-L)
A Tendon-driven Continuum Manipulator with Robust Shape Estimation by Multiple IMUs Arc Lab 2024-02-04 | Accepted for publication in the IEEE Robotics and Automation Letters (RA-L)
Mobile Robot Collision Avoidance Based on Deep Reinforcement Learning With Motion Constraints Arc Lab 2023-12-30 | Utilizing neural networks and reinforcement learning algorithms, Deep Reinforcement Learning (DRL) has become increasingly important for collision avoidance and navigation in mobile robotics. This approach allows agents to navigate through obstacles and reach destinations in complex and dynamic environments effectively. However, existing research has encountered limitations, such as being assisted by traditional algorithms, the need for multi-sensor data fusion, and application in overly simplified or non-random scenarios. To address these challenges, this letter presents a novel end-to-end DRL framework. Leveraging limited observations from the depth camera on the robot and incorporating differential robot motion constraints, our framework adeptly navigates and circumvents obstacles in dense, random, and mapless environments. Network model parameters are refined via the Proximal Policy Optimization (PPO) algorithm. Subsequent evaluations and tests against existing algorithms on the BARN_Dataset map set in simulation underscore its high success rate and generalizability in complex static environments. Furthermore, the model was deployed on a TurtleBot2 robot with a depth camera and tested in various real-world scenarios, including static, complex, and random environments, as well as with moving pedestrians, demonstrating its feasibility and robustness with exceptional results.
EVI-SAM: Tightly-coupled Event-Visual-Inertial State Estimation and 3D Dense Mapping Arc Lab 2023-12-14 | Event cameras are bio-inspired, motion-activated sensors that demonstrate substantial potential in handling challenging situations, such as motion blur and high-dynamic range. In this paper, we proposed EVI-SAM to tackle the problem of 6 DoF pose tracking and 3D reconstruction using a monocular event camera. A novel event-based hybrid tracking framework is designed to estimate the pose, leveraging the robustness of feature matching and the precision of direct alignment. Specifically, we develop an event-based 2D-2D alignment to construct the photometric constraint, and tightly integrate it with the event-based reprojection constraint. The mapping module recovers the dense and colorful depth of the scene through the image-guided event-based mapping method. Subsequently, the appearance, texture, and surface mesh of the 3D scene can be reconstructed by fusing the dense depth map from multiple viewpoints using truncated signed distance function (TSDF) fusion. To the best of our knowledge, this is the first non-learning work to realize event-based dense mapping. Numerical evaluations are performed on both publicly available and self-collected datasets, which qualitatively and quantitatively demonstrate the superior performance of our method. Our EVI-SAM effectively balances accuracy and robustness while maintaining computational efficiency, showcasing superior pose tracking and dense mapping performance in challenging scenarios.
Fast and adaptive perception and planning for UAVs in dynamic cluttered environments Arc Lab 2023-12-12 | Fast and adaptive perception and planning for UAVs in dynamic cluttered environments.Perception; motion planning; UAVs; dynamic obstacle avoidance; object segmentation; object detection and tracking; point cloud
ECMD: An Event-Centric Multisensory Driving Dataset for SLAM Arc Lab 2023-10-31 | Leveraging multiple sensors enhances complex environmental perception and increases resilience to varying luminance conditions and high-speed motion patterns, achieving precise localization and mapping. This paper proposes, ECMD, an event-centric multisensory dataset containing 81 sequences and covering over 200 km of various challenging driving scenarios including high-speed motion, repetitive scenarios, dynamic objects, etc. ECMD provides data from two sets of stereo event cameras with different resolutions (640*480, 346*260), stereo industrial cameras, an infrared camera, a top-installed mechanical LiDAR with two slanted LiDARs, two consumer-level GNSS receivers, and an onboard IMU. Meanwhile, the ground-truth of the vehicle was obtained using a centimeter-level high-accuracy GNSS-RTK/INS navigation system. All sensors are well-calibrated and temporally synchronized at the hardware level, with recording data simultaneously. We additionally evaluate several state-of-the-art SLAM algorithms for benchmarking visual and LiDAR SLAM and identifying their limitations. The dataset is available at arclab-hku.github.io/ecmd/.
FT-Net: Learning Failure Recovery and Fault-tolerant Locomotion For Quadruped Robots Arc Lab 2023-10-27 | Accepted by Robotics and Automation Letter (RA-L)
Learning Agile Flights through Narrow Gaps with Varying Angles using Onboard Sensing - v2 Arc Lab 2023-06-05 | ...
Learning Agile Flights through Narrow Gaps with Varying Angles using Onboard Sensing Arc Lab 2023-02-19 | ...
ArcLab highlights 2020-2022 Arc Lab 2023-01-26 | Some of the work done by ArcLab HKU during 2020-2022.Our lab website: https://arclab.hku.hk/
ESVIO: Event-based Stereo Visual Inertial Odometry Arc Lab 2022-12-26 | Paper Title: "ESVIO: Event-based Stereo Visual Inertial Odometry"* Project website: github.com/arclab-hku/Event_based_VO-VIO-SLAM* Source Code: github.com/arclab-hku/ESVIO* ESVIO Dataset Evaluation: b23.tv/oV885X7* Onboard Flight on HDR Scene: b23.tv/wcZiKzG* Onboard Flight on Aggressive Motion: b23.tv/nQUbMjy* ESVIO Outdoor Large-scale Evaluation: b23.tv/WHSgh6J
An Aerial Continuum Manipulator with IMU-based Kinematic Modeling and Tendon-slacking Prevention Arc Lab 2022-11-15 | ...
PL-EVIO: Robust Monocular Event-based Visual Inertial Odometry with Point and Line Features Arc Lab 2022-10-18 | More demonstrations can be seen in followings:* Project website: b23.tv/OE3QM6j * Three kinds of feature detection and tracking: b23.tv/x57n23j * Evaluation in UZH-FPV: b23.tv/f4FlU1v * Onboard quadrotor flighting evaluation: b23.tv/VqGMkyD * Outdoor large-scale evaluation: b23.tv/8KoLOCO* Our pervious EIO work (IROS 2022): b23.tv/XByDp1zAbstract: Robust and reliable state estimation in challenge situations, e.g. aggressive motion, is still an unsolved problem, especially achieving onboard state feedback control for aggressive motion. In this paper, we proposed robust and real-time event-based visual-inertial odometry (VIO) with event, image, and inertial measurement. In particular, we design the line-based event features to provide additional structure or constraint information in the human-made scene, while the point-based event and image features provide a good supplement for each other through well-design feature management. Finally, the point-based and line-based visual residual from the event camera, the point-based visual residual from the standard camera, and the residual from IMU pre-integration are tightly-coupled fused in a keyframe-based graph optimization framework to provide reliable state estimation. Experiments in the public benchmark datasets show that our method can achieve superior performance compared with the state-of-the-art image-based or event-based VIO. We also use our pipeline to demonstrate onboard closed-loop quadrotor aggressive flight and large-scale outdoor experiments.
IROS 2022 Event-based Visual Inertial Odometry (EVIO) Arc Lab 2022-07-30 | Paper Title: Monocular Event Visual Inertial Odometry based on Event-corner using Sliding Windows Graph-based OptimizationProject Website: sites.google.com/connect.hku.hk/hku-arclab-evio/home
Perception and Avoidance of Multiple Small Fast Moving Objects with Only Low-cost RGBD Camera Arc Lab 2022-06-17 | For RA-L & IROS 2022 Submission.
Event-based Visual Inertial Odometry Arc Lab 2022-02-28 | Paper Title《Monocular Event Visual Inertial Odometry based on Event-corner using Sliding Windows Graph-based Optimization》Project Website: sites.google.com/connect.hku.hk/hku-arclab-evio/home
Perception and Avoidance of Multiple Fast Moving Small Objects for Quadrotors Arc Lab 2022-02-23 | Video for RAL + IROS 2022 submission.Title: Perception and Avoidance of Multiple Fast Moving Small Objects for Quadrotors with Only Low-cost RGBD CameraWe propose a perception and avoidance system of fast moving small objects, realizing the performance with an RGBD sensor only, which is the most lightweight system.Github: github.com/arclab-hku/fast-dodging
Real-time Identification and Simultaneous Avoidance of Static and Dynamic Obstacles on Point Cloud Arc Lab 2021-12-06 | This is a supplementary video of more hardware tests for our paper "Real-time Identification and Simultaneous Avoidance of Static and Dynamic Obstacles on Point Cloud for UAVs Navigation". (Hardware flight tests)
Real-time Identification and Simultaneous Avoidance of Static and Dynamic Obstacles on Point Cloud Arc Lab 2021-09-09 | This is a supplementary video for our paper "Real-time Identification and Simultaneous Avoidance of Static and Dynamic Obstacles on Point Cloud for UAVs Navigation".Abstract:Avoiding hybrid obstacles in unknown scenarios with an efficient flight strategy is a key challenge for unmanned aerial vehicle applications. In this paper, we introduce a more robust technique to distinguish and track dynamic obstacles from static ones with only point cloud input. Then, to achieve dynamic avoidance, we propose the forbidden pyramids method to solve the desired vehicle velocity with an efficient sampling-based method in iteration. The motion primitives are generated by solving a nonlinear optimization problem with the constraint of desired velocity and the waypoint.Furthermore, we present several techniques to deal with the position estimation error for close objects, the error for deformable objects, and the time gap between different submodules. The proposed approach is implemented to run onboard in real-time and validated extensively in simulation and hardware tests, demonstrating our superiority in tracking robustness, energy cost, and calculating time.
ICRA2021 Online Recommendation-based Convolutional Features for Scale-Aware Visual Tracking Arc Lab 2021-05-31 | In this paper, we develop an online learning-based visual tracking framework that can optimize the target model and estimate the scale variation for object tracking. We propose a recommender-based tracker, which is capable of selecting the representative convolutional neural network (CNN) layers and feature maps autonomously. In addition, the proposed recommender computes the weights of these layers and feature maps. A discriminative target percept of each recommended layer is reconstructed by the weighted sum of the recommended feature maps. Then the target model of the correlation filter is updated by the weighted sum of the target percepts. Thus, a sub-network is extracted from the pre-trained CNN backbone for the tracking process of a specific target. To deal with scale changes, we propose a spatiotemporal-based min-channel method to estimate the target size variation directly from CNN features. Experimental results on 50 benchmark datasets and video data from a rescue drone demonstrate that the proposed tracker is quite competitive with the state-of-the-art CNN-based trackers in terms of accuracy, scale adaptation, and robustness for UAV-related applications.Github: github.com/arclab-hku/ICRA2021tracking.git
A Fast Planning Approach for 3D Short Trajectorywith a Parallel Framework Arc Lab 2021-05-27 | In this work, we present a parallel algorithm architecture with the map planner and the point cloud planner for UAVs trajectory planning, achieving satisfactory performance in the planning success rate, path length, and fast response ability.In this video, we will demonstrate the flight performance with a real drone in multiple environments, especially the reaction when faced with some intruding dynamic obstacles.
Identification and Avoidance of Static and Dynamic Obstacles on Point Cloud for UAVs Navigation Arc Lab 2020-11-03 | This is a short video introduction to our recently submitted paper for RAL-"Identification and Avoidance of Static and Dynamic Obstacles on Point Cloud for UAVs Navigation". It mainly demonstrates the simulation and hardware results of the estimation of moving obstacles position and velocity and the flight test for the drone flying in an environment with both static and dynamic obstacles exist. Only the point cloud is used as the information from the environment and no need to build a map first, which is quite challenging work.
SOPVO demo Arc Lab 2020-10-04 | Stereo orientation prior visual odometry (SOPVO) ROS(C++) project demo. Frontend only, no BA, no loop closure. Github: github.com/arclab-hku/SOPVOOther links:Bitbucket:bitbucket.org/arclabadmin/sopvo/src/masterKITTI results demo: youtu.be/0eGTsfJpJl4
Quadrotor flight despite the loss of two propellers Arc Lab 2020-09-08 | A preliminary video of the quadrotor flight when two of the propellers are "lost" (not spinning). Two motors are stopped and their propellers are not spinning.
