Demonstrating Agile Flight from Pixels without State Estimation (RSS 2024) @ailabRPG
Demonstrating Agile Flight from Pixels without State Estimation (RSS 2024)  @ailabRPG
Uploaded July 2024 | Updated November 2026, 1 hour ago
Quadrotors are among the most agile flying robots. Despite recent advances in learning-based control and computer vision, autonomous drones still rely on explicit state estimation. On the other hand, human pilots only rely on a first-person-view video stream from the drone onboard camera to push the platform to its limits and fly robustly in unseen environments. To the best of our knowledge, we present the first vision-based quadrotor system that autonomously navigates through a sequence of gates at high speeds while directly mapping pixels to control commands. Like professional drone-racing pilots, our system does not use explicit state estimation and leverages the same control commands humans use (collective thrust and body rates). We demonstrate agile flight at speeds up to 40km/h with accelerations up to 2g. This is achieved by training vision-based policies with reinforcement learning (RL). The training is facilitated using an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use the inner edges of the gates as a sensor abstraction. This simple yet robust, task-relevant representation can be simulated during training without rendering images. During deployment, a Swin-transformer-based gate detector is used. Our approach enables autonomous agile flight with standard, off-the-shelf hardware. Although our demonstration focuses on drone racing, we believe that our method has an impact beyond drone racing and can serve as a foundation for future research into real-world applications in structured environments.

Reference:
I. Geles*, L. Bauersfeld*, A. Romero, J. Xing, D. Scaramuzza,
"Demonstrating Agile Flight from Pixels without State Estimation",
Robotics: Science and Systems (RSS), 2024, Delft, Netherlands
PDF: https://rpg.ifi.uzh.ch/docs/RSS24_Geles.pdf

For more info about our research on:
Agile Drone Flight: http://rpg.ifi.uzh.ch/aggressive_flight.html
Drone Racing: http://rpg.ifi.uzh.ch/research_drone_racing.html
Machine Learning: http://rpg.ifi.uzh.ch/research_learning.html

Affiliation:
Ismail Geles, Leonard Bauersfeld, Angel Romero, Jiaxu Xing, and Davide Scaramuzza are with the Robotics and Perception Group, University of Zurich, Switzerland: https://rpg.ifi.uzh.ch/
Demonstrating Agile Flight from Pixels without State Estimation (RSS 2024)ICRA 2026 Keynote Tutorial - Learning Agile Vision-based Quadrotor FlightParabolic Flight Experiments | How Gravity affects Self-Motion estimation in Drone PilotsFisher Information Field: an Efficient and Differentiable Map for Perception-aware PlanningPowerline Tracking with Event Cameras (IROS 2021 Video Pitch)10-Year Anniversary of the UZH Robotics and Perception GroupEnd-to-End Learning of Representations for Asynchronous Event-Based Data (ICCV19)Time Lens++: Event-based Frame Interpolation with Parametric Flow and Multi-scale Fusion (CVPR 2022)Event-Based Angular Velocity Regression with Spiking Networks (ICRA20 Video Pitch)5th Year Anniversary of the Robotics and Perception GroupE-RAFT: Dense Optical Flow from Event Cameras (3DV 2021)TimeLens: Event-based Video Frame Interpolation (CVPR 2021)
UZH Robotics and Perception Group |

Demonstrating Agile Flight from Pixels without State Estimation (RSS 2024)

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER