Human-Level Performance with Autonomous Vision-based Drones - Davide Scaramuzza @ailabRPG
Human-Level Performance with Autonomous Vision-based Drones - Davide Scaramuzza  @ailabRPG
Uploaded November 2023 | Updated November 2026, 38 minutes ago
On November 2, 2023, at an event hosted by the Swiss Association of Aeronautical Sciences at ETH, Prof. Davide Scaramuzza presented a comprehensive overview of our latest advancements in autonomous drone technology aimed at achieving human-level performance. Autonomous drones hold immense potential in fields like inspection, agriculture, and search-and-rescue operations, offering prospects for enhanced productivity. However, the current capabilities of these drones lag behind human pilots in terms of speed, adaptability, and reliability. The key question addressed in this talk is: How can autonomous drones outperform human pilots? The presentation delves into the complexities of autonomous, agile navigation in environments without GPS, highlighting the challenges in perception, learning, planning, and control in robotics research. Prof. Scaramuzza illustrates how the integration of deep learning techniques with fast-response sensors, such as event cameras, enables drones to attain remarkable levels of speed and resilience, relying exclusively on onboard computational resources. This advancement is a significant stride toward elevating the efficiency and safety of future autonomous aerial vehicles.

00:00 - Introduction
00:28 - Global commercial drone market
02:23 - Vision-based drone navigation
05:53 - Why we want agile drones
13:34 - Previous works on agile flight
15:14 - Challenges of vision-based agile flight
18:36 - Vision-based drone control architecture
21:00 - Key questions to incorporate deep learning into drone control
22:08 - Our vision-controlled drone: "SWIFT"
23:28 - Acrobatics and high-speed flight in the wild
28:09 - Drone racing
30:22 - Autonomous drone racing
31:28 - Formal definition of the autonomous drone-racing problem
33:29 - Autonomous drone racing control approaches: MPC, MPCC, RL
36:56 - Unmodeled effects in autonomous drone racing
37:26 - State estimation of SWIFT (Nature paper)
38:36 - How SWIFT learns to race with reinforcement learning (Nature and SciRob papers)
40:52 - SWIFT against the world champion drone-racing pilots (Nature paper)
44:00 - Simulation-to-reality gap in SWIFT
46:17 - Reinforcement learning vs. optimal control (SciRob paper)
47:38 - SWIFT vs. human flight behavior
53:09 - The future of drones: how birds fly and see
54:00 - Fast sensors: event cameras
59:05 - The evolution of event cameras in academia and industry
1:00:24 - Application 1: keeping drones flying when a motor fails
1:01:04 - Application 2: Dodging dynamic objects with drones
1:01:49 - Application 3: Catching dynamic objects with legged robots
1:02:14 - Combining event cameras with RGB cameras
1:02:59 - The future of event cameras for robotics: low-latency, high-bandwidth control
1:08:51 - Q&A
1:08:56 - User-conditioned neural network control policies

About the presenter:
Davide Scaramuzza is a Professor of Robotics and Perception at the University of Zurich. He did his Ph.D. at ETH Zurich, a postdoc at the University of Pennsylvania, and was a visiting professor at Stanford University. His research focuses on autonomous, agile microdrone navigation using standard and event-based cameras. He pioneered autonomous, vision-based navigation of drones, which inspired the navigation algorithm of the NASA Mars helicopter and many drone companies. He made major contributions to visual-inertial state estimation, vision-based agile navigation of microdrones, and low-latency, robust perception with event cameras, which were transferred to many products, from drones to automobiles, cameras, AR/VR headsets, and mobile devices. In 2022, his team demonstrated that an AI-controlled, vision-based drone could outperform the world champions of drone racing, a result that was published in Nature. He is a consultant for the United Nations on disaster response and disarmament. He won many awards, including an IEEE Technical Field Award, the IEEE Robotics and Automation Society Early Career Award, the European-Research-Council Consolidator Grant, a Google Research Award, two NASA TechBrief Awards, and many paper awards. In 2015, he co-founded Zurich-Eye, today Meta Zurich, which developed the world-leading virtual-reality headset Meta Quest. In 2020, he co-founded SUIND, which builds autonomous drones for precision agriculture. Many aspects of his research have been featured in the media, such as The New York Times, The Economist, and Forbes.

Affiliation:
Robotics and Perception Group, University of Zurich, Switzerland: https://rpg.ifi.uzh.ch/
Human-Level Performance with Autonomous Vision-based Drones - Davide ScaramuzzaDeep Drone Racing: From Simulation to Reality with Domain RandomizationLearning Minimum-Time Flight in Cluttered Environments (RAL 2022)Video to Events: Recycling Video Datasets for Event Cameras (CVPR20 1-min Video Pitch)HDVIO2.0: Wind and Disturbance Estimation with Hybrid Dynamics VIO (T-RO 2025)Learning Quadruped Locomotion Using Differentiable Simulation (CORL 2024, Oral)Recurrent Vision Transformers for Object Detection with Event Cameras (CVPR 2023)Swiss Drone Days - Zurich, June 11-12, 2022 - Teaser #1Approximate Imitation Learning for Event-based Flight in Cluttered Environments (CoRL 2026)A Benchmark Comparison of Monocular Visual-Inertial Odometry Algorithms for Flying RobotsESIM: an Open Event Camera SimulatorLearning Agile Quadrotor Flight in the Real World (RSS 2026)
UZH Robotics and Perception Group |

Human-Level Performance with Autonomous Vision-based Drones - Davide Scaramuzza

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER