Uploaded July 2019 | Updated November 2026, 1 hour ago
Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (more than 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. We also demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.
Reference: Henri Rebecq, René Ranftl, Vladlen Koltun, Davide Scaramuzza. High Speed and High Dynamic Range Video with an Event Camera, IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2019.
Paper: http://rpg.ifi.uzh.ch/docs/TPAMI19_Rebecq.pdf
Project page: http://rpg.ifi.uzh.ch/E2VID.html
Code: github.com/uzh-rpg/rpg_e2vid
For more info about our research page on event based vision: http://rpg.ifi.uzh.ch/research_dvs.html
Our research page on deep learning: http://rpg.ifi.uzh.ch/research_learning.html
For event-camera datasets and simulator, see here:
http://rpg.ifi.uzh.ch/davis_data.html
http://rpg.ifi.uzh.ch/esim.html
Other resources on event cameras (publications, software, drivers, where to buy, etc.):
github.com/uzh-rpg/event-based_vision_resources
Affiliations:
H. Rebecq and D. Scaramuzza are with the Robotics and Perception Group, Dept. of Informatics, University of Zurich, and Dept. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland http://rpg.ifi.uzh.ch/
R. Ranftl and V. Koltun are with Intel Labs.
Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (more than 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. We also demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.
Reference: Henri Rebecq, René Ranftl, Vladlen Koltun, Davide Scaramuzza. High Speed and High Dynamic Range Video with an Event Camera, IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2019.
Paper: http://rpg.ifi.uzh.ch/docs/TPAMI19_Rebecq.pdf
Project page: http://rpg.ifi.uzh.ch/E2VID.html
Code: github.com/uzh-rpg/rpg_e2vid
For more info about our research page on event based vision: http://rpg.ifi.uzh.ch/research_dvs.html
Our research page on deep learning: http://rpg.ifi.uzh.ch/research_learning.html
For event-camera datasets and simulator, see here:
http://rpg.ifi.uzh.ch/davis_data.html
http://rpg.ifi.uzh.ch/esim.html
Other resources on event cameras (publications, software, drivers, where to buy, etc.):
github.com/uzh-rpg/event-based_vision_resources
Affiliations:
H. Rebecq and D. Scaramuzza are with the Robotics and Perception Group, Dept. of Informatics, University of Zurich, and Dept. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland http://rpg.ifi.uzh.ch/
R. Ranftl and V. Koltun are with Intel Labs.




![Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight (Science Robotics)
Agilicious is a co-designed hardware and software framework tailored to autonomous, agile quadrotor flight, which has been developed and used since 2016 at the Robotics and Perception Group of the University of Zurich. Agilicious is described in our Science Robotics 2022 paper, linked below. It is completely open-source and open hardware and supports both model-based and neural-network-based controllers! Also, it provides high thrust-to-weight and torque-to-inertia ratios for agility, onboard vision sensors, GPU-accelerated compute hardware for real-time perception and neural-network inference, a real-time flight controller, and a versatile software stack. In contrast to existing frameworks, Agilicious offers a unique combination of flexible software and high-performance hardware. Agilicious has been used in over 30 scientific papers at our lab, including trajectory tracking for drone racing scenarios at up to 5g and 70km/h [SciRob21_Foehn], vision-based acrobatic flight [RSS20_Kaufmann), obstacle avoidance in both structured and unstructured environments using solely onboard perception [SciRob21_Loquercio], and hardware-in-the-loop simulation in virtual-reality environments! Thanks to its versatility, we believe that Agilicious supports the next generation of scientific and industrial quadrotor research. Agilicious allows you to seamlessly develop, test, reproduce, and benchmark control algorithms such as Non-linear Model Predictive Control (MPC), differential-flatness-based control (DFBC), and INDI (Incremental Non-linear Dynamic Inversion) [TRO 2022_Sun]. Agilicious also includes our state-of-the-art, high-fidelity simulator, which uses Blade Element Momentum (BEM) theory for the propeller model and provides accurate estimates of the quadcopter’s aerodynamic characteristics across the flight envelope [RSS 2021_Bauersfeld].
Agilicious webpage: https://agilicious.dev/
References:
Agilicious main reference (all open-access PDFs and videos can be found on the Agilicious page):
Foehn, Kaufmann, Romero, Penicka, Sun, Bauersfeld, Laengle, Cioffi, Song, Loquercio, Scaramuzza, “Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight”, AAAS Science Robotics, 2022.
PDF: https://rpg.ifi.uzh.ch/docs/ScienceRobotics22_Foehn.pdf
Agilicious implements the algorithms described in the following papers:
[SciRob21_Foehn] Philipp Foehn, Angel Romero, Davide Scaramuzza, “Time-Optimal Planning for Quadrotor Waypoint Flight”, Science Robotics, 2021,
PDF: https://rpg.ifi.uzh.ch/docs/ScienceRobotics21_Foehn.pdf
[RSS20_Kaufmann] Elia Kaufmann, Antonio Loquercio, Rene Ranftl, Mark Müller, Vladlen Koltun, Davide Scaramuzza, “Deep Drone Acrobatics”, RSS: Robotics, Science, and Systems, 2020,
PDF: https://rpg.ifi.uzh.ch/docs/RSS20_Kaufmann.pdf
[SciRob21_Loquercio] Antonio Loquercio, Elia Kaufmann, Rene Ranftl, Mark Müller, Vladlen Koltun, Davide Scaramuzza, “Learning High-Speed Flight in the Wild”, Science Robotics, 2021,
PDF: https://rpg.ifi.uzh.ch/docs/Loquercio21_Science.pdf
[TRO 2022_Sun] Sihao Sun, Angel Romero, Philipp Foehn, Elia Kaufmann, Davide Scaramuzza, “A Comparative Study of Nonlinear MPC and Differential-Flatness-based Control for Quadrotor Agile Flight”, IEEE Transactions on Robotics, 2022,
PDF: https://rpg.ifi.uzh.ch/docs/Arxiv21_MPC_Sun.pdf
[RSS 2021_Bauersfeld] Leonard Bauersfeld, Elia Kaufmann, Philipp Foehn, Sihao Sun, Davide Scaramuzza, “NeuroBEM: Hybrid Aerodynamic Quadrotor Model”, RSS: Robotics, Science, and Systems, 2021,
PDF: https://rpg.ifi.uzh.ch/docs/RSS21_Bauersfeld.pdf
Matthias Faessler, Antonio Franchi, Davide Scaramuzza, “Differential Flatness of Quadrotor Dynamics Subject to Rotor Drag for Accurate Tracking of High-Speed Trajectories”, IEEE Robotics and Automation Letters, 2018.
PDF: https://rpg.ifi.uzh.ch/docs/RAL18_Faessler.pdf
Our research in Agile Drone Flight:
http://rpg.ifi.uzh.ch/aggressive_flight.html
Affiliations:
P. Foehn, E. Kaufmann, A. Romero, R. Penicka, S. Sun, L. Bauersfeld, T. Laengle, Y. Song, A. Loquercio, D. Scaramuzza are with the Robotics and Perception Group, University of Zurich, Switzerland
http://rpg.ifi.uzh.ch/
Music Credits: scottholmesmusic.com under Free Creative Commons License Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight (Science Robotics)](https://i.ytimg.com/vi/fNYxPLyJ5YY/mqdefault.jpg)





