End-to-End Learning of Representations for Asynchronous Event-Based Data (ICCV19) @ailabRPG
End-to-End Learning of Representations for Asynchronous Event-Based Data (ICCV19)  @ailabRPG
Uploaded April 2019 | Updated November 2026, 1 hour ago
Event cameras are vision sensors that record asynchronous streams of per-pixel brightness changes, referred to as “events”. They have appealing advantages over frame-based cameras for computer vision, including high temporal resolution, high dynamic range, and no motion blur. Due to the sparse, non-uniform spatiotemporal layout of the event signal, pattern recognition algorithms typically aggregate events into a grid-based representation and subsequently process it by a standard vision pipeline, e.g., Convolutional Neural Network (CNN). In this work, we introduce a general framework to convert event streams into grid-based representations through a sequence of differentiable operations. Our framework comes with two main advantages: (i) allows learning the input event representation together with the task dedicated network in an end to end manner, and (ii) lays out a taxonomy that unifies the majority of extant event representations in the literature and identifies novel ones. Empirically, we show that our approach to learning the event representation end-to-end yields an improvement of approximately 12% on optical flow estimation and object recognition over state-of-the-art methods.

Reference:
D. Gehrig, A. Loquercio, K. G. Derpanis, D. Scaramuzza
End-to-End Learning of Representations for Asynchronous Event-Based Data
International Conference on Computer Vision, Seoul, Oct. 2019
PDF: http://rpg.ifi.uzh.ch/docs/ICCV19_Gehrig.pdf
Code: github.com/uzh-rpg/rpg_event_representation_learning

See also our research page on event based vision: http://rpg.ifi.uzh.ch/research_dvs.html

For event-camera datasets and event camera simulator, see here:
http://rpg.ifi.uzh.ch/davis_data.html
http://rpg.ifi.uzh.ch/esim/index.html

Other resources on event cameras (publications, software, drivers, where to buy, etc.):
github.com/uzh-rpg/event-based_vision_resources

Affiliations: D. Gehrig, A. Loquercio, and D. Scaramuzza are with the Robotics and Perception Group, Dept. of Informatics, University of Zurich, and Dept. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland http://rpg.ifi.uzh.ch/.
Konstantinos G. Derpanis is from Ryerson University (ryersonvisionlab.github.io/) and with the Samsung AI Centre in Toronto.
End-to-End Learning of Representations for Asynchronous Event-Based Data (ICCV19)Time Lens++: Event-based Frame Interpolation with Parametric Flow and Multi-scale Fusion (CVPR 2022)Event-Based Angular Velocity Regression with Spiking Networks (ICRA20 Video Pitch)5th Year Anniversary of the Robotics and Perception GroupE-RAFT: Dense Optical Flow from Event Cameras (3DV 2021)TimeLens: Event-based Video Frame Interpolation (CVPR 2021)Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones (T-RO 2026)Data-driven Feature Tracking for Event Cameras (CVPR 2023, Award Candidate; T-PAMI25)Low Latency Automotive Vision with Event Cameras (Nature, 2024)Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning (ICRA 2021)High Speed and High Dynamic Range Video with an Event CameraLearning on the Fly: Rapid Policy Adaptation via Differentiable Simulation (RA-L 2026)
UZH Robotics and Perception Group |

End-to-End Learning of Representations for Asynchronous Event-Based Data (ICCV'19)

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER