Uploaded August 2020 | Updated November 2026, 1 hour ago
Motivated by the astonishing capabilities of natural intelligent agents and inspired by theories from psychology, this paper explores the idea that perception gets coupled to 3D properties of the world via interaction with the environment. Existing works for depth estimation require either massive
amounts of annotated training data or some form of hard-coded geometrical constraint. This paper explores a new approach to learning depth perception requiring neither of those. Specifically, we propose a novel global-local network architecture that can be trained with the data observed by a robot exploring an environment: images and extremely sparse depth measurements, down to even a single pixel per image. From a pair of consecutive images, the proposed network outputs a latent representation of the camera’s and scene’s parameters, and a dense depth map. Experiments on several datasets show that, when ground truth is available even for just one of the image pixels, the proposed network can learn monocular dense depth estimation up to 22.5% more accurately than state-of-the-art approaches. We believe that this work, in addition to its scientific interest, lays the foundations to learn depth with extremely sparse supervision, which can be valuable to all robotic systems acting under severe bandwidth or sensing constraints.
Reference:
A.Loquercio, A. Dosovitskiy, D. Scaramuzza
Robotics and Automation Letters and IROS, 2020
PDF: http://rpg.ifi.uzh.ch/docs/IROS20_Loquercio.pdf
More info on our research on Robot Learning: http://rpg.ifi.uzh.ch/research_learning.html
Affiliations:
A. Loquercio and D. Scaramuzza are with the Robotics and Perception Group, Dep. of Informatics, University of Zurich, and Dep. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland
http://rpg.ifi.uzh.ch/
Alexey Dosovitskiy is with Google Research, Berlin.
Motivated by the astonishing capabilities of natural intelligent agents and inspired by theories from psychology, this paper explores the idea that perception gets coupled to 3D properties of the world via interaction with the environment. Existing works for depth estimation require either massive
amounts of annotated training data or some form of hard-coded geometrical constraint. This paper explores a new approach to learning depth perception requiring neither of those. Specifically, we propose a novel global-local network architecture that can be trained with the data observed by a robot exploring an environment: images and extremely sparse depth measurements, down to even a single pixel per image. From a pair of consecutive images, the proposed network outputs a latent representation of the camera’s and scene’s parameters, and a dense depth map. Experiments on several datasets show that, when ground truth is available even for just one of the image pixels, the proposed network can learn monocular dense depth estimation up to 22.5% more accurately than state-of-the-art approaches. We believe that this work, in addition to its scientific interest, lays the foundations to learn depth with extremely sparse supervision, which can be valuable to all robotic systems acting under severe bandwidth or sensing constraints.
Reference:
A.Loquercio, A. Dosovitskiy, D. Scaramuzza
Robotics and Automation Letters and IROS, 2020
PDF: http://rpg.ifi.uzh.ch/docs/IROS20_Loquercio.pdf
More info on our research on Robot Learning: http://rpg.ifi.uzh.ch/research_learning.html
Affiliations:
A. Loquercio and D. Scaramuzza are with the Robotics and Perception Group, Dep. of Informatics, University of Zurich, and Dep. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland
http://rpg.ifi.uzh.ch/
Alexey Dosovitskiy is with Google Research, Berlin.










