Uploaded October 2024 | Updated November 2026, 42 minutes ago
Visual Odometry (VO) is essential to downstream mobile robotics and augmented/virtual reality tasks. Despite recent advances, existing VO methods still rely on heuristic design choices that require several weeks of hyperparameter tuning by human experts, hindering generalizability and robustness. We address these challenges by reframing VO as a sequential decision-making task and applying Reinforcement Learning (RL) to adapt the VO process dynamically. Our approach introduces a neural network, operating as an agent within the VO pipeline, to make decisions such as keyframe and grid-size selection based on real-time conditions. Our method minimizes reliance on heuristic choices using a reward function based on pose error, runtime, and other metrics to guide the system. Our RL framework treats the VO system and the image sequence as an environment, with the agent receiving observations from keypoints, map statistics, and prior poses. Experimental results using classical VO methods and public benchmarks demonstrate improvements in accuracy and robustness, validating the generalizability of our RL-enhanced VO approach to different scenarios. We believe this paradigm shift advances VO technology by eliminating the need for time-intensive parameter tuning of heuristics.
Reference:
Nico Messikommer*, Giovanni Cioffi*, Mathias Gehrig , and Davide Scaramuzza
"Reinforcement Learning Meets Visual Odometry" European Conference on Computer Vision. (ECCV), 2024
PDF: https://rpg.ifi.uzh.ch/docs/ECCV24_Messikommer.pdf
Code: github.com/uzh-rpg/rl_vo
For more info on our research on:
Visual and Inertial Odometry and SLAM: http://rpg.ifi.uzh.ch/research_vo.html
Machine Learning: https://rpg.ifi.uzh.ch/research_learning.html
Affiliation: N. Messikommer, G. Cioffi, M. Gehrig, and D. Scaramuzza are with the Robotics and Perception Group, Dept. of Informatics, University of Zurich, Switzerland
http://rpg.ifi.uzh.ch/
Visual Odometry (VO) is essential to downstream mobile robotics and augmented/virtual reality tasks. Despite recent advances, existing VO methods still rely on heuristic design choices that require several weeks of hyperparameter tuning by human experts, hindering generalizability and robustness. We address these challenges by reframing VO as a sequential decision-making task and applying Reinforcement Learning (RL) to adapt the VO process dynamically. Our approach introduces a neural network, operating as an agent within the VO pipeline, to make decisions such as keyframe and grid-size selection based on real-time conditions. Our method minimizes reliance on heuristic choices using a reward function based on pose error, runtime, and other metrics to guide the system. Our RL framework treats the VO system and the image sequence as an environment, with the agent receiving observations from keypoints, map statistics, and prior poses. Experimental results using classical VO methods and public benchmarks demonstrate improvements in accuracy and robustness, validating the generalizability of our RL-enhanced VO approach to different scenarios. We believe this paradigm shift advances VO technology by eliminating the need for time-intensive parameter tuning of heuristics.
Reference:
Nico Messikommer*, Giovanni Cioffi*, Mathias Gehrig , and Davide Scaramuzza
"Reinforcement Learning Meets Visual Odometry" European Conference on Computer Vision. (ECCV), 2024
PDF: https://rpg.ifi.uzh.ch/docs/ECCV24_Messikommer.pdf
Code: github.com/uzh-rpg/rl_vo
For more info on our research on:
Visual and Inertial Odometry and SLAM: http://rpg.ifi.uzh.ch/research_vo.html
Machine Learning: https://rpg.ifi.uzh.ch/research_learning.html
Affiliation: N. Messikommer, G. Cioffi, M. Gehrig, and D. Scaramuzza are with the Robotics and Perception Group, Dept. of Informatics, University of Zurich, Switzerland
http://rpg.ifi.uzh.ch/










