Uploaded January 2025 | Updated September 2026, 3 weeks ago
Last year @meta introduced #CoTracker — a transformer-based model that jointly tracks points in a video. After the initial release of CoTracker, the model's GitHub page received 4k stars, which was a signal that point tracking is useful not only for visual effects but also for robotics, video generation, as well as bio and medical domains.
In this talk, BuzzRobot guest, Nikita Karaev, explains how CoTracker works and shares key learnings and research insights from working on CoTracker and CoTracker3 that was introduced a few months ago.
Timestamps:
0:00 Introduction
0:17 Pixel-level motion estimation: single point tracking, optical flow and main challenges
2:52 Tracking with CoTracker
4:03 Training data: Kubric dataset
4:15 Tracking points together
4:57 TAPIR by Google DeepMind VS Meta's CoTracker
6:15 Applications of point tracking: visual effects, robotics, bio and medical domains
8:49 Applications of point tracking: controlled video generation
9:37 Expected and unexpected applications
10:45 The scientific method: prevailing theory, limitations, hypothesis and test by experiment
13:33 Architecture: hypotheses
15:55 Synthetic data: hypothesis
16:19 Scaling on real videos
18:17 CoTracker3: adopting the best hypothesis
19:16 Q&A
#ai #artificialintelligance #machinelearning #machinelearningmodel #cotracker #cotracker3 #metaai #meta #pointtracking #robotics #robots #deeplearning #llms #llm #reinforcementlearning #technology #tech #techtalk #techtalks #aitalks #aitalk #science #python #pythonprogramming #programming
Social Links:
Newsletter: buzzrobot.substack.com
X: https://x.com/sopharicks
Slack: join.slack.com/t/buzzrobot/shared_invite/zt-2s067rv7n-guPIMGe62rbp9ncxdnOUfQ
Last year @meta introduced #CoTracker — a transformer-based model that jointly tracks points in a video. After the initial release of CoTracker, the model's GitHub page received 4k stars, which was a signal that point tracking is useful not only for visual effects but also for robotics, video generation, as well as bio and medical domains.
In this talk, BuzzRobot guest, Nikita Karaev, explains how CoTracker works and shares key learnings and research insights from working on CoTracker and CoTracker3 that was introduced a few months ago.
Timestamps:
0:00 Introduction
0:17 Pixel-level motion estimation: single point tracking, optical flow and main challenges
2:52 Tracking with CoTracker
4:03 Training data: Kubric dataset
4:15 Tracking points together
4:57 TAPIR by Google DeepMind VS Meta's CoTracker
6:15 Applications of point tracking: visual effects, robotics, bio and medical domains
8:49 Applications of point tracking: controlled video generation
9:37 Expected and unexpected applications
10:45 The scientific method: prevailing theory, limitations, hypothesis and test by experiment
13:33 Architecture: hypotheses
15:55 Synthetic data: hypothesis
16:19 Scaling on real videos
18:17 CoTracker3: adopting the best hypothesis
19:16 Q&A
#ai #artificialintelligance #machinelearning #machinelearningmodel #cotracker #cotracker3 #metaai #meta #pointtracking #robotics #robots #deeplearning #llms #llm #reinforcementlearning #technology #tech #techtalk #techtalks #aitalks #aitalk #science #python #pythonprogramming #programming
Social Links:
Newsletter: buzzrobot.substack.com
X: https://x.com/sopharicks
Slack: join.slack.com/t/buzzrobot/shared_invite/zt-2s067rv7n-guPIMGe62rbp9ncxdnOUfQ
