Uploaded July 2025 | Updated September 2026, 2 weeks ago
Talk: Charlie Gerard - "Motion Control With Multimodal LLMs", JSNation 2025
What if you could use multimodal LLMs to interact with websites or IoT devices using motion control?
As advancements in multimodal AI offer new opportunities to push the boundaries of what can be done with this technology, I started wondering how it could be leveraged from the perspective of human-computer interaction.
In this talk, I will take you through my research experimenting with building motion-controlled prototypes using LLMs in JavaScript.
Watch the full talk with Q&A on GitNation website: gitnation.com/contents/motion-control-with-multimodal-llms
Follow the link to watch the full version of all the conference talks, QnA’s with speakers and hands-on workshop recordings → gitnation.com/events/jsnation-2025
Timestamps:
00:00 Introduction to Motion Control with Multimodal AI
01:24 Exploring TensorFlow.js Models for Motion Control
02:47 Enhancing User Interactions with Gesture Control
04:45 Redefining Coding Interfaces with Minimal Inputs
06:58 Leveraging LLMs for Gesture-Based Interaction
08:55 Investigating LLMs for Motion Control Experiences
10:18 Exploring Gemiini for Gesture Recognition
12:34 Webcam-Based Gesture Recognition with Gemini
14:28 Controlling Light States with Gemini Functions
15:46 Interacting with Gemini and WiFi Light Bulbs
17:44 Enabling Multi-Light Control with ID and Position
19:05 Custom Light Names and Gestural Control Logic
20:13 Live API Functions and Directional Light Control
22:57 Integrating TensorFlow.js for Color Control
24:08 Color Changes with Hand Gestures
26:07 Custom Gesture Databases and Autonomous Interfaces
28:28 Personalized Interfaces and Motion Control
🔥 Want to hear talks like this live? Join us at JSNation US 2025 in New York: jsnation.us/?utm_source=youtube&utm_medium=charliegerard
JSNation website: jsnation.com
JSNation on X: https://x.com/thejsnation
JSNation on LinkedIn: linkedin.com/showcase/jsnation-the-main-js-conference
JSNation on Bluesky: https://bsky.app/profile/jsnation.gitnation.org
JSNation on Instagram: instagram.com/thejsnation
GitNation on TikTok: tiktok.com/@gitnationorg
#JSNation #MultimodalLLMs #GitNation #JSNation2025 #MotionControl #TechTalks
Talk: Charlie Gerard - "Motion Control With Multimodal LLMs", JSNation 2025
What if you could use multimodal LLMs to interact with websites or IoT devices using motion control?
As advancements in multimodal AI offer new opportunities to push the boundaries of what can be done with this technology, I started wondering how it could be leveraged from the perspective of human-computer interaction.
In this talk, I will take you through my research experimenting with building motion-controlled prototypes using LLMs in JavaScript.
Watch the full talk with Q&A on GitNation website: gitnation.com/contents/motion-control-with-multimodal-llms
Follow the link to watch the full version of all the conference talks, QnA’s with speakers and hands-on workshop recordings → gitnation.com/events/jsnation-2025
Timestamps:
00:00 Introduction to Motion Control with Multimodal AI
01:24 Exploring TensorFlow.js Models for Motion Control
02:47 Enhancing User Interactions with Gesture Control
04:45 Redefining Coding Interfaces with Minimal Inputs
06:58 Leveraging LLMs for Gesture-Based Interaction
08:55 Investigating LLMs for Motion Control Experiences
10:18 Exploring Gemiini for Gesture Recognition
12:34 Webcam-Based Gesture Recognition with Gemini
14:28 Controlling Light States with Gemini Functions
15:46 Interacting with Gemini and WiFi Light Bulbs
17:44 Enabling Multi-Light Control with ID and Position
19:05 Custom Light Names and Gestural Control Logic
20:13 Live API Functions and Directional Light Control
22:57 Integrating TensorFlow.js for Color Control
24:08 Color Changes with Hand Gestures
26:07 Custom Gesture Databases and Autonomous Interfaces
28:28 Personalized Interfaces and Motion Control
🔥 Want to hear talks like this live? Join us at JSNation US 2025 in New York: jsnation.us/?utm_source=youtube&utm_medium=charliegerard
JSNation website: jsnation.com
JSNation on X: https://x.com/thejsnation
JSNation on LinkedIn: linkedin.com/showcase/jsnation-the-main-js-conference
JSNation on Bluesky: https://bsky.app/profile/jsnation.gitnation.org
JSNation on Instagram: instagram.com/thejsnation
GitNation on TikTok: tiktok.com/@gitnationorg
#JSNation #MultimodalLLMs #GitNation #JSNation2025 #MotionControl #TechTalks
