Uploaded August 2026 | Updated November 2026, 6 days ago
In this series, we explore using reinforcement learning algorithms (RL) to have a robot learn to balance on its own. We will deploy the trained agent to a real ESP32-based robot platform, tackle the sim-to-real gap using post-processing and domain randomization, and eventually build a full remote-controlled balance bot. The same techniques used here are what researchers use to train state-of-the-art walking robots. So, while we are starting with a simpler project, know that these skills will transfer to modern robotics.
Click here to register for Shawn’s webinar on this topic: event.on24.com/wcc/r/5384134/958EFFE494BF1C6A2AD94F9B093DC01B?partnerref=vid
To follow along with the series, you will need the following hardware:
● M5Stack BALA2 Fire robot kit: digikey.com/short/7h2fb5hd
All project files, including the FreeCAD model, MuJoCo XML descriptor, and Docker environment, are available in the following GitHub repository:
github.com/ShawnHymel/reinforcement-learning-for-robotics
Our balance bot can now stand upright and hold its position robustly thanks to domain randomization (DR), but a robot that just stands there isn't very useful. In this episode, we extend the agent to accept movement commands, teaching the robot to drive forward, backward, and turn while continuing to balance on its own.
You can read a full write-up of everything we cover in this video here:
digikey.com/en/maker/tutorials/2026/reinforcement-learning-for-robotics-part-5-adding-commands-to-the-agent
To make this work, we expand the observation vector from four elements to six, adding normalized velocity and yaw rate commands alongside the existing IMU and encoder readings. We also overhaul the reward function, replacing the origin penalty with a Gaussian tracking reward that incentivizes the agent to match the commanded velocity and yaw rate as closely as possible. A new smoothness penalty encourages clean transitions between commands rather than jerky direction changes.
Training now spans ten curriculum learning phases that interleave command learning with domain randomization. Rather than stacking commands on top of a fully domain-randomized agent (which we found didn't work well), we teach the robot to follow commands first and then gradually reintroduce the full suite of DR (sensor noise, motor noise, random pushes, mass and friction variation, axle torque noise, etc.) across the later phases. The result is an agent that can follow live remote control inputs mid-episode while remaining robust to real-world conditions.
Deployment to the real robot is coming in the final episode, where we'll build a Wi-Fi access point and web-based controller on the ESP32.
Coursera Deep Learning specialization: coursera.org/specializations/deep-learning
Coursera Reinforcement Learning specialization: coursera.org/specializations/reinforcement-learning
Introduction to Reinforcement Learning textbook by Sutton and Barto: incompleteideas.net/book/the-book-2nd.html
Reinforcement Learning math blog post series: shawnhymel.com/3316/what-is-reinforcement-learning
Introduction to Reinforcement Learning video: youtube.com/watch?v=3av8vozEczU
Learn more:
Maker.io - digikey.com/en/maker
DigiKey’s Blog – TheCircuit digikey.com/en/blog
Connect with DigiKey on Facebook facebook.com/digikey.electronics
And follow us on X: https://x.com/digikey
0:00 Intro
1:17 Overview of adding commands to the observation vector
1:52 Adding commands and new domain randomization to the environment
10:14 Training the RL agent with curriculum learning
16:22 Conclusion
In this series, we explore using reinforcement learning algorithms (RL) to have a robot learn to balance on its own. We will deploy the trained agent to a real ESP32-based robot platform, tackle the sim-to-real gap using post-processing and domain randomization, and eventually build a full remote-controlled balance bot. The same techniques used here are what researchers use to train state-of-the-art walking robots. So, while we are starting with a simpler project, know that these skills will transfer to modern robotics.
Click here to register for Shawn’s webinar on this topic: event.on24.com/wcc/r/5384134/958EFFE494BF1C6A2AD94F9B093DC01B?partnerref=vid
To follow along with the series, you will need the following hardware:
● M5Stack BALA2 Fire robot kit: digikey.com/short/7h2fb5hd
All project files, including the FreeCAD model, MuJoCo XML descriptor, and Docker environment, are available in the following GitHub repository:
github.com/ShawnHymel/reinforcement-learning-for-robotics
Our balance bot can now stand upright and hold its position robustly thanks to domain randomization (DR), but a robot that just stands there isn't very useful. In this episode, we extend the agent to accept movement commands, teaching the robot to drive forward, backward, and turn while continuing to balance on its own.
You can read a full write-up of everything we cover in this video here:
digikey.com/en/maker/tutorials/2026/reinforcement-learning-for-robotics-part-5-adding-commands-to-the-agent
To make this work, we expand the observation vector from four elements to six, adding normalized velocity and yaw rate commands alongside the existing IMU and encoder readings. We also overhaul the reward function, replacing the origin penalty with a Gaussian tracking reward that incentivizes the agent to match the commanded velocity and yaw rate as closely as possible. A new smoothness penalty encourages clean transitions between commands rather than jerky direction changes.
Training now spans ten curriculum learning phases that interleave command learning with domain randomization. Rather than stacking commands on top of a fully domain-randomized agent (which we found didn't work well), we teach the robot to follow commands first and then gradually reintroduce the full suite of DR (sensor noise, motor noise, random pushes, mass and friction variation, axle torque noise, etc.) across the later phases. The result is an agent that can follow live remote control inputs mid-episode while remaining robust to real-world conditions.
Deployment to the real robot is coming in the final episode, where we'll build a Wi-Fi access point and web-based controller on the ESP32.
Coursera Deep Learning specialization: coursera.org/specializations/deep-learning
Coursera Reinforcement Learning specialization: coursera.org/specializations/reinforcement-learning
Introduction to Reinforcement Learning textbook by Sutton and Barto: incompleteideas.net/book/the-book-2nd.html
Reinforcement Learning math blog post series: shawnhymel.com/3316/what-is-reinforcement-learning
Introduction to Reinforcement Learning video: youtube.com/watch?v=3av8vozEczU
Learn more:
Maker.io - digikey.com/en/maker
DigiKey’s Blog – TheCircuit digikey.com/en/blog
Connect with DigiKey on Facebook facebook.com/digikey.electronics
And follow us on X: https://x.com/digikey
0:00 Intro
1:17 Overview of adding commands to the observation vector
1:52 Adding commands and new domain randomization to the environment
10:14 Training the RL agent with curriculum learning
16:22 Conclusion




![Aperture Science [Maker Update] | Maker.io
This week on Maker Update, speedy thing goes in, speedy thing comes out, a sip of sadness, clocking infinity, and inverse kinematics for tentacles.
+Show Notes [Maker Update Ep.502]+
=Project of the Week=
Linked Portal Illusion by James Programming 0:28
https://www.youtube.com/watch?v=Lj5IdBTnoGk
=More Projects=
Soda Machine for Any Flavor by Luke Creatives 2:28
https://www.youtube.com/watch?v=kBb56968ixI
How to Make Lightning by Doktor Scott diabolical 3:46
https://www.youtube.com/watch?v=3GILF-XltNk
7-Segment Infinity Mirror Clock by Andy Huot Creations 4:34
https://www.youtube.com/watch?v=kG54YsSoqmE
=Tips & Tools=
Inverse Kinematics for Tentacle Robots by James Bruton 5:27
https://www.youtube.com/watch?v=BvGphB0iaHE
Best 3d Scanner on the Market by Making for Motorsport 6:07
https://www.youtube.com/watch?v=Q1tQDsWJTvg
Make Your Own Field Notes Books by Perry Daniels 6:46
https://www.youtube.com/watch?v=zYn8PBWR2lg
=DigiKey Spotlight=
Breadboarding an STM32 Microcontroller 7:26
https://www.youtube.com/watch?v=M9FHGF2F6tE
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ Aperture Science [Maker Update] | Maker.io](https://i.ytimg.com/vi/JQIsJ-VVoE8/mqdefault.jpg)
![Babelfish [Maker Update] | Maker.io
This week on Maker Update: a box for translating conversations offline, a dress for translating movement into light, a clock for translating light into time, and a toy for translating your voice into gibberish.
+Show Notes [Maker Update Ep.496]+
=Project of the Week=
Gemma Translator by Google Creative Lab 0:32
https://github.com/google-gemma/gemma-translator
=More Projects=
Polka Dot Electro-Swing Dress with NeoPixels by Erin St. Blaine 2:11
https://learn.adafruit.com/polka-dot-dress-with-neopixels
ChronoBloom by Joe S 3:07
https://github.com/Maestro8484/ChronoBloom
Voicebox FX by Ruiz Brothers and Liz Clark 3:50
https://learn.adafruit.com/voicebox-fx
MurMur by OiO now DIY Build Guide 4:17
https://github.com/oio/murmur-diy
=Tips & Tools=
Esp32-ai by Slava S. 4:55
https://github.com/slvDev/esp32-ai
Gridfinity Carousel by Nich Fugal 5:22
https://www.printables.com/model/698402-gridfinity-carousel
Gridfinity Carousel - Motorized by Nich Fugal 5:44
https://www.printables.com/model/897707-gridfinity-carousel-motorized
=DigiKey Spotlight=
Reinforcement Learning for Robotics Part 2: Train a Balance Bot with PPO by DigiKey and Shawn Hymel 5:53
https://www.youtube.com/watch?v=zsdceSTRBl4
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ Babelfish [Maker Update] | Maker.io](https://i.ytimg.com/vi/JyovE8wLhAI/mqdefault.jpg)
![Ready to Wear [Maker Update] | Maker.io
This week on Maker Update: a hard case corset, a secret door for cats, making cats out of cardboard, and how to make your 3d printing safer for your cats.
+Show Notes [Maker Update Ep.485]+
=Project of the Week=
Wooden Dress for the MET Gala by Xyla Foxlin 0:30
https://www.youtube.com/watch?v 1ejR5DSWKI
=News=
WPI XRP Medal Challenge 3:15
https://www.digikey.com/en/maker/blogs/2026/build-something-awesome-the-wpi-xrp-medal-challenge-is-here
https://www.sparkfun.com/wpi-xrp-medal-challenge
=More Projects=
Secret Cat Door by Morley Kert 3:50
https://www.youtube.com/watch?v=XAycw099agw
Cardboard Lion by Ali Spagnola 4:31
https://www.youtube.com/watch?v=GcVfh_bcs1Y
Minimalist Note Device by Paul Lagier 5:19
https://www.youtube.com/watch?v=3t0k7E7WiOQ
=Tips & Tools=
Safer Resin Printing by Katz Creates 6:03
https://www.youtube.com/watch?v=XaNxWOZJLws
3d Printing Tips for Stop Motion by Valentin Felder 6:34
https://www.youtube.com/watch?v=0OXRJwJJz3w
FIFA World Cup Scoreboard by Liz Clark 7:01
https://learn.adafruit.com/led-matrix-fifa-world-cup-scoreboard
=DigiKey Spotlight=
How do Hall Effect Sensors work? 7:24
https://www.youtube.com/watch?v=s51tHmFGm_k
Shop Hall Effect Sensors on DigiKey:
https://www.digikey.com/short/5jm0j4j9
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ Ready to Wear [Maker Update] | Maker.io](https://i.ytimg.com/vi/K6QaRlwngJE/mqdefault.jpg)


![Vegas Baby [Maker Update] | Maker.io
Vegas Baby [Maker Update] | Maker.io Vegas Baby [Maker Update] | Maker.io](https://i.ytimg.com/vi/LDPltbFL3qY/mqdefault.jpg)
![Know When to Fold Em [Maker Update] | Maker.io
This week on Maker Update: a dinosaur for your pocket, a Muse for makers, a lamp you’ll want to lick, and taking your message to the streets.
+Show Notes [Maker Update Ep.506]+
=Project of the Week=
DinoBoy by Paul from Watt IV 0:25
https://wattiv.nl/work/dinoboy/
Build guide:
https://www.schematik.io/projects/dinoboy-9cyx
=News=
Meta Brings Its Muse AI Agent to ESP32 and Raspberry Pi 2:14
https://gadgets.muse.ai/
=More Projects=
I Proved the Pop Art Concept by Wesley Treat 2:38
https://www.youtube.com/watch?v=C5HEF6SXCwY
Im Vandalizing My Community... With Kindness by byte sized engineering 3:34
https://www.digikey.com/en/maker/projects/im-vandalizing-my-community-with-kindness-part-1/be1746a322844526aacbe3875d5b4c86
pt.2
https://www.digikey.com/en/maker/projects/im-vandalizing-my-community-with-kindness-part-2/b7fca3ecb63d46928a4511cf25eec0d8
See also: Bikelangelo: the Graffiti Maker Bike by sagarrabanana
https://www.instructables.com/Bikelangelo-the-Graffiti-Maker-Bike/
StreetWriter By charlesglorioso
https://www.instructables.com/StreetWriter/
I Turned A Pickup Truck Into A PRINTER by Ryder Calm Down
https://www.youtube.com/watch?v=WKIs_u5jr1s
=Tips & Tools=
I Turned Origami Into Mechanisms by matthew lim 4:16
https://www.youtube.com/watch?v=Pz2Tpx5pkAg
I made a HAIRY combat robot in 48hrs! by Makers Muse 4:52
https://www.youtube.com/watch?v=c0_OhAfTF-Q
=DigiKey Spotlight=
Intro to LoRa Mesh Part 1: What are Meshtastic and MeshCore? by Shawn Hymel 5:42
https://www.youtube.com/watch?v=CSpjDLRIpio
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ Know When to Fold Em [Maker Update] | Maker.io](https://i.ytimg.com/vi/LFYQt6dm1y4/mqdefault.jpg)