Uploaded August 2026 | Updated November 2026, 12 hours ago
In this series, we explore using reinforcement learning algorithms (RL) to have a robot learn to balance on its own. We will deploy the trained agent to a real ESP32-based robot platform, tackle the sim-to-real gap using post-processing and domain randomization, and eventually build a full remote-controlled balance bot. The same techniques used here are what researchers use to train state-of-the-art walking robots. So, while we are starting with a simpler project, know that these skills will transfer to modern robotics.
Click here to register for Shawn’s webinar: event.on24.com/wcc/r/5384134/958EFFE494BF1C6A2AD94F9B093DC01B?partnerref=vid
To follow along with the series, you will need the following hardware:
M5Stack BALA2 Fire robot kit: digikey.com/short/7h2fb5hd
All project files, including the FreeCAD model, MuJoCo XML descriptor, and Docker environment, are available in the following GitHub repository:
github.com/ShawnHymel/reinforcement-learning-for-robotics
In the previous episode, we trained our balance bot agent entirely in simulation. Now, it's time to see how it holds up in the real world. In this episode, we deploy the trained PPO actor network to the M5Stack BALA 2 Fire robot and quickly run into the sim-to-real gap: the differences between simulation and reality that the agent never learned to handle.
You can read a full write-up of everything we cover in this video here:
digikey.com/en/maker/tutorials/2026/reinforcement-learning-for-robotics-part-3-deploy-ai-agent-to-a-robot
We start by converting the trained actor network from ONNX to a plain C header file, which gets compiled directly into the Arduino firmware. We are not accelerating the simple dense network (i.e. no TensorFlow Lite, NPU, etc.), just multiply-accumulates and a tanh activation running in 2–3ms on the ESP32. From there, we walk through the Arduino firmware that reads the IMU and wheel encoders, builds the observation vector, runs inference, and drives the motors. We keep all this within a 5ms control loop.
With the initial firmware uploaded, the robot balances briefly but eventually drifts and tips. We cover the main sources of sim-to-real error (sensor noise, motor inaccuracies, modeling errors, and calibration offsets) and then add a set of firmware fixes, including a pitch offset, motor boost, action deadband, and low-pass filter. While these fixes help the robot to balance more reliably, they require manual tuning and undermine the whole point of training a robust agent. The real solution is domain randomization, which we'll tackle in the next episode.
Coursera Deep Learning specialization: coursera.org/specializations/deep-learning
Coursera Reinforcement Learning specialization: coursera.org/specializations/reinforcement-learning
Introduction to Reinforcement Learning textbook by Sutton and Barto: incompleteideas.net/book/the-book-2nd.html
Reinforcement Learning math blog post series: shawnhymel.com/3316/what-is-reinforcement-learning
Introduction to Reinforcement Learning video: youtube.com/watch?v=3av8vozEczU
Learn more:
Maker.io - digikey.com/en/maker
DigiKey’s Blog – TheCircuit digikey.com/en/blog
Connect with DigiKey on Facebook facebook.com/digikey.electronics
And follow us on X: https://x.com/digikey
0:00 Intro
0:45 Strategy to deploy the actor network
1:30 ONNX actor to C code script
5:35 Calibrate IMU
9:34 Deploy agent to robot with Arduino
17:45 Post-processing firmware fixes
22:25 Conclusion
In this series, we explore using reinforcement learning algorithms (RL) to have a robot learn to balance on its own. We will deploy the trained agent to a real ESP32-based robot platform, tackle the sim-to-real gap using post-processing and domain randomization, and eventually build a full remote-controlled balance bot. The same techniques used here are what researchers use to train state-of-the-art walking robots. So, while we are starting with a simpler project, know that these skills will transfer to modern robotics.
Click here to register for Shawn’s webinar: event.on24.com/wcc/r/5384134/958EFFE494BF1C6A2AD94F9B093DC01B?partnerref=vid
To follow along with the series, you will need the following hardware:
M5Stack BALA2 Fire robot kit: digikey.com/short/7h2fb5hd
All project files, including the FreeCAD model, MuJoCo XML descriptor, and Docker environment, are available in the following GitHub repository:
github.com/ShawnHymel/reinforcement-learning-for-robotics
In the previous episode, we trained our balance bot agent entirely in simulation. Now, it's time to see how it holds up in the real world. In this episode, we deploy the trained PPO actor network to the M5Stack BALA 2 Fire robot and quickly run into the sim-to-real gap: the differences between simulation and reality that the agent never learned to handle.
You can read a full write-up of everything we cover in this video here:
digikey.com/en/maker/tutorials/2026/reinforcement-learning-for-robotics-part-3-deploy-ai-agent-to-a-robot
We start by converting the trained actor network from ONNX to a plain C header file, which gets compiled directly into the Arduino firmware. We are not accelerating the simple dense network (i.e. no TensorFlow Lite, NPU, etc.), just multiply-accumulates and a tanh activation running in 2–3ms on the ESP32. From there, we walk through the Arduino firmware that reads the IMU and wheel encoders, builds the observation vector, runs inference, and drives the motors. We keep all this within a 5ms control loop.
With the initial firmware uploaded, the robot balances briefly but eventually drifts and tips. We cover the main sources of sim-to-real error (sensor noise, motor inaccuracies, modeling errors, and calibration offsets) and then add a set of firmware fixes, including a pitch offset, motor boost, action deadband, and low-pass filter. While these fixes help the robot to balance more reliably, they require manual tuning and undermine the whole point of training a robust agent. The real solution is domain randomization, which we'll tackle in the next episode.
Coursera Deep Learning specialization: coursera.org/specializations/deep-learning
Coursera Reinforcement Learning specialization: coursera.org/specializations/reinforcement-learning
Introduction to Reinforcement Learning textbook by Sutton and Barto: incompleteideas.net/book/the-book-2nd.html
Reinforcement Learning math blog post series: shawnhymel.com/3316/what-is-reinforcement-learning
Introduction to Reinforcement Learning video: youtube.com/watch?v=3av8vozEczU
Learn more:
Maker.io - digikey.com/en/maker
DigiKey’s Blog – TheCircuit digikey.com/en/blog
Connect with DigiKey on Facebook facebook.com/digikey.electronics
And follow us on X: https://x.com/digikey
0:00 Intro
0:45 Strategy to deploy the actor network
1:30 ONNX actor to C code script
5:35 Calibrate IMU
9:34 Deploy agent to robot with Arduino
17:45 Post-processing firmware fixes
22:25 Conclusion
![Seeing Double [Maker Update] | Maker.io
This week on Maker Update: a lamp that’s through the looking glass, a butterfly you can steer, a 3d printed vending machine, and how to use free weights as a battery.
+Show Notes [Maker Update Ep.469]+
=Project of the Week=
Kaleidoscope Lamp by Simone Giertz 0:31
https://www.youtube.com/watch?v=GPSNT5vjSO8
=More Projects=
RC Butterfly by S-DIY 2:56
https://www.youtube.com/watch?v=eLTltRwuGW8
Making Furniture from Tiny Objects by Rachel Maksy 4:01
https://www.youtube.com/watch?v=z90eikSwvhc
3d Printed Vending Machine by DevMode Makes 5:03
https://www.youtube.com/watch?v=iorlgZoYXaQ
=Tips & Tools=
Testing Advanced 3d Printing Materials by Makers Muse 5:43
https://www.youtube.com/watch?v=Lo6UdFKQO0c
Can a Pendulum be a Battery by Tom Stanton 6:22
https://www.youtube.com/watch?v=uqmT1GzRXWI
TrueNAS Deep Dive by Zack Freedman 7:01
https://www.youtube.com/watch?v=n4wDmjEn8Vo
=DigiKey Spotlight=
Embedded Rust by Shawn Hymel 7:45
https://www.youtube.com/watch?v=mz-vW-Ar_GM
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ Seeing Double [Maker Update] | Maker.io](https://i.ytimg.com/vi/fyR3Dv0BOlo/mqdefault.jpg)


![In My Lane, Flourishing [Maker Update] | Maker.io
This week on Maker Update: bowling 300 in your kitchen, knights of the Canadian tuxedo, a dimpled display, and a control board for tiny bots.
+Show Notes [Maker Update Ep.471]+
=Project of the Week=
Mini Bowling Lane by Danny Lum 0:31
https://www.youtube.com/watch?v=sriRDDAa19A
=More Projects=
Armor Made From Jeans by Miss Twisted 2:58
https://www.youtube.com/watch?v=eIpotKXXrBI
Air Powered Display by Soiboi Soft 4:03
https://www.youtube.com/watch?v=VZ2ZcOzLnGg
R2-Cthru by Swooshdave 5:15
https://www.youtube.com/watch?v=PemdlFmORNs
=Tips & Tools=
Smallest Robotic Drive by Microbots 6:01
https://www.youtube.com/watch?v=uoak2QaHMsI
Design for Any Thread by Salveo3d 6:41
https://www.youtube.com/shorts/n8MxYaiIvno
3d Printed Furniture by Paul Lagier 7:13
https://www.youtube.com/watch?v=f3j8getccfA
=DigiKey Spotlight=
Making Sound Reactive Projects by Becky Stern 7:56
https://www.youtube.com/watch?v=TnpJTVUOYdA
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ In My Lane, Flourishing [Maker Update] | Maker.io](https://i.ytimg.com/vi/gBiG8B-54S4/mqdefault.jpg)

![Headfirst [Maker Update] | Maker.io
Headfirst [Maker Update] | Maker.io Headfirst [Maker Update] | Maker.io](https://i.ytimg.com/vi/g_b9DkZBczE/mqdefault.jpg)
![Playing by the Rules [Maker Update] | Maker.io
This week on Maker Update: an instrument that rules, street music, a mouse that raves, and a couple new boards worth knowing about.
+Show Notes [Maker Update Ep.501]+
=Project of the Week=
Ruler Drive - Automatic Ruler Playing Robot by XA10C4 0:25
https://www.youtube.com/watch?v=1HWUR4l9Tc4
Also see: Slide Whistle Orchestra by Mitxela
https://www.youtube.com/watch?v=6JeV53LZ8ag
=More Projects=
The 101 Plays Itself by Jay Judah 2:41
https://jayjudah.com/101/#theremin,glass
Party Mode Mouse Droid By heychaostheory 3:42
https://www.instructables.com/Autonomous-Mouse-Droid-With-Party-Mode
=Tips & Tools=
Maker Update Full Online Project Archive 4:40
https://donaldbell.github.io/maker-update-archive/
Maker Update Project/Tip Randomizer 5:08
https://donaldbell.github.io/maker-update-archive/discover.html
M5Stack Faces Series 5:26
https://www.youtube.com/watch?v=rB1nRhG64yg
M5STACK Faces 3 Bottom Board:
https://www.digikey.com/en/products/detail/m5stack-technology-co-ltd/A168/30320687
M5STACK Faces Gamepad Panel v3.0:
https://www.digikey.com/en/products/detail/m5stack-technology-co-ltd/A004-V3/30320690
M5STACK Faces Calculator Panel v3.0:
https://www.digikey.com/en/products/detail/m5stack-technology-co-ltd/A005-V3/30320688
M5STACK Faces KeyBoard Panel v3.0:
https://www.digikey.com/en/products/detail/m5stack-technology-co-ltd/A003-V3/30320691
LattePanda Mu Ultra 6:00
https://www.lattepanda.com/lattepanda-mu-ultra
Find on DigiKey:
https://www.digikey.com/en/products/detail/dfrobot/DFR1294/30277222
=DigiKey Spotlight=
Arduino Project to Product by Shawn Hymel 6:38
https://youtube.com/playlist?list=PLEBQazB0HUyQd6Fsf5NQ75M9llbi1_j_8&si=PpJsuLgViBPC73DR
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ Playing by the Rules [Maker Update] | Maker.io](https://i.ytimg.com/vi/gvPlZ3dis-0/mqdefault.jpg)
![Portable Worlds [Maker Update] | Maker.io
This week on Maker Update: a world in the palm of your hand, Modulinos for motors, a brass helicopter, and a wooden rocket.
+Show Notes [Maker Update Ep.490]+
=Project of the Week=
Mur Mur by OiO and Mattering 0:25
https://www.murmur.living/
https://www.creativeapplications.net/member/mur-mur-a-world-inside-a-speaker/
=News=
Three new Arduino Modulino modules 2:57
https://blog.arduino.cc/2026/06/26/three-new-arduino-modulino-modules-are-here-bigger-ideas-now-come-with-zero-added-stress/
Shop Arduino on DigiKey:
https://www.digikey.com/en/supplier-centers/arduino
=More Projects=
DIY Helicopter Lamp With a Real Spinning Rotor by Huy Vector 3:46
https://www.youtube.com/watch?v=ivO_DXM4fLc
But Its Still Better Than the New Tech by Wesley Treat 5:15
https://www.youtube.com/watch?v=yVdXJoRjJrk
=Tips & Tools=
WLED 16: Whats New by Erin St. Blaine 5:47
https://learn.adafruit.com/wled-16-what-is-new
10 Ways to Make Perfect 3D Printed Holes by Slant 3D 6:23
https://www.youtube.com/watch?v=3BIvSiuQVWQ
=DigiKey Spotlight=
Electronics with Becky Stern 7:05
https://www.youtube.com/playlist?list=PLEBQazB0HUyTF7kB7v1S77F9QK0gT Kt
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ Portable Worlds [Maker Update] | Maker.io](https://i.ytimg.com/vi/gxVZIdy60wc/mqdefault.jpg)

![Perfect Pitch Printer [Maker Update] | Maker.io
This week on Maker Update: a song you can print in plastic, a tape measure for football fields, a custom control panel, and adding textures to your 3d prints.
+Show Notes [Maker Update Ep.479]+
=Project of the Week=
3d Printing a Song by Super Valid Designs 0:27
https://www.youtube.com/watch?v=QZZD3ODLs_Y
=More Projects=
Guinness World Record Tape Measure by Ali Spagnola 3:02
https://www.youtube.com/watch?v GptQ0BmwkI
Rocket Selfie Camera by BPS.Space 4:00
https://www.youtube.com/watch?v=G61T-5d9jiA
Lego-Style Sim Racing Dash by Xshift 4:52
https://www.youtube.com/watch?v=qbU20-ULeZM
=Tips & Tools=
Every Hardware Protocol Explained in 6 Minutes by Tech Explained 5:30
https://www.youtube.com/watch?v=0rlpwVNyBO8
Any Texture On Your 3d Prints by CNC Kitchen 6:00
https://www.youtube.com/watch?v=rTBkjR7JvzI
https://bumpmesh.com/
3d Printed Concrete Molds by CNC Bratan 6:31
https://www.youtube.com/watch?v=ig0CmUef3sQ
=DigiKey Spotlight=
Product Unboxing: M5Stack Tab5 IoT Dev Kit 7:11
https://www.youtube.com/watch?v=ok58Z6icQF4
Find the M5Stack Tab5 IoT Dev Kit on DigiKey
https://www.digikey.com/en/products/detail/m5stack-technology-co-ltd/K145/26740594
#makerupdate
Maker Update is a weekly video series covering news, projects, tips, and events that interest the maker and DIY community.
Find more online:
Blog: http://makerprojectlab.com
Instagram: https://www.instagram.com/makerprojectlab/ Perfect Pitch Printer [Maker Update] | Maker.io](https://i.ytimg.com/vi/hWQGnH9OBa8/mqdefault.jpg)
