Reinforcement Learning for Robotics Part 2: Train a Balance Bot with PPO | DigiKey @digikey
Reinforcement Learning for Robotics Part 2: Train a Balance Bot with PPO | DigiKey  @digikey
Uploaded August 2026 | Updated November 2026, 13 hours ago
In this series, we explore using reinforcement learning algorithms (RL) to have a robot learn to balance on its own. We will deploy the trained agent to a real ESP32-based robot platform, tackle the sim-to-real gap using post-processing and domain randomization, and eventually build a full remote-controlled balance bot. The same techniques used here are what researchers use to train state-of-the-art walking robots. So, while we are starting with a simpler project, know that these skills will transfer to modern robotics.

To follow along with the series, you will need the following hardware:
M5Stack BALA2 Fire robot kit: digikey.com/short/7h2fb5hd

All project files, including the FreeCAD model, MuJoCo XML descriptor, and Docker environment, are available in the following GitHub repository:
github.com/ShawnHymel/reinforcement-learning-for-robotics

In this episode, we develop a Farama Gymnasium environment wrapper for our MuJoCo simulation and use Proximal Policy Optimization (PPO) to train a balance bot agent entirely in simulation. We cover how to design a reward function, build a custom Gymnasium environment, and use curriculum learning to shape the agent's behavior across two training phases.

You can read a full write-up of everything we cover in this video here:
digikey.com/en/maker/tutorials/2026/reinforcement-learning-for-robotics-part-2-train-a-balance-bot-with-ppo

This episode builds directly on episode 1, where we imported our robot into MuJoCo. By the end, we have a trained agent that balances reliably in simulation, ready to be deployed to real hardware in the next episode.

We start with a quick refresher on the core RL loop (observations, actions, rewards, and the agent/environment distinction) before diving into reward function design. From there, we build the Gymnasium environment wrapper, walk through the PPO training algorithm and its actor-critic architecture, and introduce curriculum learning as a way to avoid reward hacking. We finish by training the agent across two phases and inspecting the resulting actor network in Netron.

You can read a full write-up of everything we cover in this video here:
digikey.com/en/maker/tutorials/2026/reinforcement-learning-for-robotics-part-2-train-a-balance-bot-with-ppo

Click here to register for Shawn’s webinar: event.on24.com/wcc/r/5384134/958EFFE494BF1C6A2AD94F9B093DC01B?partnerref=vid

Coursera Deep Learning specialization: coursera.org/specializations/deep-learning
Coursera Reinforcement Learning specialization: coursera.org/specializations/reinforcement-learning
Introduction to Reinforcement Learning textbook by Sutton and Barto: incompleteideas.net/book/the-book-2nd.html
Reinforcement Learning math blog post series: shawnhymel.com/3316/what-is-reinforcement-learning
Introduction to Reinforcement Learning video: youtube.com/watch?v=3av8vozEczU

Maker.io - digikey.com/en/maker
DigiKey’s Blog – TheCircuit digikey.com/en/blog
Connect with DigiKey on Facebook facebook.com/digikey.electronics
And follow us on X: https://x.com/digikey

Product:
K014-E M5Stack Technology Co., Ltd. | Robotics Kits | DigiKey: digikey.com/en/products/detail/m5stack-technology-co-ltd/K014-E/16679752
K038-B M5Stack Technology Co., Ltd. | Robotics Kits | DigiKey:
digikey.com/en/products/detail/m5stack-technology-co-ltd/K038-B/16679760

0:00 Intro
1:10 Overview of reinforcement learning
4:28 Gymnasium environment
15:02 Testing the environment
19:03 PPO overview
20:42 CleanRL PPO implementation
44:13 Phase 1 training
49:43 TensorBoard plots
55:15 Phase 2 training
57:16 TensorBoard plots and evaluation
59:56 Export actor network to ONNX
1:01:22 Conclusion
Reinforcement Learning for Robotics Part 2: Train a Balance Bot with PPO | DigiKey
DigiKey |

Reinforcement Learning for Robotics Part 2: Train a Balance Bot with PPO | DigiKey

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER