I Trained Qwen 4B With RL to Catch Qwen 27B @engineerprompt
I Trained Qwen 4B With RL to Catch Qwen 27B  @engineerprompt
Uploaded October 2026 | Updated November 2026, 1 day ago
Can RL Make a 4B Model Compete with 27B? My Open RL Environment Experiments (GRPO, LoRA, DGX)

In this video I test whether open-source RL environments can make a small model perform closer to a much larger one by training Qwen 3.5 4B (with a LoRA) against Kaggle-style “small data” RL tasks on two DGX Sparks and comparing to Qwen 27B on a held-out exam. I explain what RL environments are (sandbox, tools, checker), why you need a strict checker to avoid reward hacking, and how GRPO works and only learns from tasks the model sometimes gets right. After weak gains from random tasks, I filter for “edge” tasks and get clear improvement, closing over half the gap, mostly by teaching the model to finish and answer correctly. Further RL hits a ceiling, so I try distillation from 27B, then RL on top, which gives the best results on unseen datasets. I share takeaways and put code, checker, and recipes on GitHub.

Thanks to FireCrawl for sponsoring this video:


Firecrawl, the context API for AI agents (free tier to get started)
firecrawl.link/promptengineer
💻 Code: harness, strict checker, training scripts and results
github.com/PromtEngineer/smoldataenvs-grpo-spark

📚 Resources
SmolDataEnvs (the RL environments): huggingface.co/datasets/FineEnvs/SmolDataEnvs
FineEnvs: github.com/adithya-s-k/FineEnvs
Qwen3.5-4B: huggingface.co/Qwen/Qwen3.5-4B
Qwen3.5-27B: huggingface.co/Qwen/Qwen3.5-27B
TRL GRPO Trainer: huggingface.co/docs/trl/grpo_trainer
NVIDIA DGX Spark: nvidia.com/en-us/products/workstations/dgx-spark
Xiaomi MiMo-V2.6 (open RL environments): mimo.mi.com/docs/en-US/news/latest/v2-6
TechCrunch on RL environments: techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents


Website: engineerprompt.ai
RAG Beyond Basics Course:
prompt-s-site.thinkific.com/courses/rag
Let's Connect:
🦾 Discord: discord.com/invite/t4eYQRUcXB
☕ Buy me a Coffee: ko-fi.com/promptengineering
|🔴 Patreon: patreon.com/PromptEngineering
💼Consulting: calendly.com/engineerprompt/consulting-call
📧 Business Contact: engineerprompt@gmail.com
Become Member: tinyurl.com/y5h28s6h
💻 Pre-configured localGPT VM: bit.ly/localGPT (use Code: PromptEngineering for 50% off).

Signup for Newsletter, localgpt:
https://tally.so/r/3y9bb0


00:00 Can RL Shrink the Gap
01:24 What RL Environments Are
02:45 Reward Hacking and Checkers
03:37 Where RL Fits in Training
04:32 GRPO and Learning Signal
05:17 Hardware Setup and Baselines
06:18 Sponsor Firecrawl
07:51 Training on Edge Tasks
09:39 Before and After Runs
11:37 Why Progress Stalls
12:38 Distillation from 27B
13:51 Best Combo Model Example
14:52 Takeaways and Wrap Up
I Trained Qwen 4B With RL to Catch Qwen 27BQwen Calling Home?Stop Building Agent Loops. This Is Replacing ThemNew Claude Features For DevelopersCodex Just Became the Everything AppGemini Omni Flash: Anything to Anything model from GoogleExploration is All You Need!JEV vs Thinking Models: 100% to 0%Training Your Own Embedding Model Is Not As Hard As You ThinkMinimax-Agent: The Ultimate Open-Source Workhorse ModelGoogles New Agent Platform Explainedllama.cpp Web UI: Run Local LLMs Without Open WebUI or Ollama
Prompt Engineering |

I Trained Qwen 4B With RL to Catch Qwen 27B

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER