RL

I have been following the discourse around RL, hearing a lot of terminology (like async RL, sample efficiency etc) and how people are using it to post-train models, and I did RL on smaller models like Qwen by following the docs of trl, unsloth. I could see the rewards going up, sometimes wiggling, sometimes 0, at times answer still being wrong at the end of training. I knew the math, the code, but I wanted to go deeper and had a lot of questions, like: what exactly changes in a model’s behaviour after RL? does it depend on how the pretraining was done? how stale can the rollouts be? - and many more. So I did some experiments and now writing this post to present those. ...

After learning about RL for LLMs in research papers, some nice youtube playlists and reading some amazing blogs, I also wanted to write one, which can introduce RL for LLMs for someone who already knows LLMs, and one which I can refer anytime as a refresher. I have tried to build this blog right from softmax all the way to GRPO, and also tried to show calculations on a five-token toy example(which I will skip in one of the sections). ...