X1

I have been following the discourse around RL, hearing a lot of terminology (like async RL, sample efficiency etc) and how people are using it to post-train models, and I did RL on smaller models like Qwen by following the docs of trl, unsloth. I could see the rewards going up, sometimes wiggling, sometimes 0, at times answer still being wrong at the end of training. I knew the math, the code, but I wanted to go deeper and had a lot of questions, like: what exactly changes in a model’s behaviour after RL? does it depend on how the pretraining was done? how stale can the rollouts be? - and many more. So I did some experiments and now writing this post to present those. ...