https://sky-2002.github.io/tags/x2/Recent content in X2 on Aakash ThatteHugoen-usSun, 13 Sep 2026 00:00:00 +0000https://sky-2002.github.io/posts/2026-09-13-tiny-rlvr-02/Sun, 13 Sep 2026 00:00:00 +0000https://sky-2002.github.io/posts/2026-09-13-tiny-rlvr-02/<p>This is the second post I am writing to discuss about a series of experiments. The first post tries to answer what happens to a model after RL. This post focuses on another question - <strong>How much can RL extract from a model, and is that amount already fixed by what pretraining supplied?</strong></p> <p>The question comes from an observation(mentioned in the <a href="https://sky-2002.github.io/posts/2026-08-30-tiny-rlvr-01/">first post</a>) that in the last 20 updates of my runs, around 53% of GRPO groups returned identical rewards - all G completions correct, or all G wrong. Identical rewards mean zero advantage, zero gradient, no learning. The all-correct case indicates nothing left to improve, and the all-wrong case indicates that the model is not sampling anything to reinforce on(neither positive nor negative), for learning correct behaviour, there should at least be one correct completion in the G(8 in my case). If that is the mechanism, then RL can only ever reinforce what the base model already reaches within G tries.</p>