https://sky-2002.github.io/tags/policy-gradients/Recent content in Policy-Gradients on Aakash ThatteHugoen-usSat, 01 Aug 2026 00:00:00 +0000https://sky-2002.github.io/posts/2026-08-01-llm-rl-starter/Sat, 01 Aug 2026 00:00:00 +0000https://sky-2002.github.io/posts/2026-08-01-llm-rl-starter/<p>Working through RL for LLMs from softmax all the way to a full GRPO update, on a five-token toy where the numbers stay hand-checkable. One section steps off the toy to sketch multi-step credit assignment.</p> <p>Goal is that we have a model that sometimes solves a task → make it solve the task more often.</p> <h2 id="language-model-is-a-probability-distribution">Language model is a probability distribution</h2> <p>During each step of autoregressive generation, an LLM produces a score (logit) for each token in the vocabulary, then converts those to probabilities with softmax. Consider the prompt <code>2+3=</code> given to our toy model whose vocabulary is <code>{0, 4, 5, 6, 15}</code>. Logits are deliberate multiples of $\ln 2$ so the softmax stays as fractions:</p>