RL

Working through RL for LLMs from softmax all the way to a full GRPO update, on a five-token toy where the numbers stay hand-checkable. One section steps off the toy to sketch multi-step credit assignment. Goal is that we have a model that sometimes solves a task → make it solve the task more often. Language model is a probability distribution During each step of autoregressive generation, an LLM produces a score (logit) for each token in the vocabulary, then converts those to probabilities with softmax. Consider the prompt 2+3= given to our toy model whose vocabulary is {0, 4, 5, 6, 15}. Logits are deliberate multiples of $\ln 2$ so the softmax stays as fractions: ...