Small, self-contained experiments on how to score model behaviour and train on those scores. Each folder is one idea, kept short enough to read in one sitting.
A language model can produce many different answers to the same input. To train it with reinforcement learning you need a number that says how good each answer was. That number is called the reward. These experiments cover three ways to get that number, and one way to use it.
- Learn the reward from human preference data (
reward-model). - Ask a strong model to compare answers and score them (
llm-judge-scorer). - Use a judge inside a real training loop on a tool-using agent (
sql-agent).
A recurring idea in all three: it is easier and more reliable to compare answers against each other than to grade one answer on its own. Absolute scores drift. Relative rankings stay stable.
| Folder | What it does |
|---|---|
| reward-model/ | Trains a reward model on pairs of preferred and rejected answers, using the Bradley-Terry loss. |
| llm-judge-scorer/ | Uses GPT-4o as a judge to score a group of answers relative to each other, then turns the scores into GRPO advantages. |
| sql-agent/ | Trains a small model to answer questions over a SQLite database by calling tools, with rewards from an LLM judge. |
- Trajectory / rollout: one full attempt by the model at a task, including every tool call and result along the way.
- Reward: a single number for how good one attempt was.
- Group: several attempts at the same task, sampled from the same model. Scoring within a group is what makes comparison possible.
- Advantage: the reward minus the group average, divided by the group standard deviation. It says whether an attempt was better or worse than the model's usual behaviour on that task. This is what GRPO trains on.
- Judge: a separate, usually stronger model that reads attempts and scores them.
The notebooks were written for a Colab T4 GPU. The llm-judge-scorer script
runs on CPU and only needs an API key. Each folder has its own README with the
details.