Course: Course 3 — LLM Fine-Tuning Masterclass
Module: FTDD-04 — TRL (Transformers Reinforcement Learning)
Duration: ~45 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
This is deep-dive FTDD-04 — TRL, Transformers Reinforcement Learning. Forty-five minutes. By the end you will understand the single library the entire open-weights post-training world is built on, and why every other tool in these deep-dives is defined by how it relates to TRL.
If you have been through module FT eleven, the training loop, you have already met TRL in passing. Here we go deep.
[SLIDE 2 — The substrate]
When you reach layer three of the Steering Stack — the actual fine-tuning — there is one library the ecosystem converges on. TRL, maintained by HuggingFace. As of its one-point-zero release on March thirty-first, twenty-twenty-six, it is downloaded three million times a month and implements more than seventy-five post-training methods. Three million a month. That is not a research toy. That is infrastructure.
Every higher-level tool you will meet wraps or competes with TRL. Axolotl wraps it. Unsloth competes by replacing its kernels. If you internalize the TRL trainer API, you have the Rosetta Stone for the whole field.
[SLIDE 3 — The thin-wrapper principle]
Why did TRL win? It is structural, not marketing. Every TRL trainer is a thin wrapper over the HuggingFace Trainer. It does not reimplement the training loop. It subclasses Trainer, overrides compute-loss to inject the SFT, DPO, or GRPO objective, and inherits everything else. The optimizer. The schedulers. The data collation and chat templates. And — critically — the distributed story: DDP, DeepSpeed ZeRO, FSDP, all out of the box.
That is why it scales to four-hundred-and-five-billion-parameter full-parameter training on clusters without falling over. The scaling is DeepSpeed and FSDP's problem. TRL gets out of their way.
[SLIDE 4 — The Stability Contract]
Before one-point-zero, TRL was an experiment. Trainer signatures, config keys, defaults shifted between minor versions. That churn is precisely why downstream projects wrapped it — partly for ergonomics, partly as a stability firewall.
The one-point-zero release shipped an explicit Stability Contract. Trainer class names, constructor signatures, config and YAML keys, and CLI flags are now a maintained surface. Breaking changes require a major version bump and a migration guide. That is what let TRL graduate from "thing researchers import" to "thing production teams pin in a requirements file."
But note the boundary. The contract covers the API surface. It does not freeze the defaults. TRL may improve a default learning rate or scheduler between minor versions, with a deprecation cycle. So in production, pin every hyperparameter explicitly. Never let a value ride on whatever the default is this week.
[SLIDE 5 — The six trainers]
Six classes. Each steers a different thing. This is the core of the module.
SFT trainer — the baseline. Steers format and instruction-following. Input is a prompt-completion dataset. This is the workhorse; about ninety percent of real jobs.
DPO trainer — preference. Input is chosen-versus-rejected pairs. Direct Preference Optimization. No reward model; it derives the gradient directly from the policy with a reference-model anchor.
KTO trainer — preference but with unpaired data. Only needs thumbs-up, thumbs-down per response. This matters because paired preference data is expensive to collect; binary feedback is what you actually get from a production thumbs-up button.
GRPO and RLOO — reasoning on verifiable rewards. Both generate multiple completions and reinforce the good ones. GRPO is the trainer behind most of the open-weights reasoning boom — it is what you reach for when the reward is "did the math check out" or "did the code pass." RLOO is the simpler, more sample-efficient alternative.
Reward trainer — trains a learned reward model for classical RLHF, the PPO path with a learned reward rather than a verifiable one.
[SLIDE 6 — The decision shortcut]
For most real work the decision tree is short. New format or instruction style — SFT. Better-or-worse preferences with pairs — DPO, or ORPO if you want to skip the SFT stage. Preferences but only thumbs up and down — KTO. Verifiable reward and you want reasoning — GRPO, or RLOO. Learned reward for classical RLHF — reward trainer plus PPO.
And the warning that matters most: GRPO reinforces whatever the base can already do. It does not install capability. If your base produces incoherent reasoning, GRPO will faithfully reinforce incoherent reasoning. SFT first to establish format and basic competence, then GRPO to sharpen. We cover this in module fourteen.
[SLIDE 7 — The production CLI]
Two equivalent surfaces into the same engine. The Python API — instantiate a config, a trainer, call train. Full control. And the CLI — trl SFT, trl DPO, trl GRPO, each with a YAML config. Both call the same trainer classes with the same config schema.
The CLI is not a toy. It is the production path for most jobs, for three reasons. One: no training code to maintain. A YAML file is config, not code — no imports, no dependency drift. Two: reproducibility is a file. The entire recipe lives in one YAML you can diff and version and hand to a colleague. Three: CI-friendly. You can lint and schema-validate a YAML before it touches a GPU. You cannot do that with a two-hundred-line training script.
The trade-off: the CLI is configured, not programmed. The moment you need a custom reward function for GRPO, or a custom data transform, or an interleaved eval loop, you drop down to the Python API. The CLI's scope is the eighty percent of jobs that fit standard recipes.
[SLIDE 8 — TRL in the ecosystem]
TRL is the substrate. The two tools most practitioners actually use are defined by how they relate to it.
Axolotl wraps TRL. You write a YAML config; Axolotl translates it into a TRL run and adds two things TRL alone does not give you: opinionated defaults across model families, and battle-tested multi-GPU orchestration. The FSDP and DeepSpeed configs that are fiddly to get right by hand. Underneath, the trainer is still SFTTrainer or DPOTrainer from TRL.
Unsloth takes a different tack. Rather than wrap TRL, it rewrites the performance-critical kernels — attention, the LoRA backward pass — in hand-tuned Triton, and exposes a TRL-compatible API so your script barely changes. The payoff is dramatically higher throughput on a single GPU. Often two times faster, half the VRAM. Unsloth is the answer for the single-GPU, sub-thirty-B, speed-critical case. Its multi-GPU story is limited, and it trails TRL on the newest methods.
When is raw TRL the answer? When you need full control — custom rewards, custom data, research objectives not in any wrapper. When you are doing research, not production. When you want the freshest methods — new trainers land in TRL first. And when your job fits a standard recipe and you want zero code — trl SFT dash dash config is the shortest path with no wrapper dependency.
[SLIDE 9 — Anti-patterns]
Three anti-patterns.
First, treating TRL as research-grade only. Because of its name and origins, some teams reach for a wrapper for production. Post one-point-zero and the Stability Contract, that is outdated. For standard recipes, the TRL CLI is the production path.
Second, reaching for GRPO before SFT. The reasoning-RL boom makes GRPO glamorous. But GRPO amplifies whatever the base can do. SFT first.
Third, choosing a trainer by novelty. DPO is not better than SFT because it is newer. KTO is not better than DPO. Each optimizes a different objective for a different data shape. Choose by goal and data, not fashion.
[SLIDE 10 — What you can now do]
You can now explain why TRL is the substrate and what the one-point-zero Stability Contract guarantees. You can map the six trainers to steering goals and pick the right one for a dataset. You can contrast the CLI and the Python API and justify the CLI for production. And you can place TRL in the ecosystem — Axolotl wraps, Unsloth competes, raw TRL controls.
Next, deep-dive FTDD-zero-five: Axolotl. The declarative wrapper over TRL, and the production path for multi-GPU. Let's see what a wrapper buys you.
End of module FTDD-04. Duration: approximately forty-three minutes at one-hundred-forty words per minute.
# Teaching Script — Module FTDD-04: TRL **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FTDD-04 — TRL (Transformers Reinforcement Learning) **Duration**: ~45 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] This is deep-dive FTDD-04 — TRL, Transformers Reinforcement Learning. Forty-five minutes. By the end you will understand the single library the entire open-weights post-training world is built on, and why every other tool in these deep-dives is defined by how it relates to TRL. If you have been through module FT eleven, the training loop, you have already met TRL in passing. Here we go deep. [SLIDE 2 — The substrate] When you reach layer three of the Steering Stack — the actual fine-tuning — there is one library the ecosystem converges on. TRL, maintained by HuggingFace. As of its one-point-zero release on March thirty-first, twenty-twenty-six, it is downloaded three million times a month and implements more than seventy-five post-training methods. Three million a month. That is not a research toy. That is infrastructure. Every higher-level tool you will meet wraps or competes with TRL. Axolotl wraps it. Unsloth competes by replacing its kernels. If you internalize the TRL trainer API, you have the Rosetta Stone for the whole field. [SLIDE 3 — The thin-wrapper principle] Why did TRL win? It is structural, not marketing. Every TRL trainer is a thin wrapper over the HuggingFace Trainer. It does not reimplement the training loop. It subclasses Trainer, overrides compute-loss to inject the SFT, DPO, or GRPO objective, and inherits everything else. The optimizer. The schedulers. The data collation and chat templates. And — critically — the distributed story: DDP, DeepSpeed ZeRO, FSDP, all out of the box. That is why it scales to four-hundred-and-five-billion-parameter full-parameter training on clusters without falling over. The scaling is DeepSpeed and FSDP's problem. TRL gets out of their way. [SLIDE 4 — The Stability Contract] Before one-point-zero, TRL was an experiment. Trainer signatures, config keys, defaults shifted between minor versions. That churn is precisely why downstream projects wrapped it — partly for ergonomics, partly as a stability firewall. The one-point-zero release shipped an explicit Stability Contract. Trainer class names, constructor signatures, config and YAML keys, and CLI flags are now a maintained surface. Breaking changes require a major version bump and a migration guide. That is what let TRL graduate from "thing researchers import" to "thing production teams pin in a requirements file." But note the boundary. The contract covers the API surface. It does not freeze the defaults. TRL may improve a default learning rate or scheduler between minor versions, with a deprecation cycle. So in production, pin every hyperparameter explicitly. Never let a value ride on whatever the default is this week. [SLIDE 5 — The six trainers] Six classes. Each steers a different thing. This is the core of the module. SFT trainer — the baseline. Steers format and instruction-following. Input is a prompt-completion dataset. This is the workhorse; about ninety percent of real jobs. DPO trainer — preference. Input is chosen-versus-rejected pairs. Direct Preference Optimization. No reward model; it derives the gradient directly from the policy with a reference-model anchor. KTO trainer — preference but with unpaired data. Only needs thumbs-up, thumbs-down per response. This matters because paired preference data is expensive to collect; binary feedback is what you actually get from a production thumbs-up button. GRPO and RLOO — reasoning on verifiable rewards. Both generate multiple completions and reinforce the good ones. GRPO is the trainer behind most of the open-weights reasoning boom — it is what you reach for when the reward is "did the math check out" or "did the code pass." RLOO is the simpler, more sample-efficient alternative. Reward trainer — trains a learned reward model for classical RLHF, the PPO path with a learned reward rather than a verifiable one. [SLIDE 6 — The decision shortcut] For most real work the decision tree is short. New format or instruction style — SFT. Better-or-worse preferences with pairs — DPO, or ORPO if you want to skip the SFT stage. Preferences but only thumbs up and down — KTO. Verifiable reward and you want reasoning — GRPO, or RLOO. Learned reward for classical RLHF — reward trainer plus PPO. And the warning that matters most: GRPO reinforces whatever the base can already do. It does not install capability. If your base produces incoherent reasoning, GRPO will faithfully reinforce incoherent reasoning. SFT first to establish format and basic competence, then GRPO to sharpen. We cover this in module fourteen. [SLIDE 7 — The production CLI] Two equivalent surfaces into the same engine. The Python API — instantiate a config, a trainer, call train. Full control. And the CLI — trl SFT, trl DPO, trl GRPO, each with a YAML config. Both call the same trainer classes with the same config schema. The CLI is not a toy. It is the production path for most jobs, for three reasons. One: no training code to maintain. A YAML file is config, not code — no imports, no dependency drift. Two: reproducibility is a file. The entire recipe lives in one YAML you can diff and version and hand to a colleague. Three: CI-friendly. You can lint and schema-validate a YAML before it touches a GPU. You cannot do that with a two-hundred-line training script. The trade-off: the CLI is configured, not programmed. The moment you need a custom reward function for GRPO, or a custom data transform, or an interleaved eval loop, you drop down to the Python API. The CLI's scope is the eighty percent of jobs that fit standard recipes. [SLIDE 8 — TRL in the ecosystem] TRL is the substrate. The two tools most practitioners actually use are defined by how they relate to it. Axolotl wraps TRL. You write a YAML config; Axolotl translates it into a TRL run and adds two things TRL alone does not give you: opinionated defaults across model families, and battle-tested multi-GPU orchestration. The FSDP and DeepSpeed configs that are fiddly to get right by hand. Underneath, the trainer is still SFTTrainer or DPOTrainer from TRL. Unsloth takes a different tack. Rather than wrap TRL, it rewrites the performance-critical kernels — attention, the LoRA backward pass — in hand-tuned Triton, and exposes a TRL-compatible API so your script barely changes. The payoff is dramatically higher throughput on a single GPU. Often two times faster, half the VRAM. Unsloth is the answer for the single-GPU, sub-thirty-B, speed-critical case. Its multi-GPU story is limited, and it trails TRL on the newest methods. When is raw TRL the answer? When you need full control — custom rewards, custom data, research objectives not in any wrapper. When you are doing research, not production. When you want the freshest methods — new trainers land in TRL first. And when your job fits a standard recipe and you want zero code — trl SFT dash dash config is the shortest path with no wrapper dependency. [SLIDE 9 — Anti-patterns] Three anti-patterns. First, treating TRL as research-grade only. Because of its name and origins, some teams reach for a wrapper for production. Post one-point-zero and the Stability Contract, that is outdated. For standard recipes, the TRL CLI is the production path. Second, reaching for GRPO before SFT. The reasoning-RL boom makes GRPO glamorous. But GRPO amplifies whatever the base can do. SFT first. Third, choosing a trainer by novelty. DPO is not better than SFT because it is newer. KTO is not better than DPO. Each optimizes a different objective for a different data shape. Choose by goal and data, not fashion. [SLIDE 10 — What you can now do] You can now explain why TRL is the substrate and what the one-point-zero Stability Contract guarantees. You can map the six trainers to steering goals and pick the right one for a dataset. You can contrast the CLI and the Python API and justify the CLI for production. And you can place TRL in the ecosystem — Axolotl wraps, Unsloth competes, raw TRL controls. Next, deep-dive FTDD-zero-five: Axolotl. The declarative wrapper over TRL, and the production path for multi-GPU. Let's see what a wrapper buys you. --- *End of module FTDD-04. Duration: approximately forty-three minutes at one-hundred-forty words per minute.*