TRL — Transformers Reinforcement Learning
Module FTDD-04 · Course 3 — LLM Fine-Tuning Masterclass
45 minutes · 4 sub-sections: The Substrate · The Six Trainers · The Production CLI · TRL in the Ecosystem
The library the entire open-weights post-training world is built on.
Deep-Dives
The substrate of the ecosystem
When you reach Layer 3 of the Steering Stack — the actual fine-tuning — one library the ecosystem converges on: TRL.
75+
post-training methods
v1.0 released March 31, 2026 — shipped a Stability Contract and a production CLI.
Why it won: the thin-wrapper principle
Every TRL trainer is a thin wrapper over the HuggingFace Trainer.
It does not reimplement the training loop. It subclasses Trainer, overrides compute_loss, and inherits everything else.
For free, you inherit:
- The optimizer (AdamW, fused / 8-bit)
- The schedulers, data collation, chat templates
- DDP · DeepSpeed ZeRO · FSDP — out of the box
- Checkpointing, logging, callbacks
This is why it scales to 405B full-param on clusters. Scaling is DeepSpeed/FSDP's problem; TRL gets out of the way.
The v1.0 Stability Contract
Trainer class names, constructor signatures, config keys, and CLI flags are now a maintained surface.
What is covered
- Trainer names (SFTTrainer, DPOTrainer...)
- Config / YAML keys
- CLI flags
- Constructor signatures
What is NOT covered
- Default values (may improve)
- Experimental trainers
In production, pin every hyperparameter explicitly. Never let a value ride on "whatever the default is this week."
The six trainers
Each steers a different thing. Know which maps to which goal.
| Trainer | Steers | Data shape |
|---|
| SFTTrainer | format, instructions | prompt + completion |
| DPOTrainer | preference (better/worse) | chosen vs rejected pairs |
| KTOTrainer | preference (unpaired) | thumbs up / down |
| GRPOTrainer | reasoning (verifiable reward) | reward function |
| RLOOTrainer | reasoning (verifiable reward) | reward function |
| RewardTrainer | learned reward for RLHF | preference -> scorer |
The decision shortcut
| Your goal / data | Reach for |
|---|
| New format or instruction style | SFTTrainer |
| "Better/worse" preferences, with pairs | DPOTrainer (or ORPO to skip SFT) |
| Preferences, unpaired thumbs up/down | KTOTrainer |
| Verifiable reward (math, code, format) | GRPOTrainer (or RLOO) |
| Learned reward for classical RLHF | RewardTrainer + PPO / async RL |
GRPO reinforces whatever the base can already do. It does not install capability. SFT first, then GRPO to sharpen.
The production CLI
The same trainers. No training code.
trl sft --config sft.yml # SFTTrainer
trl dpo --config dpo.yml # DPOTrainer
trl grpo --config grpo.yml # GRPOTrainer
Both the CLI and the Python API call the same trainer classes with the same config schema.
Why the CLI wins in production: no training code to maintain · reproducibility is a file · CI-friendly (lint + schema-validate the YAML). The CLI is the 80% path; drop to the API for custom rewards or non-standard loops.
TRL in the ecosystem
| Tool | Relationship to TRL | Best for |
|---|
| Axolotl | Wraps TRL (config + multi-GPU) | Production, multi-GPU, reproducibility |
| Unsloth | Replaces kernels (Triton, TRL-compatible API) | Single-GPU speed, sub-30B |
| Raw TRL | Is the substrate (API or CLI) | Full control, research, freshest methods |
Knowing the TRL API is the Rosetta Stone for the whole ecosystem. New trainers land here first; the wrappers follow.
Anti-patterns
Treating TRL as research-grade only. Post-v1.0 + the Stability Contract, the CLI is the production path. Wrappers are optional ergonomics, not safety requirements.
Reaching for GRPO before SFT. GRPO amplifies whatever the base can do. If the base is incoherent, GRPO reinforces incoherent reasoning. SFT first, then GRPO to sharpen.
Choosing a trainer by novelty. DPO is not "better" than SFT because it is newer. Each optimizes a different objective for a different data shape. Choose by goal + data, not fashion.
What you can now do
- Explain why TRL is the substrate and what the v1.0 Stability Contract guarantees.
- Map the six trainers to steering goals and pick the right one for a given dataset.
- Contrast the CLI vs the Python API and justify the CLI for production.
- Place TRL in the ecosystem: Axolotl wraps, Unsloth competes, raw TRL controls.
Next: FTDD-05 — Axolotl · The declarative wrapper over TRL