Course: Course 3 — LLM Fine-Tuning Masterclass Module: FTDD-04 — TRL Duration: 45 minutes Level: Senior Engineer and above Prerequisites: FT00 (The Steering Stack), FT11 (The Training Loop)
After completing this module, you will be able to:
trl sft / dpo / grpo), and explain why the CLI is how most production jobs are launched.The library the entire open-weights post-training world is built on. Understand why it won, and the rest of the ecosystem becomes legible.
When you reach Layer 3 of the Steering Stack — the actual fine-tuning — there is one library the ecosystem converges on: TRL (Transformers Reinforcement Learning), maintained by HuggingFace. As of its v1.0 release on March 31, 2026, it is downloaded three million times a month and implements more than seventy-five post-training methods. Every higher-level tool you will meet in these deep-dives either wraps TRL (Axolotl) or competes with it by replacing pieces of it (Unsloth). If you internalize TRL's trainer API, you have the Rosetta Stone for the whole field.
The name is now slightly misleading. TRL began as a Reinforcement Learning library (it shipped PPO for LLMs before almost anyone else), but it long ago absorbed supervised fine-tuning and the entire preference-optimization family. "SFT is in a library called Reinforcement Learning" is a standing forum joke. The accurate description today: TRL is the full-stack post-training library. SFT, the DPO family, RL on verifiable rewards, reward modeling, and distillation variants all live here.
The reason TRL won is structural, not marketing. Every TRL trainer is a thin wrapper over the HuggingFace Trainer (and its modern successor, Transformers). That means you inherit, for free:
A TRL trainer does not reimplement the training loop. It subclasses Trainer, overrides compute_loss to inject the SFT / DPO / GRPO objective, and otherwise hands control back to the battle-tested Transformers machinery. This is why it scales to 405B-parameter full-parameter training on clusters without falling over: the scaling is DeepSpeed and FSDP's problem, and TRL gets out of their way.
Before v1.0, TRL was treated as an experiment. Trainer signatures, config keys, and default behaviors shifted between minor versions, which is precisely why downstream projects (Axolotl, Unsloth) wrapped it — partly for ergonomics, partly as a stability firewall against TRL's churn.
The v1.0 release (March 31, 2026) changed the framing. It shipped an explicit Stability Contract: trainer class names, their constructor signatures, the TRLConfig / YAML keys, and the CLI flags are now maintained as a stable surface across releases. Breaking changes require a major version bump and a migration guide. This is what allowed TRL to graduate from "thing researchers import" to "thing production teams pin in a requirements.txt." The three-million-monthly-downloads figure is the market ratifying that decision: when a library is downloaded that often, it is infrastructure.
The contract's boundary. Stability covers the API surface (trainer names, config keys, CLI flags). It does not freeze the defaults. TRL may change a default learning rate or a default scheduler between minor versions if the new default is strictly better, with a deprecation cycle. Pin your config explicitly — never rely on an unspecified default in production.
Six classes. Each steers a different thing. Knowing which trainer maps to which steering goal is the core of this module.
All six live in trl. Each takes a model, a dataset, and a config; each overrides compute_loss with a different objective; each returns a trained model (full-parameter or adapter, your choice via PEFT). Here is the map, ordered by where they sit on the Steering Stack.
Steers format and instruction-following. Input: a prompt-completion or chat dataset. Objective: plain next-token cross-entropy, but only on the completion tokens (the prompt is masked). This is the workhorse — it is what ~90% of real fine-tuning jobs use. When in doubt, start here (Module FT12).
SFTTrainer also handles the messy ergonomics that vanilla Transformers SFT does not: applying the chat template, packing short sequences for throughput, masking the prompt, and handling multi-turn conversations correctly. These details are where hand-rolled SFT loses hours.
Steers preference — "this response is better than that one." Input: a preference dataset of (prompt, chosen, rejected) triples. Objective: the Direct Preference Optimization loss (Rafailov et al., 2023), which needs no reward model — it derives the preference gradient directly from the policy with a reference-model KL anchor. This is the default for "make the model prefer concise answers" / "prefer safer answers" / "prefer the style in my dataset" (Module FT13).
The DPO family, all in TRL, includes the major variants: IPO (identity preference optimization, fixes DPO's overfitting on noisy preferences), SIMPO (length-normalized, reference-free), CPO (contrastive), and ORPO (combines SFT and preference in one pass, no reference model). If DPO is your default, ORPO is worth knowing — it removes the separate SFT stage.
Steers preference but with unpaired data. Kahneman-Tversky Optimization (Ethayarajh et al., 2024) only needs binary thumbs-up / thumbs-down labels per response, not chosen/rejected pairs. This matters because paired preference data is expensive to collect (you need two responses and a ranking); binary feedback is what you actually get from a thumbs-up button in production. KTO turns that signal into a preference gradient.
Steers reasoning and behavior where you have a verifiable reward. Both are RL methods that generate multiple completions per prompt and reinforce the good ones.
Both take a reward function (Python callable) rather than a reward model. This is a deliberate design: verifiable rewards (unit tests, math verification, format checkers) are more reliable than a learned reward model, which is itself a model that can be reward-hacked.
RewardTrainer trains a reward model — a classifier head on top of the base that scores response quality — for use in classical RLHF (PPO with a learned reward). You reach for this when you do not have a verifiable reward function and must learn one from preference data.
v1.0 also introduced the async RL path for the PPO-style loop, decoupling generation from the gradient update so you can overlap rollouts and training across devices. This is research-grade machinery; most practitioners will use GRPO/RLOO with a verifiable reward before they touch learned-reward PPO.
For the vast majority of real work, the decision tree is short:
The same trainers, no training code. This is how most real jobs are launched.
TRL exposes two equivalent surfaces. The Python API is what research code and tutorials show: instantiate a SFTConfig, a SFTTrainer, call .train(). You have full control — custom reward functions, custom data collation, interleaving with your own evaluation harness.
The CLI, new in recent releases and a headline feature of v1.0, runs the identical trainers from a YAML config:
trl sft --config sft.yml # runs SFTTrainer
trl dpo --config dpo.yml # runs DPOTrainer
trl grpo --config grpo.yml # runs GRPOTrainer
Both paths call the same trainer classes with the same config schema. The CLI is not a toy. It is the production path for the majority of jobs, for three reasons.
The trade-off: the CLI is configured, not programmed. The moment you need a custom reward function for GRPO (anything beyond a built-in), or a custom data transform, or an interleaved eval loop, you drop down to the Python API. The CLI's scope is the 80% of jobs that fit standard recipes.
This is the part the v1.0 Stability Contract makes load-bearing. The keys in your YAML — model_name_or_path, dataset_text_field, per_device_train_batch_size, lora_r, bf16, deepspeed — are stable across releases. You can pin trl>=1.0,<2.0 and your configs keep working. Before v1.0 this was not true, and it is the single biggest reason teams used Axolotl as a stability firewall. Post-v1.0, the CLI is increasingly the path of least resistance for standard jobs.
TRL is the substrate. The two tools most practitioners actually use — Axolotl and Unsloth — are defined entirely by how they relate to it.
Axolotl (FTDD-05) is a declarative wrapper over TRL. You write a YAML config, Axolotl translates it into a TRL training run, and adds two things TRL alone does not give you: opinionated defaults that work across many model families, and battle-tested multi-GPU orchestration (FSDP and DeepSpeed configs that are fiddly to get right by hand). For production multi-GPU jobs, Axolotl-on-TRL is the dominant pattern. Underneath, the trainer is still SFTTrainer or DPOTrainer from TRL.
The cost of Axolotl's ergonomics is a layer of abstraction and slightly slower iteration on bleeding-edge methods — when a new trainer lands in TRL, it takes time to surface cleanly in Axolotl's config schema.
Unsloth (FTDD-03) takes a different tack. Rather than wrap TRL, it rewrites the performance-critical kernels (the attention, the LoRA backward pass) in hand-tuned Triton, and exposes a TRL-compatible API so your training script barely changes. The payoff is dramatically higher throughput on a single GPU — often 2x faster, half the VRAM. Unsloth is the right answer for the single-GPU, sub-30B, speed-critical case.
The trade-off: Unsloth's multi-GPU story is limited, and it trails TRL on the newest methods. It is an accelerator for the SFT/LoRA common case, not a general replacement.
Reach for raw TRL (Python API or CLI) when:
trl sft --config x.yml is the shortest path from YAML to a trained model, with no wrapper dependency at all.Because of its name and origins, some teams assume TRL is for RL researchers and reach for a wrapper for "production." Post-v1.0 and the Stability Contract, this is outdated. For standard SFT/DPO/GRPO recipes, the TRL CLI is the production path; the wrappers are optional ergonomics, not safety requirements.
The reasoning-RL boom makes GRPO glamorous. But GRPO amplifies whatever the base can already do — it does not install capability. If your base produces incoherent reasoning, GRPO will faithfully reinforce incoherent reasoning. SFT first to establish format and basic competence, then GRPO to sharpen. (Module FT14.)
The Stability Contract covers the API surface, not the defaults. A default learning rate or scheduler may improve between versions. In production, pin every hyperparameter explicitly in your config; never let a value ride on "whatever the default is this week."
DPO is not "better" than SFT because it is newer; KTO is not "better" than DPO. Each trainer optimizes a different objective for a different data shape. Choose by the steering goal and the data you have, not by what is fashionable. The decision shortcut in FTDD-04.2 exists because the wrong trainer on the right data still fails.
| Term | Definition |
|---|---|
| TRL | Transformers Reinforcement Learning — HuggingFace's full-stack post-training library; the substrate of the open-weights ecosystem |
| Stability Contract | TRL v1.0's promise that trainer names, constructor signatures, config keys, and CLI flags are maintained across releases |
| SFTTrainer | The baseline trainer; steers format and instruction-following; masks the prompt, handles chat templates and packing |
| DPOTrainer | Direct Preference Optimization; steers preference from (prompt, chosen, rejected) triples; no reward model |
| KTOTrainer | Kahneman-Tversky Optimization; preference from unpaired binary feedback (thumbs up/down) |
| GRPOTrainer | Group Relative Policy Optimization; RL with a verifiable reward function; the reasoning-model workhorse |
| RLOOTrainer | REINFORCE Leave-One-Out; sample-efficient RL alternative to PPO |
| RewardTrainer | Trains a learned reward model for classical RLHF (PPO) |
| The thin-wrapper principle | Every TRL trainer subclasses the HuggingFace Trainer, inheriting optimizers, schedulers, and DDP/DeepSpeed/FSDP |
trl sft / dpo / grpo |
The production CLI; runs the same trainers as the Python API from a YAML config |
See 07-lab-spec.md. The "TRL in 20 minutes" lab: run an SFT job on a 1B model via the Python API, then re-run the identical job via trl sft from a YAML config, and confirm the two recipes are equivalent. Feel the API and the CLI as two doors into the same engine.
# Deep-Dive FTDD-04 — TRL (Transformers Reinforcement Learning) **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FTDD-04 — TRL **Duration**: 45 minutes **Level**: Senior Engineer and above **Prerequisites**: FT00 (The Steering Stack), FT11 (The Training Loop) --- ## Learning Objectives After completing this module, you will be able to: 1. Explain why TRL is the canonical post-training library and what its v1.0 Stability Contract guarantees (and does not). 2. Map the six core trainers to the Steering Stack layers they operate at, and state when each is the right tool. 3. Contrast the Python-API path against the production CLI (`trl sft` / `dpo` / `grpo`), and explain why the CLI is how most production jobs are launched. 4. Place TRL in the ecosystem: how Axolotl wraps it, how Unsloth competes with it, and when raw TRL is the correct choice. --- # FTDD-04.1 — Why TRL Is the Substrate *The library the entire open-weights post-training world is built on. Understand why it won, and the rest of the ecosystem becomes legible.* ## The one library you cannot avoid When you reach Layer 3 of the Steering Stack — the actual fine-tuning — there is one library the ecosystem converges on: **TRL (Transformers Reinforcement Learning)**, maintained by HuggingFace. As of its v1.0 release on March 31, 2026, it is downloaded **three million times a month** and implements **more than seventy-five post-training methods**. Every higher-level tool you will meet in these deep-dives either wraps TRL (Axolotl) or competes with it by replacing pieces of it (Unsloth). If you internalize TRL's trainer API, you have the Rosetta Stone for the whole field. The name is now slightly misleading. TRL began as a Reinforcement Learning library (it shipped PPO for LLMs before almost anyone else), but it long ago absorbed supervised fine-tuning and the entire preference-optimization family. "SFT is in a library called Reinforcement Learning" is a standing forum joke. The accurate description today: **TRL is the full-stack post-training library.** SFT, the DPO family, RL on verifiable rewards, reward modeling, and distillation variants all live here. ### The thin-wrapper principle The reason TRL won is structural, not marketing. Every TRL trainer is a **thin wrapper over the HuggingFace `Trainer`** (and its modern successor, `Transformers`). That means you inherit, for free: - the same optimizer (AdamW, and the fused/8-bit variants), - the same learning-rate schedulers, - the same data collation and tokenization pipeline, - the same distributed training story — **DDP, DeepSpeed ZeRO, and FSDP** — out of the box, - the same checkpointing, logging, and callback hooks. A TRL trainer does not reimplement the training loop. It subclasses `Trainer`, overrides `compute_loss` to inject the SFT / DPO / GRPO objective, and otherwise hands control back to the battle-tested Transformers machinery. This is why it scales to 405B-parameter full-parameter training on clusters without falling over: the scaling is DeepSpeed and FSDP's problem, and TRL gets out of their way. ### What v1.0 actually changed Before v1.0, TRL was treated as an experiment. Trainer signatures, config keys, and default behaviors shifted between minor versions, which is precisely why downstream projects (Axolotl, Unsloth) wrapped it — partly for ergonomics, partly as a **stability firewall** against TRL's churn. The v1.0 release (March 31, 2026) changed the framing. It shipped an explicit **Stability Contract**: trainer class names, their constructor signatures, the `TRLConfig` / YAML keys, and the CLI flags are now maintained as a stable surface across releases. Breaking changes require a major version bump and a migration guide. This is what allowed TRL to graduate from "thing researchers import" to "thing production teams pin in a `requirements.txt`." The three-million-monthly-downloads figure is the market ratifying that decision: when a library is downloaded that often, it is infrastructure. > **The contract's boundary.** Stability covers the API surface (trainer names, config keys, CLI flags). It does **not** freeze the defaults. TRL may change a default learning rate or a default scheduler between minor versions if the new default is strictly better, with a deprecation cycle. Pin your config explicitly — never rely on an unspecified default in production. --- # FTDD-04.2 — The Six Trainers *Six classes. Each steers a different thing. Knowing which trainer maps to which steering goal is the core of this module.* All six live in `trl`. Each takes a model, a dataset, and a config; each overrides `compute_loss` with a different objective; each returns a trained model (full-parameter or adapter, your choice via PEFT). Here is the map, ordered by where they sit on the Steering Stack. ## SFTTrainer — the baseline Steers **format and instruction-following.** Input: a prompt-completion or chat dataset. Objective: plain next-token cross-entropy, but only on the completion tokens (the prompt is masked). This is the workhorse — it is what ~90% of real fine-tuning jobs use. When in doubt, start here (Module FT12). SFTTrainer also handles the messy ergonomics that vanilla Transformers SFT does not: applying the chat template, packing short sequences for throughput, masking the prompt, and handling multi-turn conversations correctly. These details are where hand-rolled SFT loses hours. ## DPOTrainer — the preference family anchor Steers **preference** — "this response is better than that one." Input: a preference dataset of `(prompt, chosen, rejected)` triples. Objective: the Direct Preference Optimization loss (Rafailov et al., 2023), which needs no reward model — it derives the preference gradient directly from the policy with a reference-model KL anchor. This is the default for "make the model prefer concise answers" / "prefer safer answers" / "prefer the style in my dataset" (Module FT13). The DPO family, all in TRL, includes the major variants: **IPO** (identity preference optimization, fixes DPO's overfitting on noisy preferences), **SIMPO** (length-normalized, reference-free), **CPO** (contrastive), and **ORPO** (combines SFT and preference in one pass, no reference model). If DPO is your default, ORPO is worth knowing — it removes the separate SFT stage. ## KTOTrainer — preference data without pairs Steers **preference but with unpaired data.** Kahneman-Tversky Optimization (Ethayarajh et al., 2024) only needs binary thumbs-up / thumbs-down labels per response, not chosen/rejected pairs. This matters because **paired preference data is expensive to collect** (you need two responses and a ranking); binary feedback is what you actually get from a thumbs-up button in production. KTO turns that signal into a preference gradient. ## RLOOTrainer and GRPOTrainer — reasoning on verifiable rewards Steers **reasoning and behavior where you have a verifiable reward.** Both are RL methods that generate multiple completions per prompt and reinforce the good ones. - **RLOO** (REINFORCE Leave-One-Out, Ahmadian et al., 2024) is the simpler, more sample-efficient alternative to PPO. It computes a baseline by averaging over the other sampled completions, reducing variance without a separate value model. - **GRPO** (Group Relative Policy Optimization, popularized by DeepSeek-R1) samples a *group* of completions, normalizes rewards within the group, and reinforces accordingly. GRPO is the trainer behind most of the open-weights reasoning-model boom — it is what you reach for when the reward is "did the math check out" or "did the code pass the tests" (Module FT14). Both take a **reward function** (Python callable) rather than a reward model. This is a deliberate design: verifiable rewards (unit tests, math verification, format checkers) are more reliable than a learned reward model, which is itself a model that can be reward-hacked. ## Reward Modeling + the async RL path `RewardTrainer` trains a **reward model** — a classifier head on top of the base that scores response quality — for use in classical RLHF (PPO with a learned reward). You reach for this when you do **not** have a verifiable reward function and must learn one from preference data. v1.0 also introduced the **async RL** path for the PPO-style loop, decoupling generation from the gradient update so you can overlap rollouts and training across devices. This is research-grade machinery; most practitioners will use GRPO/RLOO with a verifiable reward before they touch learned-reward PPO. ### The decision shortcut For the vast majority of real work, the decision tree is short: - New format or instruction style → **SFTTrainer**. - "Better/worse" preferences, with pairs → **DPOTrainer** (or ORPO to skip SFT). - Preferences, unpaired thumbs up/down → **KTOTrainer**. - Verifiable reward (math, code, format) and you want reasoning → **GRPOTrainer** (or RLOO). - Learned reward model for classical RLHF → **RewardTrainer** + PPO/async RL. --- # FTDD-04.3 — The Production CLI *The same trainers, no training code. This is how most real jobs are launched.* ## The Python API versus the CLI TRL exposes two equivalent surfaces. The **Python API** is what research code and tutorials show: instantiate a `SFTConfig`, a `SFTTrainer`, call `.train()`. You have full control — custom reward functions, custom data collation, interleaving with your own evaluation harness. The **CLI**, new in recent releases and a headline feature of v1.0, runs the identical trainers from a YAML config: ```bash trl sft --config sft.yml # runs SFTTrainer trl dpo --config dpo.yml # runs DPOTrainer trl grpo --config grpo.yml # runs GRPOTrainer ``` Both paths call the same trainer classes with the same config schema. **The CLI is not a toy.** It is the production path for the majority of jobs, for three reasons. ### Why the CLI wins in production 1. **No training code to maintain.** A YAML file is config, not code. It has no imports, no dependency drift, no "this worked on the old TRL version." When TRL ships a fix, your config still runs. 2. **Reproducibility is a file.** The entire recipe — model, dataset, hyperparameters, PEFT settings, distributed strategy — lives in one YAML you can diff, version, and hand to a colleague. "Reproduce my run" becomes "run this file." 3. **CI-friendly.** A YAML config can be linted, schema-validated, and tested in CI before it ever touches a GPU. A 200-line Python training script cannot. The trade-off: the CLI is configured, not programmed. The moment you need a **custom reward function** for GRPO (anything beyond a built-in), or a custom data transform, or an interleaved eval loop, you drop down to the Python API. The CLI's scope is the 80% of jobs that fit standard recipes. ### The config-key stability promise This is the part the v1.0 Stability Contract makes load-bearing. The keys in your YAML — `model_name_or_path`, `dataset_text_field`, `per_device_train_batch_size`, `lora_r`, `bf16`, `deepspeed` — are stable across releases. You can pin `trl>=1.0,<2.0` and your configs keep working. Before v1.0 this was not true, and it is the single biggest reason teams used Axolotl as a stability firewall. Post-v1.0, the CLI is increasingly the path of least resistance for standard jobs. --- # FTDD-04.4 — TRL in the Ecosystem *TRL is the substrate. The two tools most practitioners actually use — Axolotl and Unsloth — are defined entirely by how they relate to it.* ## Axolotl: TRL, configured Axolotl (FTDD-05) is a **declarative wrapper over TRL**. You write a YAML config, Axolotl translates it into a TRL training run, and adds two things TRL alone does not give you: opinionated defaults that work across many model families, and battle-tested **multi-GPU orchestration** (FSDP and DeepSpeed configs that are fiddly to get right by hand). For production multi-GPU jobs, Axolotl-on-TRL is the dominant pattern. Underneath, the trainer is still `SFTTrainer` or `DPOTrainer` from TRL. The cost of Axolotl's ergonomics is a layer of abstraction and slightly slower iteration on bleeding-edge methods — when a new trainer lands in TRL, it takes time to surface cleanly in Axolotl's config schema. ## Unsloth: TRL's kernels, replaced Unsloth (FTDD-03) takes a different tack. Rather than wrap TRL, it **rewrites the performance-critical kernels** (the attention, the LoRA backward pass) in hand-tuned Triton, and exposes a TRL-compatible API so your training script barely changes. The payoff is dramatically higher throughput on a **single GPU** — often 2x faster, half the VRAM. Unsloth is the right answer for the single-GPU, sub-30B, speed-critical case. The trade-off: Unsloth's multi-GPU story is limited, and it trails TRL on the newest methods. It is an accelerator for the SFT/LoRA common case, not a general replacement. ## When raw TRL is the answer Reach for raw TRL (Python API or CLI) when: - **You need full control.** Custom reward functions, custom data transforms, interleaved evaluation, research objectives not yet in any wrapper. The wrapper tools are configured, not programmed; the moment your loop is non-standard, you are in raw TRL. - **You are doing research, not production.** A new loss, a new sampling strategy, an ablation. Wrappers optimize for the known-good recipes; research is the opposite. - **You want the freshest methods.** New trainers land in TRL first. Axolotl and Unsloth follow. - **Your job fits a standard recipe and you want zero code.** `trl sft --config x.yml` is the shortest path from YAML to a trained model, with no wrapper dependency at all. --- ## Anti-Patterns ### Treating TRL as research-grade only Because of its name and origins, some teams assume TRL is for RL researchers and reach for a wrapper for "production." Post-v1.0 and the Stability Contract, this is outdated. For standard SFT/DPO/GRPO recipes, the TRL CLI is the production path; the wrappers are optional ergonomics, not safety requirements. ### Reaching for GRPO before SFT The reasoning-RL boom makes GRPO glamorous. But GRPO amplifies whatever the base can already do — it does not install capability. If your base produces incoherent reasoning, GRPO will faithfully reinforce incoherent reasoning. SFT first to establish format and basic competence, then GRPO to sharpen. (Module FT14.) ### Relying on unspecified defaults in production The Stability Contract covers the API surface, not the defaults. A default learning rate or scheduler may improve between versions. In production, pin every hyperparameter explicitly in your config; never let a value ride on "whatever the default is this week." ### Choosing a trainer by novelty DPO is not "better" than SFT because it is newer; KTO is not "better" than DPO. Each trainer optimizes a different objective for a different data shape. Choose by the steering goal and the data you have, not by what is fashionable. The decision shortcut in FTDD-04.2 exists because the wrong trainer on the right data still fails. --- ## Key Terms | Term | Definition | | --- | --- | | **TRL** | Transformers Reinforcement Learning — HuggingFace's full-stack post-training library; the substrate of the open-weights ecosystem | | **Stability Contract** | TRL v1.0's promise that trainer names, constructor signatures, config keys, and CLI flags are maintained across releases | | **SFTTrainer** | The baseline trainer; steers format and instruction-following; masks the prompt, handles chat templates and packing | | **DPOTrainer** | Direct Preference Optimization; steers preference from (prompt, chosen, rejected) triples; no reward model | | **KTOTrainer** | Kahneman-Tversky Optimization; preference from unpaired binary feedback (thumbs up/down) | | **GRPOTrainer** | Group Relative Policy Optimization; RL with a verifiable reward function; the reasoning-model workhorse | | **RLOOTrainer** | REINFORCE Leave-One-Out; sample-efficient RL alternative to PPO | | **RewardTrainer** | Trains a learned reward model for classical RLHF (PPO) | | **The thin-wrapper principle** | Every TRL trainer subclasses the HuggingFace Trainer, inheriting optimizers, schedulers, and DDP/DeepSpeed/FSDP | | **`trl sft` / `dpo` / `grpo`** | The production CLI; runs the same trainers as the Python API from a YAML config | --- ## Lab Exercise See `07-lab-spec.md`. The "TRL in 20 minutes" lab: run an SFT job on a 1B model via the Python API, then re-run the identical job via `trl sft` from a YAML config, and confirm the two recipes are equivalent. Feel the API and the CLI as two doors into the same engine. --- ## References 1. **HuggingFace TRL** — *TRL v1.0: Post-Training Library Built to Move with the Field*. Official v1.0 announcement, March 2026. huggingface.co/blog/trl-v1 2. **Rafailov et al. (2023)** — *Direct Preference Optimization*. arXiv:2305.18290. The DPO objective. 3. **Ethayarajh et al. (2024)** — *KTO: Model Alignment as Prospect Theoretic Optimization*. arXiv:2402.01306. Unpaired preference optimization. 4. **Ahmadian et al. (2024)** — *Back to Basics: Revisiting REINFORCE Style Optimization for LLMs*. arXiv:2402.14740. RLOO. 5. **Shao et al. (2024)** — *DeepSeekMath / GRPO*. arXiv:2402.03300. Group Relative Policy Optimization. 6. **TRL documentation** — huggingface.co/docs/trl. Trainer API and CLI reference. 7. **Course 3, FT11** — *The Training Loop*. Where TRL's role in the stack is introduced. 8. **Course 3, FTDD-05** — *Axolotl*. The declarative wrapper over TRL. 9. **Course 3, FTDD-03** — *Unsloth*. The kernel-level competitor for single-GPU speed.