Diagrams — Module FTDD-04: TRL

Module: FTDD-04 — TRL (Transformers Reinforcement Learning) Diagram count: 4 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.

Diagram 1 — TRL as the Substrate Layer

Type: Layered architecture Purpose: The central picture. TRL sits on the HuggingFace Trainer and is itself the substrate that Axolotl, Unsloth, and HuggingFace Jobs build on. Reading the diagram: Bottom-up. The Transformers Trainer is the foundation; TRL's trainers are thin wrappers; the ecosystem tools sit above. The arrows mean "builds on / wraps."

block-beta
  columns 1
  Ecosystem["THE ECOSYSTEM\nAxolotl (config + multi-GPU)\nUnsloth (kernel replacement)\nHuggingFace Jobs"]
  TRL["TRL v1.0\n75+ post-training methods · 3M downloads/month\nStability Contract + production CLI"]
  Trainer["HuggingFace Trainer / Transformers\noptimizer · schedulers · DDP · DeepSpeed ZeRO · FSDP"]

  Trainer --> TRL
  TRL --> Ecosystem

  style Trainer fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style TRL fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Ecosystem fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8

Diagram 2 — The Six Trainers, Mapped to Steering Goals

Type: Mapping / decision table-as-diagram Purpose: Which trainer steers which thing. This is the core of the module. Reading the diagram: Left = trainer. Middle = what it steers. Right = the data shape it expects.

flowchart LR
  SFT["SFTTrainer"] -->|"format, instruction-following"| D1["prompt + completion"]
  DPO["DPOTrainer"] -->|"preference (better/worse)"| D2["chosen vs rejected pairs"]
  KTO["KTOTrainer"] -->|"preference (unpaired)"| D3["thumbs up / down"]
  RLOO["RLOOTrainer"] -->|"reasoning (verifiable reward)"| D4["reward function"]
  GRPO["GRPOTrainer"] -->|"reasoning (verifiable reward)"| D4
  RM["RewardTrainer"] -->|"learned reward for RLHF"| D5["preference pairs -> scorer"]

  style SFT fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style DPO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style KTO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style RLOO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style GRPO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style RM fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style D1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style D2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style D3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style D4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style D5 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0

Diagram 3 — Python API vs Production CLI

Type: Comparison / two-door Purpose: Two equivalent surfaces into the same engine. When to use which. Reading the diagram: Both doors run the same trainer classes. The CLI is configured; the API is programmed. The CLI wins for standard recipes and reproducibility; the API wins when you need custom logic.

flowchart TD
  Engine["TRL Trainers\nSFTTrainer · DPOTrainer · GRPOTrainer ..."]
  CLI["CLI path\ntrl sft --config x.yml\ntrl dpo --config x.yml\ntrl grpo --config x.yml"]
  API["Python API path\nSFTConfig + SFTTrainer\nfull control, custom rewards"]

  CLI -->|"same trainers, no code"| Engine
  API -->|"same trainers, full control"| Engine

  CLI -.->|"standard recipes, reproducible, CI-friendly"| Use1["Production jobs"]
  API -.->|"custom reward fns, research, non-standard loop"| Use2["Research + custom"]

  style Engine fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style CLI fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style API fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Use1 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style Use2 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0

Diagram 4 — TRL in the Ecosystem: Wraps, Competes, Controls

Type: Three-relationship map Purpose: How Axolotl and Unsloth relate to TRL, and when raw TRL is the answer. Reading the diagram: Three relationships to the same substrate. Axolotl WRAPS (adds config + multi-GPU). Unsloth REPLACES kernels (single-GPU speed). Raw TRL is full control.

flowchart LR
  TRL["TRL\nthe substrate"]

  Axolotl["Axolotl\nWRAPS TRL\nYAML config + multi-GPU\n(FSDP/DeepSpeed orchestration)"]
  Unsloth["Unsloth\nREPLACES kernels\nTriton kernels, single-GPU speed\nTRL-compatible API"]
  Raw["Raw TRL\nFULL CONTROL\nPython API or CLI\nresearch, custom rewards, freshest methods"]

  Axolotl -->|"wraps"| TRL
  Unsloth -->|"competes: replaces kernels"| TRL
  Raw -->|"is"| TRL

  style TRL fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Axolotl fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Unsloth fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Raw fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8

Validation notes

All four diagrams use the course design system colors: #14141f panel fill, #5eead4 accent for primary, rgba(255,255,255,0.08)/rgba(94,234,212,0.3) for secondary borders, #e4e4e8 / #9494a0 for text.
Paste each into Mermaid Live Editor to render. All use stable Mermaid syntax (block-beta, flowchart) supported in current Mermaid (v10.4+).
For the slide deck (artifact 03), these are rendered as static SVG/PNG captures from Mermaid Live, inlined into reveal.js.

# Diagrams — Module FTDD-04: TRL **Module**: FTDD-04 — TRL (Transformers Reinforcement Learning) **Diagram count**: 4 **Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live). --- ## Diagram 1 — TRL as the Substrate Layer **Type**: Layered architecture **Purpose**: The central picture. TRL sits on the HuggingFace `Trainer` and is itself the substrate that Axolotl, Unsloth, and HuggingFace Jobs build on. **Reading the diagram**: Bottom-up. The Transformers Trainer is the foundation; TRL's trainers are thin wrappers; the ecosystem tools sit above. The arrows mean "builds on / wraps." ```mermaid block-beta columns 1 Ecosystem["THE ECOSYSTEM\nAxolotl (config + multi-GPU)\nUnsloth (kernel replacement)\nHuggingFace Jobs"] TRL["TRL v1.0\n75+ post-training methods · 3M downloads/month\nStability Contract + production CLI"] Trainer["HuggingFace Trainer / Transformers\noptimizer · schedulers · DDP · DeepSpeed ZeRO · FSDP"] Trainer --> TRL TRL --> Ecosystem style Trainer fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style TRL fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Ecosystem fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 ``` --- ## Diagram 2 — The Six Trainers, Mapped to Steering Goals **Type**: Mapping / decision table-as-diagram **Purpose**: Which trainer steers which thing. This is the core of the module. **Reading the diagram**: Left = trainer. Middle = what it steers. Right = the data shape it expects. ```mermaid flowchart LR SFT["SFTTrainer"] -->|"format, instruction-following"| D1["prompt + completion"] DPO["DPOTrainer"] -->|"preference (better/worse)"| D2["chosen vs rejected pairs"] KTO["KTOTrainer"] -->|"preference (unpaired)"| D3["thumbs up / down"] RLOO["RLOOTrainer"] -->|"reasoning (verifiable reward)"| D4["reward function"] GRPO["GRPOTrainer"] -->|"reasoning (verifiable reward)"| D4 RM["RewardTrainer"] -->|"learned reward for RLHF"| D5["preference pairs -> scorer"] style SFT fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style DPO fill:#14141f,stroke:#5eead4,color:#e4e4e8 style KTO fill:#14141f,stroke:#5eead4,color:#e4e4e8 style RLOO fill:#14141f,stroke:#5eead4,color:#e4e4e8 style GRPO fill:#14141f,stroke:#5eead4,color:#e4e4e8 style RM fill:#14141f,stroke:#5eead4,color:#e4e4e8 style D1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0 style D2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0 style D3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0 style D4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0 style D5 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0 ``` --- ## Diagram 3 — Python API vs Production CLI **Type**: Comparison / two-door **Purpose**: Two equivalent surfaces into the same engine. When to use which. **Reading the diagram**: Both doors run the same trainer classes. The CLI is configured; the API is programmed. The CLI wins for standard recipes and reproducibility; the API wins when you need custom logic. ```mermaid flowchart TD Engine["TRL Trainers\nSFTTrainer · DPOTrainer · GRPOTrainer ..."] CLI["CLI path\ntrl sft --config x.yml\ntrl dpo --config x.yml\ntrl grpo --config x.yml"] API["Python API path\nSFTConfig + SFTTrainer\nfull control, custom rewards"] CLI -->|"same trainers, no code"| Engine API -->|"same trainers, full control"| Engine CLI -.->|"standard recipes, reproducible, CI-friendly"| Use1["Production jobs"] API -.->|"custom reward fns, research, non-standard loop"| Use2["Research + custom"] style Engine fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style CLI fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style API fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style Use1 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0 style Use2 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0 ``` --- ## Diagram 4 — TRL in the Ecosystem: Wraps, Competes, Controls **Type**: Three-relationship map **Purpose**: How Axolotl and Unsloth relate to TRL, and when raw TRL is the answer. **Reading the diagram**: Three relationships to the same substrate. Axolotl WRAPS (adds config + multi-GPU). Unsloth REPLACES kernels (single-GPU speed). Raw TRL is full control. ```mermaid flowchart LR TRL["TRL\nthe substrate"] Axolotl["Axolotl\nWRAPS TRL\nYAML config + multi-GPU\n(FSDP/DeepSpeed orchestration)"] Unsloth["Unsloth\nREPLACES kernels\nTriton kernels, single-GPU speed\nTRL-compatible API"] Raw["Raw TRL\nFULL CONTROL\nPython API or CLI\nresearch, custom rewards, freshest methods"] Axolotl -->|"wraps"| TRL Unsloth -->|"competes: replaces kernels"| TRL Raw -->|"is"| TRL style TRL fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Axolotl fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style Unsloth fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8 style Raw fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 ``` --- ## Validation notes - All four diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `rgba(255,255,255,0.08)`/`rgba(94,234,212,0.3)` for secondary borders, `#e4e4e8` / `#9494a0` for text. - Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`block-beta`, `flowchart`) supported in current Mermaid (v10.4+). - For the slide deck (artifact 03), these are rendered as static SVG/PNG captures from Mermaid Live, inlined into reveal.js.