Lab Specification — Module FTDD-04: TRL

python3.11 -m venv ftdd04-env && source ftdd04-env/bin/activate
pip install -q "trl>=1.0" transformers accelerate peft torch datasets

python -c "import trl; print('TRL', trl.__version__)"
trl --help          # should list sft, dpo, grpo, ...

import torch
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "openbmb/MiniCPM5-1B"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True,
)

# A small instruction dataset; use a hosted slice so no local prep is needed.
ds = load_dataset("HuggingFaceTB/smoltalk", "all", split="train").select(range(2000))

cfg = SFTConfig(
    output_dir="./out-api",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,           # PINNED explicitly — never rely on defaults
    logging_steps=10,
    save_strategy="no",
    bf16=False,                    # set True on newer GPUs; FP16 here for broad compat
    report_to="none",
)

trainer = SFTTrainer(model=model, args=cfg, train_dataset=ds, processing_class=tokenizer)
trainer.train()
print("Python-API run complete.")

# sft.yml — the production CLI path
model_name_or_path: openbmb/MiniCPM5-1B
dataset_name: HuggingFaceTB/smoltalk
dataset_config_name: all
dataset_train_split: train
max_seq_length: 2048

output_dir: ./out-cli
num_train_epochs: 1
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 2e-4
logging_steps: 10
save_strategy: "no"
bf16: false
report_to: none

trl sft --config sft.yml

trl sft --help           # lists every CLI flag, which mirrors the YAML keys
python -c "from trl import SFTConfig; import dataclasses; print([f.name for f in dataclasses.fields(SFTConfig)])"

# Lab Specification — Module FTDD-04: TRL

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FTDD-04 — TRL (Transformers Reinforcement Learning)
**Duration**: 30–40 minutes
**Environment**: Python 3.11+. A consumer GPU (RTX 4090 / 16–24GB) OR Apple Silicon (M-series) OR free Google Colab T4. ~6GB free disk for the model + dataset.

---

## Learning objectives

By the end of this lab you will have:

1. **Run an SFT job via the TRL Python API** — `SFTConfig` + `SFTTrainer` on a 1B base model — and felt the trainer-as-thin-wrapper.
2. **Re-run the identical job via the production CLI** (`trl sft --config ...`) and confirmed the two recipes are equivalent — the same engine, two doors.
3. **Inspected the YAML config schema** and understood why a config file (not training code) is the production path: reproducibility-as-a-file, CI-validatable.
4. **Stated, in your own words, why TRL is the substrate** — and what dropping to the Python API buys you that the CLI cannot.

This lab is deliberately a *pair* of runs. The point is to feel the API and the CLI as two surfaces into the same engine before you spend the rest of the ecosystem deep-dives learning what the wrappers add.

---

## Phase 0 — Environment setup (5 min)

```bash
python3.11 -m venv ftdd04-env && source ftdd04-env/bin/activate
pip install -q "trl>=1.0" transformers accelerate peft torch datasets
```

Verify the stack and confirm the CLI is on your PATH (it ships with TRL v1.0):

```bash
python -c "import trl; print('TRL', trl.__version__)"
trl --help          # should list sft, dpo, grpo, ...
```

If `trl --help` fails, ensure `trl>=1.0` installed correctly and that your venv's `bin/` is on `$PATH`.

---

## Phase 1 — SFT via the Python API (10 min)

Use a small open base — `openbmb/MiniCPM5-1B` — and a tiny instruction slice. The goal is a *working run*, not a great model.

```python
import torch
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "openbmb/MiniCPM5-1B"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True,
)

# A small instruction dataset; use a hosted slice so no local prep is needed.
ds = load_dataset("HuggingFaceTB/smoltalk", "all", split="train").select(range(2000))

cfg = SFTConfig(
    output_dir="./out-api",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,           # PINNED explicitly — never rely on defaults
    logging_steps=10,
    save_strategy="no",
    bf16=False,                    # set True on newer GPUs; FP16 here for broad compat
    report_to="none",
)

trainer = SFTTrainer(model=model, args=cfg, train_dataset=ds, processing_class=tokenizer)
trainer.train()
print("Python-API run complete.")
```

**Record**: the final training loss. Note how little code you wrote — `SFTTrainer` handled chat templates, prompt masking, and packing for you. This is the thin-wrapper principle: you configured, you did not reimplement the loop.

> **What just happened (the teaching moment):** You wrote no training loop. `SFTTrainer` subclassed the HuggingFace `Trainer`, overrode `compute_loss` with the SFT objective, and inherited the optimizer, scheduler, and device placement. That is the substrate.

---

## Phase 2 — The identical job via the production CLI (10 min)

Now re-run the *same recipe* from a YAML config, with no Python at all. Create `sft.yml`:

```yaml
# sft.yml — the production CLI path
model_name_or_path: openbmb/MiniCPM5-1B
dataset_name: HuggingFaceTB/smoltalk
dataset_config_name: all
dataset_train_split: train
max_seq_length: 2048

output_dir: ./out-cli
num_train_epochs: 1
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 2e-4
logging_steps: 10
save_strategy: "no"
bf16: false
report_to: none
```

Run it:

```bash
trl sft --config sft.yml
```

**Record**: the final training loss from the CLI run.

> **Compare:** the two losses should be effectively identical (same model, same data slice, same hyperparameters, same underlying `SFTTrainer`). Any small difference is RNG seed / data-order nondeterminism, not a difference in method. You have now felt the API and the CLI as two doors into the same engine.

---

## Phase 3 — Why the CLI wins in production (5 min)

No code. Answer in 3–5 sentences:

1. The YAML in Phase 2 is your entire training recipe. List three things about it that make it more reproducible / CI-friendly than the Python script in Phase 1.
2. What is the ONE thing in Phase 1 that you *cannot* do from a static YAML in Phase 2? (Hint: think about GRPO with a custom reward function.) When would that force you back to the Python API?
3. The Stability Contract guarantees your `sft.yml` keeps working across TRL 1.x releases. What is the ONE thing it does *not* guarantee that you must defend against in production?

---

## Phase 4 — Inspect the config schema (optional, 5 min)

Confirm the config-key stability promise is real — list the keys the CLI accepts, and confirm the ones you used are documented:

```bash
trl sft --help           # lists every CLI flag, which mirrors the YAML keys
python -c "from trl import SFTConfig; import dataclasses; print([f.name for f in dataclasses.fields(SFTConfig)])"
```

Skim the output. You should see `model_name_or_path`, `learning_rate`, `per_device_train_batch_size`, `lora_r`, `deepspeed`, and dozens more. These are the stable surface the Stability Contract protects.

---

## Deliverables

Submit `ftdd04-lab-report.md`:

- [ ] Phase 1: the Python-API run's final training loss; a one-line note on what `SFTTrainer` handled for you (chat template, prompt masking, packing).
- [ ] Phase 2: the CLI run's final training loss; a note confirming it matches Phase 1 within RNG noise.
- [ ] Phase 3: your 3–5 sentence answers to the three questions.
- [ ] (Optional) Phase 4: three config keys you found via `trl sft --help`.

---

## Solution key

- **Phase 1**: a successful run produces a decreasing loss curve and a final loss in a sane range (exact value depends on the data slice; the point is that it *runs* and *decreases*). The teaching note should mention that `SFTTrainer` handled the chat template, prompt masking, and sequence packing — the ergonomics that hand-rolled SFT loses hours on.
- **Phase 2**: the CLI run's final loss should match Phase 1 within small RNG/data-order noise. If the losses diverge materially, the student changed a hyperparameter between runs (check `learning_rate`, `batch_size`, `gradient_accumulation_steps`) or used a different data slice. Both runs call the same `SFTTrainer`.
- **Phase 3** (model answers):
  1. Reproducibility/CI: (a) the YAML is config not code — no imports/dependency drift; (b) it is one diff-able, version-able file capturing the whole recipe; (c) it can be linted/schema-validated in CI before touching a GPU.
  2. A GRPO job with a *custom* reward function (anything beyond a built-in) cannot be expressed in a static YAML — the reward is Python code. That forces a drop to the Python API. Same for custom data transforms or interleaved eval.
  3. The Stability Contract does NOT freeze DEFAULTS. A default LR or scheduler may improve between minor versions. Pin every hyperparameter explicitly; never ride an unspecified default.
- **Phase 4**: any three of `model_name_or_path`, `learning_rate`, `per_device_train_batch_size`, `lora_r`, `deepspeed`, `num_train_epochs`, `bf16`, etc.

---

## Stretch goals

1. **Swap to DPO via the CLI.** Write a `dpo.yml` over a small preference dataset (e.g., `trl-lib/ultrafeedback_binarized`) and run `trl dpo --config dpo.yml`. Observe that the CLI surface is uniform across trainers — the same ergonomics, a different objective. (Sets up FTDD-05 / Axolotl.)
2. **Enable LoRA via the config.** Add `use_peft: true`, `lora_r: 8`, `lora_alpha: 16` to `sft.yml` and re-run. Confirm the steerable-params percentage is under 1% — the Steering Stack thesis (FT00), felt through TRL. (Sets up FT08.)
3. **Compare against Unsloth.** Re-run the Phase 1 SFT job using Unsloth's TRL-compatible API and measure throughput (tokens/sec or steps/sec). Quantify the single-GPU speedup that motivates FTDD-03.