"Question"	"Answer"	"Tag"
"What is TRL, and why is it described as the substrate of the open-weights post-training ecosystem?"	"TRL (Transformers Reinforcement Learning) is HuggingFace's full-stack post-training library. It is the substrate because every higher-level tool either wraps it (Axolotl) or competes by replacing pieces of it (Unsloth). 3M downloads/month, 75+ methods. Knowing its trainer API is the Rosetta Stone for the whole field."	c3::ftdd04::recall
"What does TRL's name refer to, and why is the name now slightly misleading?"	"It began as a Reinforcement Learning library (it shipped PPO for LLMs early), but long ago absorbed SFT and the entire preference-optimization family. The accurate description today: TRL is the full-stack post-training library. 'SFT in a library called Reinforcement Learning' is a standing forum joke."	c3::ftdd04::recall
"Explain the thin-wrapper principle behind TRL's trainers."	"Every TRL trainer is a thin wrapper over the HuggingFace Trainer. It does not reimplement the training loop — it subclasses Trainer, overrides compute_loss to inject the SFT/DPO/GRPO objective, and inherits everything else: optimizer, schedulers, data collation, chat templates, and DDP/DeepSpeed ZeRO/FSDP."	c3::ftdd04::recall
"What does the TRL v1.0 Stability Contract guarantee, and what does it NOT cover?"	"It guarantees trainer class names, constructor signatures, config/YAML keys, and CLI flags are a maintained surface across releases (breaking changes need a major version bump + migration guide). It does NOT freeze DEFAULTS — a default LR or scheduler may improve between minor versions with a deprecation cycle. Pin every hyperparameter explicitly in production."	c3::ftdd04::recall
"Name the six core TRL trainers and what each steers."	"SFTTrainer (format/instructions), DPOTrainer (preference, chosen/rejected pairs), KTOTrainer (preference, unpaired thumbs up/down), RLOOTrainer (reasoning, verifiable reward), GRPOTrainer (reasoning, verifiable reward), RewardTrainer (trains a learned reward model for RLHF/PPO)."	c3::ftdd04::recall
"What data shape does DPOTrainer expect, and what is its objective?"	"Chosen-vs-rejected pairs: (prompt, chosen, rejected) triples. Objective: Direct Preference Optimization (Rafailov 2023) — no reward model; it derives the preference gradient directly from the policy with a reference-model KL anchor."	c3::ftdd04::recall
"Why does KTOTrainer exist, and how does its data differ from DPO?"	"KTO (Kahneman-Tversky Optimization) only needs unpaired BINARY feedback (thumbs up/down per response), not chosen/rejected pairs. This matters because paired preference data is expensive to collect; binary feedback is what you actually get from a production thumbs-up button. KTO turns that signal into a preference gradient."	c3::ftdd04::recall
"What distinguishes GRPOTrainer and RLOOTrainer from the DPO family?"	"They are RL methods that take a verifiable REWARD FUNCTION (a Python callable), not a preference dataset. They generate multiple completions per prompt and reinforce the good ones. GRPO normalizes rewards within a group; RLOO uses a leave-one-out baseline. Both are the reasoning/verifiable-reward path (e.g. 'did the math check out')."	c3::ftdd04::application
"For a job where you have unit-test pass/fail as the reward and want to sharpen reasoning, which trainer do you reach for, and why not DPO?"	"GRPOTrainer (or RLOOTrainer). You have a verifiable reward function (the tests), which is exactly what GRPO/RLOO consume. DPO is wrong because it needs chosen/rejected PAIRS and optimizes preference, not a verifiable reward. GRPO reinforces completions that pass the tests."	c3::ftdd04::application
"Give the decision shortcut: which trainer for (a) new output format, (b) preference pairs, (c) unpaired thumbs up/down, (d) verifiable math reward, (e) learned reward for RLHF?"	"(a) SFTTrainer; (b) DPOTrainer (or ORPO to skip SFT); (c) KTOTrainer; (d) GRPOTrainer (or RLOO); (e) RewardTrainer + PPO/async RL."	c3::ftdd04::application
"Why is reaching for GRPO before SFT an anti-pattern?"	"GRPO amplifies whatever the base can already do — it does not install capability. If the base produces incoherent reasoning, GRPO faithfully reinforces incoherent reasoning. SFT first to establish format and basic competence, then GRPO to sharpen reasoning."	c3::ftdd04::application
"Contrast the TRL Python API path against the production CLI. When does each win?"	"Both call the same trainer classes with the same config schema. The CLI (trl sft/dpo/grpo --config) wins for standard recipes: no training code to maintain, reproducibility-is-a-file, CI-friendly. The Python API wins when you need custom reward functions, custom data transforms, interleaved eval, or non-standard loops. CLI = the 80% path; API = full control."	c3::ftdd04::application
"State three reasons the TRL CLI wins for production jobs over a hand-written training script."	"(1) No training code to maintain — a YAML is config, not code; no imports/dependency drift. (2) Reproducibility is a file — the entire recipe (model, data, hyperparams, PEFT, distributed) is one diff-able, version-able YAML. (3) CI-friendly — a YAML can be linted and schema-validated before it touches a GPU; a 200-line script cannot."	c3::ftdd04::analysis
"How does Axolotl relate to TRL, and what does the wrapper add that raw TRL does not?"	"Axolotl WRAPS TRL — underneath, the trainer is still SFTTrainer/DPOTrainer. The wrapper adds: (1) opinionated defaults that work across model families, and (2) battle-tested multi-GPU orchestration (FSDP/DeepSpeed configs fiddly to get right by hand). Cost: a layer of abstraction and slower iteration on bleeding-edge methods."	c3::ftdd04::analysis
"How does Unsloth relate to TRL, and what is its trade-off?"	"Unsloth does NOT wrap TRL — it REPLACES the performance-critical kernels (attention, LoRA backward) with hand-tuned Triton, and exposes a TRL-compatible API. Payoff: ~2x throughput, ~half the VRAM on a single GPU. Trade-off: limited multi-GPU story and trails TRL on the newest methods. It's an accelerator for the single-GPU sub-30B SFT/LoRA common case."	c3::ftdd04::analysis
"Why does the thin-wrapper principle explain TRL's ability to scale to 405B full-parameter training on clusters?"	"Because scaling is DeepSpeed ZeRO and FSDP's job, and TRL inherits both for free by subclassing the HuggingFace Trainer. TRL's trainers get out of the way — they only override compute_loss. The model-parallel / sharding / gradient-accumulation machinery is the Transformers library's, already battle-tested, so TRL rides it to cluster scale without reimplementation."	c3::ftdd04::analysis
"Why is 'choosing a trainer by novelty' an anti-pattern? Give a concrete example."	"Each trainer optimizes a DIFFERENT objective for a DIFFERENT data shape; newer is not better for your problem. Example: DPO is not 'better' than SFT because newer — DPO needs preference pairs and steers preference; SFT steers format. If your goal is reliable JSON output, SFT is correct and DPO is wrong, regardless of novelty. Choose by steering goal + data, not fashion."	c3::ftdd04::analysis
"When is raw TRL (not a wrapper) the correct choice? List four cases."	"(1) Full control — custom reward functions, custom data transforms, interleaved eval, research objectives not in any wrapper. (2) Research, not production — new losses, sampling strategies, ablations. (3) Freshest methods — new trainers land in TRL first; wrappers follow. (4) Standard recipe with zero code — 'trl sft --config x.yml' is the shortest path with no wrapper dependency."	c3::ftdd04::analysis