Kush v8.2: Training Pipeline Hardening — What We Learned

Kush v8.2: Training Pipeline Hardening

Kush v7 has been the production model since May 2026 — a clean SFT-only build on Llama 3.1 8B that scored 83.9% persona alignment. This is the story of what happened when we tried to improve it with SimPO preference alignment, how three separate configuration regressions produced a model that scored 13.2%, and the twelve engineering lessons that came out of the debugging.

This is not a release announcement. V8.2 is not deployed. V7 remains production. This is a postmortem and engineering reference for anyone building preference-aligned LLMs with TRL and Unsloth.

What We Tried to Do

The plan was straightforward:

1. Take V7’s clean SFT training data (~3,500 rows)
2. Add semantic safety filters (remove anti-vax, conspiracy, persona bloat)
3. Add new cultural data (Grifties commentary, YouTube transcripts)
4. Run SFT + SimPO (Simple Preference Optimization)
5. Export GGUF for Ollama

SimPO was chosen because it’s reference-free (saves 50% VRAM) and the SimPO paper (Meng et al., 2024) reported +6.4 points on AlpacaEval 2 over DPO.

What Actually Happened

The deployed model was catastrophically broken. E2E testing showed:

The model output podcast transcript fragments, raw tweet artifacts, and template patterns instead of coherent persona responses.

Root Cause 1: Wrong Trainer (DPOTrainer instead of CPOTrainer)

V8 inherited a configuration from a notebook that never ran (the V8 pipeline failed on all 16 tasks due to API rate limits). It used DPOTrainer with loss_type='simpo', which sounds correct but isn’t.

We verified in TRL 0.24.0 source code:

# DPOTrainer.dpo_loss() — line 211-216:
else:
    raise ValueError(
        f"Unknown loss type: {self.loss_type}. Should be one of "
        "['sigmoid', 'hinge', 'ipo', 'exo_pair', 'nca_pair', 'robust', "
        "'bco_pair', 'sppo_hard', 'aot', 'aot_pair', 'discopop', "
        "'apo_zero', 'apo_down', 'sft']"
    )

'simpo' is not in that list. DPOTrainer has zero references to “simpo” in its entire source. The SimPO loss is implemented exclusively in CPOTrainer (6 references). The simpo_gamma parameter exists only on CPOConfig.

The fix: Use CPOTrainer + CPOConfig(loss_type='simpo').

The irony: V6 Colab — the proven successful run — used the now-deprecated SimPOTrainer/SimPOConfig classes. TRL removed those in 0.24 and merged SimPO into CPO. V8’s author changed the loss type but not the trainer, inheriting a config from a model that was never trained.

Root Cause 2: Wrong beta (0.1 instead of 2.5)

DPO’s beta=0.1 controls KL penalty strength against a reference model. SimPO’s beta is a completely different parameter — it’s a reward scaling factor. The paper recommends β=2.0–2.5 for Llama 3.

Using β=0.1 for SimPO reduces the preference signal by 25×. Even if the trainer had been correct, the alignment would have been negligible.

The proven values from V6 Colab (the successful SimPO run on this exact base model):

Root Cause 3: Data Contamination (531 toxic rows)

The SFT payload had 4,565 rows, but 531 (11.6%) were toxic:

• 379 code-fix GitHub PR rows — Python code diffs with “Fix the following issue in backend/…” prompts, completely wrong domain
• 91 template fragment rows — short staccato responses like “Understand this… My sister. History proves…” averaging 217 chars
• 45 raw Twitter tweets — verbatim @TheGrifties posts with URLs, hashtags, and political figures
• 5 raw podcast transcripts — verbatim speech-to-text with filler words (“uh”, “my my”) and no punctuation
• 17 short responses under 50 characters

Why 2.5% toxic content was enough to break the model: The toxic rows averaged 90–217 characters while clean data averaged 1,474 characters. At the same batch size, short toxic rows were seen ~7× more frequently per epoch. The poison was massively overrepresented despite being a small percentage of total rows.

The fix: Removed all 531 rows. Added an is_low_quality() filter to the prepare script to prevent regression. Final dataset: 4,046 clean rows.

Root Cause 4: Unsloth’s Modelfile Has temperature=1.5

When Unsloth exports a GGUF, it auto-generates a Modelfile with temperature 1.5 and min_p 0.1. This is scorching hot and causes random, incoherent token generation.

Our notebook-generated Modelfile had the correct parameters (temperature 0.7, repeat_penalty 1.1) but lacked a TEMPLATE directive, so Ollama didn’t use it for chat formatting. The result: the model ran at temperature 1.5 with no chat template.

The fix: The notebook now writes a complete Modelfile with:

• Correct Llama 3.1 Go template ({{ .Messages }} with <|start_header_id|> tokens)
• temperature 0.7 (not 1.5)
• All 4 stop tokens
• num_predict 512 (was 2048)
• repeat_penalty 1.1 (was 1.5 — another inherited bug)

The Three-Way Config Verification

To prove the fix was correct (not assumed), we verified against three sources:

V6 Colab is the gold standard because it used the same base model (Llama 3.1 8B Instruct). V15 Qwen used Qwen2.5-7B — different architecture, different optimal params.

What V7 Got Right (And Why It’s Still Production)

V7 succeeded with simplicity:

• SFT-only — no preference alignment to get wrong
• Clean data — curated 5-source mix, no contamination
• Conservative params — repeat_penalty 1.1, num_predict 256
• Correct Modelfile — proper template, temperature 0.7

V7 scored 83.9% persona alignment in E2E testing and has been stable in production for 6 weeks. Until V8.2-retrained passes the full E2E suite, V7 stays.

Lessons for Anyone Building Aligned LLMs

1. SimPO requires CPOTrainer. DPOTrainer doesn’t implement it. Check your TRL version.
2. SimPO beta ≠ DPO beta. They’re semantically different parameters despite sharing a name.
3. SimPO LR is 200× lower than SFT LR. If SFT uses 1e-4, SimPO uses 5e-7.
4. Data quality beats quantity. 2.5% toxic rows broke a model. Short rows are overrepresented.
5. Never trust Unsloth’s auto-generated Modelfile. It uses temperature 1.5.
6. repeat_penalty > 1.3 causes degenerate output on Llama 3.1. Use 1.1.
7. Always include a TEMPLATE directive in Ollama Modelfiles. Without it, chat formatting breaks.
8. Check TRL API changes on upgrade. SimPOTrainer was removed; tokenizer= became processing_class=.

What Comes Next

The notebook is patched, the data is cleaned, and the payload is packaged. The next Colab A100 run should produce a working model. The E2E test harness is ready to validate it.

V7 remains production. V8.2 is not a release — it’s a lesson.