Self-distillation inherits the reversal curse

marc

05 November 2026

2 min

Recently I've been exploring the idea that language models need something like a sleep phase, a time when they keep working through what they've recently learnt and fold it into their weights. That needs some way of moving knowledge from the model's context into its weights, and one existing way is self-distillation. With the facts in front of it, a model can answer questions about them either way round. I wanted to know whether self-distillation carries that over into the weights, or only the facts as they were given. That question turned into a small study of its own, and the results seemed worth sharing.

I explored this with Claude's help. Claude designed most of the experiments and controls, wrote the analysis code and drafted the write-up. I ran the experiments, decided what to test next and checked the results. Along the way our first explanation turned out to be wrong, and it was the experiments that showed it.

What was found. A popular way to give a model new knowledge is self-distillation: a teacher copy of the model sees the new facts, the student (the same model without them) writes about them, and the student is trained to match the teacher word by word. In a controlled test with invented films, directors and studios, the student learnt the facts in the direction they were given and almost nothing else. It inherits the reversal curse. Learning "Dessa Veypex directed The Scarlet Letter" didn't teach it who directed The Scarlet Letter, although the teacher knew. And the student can write the reversal correctly and still not learn it:

Asked to write about one film, director or studio at a time, the student mostly guesses wrong, the teacher corrects the guess, and the reversal is learnt (0.85, against 0.91 with the facts in front of it). List the other names in that prompt and the gain vanishes, even though the student now writes the reversal correctly half the time: with the names there it can recognise the answer instead of recalling it, so there's nothing to learn. A second, separate limit comes from the training objective: under reverse KL, which much self-distillation uses, answers of a type that never comes after anything in the learnt facts, here people's names, stay near chance even when every cue is covered. Throughout, ordinary recall stayed at 95–100%, so none of this shows in the usual evaluations.

What our results suggest, if you consolidate new facts this way:

1. Make the student write from each cue a later question starts from (the film, the studio…) through to the relation, and keep the answers out of its prompt.

2. Use forward KL for facts.

3. Test reversed and multi-step questions, not just recall.

The details

It's a small synthetic world on two 7–8B models, and I'd be interested to know whether it holds more widely. The full write-up has every table, the controls and the predictions we wrote down before running them, confidence intervals and the caveats.

- Full write-up: https://github.com/Mojeek/self-distillation-reversal-curse/blob/main/self-distillation-reversal-curse.pdf

- Code and result files: https://github.com/Mojeek/self-distillation-reversal-curse

- Related: Tang & Yang, On-Policy Distillation Teaches New Skills but Not New Knowledge (2026); Berglund et al., The Reversal Curse (2024); Shenfeld et al., Self-Distillation Enables Continual Learning (2026); Behrouz et al., Language Models Need Sleep (2026).

marc

05 November 2026

2 min