Distillation

Distillation is a way to compress a large “teacher” AI model into a smaller “student” model by training the student on the teacher’s outputs.

Key points:

• Developers query a large teacher model and collect its answers (sometimes checking them against a labeled dataset).
• Those answers become training data for a smaller student model that learns to mimic the teacher.
• This produces a more efficient model with performance close to the original, enabling faster/cheaper deployment (e.g., GPT‑4 Turbo vs GPT‑4).
• All major AI companies use distillation internally; using it on a competitor’s model via their API typically violates their terms of service.