Knowledge Distillation
Transfer knowledge from a large teacher model to a smaller student model using temperature scaling.
Free theory articleTransfer knowledge from a large teacher model to a smaller student model using temperature scaling.
Concept & interpretation
Knowledge Distillation is a model compression technique in which a small "student" model is trained to reproduce the behavior of a larger, pre-trained "teacher" model. Instead of solely learning from the hard ground-truth labels, the student learns from the "soft targets" produced by the teacher. These soft targets contain "dark knowledge"—the relative probabilities of incorrect classes—which reveals the teacher's understanding of similarities between different classes, leading to faster convergence and better generalization for the student.