Explore modules

Knowledge Distillation

Transfer knowledge from a large teacher model to a smaller student model using temperature scaling.

Free theory article

Transfer knowledge from a large teacher model to a smaller student model using temperature scaling.

Concept & interpretation

Knowledge Distillation is a model compression technique in which a small "student" model is trained to reproduce the behavior of a larger, pre-trained "teacher" model. Instead of solely learning from the hard ground-truth labels, the student learns from the "soft targets" produced by the teacher. These soft targets contain "dark knowledge"—the relative probabilities of incorrect classes—which reveals the teacher's understanding of similarities between different classes, leading to faster convergence and better generalization for the student.

Read the full free theory article