Model distillation (or knowledge distillation) consist in making a student model learn from a teacher model in order to eventually use the student model as an alternative to the original teacher model. The student model is usually designed to be more efficient, so they are are for example made to have fewer parameters or to be a quantized version of the teacher model. In general, both models are deep neural networks although perhaps other kinds of models can be distilled.

Mathematically speaking, the distillation loss used for a batch of data will be something like $$\mathcal{L}=d(f_s(x),f_t(x))$$ where $f_s$ is the student model, $f_t$ is the teacher model and $d$ is some "distance" function, such as cross-entropy, the squared L2 norm or a even a smooth L1 norm.

However, we are not specifying how we obtain the data points $x$ in this formulation, so what is the actual underlying problem to be optimized?

Is it having the least distillation error on a specific dataset, such as the one used for training the teacher model? Is it the dataset used for a downstream task? Or do we want the student model to somehow best fit the teacher on all possible data points, which would mean the actual distillation problem is

\begin{equation} \min \int_X \mathcal{L}(x)dx \tag{1} \label{intequation} \end{equation}

It seems to me that the properties of the distilled model will be vastly different depending on what particular distribution of $x$ we choose. In particular, if we choose the distribution to be a different one than the distribution used for training the teacher, then the student will have a higher chance of outperforming the teacher on this particular distribution as the student might "erase" the prediction artifacts of the teacher model.

So my questions are:

  1. what is the true distillation problem, if it exists (or is it just this family of problems as a whole) ?
  2. Shouldn't we try to directly solve $\ref{intequation}$, since it does not depend on the distribution of $x$? (It also seems closer to the intuition that the student model has to approximate the teacher as best as possible)
  3. maybe the trained teacher model $f_t$ implies a distribution over $x$ that we should try to recover and use to train the student, or would this be also an ill-defined problem?

1 Answer 1


Your defined loss function measures how well the student agrees with the teacher, so the student can never outperform the teacher with this definition.

In reality, a student with a better bias can sometimes outperform a teacher. The role of the teacher here is to provide a synthetic training set, which you might be doing for reasons of confidentiality, or the expense of generating real data points.

In regards to your equation (1), of course the distribution X that really matters is the future distribution. Optimising for every possible distribution might be an inefficient way to anticipate the likely futures.

  • $\begingroup$ I suppose the student can outperform the teacher if the student is generalizing better. But in the case of using the target distribution for the distillation, doesn't that make the problem more akin to transfer learning than really copying the teacher function (as in having the same model/function, but smaller)? I think it is also likely that in this particular setting the student will consistently outperform the teacher, since the student will be the only model trained on the target distribution. $\endgroup$
    – Ben
    Commented Nov 15, 2023 at 17:13
  • $\begingroup$ True. But the proposed loss function as the minimum over all possible distributions includes the constant distributions. So it is actually the minimum over all possible datapoints. Some of these are more important than others. For example for a forensic application, this does not include the value that "it it better to free the guilty than to imprison the innocent". A loss function should be based on domain considerations. $\endgroup$ Commented Nov 17, 2023 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.