# Is model distillation an ill-defined problem?

Model distillation (or knowledge distillation) consist in making a student model learn from a teacher model in order to eventually use the student model as an alternative to the original teacher model. The student model is usually designed to be more efficient, so they are are for example made to have fewer parameters or to be a quantized version of the teacher model. In general, both models are deep neural networks although perhaps other kinds of models can be distilled.

Mathematically speaking, the distillation loss used for a batch of data will be something like $$\mathcal{L}=d(f_s(x),f_t(x))$$ where $$f_s$$ is the student model, $$f_t$$ is the teacher model and $$d$$ is some "distance" function, such as cross-entropy, the squared L2 norm or a even a smooth L1 norm.

However, we are not specifying how we obtain the data points $$x$$ in this formulation, so what is the actual underlying problem to be optimized?

Is it having the least distillation error on a specific dataset, such as the one used for training the teacher model? Is it the dataset used for a downstream task? Or do we want the student model to somehow best fit the teacher on all possible data points, which would mean the actual distillation problem is

$$$$\min \int_X \mathcal{L}(x)dx \tag{1} \label{intequation}$$$$

It seems to me that the properties of the distilled model will be vastly different depending on what particular distribution of $$x$$ we choose. In particular, if we choose the distribution to be a different one than the distribution used for training the teacher, then the student will have a higher chance of outperforming the teacher on this particular distribution as the student might "erase" the prediction artifacts of the teacher model.

So my questions are:

1. what is the true distillation problem, if it exists (or is it just this family of problems as a whole) ?
2. Shouldn't we try to directly solve $$\ref{intequation}$$, since it does not depend on the distribution of $$x$$? (It also seems closer to the intuition that the student model has to approximate the teacher as best as possible)
3. maybe the trained teacher model $$f_t$$ implies a distribution over $$x$$ that we should try to recover and use to train the student, or would this be also an ill-defined problem?