12
$\begingroup$

There is a famous paper "distilling the knowledge in a neural network" from Hinton about training a small NN to represent a large deep NN.

Is that possible to do the same thing for a stacked Ensemble model? For example, if I have a three-layer stacking model. The first layer contains a combination of XGBoost, Random Forest, glmnet, SVM Radial Kernel, and KNN; The second layer contains XGBoost and glmnet; The third layer contains one Random Forest model.

Is that possible to represent this model with a much smaller model?

$\endgroup$
1
  • 1
    $\begingroup$ I think it's possible the same way it's done for neural nets. Probably add in softmax at the end of your model and collect the output right before the softmax (logits) and train the smaller model to obtain these values. I'm also keen to know if it's possible. $\endgroup$
    – Ravil
    Jan 30, 2019 at 9:28

2 Answers 2

6
$\begingroup$

In fact Hinton et al (2014) in their paper, as inspiration refer to paper by Caruana et al (2006), who described distilling knowledge from an ensemble of models. Moreover, distilling is a very basic and versatile approach that can be applied to any model. The point is that you have noisy data and train a big model on this data, this model learns some kind of approximate representation of the data, that is smoother then the raw data. This representation should be less noisy then the data, so it should be easier to learn from it, then from noisy data.

Among other, later, interpretations of why distillation works, is that what we often observe is that the big models like deep neural networks, among huge number of parameters they have, only a handful "does most of the work". One of the interpretations of why huge, deep neural networks work, is that the models with huge number of parameters have a very big search space to find among them such combinations that lead to useful functions. If that is the case, then we don't need the complicated model if only we could extract the small fraction of it that learned the most important characteristics of the problem. This is probably not only the case for neural networks, but also for other modern machine learning algorithms.

From practical point of view, as noticed by Hinton et al, it is easier to learn from smoother representation of the knowledge, so distillation should work better for things like predicted probabilities, or logits, then the hard classifications. Moreover, what Hinton et al proposed, is to further smooth the outputs of the model by including the temperature parameter in the softmax function. This may be more or less useful depending on how well callibrated are the probabilities returned by your model. If your model returns values that are very clustered over high and low probabilities, then the values are not very much more discriminative then the hard classifications. On another hand, with "smooth" outputs the knowledge of the model about the variability of the data is better preserved.

Finally, the idea is pretty simple, so you could give it a try and see if the result achieved by the small, "student" model is close to the big model. The whole point of the small model is that is is lighter then the big one, so should also be faster to train, what makes experimenting much easier. As a word of caution, when looking at the results remember to look not only on the general performance metrics, but also at the tails, at how the small vs big model does handle the atypical cases. There is always risk that the simpler model would learn to classify correctly the "average" cases, but will not do well on the edge cases, so you need to double check this (it is not easy).

$\endgroup$
5
  • $\begingroup$ @ Tim: Great answer! Can you please explain what exactly is meant by "Distilling"? Thank you! $\endgroup$
    – stats_noob
    Nov 27, 2021 at 2:59
  • 1
    $\begingroup$ @stats555 it’s partially explained: distilling is training a (simpler) model on predictions of another (more complicated) model. $\endgroup$
    – Tim
    Nov 27, 2021 at 6:36
  • $\begingroup$ Oh OK, I think I have heard about this in other words! For instance, I have heard of people taking the output probabilities from a random forest model, and feeding these probabilities into a logistic regression model. $\endgroup$
    – stats_noob
    Nov 27, 2021 at 6:42
  • $\begingroup$ Can this also be done for regression models? $\endgroup$
    – stats_noob
    Nov 27, 2021 at 6:42
  • $\begingroup$ @stats555 as the answer says, any model. $\endgroup$
    – Tim
    Nov 27, 2021 at 6:53
0
$\begingroup$

yes, it's possible. as long as you have logits (the inputs to the final softmax layer), you can represent your stacked ensemble model with a much smaller model (which could be a neural net, svm, etc.)

initially, the smaller model ("student") was designed to predict raw unnormalized logits. nowadays, it's more common to train the student to predict softmax outputs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.