In fact Hinton et al (2014) in their paper, as inspiration refer to paper by Caruana et al (2006), who described distilling knowledge from an ensemble of models. Moreover, distilling is a very basic and versatile approach that can be applied to any model. The point is that you have noisy data and train a big model on this data, this model learns some kind of approximate representation of the data, that is smoother then the raw data. This representation should be less noisy then the data, so it should be easier to learn from it, then from noisy data.
Among other, later, interpretations of why distillation works, is that what we often observe is that the big models like deep neural networks, among huge number of parameters they have, only a handful "does most of the work". One of the interpretations of why huge, deep neural networks work, is that the models with huge number of parameters have a very big search space to find among them such combinations that lead to useful functions. If that is the case, then we don't need the complicated model if only we could extract the small fraction of it that learned the most important characteristics of the problem. This is probably not only the case for neural networks, but also for other modern machine learning algorithms.
From practical point of view, as noticed by Hinton et al, it is easier to learn from smoother representation of the knowledge, so distillation should work better for things like predicted probabilities, or logits, then the hard classifications. Moreover, what Hinton et al proposed, is to further smooth the outputs of the model by including the temperature parameter in the softmax function. This may be more or less useful depending on how well callibrated are the probabilities returned by your model. If your model returns values that are very clustered over high and low probabilities, then the values are not very much more discriminative then the hard classifications. On another hand, with "smooth" outputs the knowledge of the model about the variability of the data is better preserved.
Finally, the idea is pretty simple, so you could give it a try and see if the result achieved by the small, "student" model is close to the big model. The whole point of the small model is that is is lighter then the big one, so should also be faster to train, what makes experimenting much easier. As a word of caution, when looking at the results remember to look not only on the general performance metrics, but also at the tails, at how the small vs big model does handle the atypical cases. There is always risk that the simpler model would learn to classify correctly the "average" cases, but will not do well on the edge cases, so you need to double check this (it is not easy).
