In Section 5.3 of the paper distilling the knowledge in a neural network, it says
we apply a clustering algorithm to the covariance matrix of the predictions of our generalist model, so that a set of classes $S^m$ that are often predicted together will be used as targets for one of our specialist models, $m$. We applied an online version of the $K$-means algorithm to the columns of the covariance matrix, and obtained reasonable clusters (shown in Table 2). We tried several clustering algorithms which produced similar results.
I know how $K$-means is applied to a bunch of data, but here it says applying $K$-means to the columns of the covariance matrix, which is quite confusing to me.
My questions are:
- What is the covariance matrix here? (or what are the random variables related to the covariance matrix?)
- How is it computed?
- How is the online $K$-means applied to the covariance matrix?
- Why would they do that?
My personal answer to the first question is that the random variables related to the covariance matrix are the probabilities of each class. If that's the case, then the covariance matrix can be computed from samples (according to this). But this is not an online method and seems to take an extremely large amount of computation for a large dataset.
Any comments or partial answers that can help me make progress are appreciated.