Deep Embedded Clustering, originally by Xie et al. and in a question on CV as well, assigns cluster memberships with a $t$-distribution, hardens them with a second power, and computes the KL divergence to optimise centroids and data representation in the latent layer of an autoencoder. My question is simply this: is it appropriate to use KL divergence in this situation, given that the variables are dependent and sum to any positive integer, not just one? Let me expand a bit more.
KL divergence can be interpreted as the dissimilarity of two distributions, $Q$ and $P$, where the entropy of their probability density function is compared: $$\text{KL}(P||Q) = \sum_{x\in X} P(x)\log\frac{P(x)}{Q(x)}$$ However, in the article Xie et al. use it with fuzzy cluster assignments. For example with two data points and two centroids, a matrix of cluster assignments $Q$ and the hardened assignments $P$ could look like this: $$Q=\begin{bmatrix}0.8&0.3\\0.2&0.7\end{bmatrix}\quad P=\begin{bmatrix}0.9&0.2\\0.1&0.8\end{bmatrix}$$ where the columns of the matrices represent data points belonging to clusters as rows. The columns are normalised to sum to 1. The hardening relates to the optimisation task, but it's not important here. What's important is that we have two different assignments for the same data.
Xie et al. proceed to calculate the KL divergence from $Q$ and $P$. The output is certainly valid, reflecting the dissimilarity of the assignments. However, as I stated above, I'm suspicious. Two things concern me:
- Dependent variables: as the assignments of each data point are normalised, the variables are dependent. Does this affect the calculation?
- Not a proper probability density function: the values sum to the number of data points instead of 1. Does the distribution need to be always normalised to 1?
