Deep Embedded Clustering, originally by Xie et al. and in a question on CV as well, assigns cluster memberships with a $t$-distribution, hardens them with a second power, and computes the KL divergence to optimise centroids and data representation in the latent layer of an autoencoder. My question is simply this: is it appropriate to use KL divergence in this situation, given that the variables are dependent and sum to any positive integer, not just one? Let me expand a bit more.

KL divergence can be interpreted as the dissimilarity of two distributions, $Q$ and $P$, where the entropy of their probability density function is compared: $$\text{KL}(P||Q) = \sum_{x\in X} P(x)\log\frac{P(x)}{Q(x)}$$ However, in the article Xie et al. use it with fuzzy cluster assignments. For example with two data points and two centroids, a matrix of cluster assignments $Q$ and the hardened assignments $P$ could look like this: $$Q=\begin{bmatrix}0.8&0.3\\0.2&0.7\end{bmatrix}\quad P=\begin{bmatrix}0.9&0.2\\0.1&0.8\end{bmatrix}$$ where the columns of the matrices represent data points belonging to clusters as rows. The columns are normalised to sum to 1. The hardening relates to the optimisation task, but it's not important here. What's important is that we have two different assignments for the same data.

Xie et al. proceed to calculate the KL divergence from $Q$ and $P$. The output is certainly valid, reflecting the dissimilarity of the assignments. However, as I stated above, I'm suspicious. Two things concern me:

  • Dependent variables: as the assignments of each data point are normalised, the variables are dependent. Does this affect the calculation?
  • Not a proper probability density function: the values sum to the number of data points instead of 1. Does the distribution need to be always normalised to 1?

1 Answer 1


When researching this question, I think I've found some answers. I'm by no means an expert, so feel free to post other answers.

I think using $\text{KL}$ is appropriate here.

Dependent variables

As the distribution has a constant sum, all the variables are dependent to a certain degree. I doubt that the fact that subsets of the variables are further dependent affect the calculation.

Not a proper PDF

Normalising the distributions again would amount to multiplying them both with some value. Let's see it through in the $\text{KL}$ formula. $$\text{KL}(cP||cQ) = \sum_{x\in X} cP(x)\log\frac{cP(x)}{cQ(x)} = c\sum_{x\in X} P(x)\log\frac{P(x)}{Q(x)}$$ So turns out that the factor just scales the overall value of $\text{KL}$, having no practical effect in the optimisation process. Though scaling could be done just to be pedantic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.