Codebook Perplexity in VQ-VAE

Question

I am working on VQ-VAE experiments, and I have noticed that perplexity has been used as an evaluation measure for VQ codebook. Also, most of the work including codebook perplexity as a evaluation measure assumes higher perplexity better, though I feel higher perplexity is not very wanted in my intuition. For example, lower perplexity indicates a better language model in general cases.

The questions are (1) What exactly are we measuring when we calculate the codebook perplexity in VQ models? (2) Why would we want to have large codebook perplexity? What is the ideal perplexity for VQ models?

Sorry if my questions are unclear.

Answer 1

Consider the example

x = np.random.randint(low=0, high=100, size=(2, 256))
print(x.shape)
z = get_one_hot(x, 1024)
avg_probs = np.mean(z, axis=0)
entropy = avg_probs * np.log(avg_probs + 1e-10)
print(-np.sum(entropy))
perplexity = np.exp(-np.sum(entropy))
print(perplexitity)

1. When calculating perplexity, we are effectively calculating the codebook utilization. In the example above, if you change the low and high to a narrow range, then out of the 1024 codebook entries that we could have picked/predicted by our model, we only ended up picking a small range. Therefore, across the mini-batch, the avg_probs would have > 0 for indexes within the low-high range. Because the avg_probs are less random (think less diffused), it would have a lower entropy and thus lower perplexity.
2. We want a high perplexity because that means that more of the indices in our codebook were used. When more indices are used, we get more perplexity due to more uniformity in avg_probs.
3. This depends on the number of tokens and the size of the codebook. I don't have a good answer for this but I think we get the best perplexity with low=0 and high=codebook_size