1
$\begingroup$

I was reading about the concept of Perplexity and was thinking whether there's a connexion with the law of law of total variance, but I couldn't find any reference.

The law of total variance is:

$$Var(Y) = E[Var(Y|X)] + Var(E[Y|X])$$

which can be interpreted as saying that the total variance of $Y$ can be decomposed into a part that is explained by $X$ and the part that is left unexplained (see algo Explained Variation for descriptions in terms of "information gained")

Let's say $Y$ is value of the next letter and $X$ contains all the previous letters written. Then a high ratio of explained vs unexplained variation would mean that the previous letters make it very likely that I predict the next letter with accuracy. This sounds very similar to the idea behind perplexity, so can you think of a way to connect them? (formally or not)

---- Addendum ----

@GeoMatt22 suggested looking at mutual information.

Perplexity can be defined as:

$$ b^{{-{\frac {1}{N}}\sum _{{i=1}}^{N}\log _{b}q(x_{i})}}$$

where the exponent can be regarded as Cross entropy.

I still don't quite get the relationship between the law of total variance and conditional entropy, but it seems they point to the same idea.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Perhaps mutual information is what you are looking for? (note that entropy is just log perplexity) $\endgroup$
    – GeoMatt22
    Commented Apr 28, 2017 at 18:41
  • $\begingroup$ I think you're on the right track! I guess the connection is between mutual information and cross entropy, which is a way to interpret perplexity. I'll think about it a bit more, let's see if we get any more answers $\endgroup$
    – cd98
    Commented Apr 28, 2017 at 21:41

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.