0
$\begingroup$

If we bound a continuous random variable's probability distribution from below with $a$ and above with $b$, would the differential entropy of this truncated portion of the pdf just be

$$h(X)_{trunc} = -\int_a^b f(x) \ln f(x) dx ?$$

What might be wrong with this formula, and why isn't truncated entropy ever used?

$\endgroup$

1 Answer 1

3
$\begingroup$

Suppose we have a continuous random variable $X$ with pdf $f$ and we truncate it to some interval $I := [a,b]$ with $P(X\in I) > 0$ (I'm assuming this is what you mean because if we just clip a pdf to some interval in the sense of looking at $f\mathbf 1_{I}$ then it won't in general remain a pdf). Then we can use the CDF $F$ of $X$ to get the new density, so if $x\in I$ we have the truncated CDF as $$ F_T(x) = P(X \leq x \mid X \in I) = \frac{P(X\leq x \cap X\in I)}{P(X\in I)} \\ = \frac{P(a \leq X \leq x)}{P(a \leq X \leq b)} = \frac{F(x) - F(a)}{F(b) - F(a)} $$ therefore the truncated density is $$ f_T(x) = \frac{f(x)}{F(b) - F(a)} $$ (I was casual about $\leq$ vs $<$ since $X$ is continuous). Outside of $I$ the density is zero. Taking $\Delta = F(b) - F(a)$, this means overall the entropy is $$ h[f_T] = -\int_a^b f_T(a) \log f_T(x) \,\text dx \\ = - \frac{1}{\Delta}\int_a^b f(a) \left[\log f(x) - \log \Delta\right] \,\text dx \\ = -\frac 1\Delta \int_a^b f(x)\log f(x)\,\text dx + \log \Delta $$ so it's not quite your formula.

Regarding its use, truncations of random variables come up a lot, both in contexts like censoring but also in probability theory (e.g. Durrett's proof of the weak law of large numbers of his Probability: Theory and Examples) so I'm sure people are doing this somewhere.


If you actually are looking at $$ -\int_a^b f(x)\log f(x)\,\text dx $$ you could note that this is $$ -\text {E} \left[ \mathbf 1_{[a,b]} \log f(X) \right] $$ so we can still view it in terms of truncations. If you have a random variable with possibly infinite entropy then perhaps looking at truncations like this is useful, maybe in an asymptotic way, but I haven't personally seen that.

$\endgroup$
5
  • $\begingroup$ is there a name for $\Delta$? and is there a textbook that gives the formula "that's not quite my formula"? $\endgroup$
    – develarist
    Oct 7, 2020 at 17:55
  • $\begingroup$ @develarist $\Delta = P(X\in I)$ so I'd just describe it that way. I don't know of an official term for it other than that. And I also don't know of an official reference for the formula I gave but I'm just computing the entropy of a truncated density which is a standard enough thing that probably a lot of textbooks would at least discuss that $\endgroup$
    – jld
    Oct 7, 2020 at 20:55
  • $\begingroup$ how might truncated entropy be useful? is there a demand or applications for measuring the disorder/unpredictability in specific segments of a pdf, for example, in its tails? $\endgroup$
    – develarist
    Oct 7, 2020 at 21:00
  • $\begingroup$ @develarist i was just googling it and came across this which might be a good place to start or get some other references atlantis-press.com/journals/jsta/125941168/view. They do pretty much the exact same computation I did here $\endgroup$
    – jld
    Oct 7, 2020 at 21:03
  • $\begingroup$ thanks, as for practical implementations of truncated entropy, since histogram discretization is the norm when estimating entropy, does truncated histogram estimation present issues not encountered in regular, full-pdf histogram estimation? $\endgroup$
    – develarist
    Oct 7, 2020 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.