# Truncated entropy

If we bound a continuous random variable's probability distribution from below with $$a$$ and above with $$b$$, would the differential entropy of this truncated portion of the pdf just be

$$h(X)_{trunc} = -\int_a^b f(x) \ln f(x) dx ?$$

What might be wrong with this formula, and why isn't truncated entropy ever used?

Suppose we have a continuous random variable $$X$$ with pdf $$f$$ and we truncate it to some interval $$I := [a,b]$$ with $$P(X\in I) > 0$$ (I'm assuming this is what you mean because if we just clip a pdf to some interval in the sense of looking at $$f\mathbf 1_{I}$$ then it won't in general remain a pdf). Then we can use the CDF $$F$$ of $$X$$ to get the new density, so if $$x\in I$$ we have the truncated CDF as $$F_T(x) = P(X \leq x \mid X \in I) = \frac{P(X\leq x \cap X\in I)}{P(X\in I)} \\ = \frac{P(a \leq X \leq x)}{P(a \leq X \leq b)} = \frac{F(x) - F(a)}{F(b) - F(a)}$$ therefore the truncated density is $$f_T(x) = \frac{f(x)}{F(b) - F(a)}$$ (I was casual about $$\leq$$ vs $$<$$ since $$X$$ is continuous). Outside of $$I$$ the density is zero. Taking $$\Delta = F(b) - F(a)$$, this means overall the entropy is $$h[f_T] = -\int_a^b f_T(a) \log f_T(x) \,\text dx \\ = - \frac{1}{\Delta}\int_a^b f(a) \left[\log f(x) - \log \Delta\right] \,\text dx \\ = -\frac 1\Delta \int_a^b f(x)\log f(x)\,\text dx + \log \Delta$$ so it's not quite your formula.

Regarding its use, truncations of random variables come up a lot, both in contexts like censoring but also in probability theory (e.g. Durrett's proof of the weak law of large numbers of his Probability: Theory and Examples) so I'm sure people are doing this somewhere.

If you actually are looking at $$-\int_a^b f(x)\log f(x)\,\text dx$$ you could note that this is $$-\text {E} \left[ \mathbf 1_{[a,b]} \log f(X) \right]$$ so we can still view it in terms of truncations. If you have a random variable with possibly infinite entropy then perhaps looking at truncations like this is useful, maybe in an asymptotic way, but I haven't personally seen that.

• is there a name for $\Delta$? and is there a textbook that gives the formula "that's not quite my formula"? Commented Oct 7, 2020 at 17:55
• @develarist $\Delta = P(X\in I)$ so I'd just describe it that way. I don't know of an official term for it other than that. And I also don't know of an official reference for the formula I gave but I'm just computing the entropy of a truncated density which is a standard enough thing that probably a lot of textbooks would at least discuss that
– jld
Commented Oct 7, 2020 at 20:55
• how might truncated entropy be useful? is there a demand or applications for measuring the disorder/unpredictability in specific segments of a pdf, for example, in its tails? Commented Oct 7, 2020 at 21:00
• @develarist i was just googling it and came across this which might be a good place to start or get some other references atlantis-press.com/journals/jsta/125941168/view. They do pretty much the exact same computation I did here
– jld
Commented Oct 7, 2020 at 21:03
• thanks, as for practical implementations of truncated entropy, since histogram discretization is the norm when estimating entropy, does truncated histogram estimation present issues not encountered in regular, full-pdf histogram estimation? Commented Oct 7, 2020 at 21:08