As this Wikipedia article describes, we can define probability density functions (pdfs) for discrete random variables using Dirac delta functions, which is called "generalized pdf".

I considered the following pdf and tried to calculate its entropy (differential entropy): $$ f(t) = \sum\limits_{i = 1}^n {{p_i}\delta (t - {x_i})} $$

I obtained $-\infty$ by solving the integral (actually because $\int\delta(0)log(\delta(0))dt$ terms appeared). But this result seems odd to me, as I found the entropy is always $-\infty$ regardless of the parameters $n$ and $\{p_i\}$. I think it is not intuitive, since if we compare it to the definition of entropy for discrete distributions, we expect that the entropy gets higher as the distribution gets closer to uniform.

Have I made a mistake in solving the integral? Or the entropy is really independent of distribution parameters in this case?

  • $\begingroup$ The engineer answer is define the integral only on the support of the distirbution. Secondly, the limit x log(x) where x goes to 0 is 0 (you can see it with lupital theorem) $\endgroup$ – Cherny Apr 10 '18 at 8:48
  • $\begingroup$ This is not xlog(x). It is delta(x)log(delta(x)) where x goes to 0. $\endgroup$ – A.Ahmadian Apr 14 '18 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.