Linked Questions
24 questions linked to/from How does entropy depend on location and scale?
0
votes
1
answer
125
views
Is entropy the same for a shifted-mean distribution? [duplicate]
The image below shows two identically shaped (Normal) distributions with the second only different by its mean. If I calculate the differential entropy of both separately, would the entropies of the ...
- 4,049
1
vote
0
answers
43
views
What is the interpretation of negative differential entropy? [duplicate]
I'm currently trying to understand the meaning of negative entropy. With the given equation:
$$
H_{cont}\ (X) = -\int_{-\infty}^{+\infty} p(x) \centerdot ln(p(x)) \ dx
$$
Here, the The_Sympathizer ...
- 11
29
votes
4
answers
30k
views
What is the role of temperature in Softmax?
I'm recently working on CNN and I want to know what is the function of temperature in the softmax formula? and why should we use high temperatures to see a softer norm in probability distribution?
The ...
- 393
12
votes
2
answers
7k
views
When is the differential entropy negative?
The definition of entropy for a continuous signal is:
$$h[f] = \operatorname{E}[-\ln (f(X))] = -\int\limits_{-\infty}^{\infty} f(x) \ln (f(x))\, dx$$
According to Wikipedia, it can be negative. When ...
- 4,598
7
votes
6
answers
2k
views
Does higher variance usually mean lower probability density?
Does higher variance usually mean lower probability density? Despite the type of distribution. Thank you.
Update：
Sorry for confusion. Please allow me to clarify. If I sample the same number of data ...
- 101
13
votes
1
answer
3k
views
Differential Entropy drops when any random variable is normalized to unit variance
Differential entropy of Gaussian R.V. is $\log_2(\sigma \sqrt{2\pi e})$. This is dependent on $\sigma$, which is the standard deviation.
If we normalize the random variable so that it has unit ...
- 4,504
7
votes
2
answers
2k
views
Entropy of Cauchy (Lorentz) Distribution
Entropy is defined as
$H$ = $- \int_\chi p(x)$ $\log$ $p(x)$ $dx$
The Cauchy Distribution is defined as
$f(x)$ = $\frac{\gamma}{\pi}$ $\frac{1}{\gamma^2 + x^2} $
I kindly ask to show the steps to ...
- 73
5
votes
2
answers
2k
views
Does the entropy of a random variable change under a linear transformation?
Let $X$ be a random variable. If $Y=aX+b$, where $a,b \in \mathbb{R}$, is the entropy of $Y$ the same as the entropy of $X$?
- 5,180
6
votes
2
answers
793
views
Equivalent ways of parametrizing Gamma distribution
From scipy documentation at https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gamma.html, Gamma distribution is written as
$$f(x, \alpha) = \frac{x^{\alpha - 1} e^{-x}}{\Gamma(\alpha)}$$...
- 1,176
3
votes
3
answers
546
views
Is entropy conserved under invertible mappings?
Suppose for a random variable $ X\colon \Omega \to E $, I have an invertible mapping $ Z = f(X) $.
Is the Shannon Entropy for each variable equivalent?
$$ H(X) = H(Z) $$
If not, can anything ...
user174574
7
votes
2
answers
2k
views
Can the differential entropy be negative infinity?
Define the (differential) entropy for density $f$ as
$$ H(f) :=-\int_{0}^{1} f(x) \log_{2}(f(x)) dx \, .$$
I am trying to find a Lebesgue measurable $f$ defined on $[0,1]$ such that $f\geq 0, \...
1
vote
2
answers
2k
views
For real variables, variance is to entropy, what the mean is to -?
If $X$ is a real random variable with a pdf, variance/standard deviation is a measure of $X$'s dispersion about the pdf's central tendency, which in turn is referred to as the mean of $X$. For many, ...
- 4,049
3
votes
2
answers
579
views
How does a distribution's differential entropy correspond to its moments?
The Gaussian distribution maximizes entropy for the following functional constraints
$$E(x) = \mu$$
and
$$E((x-\mu)^2) = \sigma^2$$
which are just its first and second statistical moments (true ...
- 4,049
6
votes
1
answer
145
views
Does Shannon Entropy uniquely characterise distribution function $f$?
If I have a distribution $f(x)$ over the real line where the support is the whole line, does the Shannon Entropy uniquely characterise $f$? I.e., do we have $H(f) = H(f^*)$ implies $f = f^*$? (The ...
- 85
0
votes
0
answers
803
views
KL Divergence Normal and Laplace densities
I want to calculate the KL-Divergence between a Laplacian density g and a normal density f. I can decompose $KL(G|F)$ to $\mathbb{E}_g[\log g(X)]-\mathbb{E}_g[\log f(X)]$. I am already stuck with my ...
- 105