Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
549
questions
0
votes
0answers
17 views
Shannon entropy and Gini impurity are interchangeable in practice yet different?
Gini impurity, not to be confused with the Gini coefficient,
is also an information theoretic measure and corresponds to Tsallis Entropy with deformation coefficient $q=2$, which in physics is ...
1
vote
0answers
14 views
When would you use purity as a measure of external validity over entropy?
This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
0
votes
0answers
11 views
How does Information Gain work?
I know the algorithm upon which the information gain is based. But how are we sure that after a split, its entropy will always decrease i.e. on dividing the data into smaller chunks, the entropy will ...
1
vote
1answer
42 views
Maximum Entropy Discrete Distribution
In Pattern Recognition and Machine Learning the author uses Lagrange multipliers to find the discrete distribution with maximum entropy.
Entropy is defined by;
$$H=-\sum_i p(x_i)\ln(p(x_i))$$
and the ...
0
votes
0answers
8 views
What is the effect of number of points on calculation of mutual information between two variables?
I was trying to calculate mutual information between two variables as follows:
mutual_info_score(x,y)
This essentially creates a 2d histogram and evaluates the ...
0
votes
0answers
11 views
How would the entropy of a random walk change over time? [closed]
Generate a stochastic process random walk (with drift) with $T$ observations. We can further split the random walk into:
expanding windows ($10$ observations, for example, are added to the window as ...
0
votes
0answers
17 views
If entropy is the underlying measure for KL-divergence, what is the underlying measure for the Wasserstein distance?
If entropy is the basis measure underlying KL-divergence (aka relative entropy), what is the basis measure underlying the Wasserstein distance?
0
votes
0answers
11 views
If a zero entropy distribution implies high information a priori, what does it mean ex posteriori?
The following counteracts the statements made for the maximum entropy principle case in order to posit a pseudo "minimum entropy principle" case that is simply the polar opposite of the ...
0
votes
1answer
20 views
Truncated entropy
If we bound a continuous random variable's probability distribution from below with $a$ and above with $b$, would the differential entropy of this truncated portion of the pdf just be
$$h(X)_{trunc} = ...
0
votes
1answer
29 views
Which has minimum concentration: the uniform distribution or the maximum entropy distribution?
For a continuous random variable, the uniform distribution has high entropy because it demonstrates the greatest level uncertainty.
However, this conflicts with the maximum entropy principle, which ...
2
votes
1answer
15 views
Entropy of RV that is function of coin flips
Suppose that random variables $X_1,\dots,X_n$ correspond to independent and fair coin flips, i.e., it holds for their entropy that $H[ X_i ] = 1$, for all $i$.
Let $Y = f(X_1,\dots,X_n)$ be some ...
0
votes
0answers
10 views
General implications of low entropy for a dataset
I am fairly familiar with entropy, which quantifies uncertainty/surprisal of a random variable. In my case, I have a corpus where I can use empirical word frequencies to estimate entropy of the entire ...
0
votes
0answers
12 views
Survival entropy
If continuous random variable $X$ has a probability distribution $f(x)$, the entropy of $X$ in its entirety (all of $f(x)$) can be calculated as $H(X)$.
If I am more interested in measuring the ...
1
vote
0answers
20 views
What is the entropy of multivariate data multiplied by a vector?
It is a general rule that for multivariate data $\boldsymbol{X}$ and a matrix $\boldsymbol{A}$, their entropy is
$$h(\boldsymbol{A} \boldsymbol{X}) = h(\boldsymbol{X}) + \ln |\det \boldsymbol{A}|$$
(...
1
vote
0answers
15 views
Entropy of large sample $X$ and small sample $Y$ comparison
Distribution $X$ of a real continuous random variable is unimodal and slightly non-normal with $1,000$ observations.
Distribution $Y$ has the same characteristics (not perfectly identical in moments ...
1
vote
1answer
15 views
Is entropy the same for a shifted-mean distribution? [duplicate]
The image below shows two identically shaped (Normal) distributions with the second only different by its mean. If I calculate the differential entropy of both separately, would the entropies of the ...
0
votes
0answers
14 views
Can $f$-divergences narrow the discrepancy between train and test fits in machine learning?
Machine learning models whose task is to predict unseen test data would work best if the test data's distribution turns out to be the same as the training data's distribution. Real data seldom works ...
2
votes
1answer
22 views
What is the entropy of a riskless random variable?
Variance and standard deviation are often used as proxies for risk and volatility. I make the analogy to information theory as follows, correct if it's wrong: a random variable $x\in \mathbb{R}$ that ...
2
votes
1answer
59 views
How come differential entropy can be negative but mutual information can't?
Differential entropy,
$$H(X) = -\int_{-\infty}^{\infty} p(x) \ln p(x) dx,$$
ordinarily is positive (the negative sign in front actually makes the entire expression positive). However, it can be ...
1
vote
0answers
51 views
For real variables, variance is to entropy, what the mean is to -?
If $X$ is a real random variable with a pdf, variance/standard deviation is a measure of $X$'s dispersion about the pdf's central tendency, which in turn is referred to as the mean of $X$. For many, ...
0
votes
0answers
25 views
Is conditional entropy noise entropy?
Can the conditional entropy, $H(X|Y)$, of two random variables $X$ and $Y$ be thought of as noise entropy in the formula for mutual information,
$$I(X,Y) = H(X) - H(X|Y)$$
making $I(X,Y)$ the full ...
-1
votes
1answer
27 views
Do identical entropies imply perfect correlation?
If two random variables $X$ and $Y$ have the same Shannon entropy, $$H(X) = H(Y)$$
can it be said that their Pearson correlation $\rho(X,Y) = 1$?
Is it true always or just in some cases, which cases
0
votes
1answer
39 views
Can two variables' entropies be equal, as well as their joint entropy?
If two random variables $X$ and $Y$ have the same Shannon entropy, $$H(X) = H(Y)$$
can their joint entropy ever be equal to both? $$H(X,Y) = H(X) = H(Y)$$
2
votes
0answers
31 views
Mutual Information and the informational coefficient of correlation
Linfoot (1957) introduced the informational coefficient of correlation, $\mathit{IC}$:
$$IC=\sqrt{1-e^{-2\cdot{I(X;Y)}}}$$
Where $\mathit{I(X;Y)}$ is the mutual information, the measure of the shared ...
1
vote
0answers
21 views
Entropy regularization versus L2 norm regularization?
In multiple regression problems, the decision variable, coefficients $\beta$, can be regularized by its L2 (Euclidean) norm, shown below (in the second term) for least squares regression. This type of ...
2
votes
2answers
59 views
How does a distribution's differential entropy correspond to its moments?
The Gaussian distribution maximizes entropy for the following functional constraints
$$E(x) = \mu$$
and
$$E((x-\mu)^2) = \sigma^2$$
which are just its first and second statistical moments (true ...
1
vote
0answers
29 views
Is KL-divergence just the multiplication rule for independent events, reformulated in terms of entropy?
We know KL-divergence is sometimes expressed like this:
which shows it's capturing the deviation between the joint distribution of X and Y, and the product of marginals for X and Y. This suggests KL-...
4
votes
2answers
144 views
How to measure whether a discrete distribution is uniform or not?
Say I have two vectors [1,2,1,2,2] and [1,2,1,1,1]. The number at each dimension is the frequency of one element. How do I measure whether these two vectors are close to the uniform distribution? I ...
1
vote
1answer
43 views
How to interpret correlation coefficient values of many experiments together? [closed]
In a research article which I was reading I found that, the authors calculated the correlation coefficients between two variables.
The two variables are “absolute entropy” and “change in entropy”. ...
1
vote
0answers
29 views
What is exponential mutual information?
Mutual information of two random variables $X$ and $Y$ is
$$
I(X,Y) = - \int_{-\infty}^\infty f(x,y) \log \frac{f(x,y)}{f(x) f(y)} \mathrm{d}x \mathrm{d}y,
$$
where $f(x)$ is a marginal and $f(x,y)$ ...
1
vote
0answers
12 views
0
votes
0answers
54 views
If entropy has log probabilities, why is exponential entropy used? [duplicate]
For random variable $X$, entropy is calculated as $H(X) =-\sum_i p(x_i) \ln p(x_i)$. Differential entropy, not shown, is its continuous counterpart. Both use log probabilities which convert the ...
1
vote
2answers
74 views
How to include the observed values, not just their probabilities, in information entropy?
Shannon entropy measures the unpredictability in a random variable's outcome as the weighted average of the probabilities of that variable's outcomes or observed values. However, it discards the ...
1
vote
0answers
18 views
What is the volatility of mutual information?
For Gaussian data, it is known that the standard deviation or volatility of entropy, $H(X)$ is
$$\sigma_H = \frac{\exp^{H-1/2}}{\sqrt{2\pi}} $$
I'm not sure how the above is derived, but could someone ...
0
votes
0answers
17 views
Finding the lower bound for relative entropy $D(f||g)$ where, $f$, $g$ are two different distribution?
I am trying to find a tighter bound of the relative entropy $D(f||g)$.
Problem statement:
Let, $f$ and $g$ be two discrete probability distribution.
$f \in \left[ {Q(x + a),Q( - x - a)} \right]$ and $...
2
votes
1answer
28 views
What is the variation of information of a variable with its self?
Variation of information measures the uncertainty we expect in one variable if we are told the value of another variable. It is computed as
$$VI(X,Y) = H(X) + H(Y) - 2 I(X,Y)$$
or
$$VI(X,Y) = H(X,Y) - ...
1
vote
0answers
14 views
How to decompose entropy into linear and non-linear components?
Mutual information is the entropy between two random variables, $X$ and $Y$, based on their probabilities. It captures both the linear and non-linear interactions between $X$ and $Y$, whereas ...
0
votes
0answers
6 views
Is transfer entropy the same as conditional mutual information?
This encyclopedia article says that transfer entropy is conditional mutual information. But the first measure makes no mention of a third variable being required unlike the second measure, and is the ...
4
votes
1answer
49 views
What's an intuitive proof for why Shannon's entropy poses a lower bound on the expected number of bits encoding a distribution?
As I understand it, Shannon's entropy is $-\sum p\log p$ which represents an expectation of the self information $-\log p$ over a distribution.
The log function was chosen to fit 3 properties:
A ...
0
votes
0answers
15 views
Minimize the limit of K-L (Kullback Leibler) divergence for a given conditional probability $p(y|x)$ distribution?
Let, $p(x);p(y)$ are the probability distribution function of random variable $X$, $Y$ and the Conditional probability $p(y|x)$ is given e.g.
$p(y|x)=Q(x+2y)$.
where, $Q(x) = \frac{1}{{\sqrt {2\pi } }}...
0
votes
0answers
18 views
Diagonal elements of a mutual information matrix
Mutual information is a metric from information theory that quantifies the non-linear co-dependencies between a pair of random variables. When extended to more than one pair, a mutual information ...
2
votes
0answers
22 views
Is limiting density of discrete points (LDDP) equivalent to negative KL-divergence?
Is limiting density of discrete points (LDDP), which is a corrected version of differential entropy, equivalent to the negative KL-divergence (or relative entropy) between a density function $m(x)$ ...
0
votes
0answers
22 views
Why do continuous data distributions have entropy of negative infinity?
Entropy is intended for discrete random variables, while differential entropy is used on continuous r.v.'s. This question is the opposite of another similarly titled question about discrete data and ...
0
votes
0answers
24 views
Why we use squared probabilities in the gini impurity
Why we are using squared probabilities instead of normal probabilities in gini impurity . Probabilities will always be positive , so why to square those ? Any leads would be highly apriciated , ...
3
votes
1answer
48 views
Is relative entropy equal to cross-entropy during optimization?
I came across a saying that estimates of KL divergence, otherwise known as relative entropy, of the truth of a random variable and its prediction ($y$ and $\hat{y}$) is equal to their cross entropy ...
0
votes
1answer
24 views
Does entropy have less estimation error than mean and variance estimates?
Estimating the mean or expected value of a continuous random variable's (r.v.) empirical distribution is known to be difficult, moreso than estimating the variance. Estimates of the mean and variance ...
5
votes
4answers
240 views
What is exponential entropy?
Differential entropy (the continuous version of Shannon's entropy measure) is
$$
H = - \int_{-\infty}^\infty f(x) \log f(x) \mathrm{d}x,
$$
where $f(x)$ is a probability density function.
What is the ...
0
votes
0answers
9 views
Are entropy and differential entropy good risk measures?
Standard deviation is often used to measure volatility or risk. But not all continuous random variables have a known statistical distribution and therefore might not even have a mean, variance or ...
2
votes
1answer
29 views
Can entropy be used to minimize prediction surprises in machine learning?
Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or ...
0
votes
0answers
19 views
Is the maximum entropy probability distribution only determined through comparison?
The maximum entropy probability distribution has entropy at least as great as that of all other members of a specified class of probability distributions (pdf's). Does that mean that the pdf with ...
