Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
568
questions
0
votes
0answers
5 views
Is the generalized entropy of certain events alway null?
In their paper: Novel Decompositions of Proper Scoring Rules for Classification, Kull and Flach wrote in section 2.2 Divergence, Entropy and Properness that when $y$ is the true class $d(p,y)=\phi(p,...
1
vote
0answers
12 views
Name for a particular family of distributions on unit simplex [closed]
Let $\Delta^n$ be the unit simplex, and let $\alpha\ge 0$ be a parameter. Is there a standard name for the following probability density function supported on $\Delta^n$?
$$f_{\alpha}(p)=C(\alpha)e^{-\...
3
votes
1answer
27 views
How to conceptualise statistical entropy
Preamble
There's this great YouTube video explaining the entropy of a probability distribution using the idea of "what is the minimum number of questions I need to ask?".
So as an example, ...
0
votes
0answers
16 views
Measuring entropy/concentration in multivariate datasets
I am interested in measuring how concentrated or widely dispersed a certain binary property is among a population, which is defined by a number of categorical variables.
For instance, let's say I have ...
0
votes
0answers
8 views
Change in Shannon Entropy in Markov Chain Process
What are the known results on the change in Shannon entropy $\Delta H_{k} = H(\vec{p}_{k}) - H(\vec{p}_{k-1})$ of the $k$-th step in a process governed by a finite state discrete time Markov chain ...
0
votes
1answer
29 views
Does logit transformation of information entropy values make sense?
I have a vector of information entropy values that range between 0 and 1 which I want to explain with some explanatory variables.
I realized that the distribution of the entropy values in my dataset ...
0
votes
0answers
12 views
Relationship between log-likelihood function and entropy (instead of cross-entropy)
Negative log-likelihood function
$$ -\ln f(X\mid\Theta)=-\int_{-\infty}^{\infty}\ln
f(x\mid\Theta) \, \mathrm{d}x$$
Differential entropy
$$ h(X) = -\int_{-\infty}^{\infty} f(x) \ln f(x) \, \mathrm{d}...
0
votes
1answer
27 views
Constructing a joint distribution from pairwise marginals
Consider a set of random variables $\{X_i\}$ with joint pdf $f(x_1 ... x_n)$. Given the marginal pdfs $f_i(x_i)$, we can construct a joint distribution
$$g(x_1 ... x_n) = \prod_i f_i(x_i)$$
which has ...
2
votes
1answer
19 views
Information Entropy Problem
I cannot figure out this simple entropy problem and it is driving me crazy!
From McElreath's Statistical Rethinking:
Imagine instead 5 buckets and a pile of 10 individually numbered pebbles. You stand ...
1
vote
0answers
24 views
“Information” Correlation
(Let $X$ and $Y$ be random variables, sufficiently nice for my question to make sense.)
$$
\text{Correlation}
$$
$$
\rho(X, Y) = \dfrac{\text{cov}(X, Y)}{\sqrt{\text{var}(X)}\sqrt{\text{var}(Y)}}
$$
...
2
votes
1answer
52 views
Information gain of the root node
Recently I saw this question and answer as attached in following image
Anyone can add details how this solution achieved?
1
vote
0answers
21 views
Can a likelihood's relative entropy be related to its predictive accuracy?
Suppose I have some prior $\pi(\theta)$, from which I draw $N$ samples, each having parameter $\theta_i$. These $\theta_i$'s are known to me. Suppose that one of these samples (unknown to me which) ...
0
votes
0answers
8 views
Bounds on the difference between entropies of two distributions?
Have there been any results regarding bounds on $\left| H(\hat{\pi})-H(\pi) \right|$ where $\hat{\pi}$ and $\pi$ are the empirical and true probability distribution?
0
votes
1answer
36 views
Clarification of variational autoencoders
I've been spending almost a month trying to understand VAE. I was reading a bunch of tutorials, and first it made sense, and seemed straightforward. Then I was experimenting with it, it produced weird ...
0
votes
1answer
13 views
Measure of segregation that takes population-level proportions into account (modified entropy/Theil index)
I am interested in a measure of segregation which allows for multiple groups and is maximized when group levels reflect the population as a whole (rather than equality of group size).
The best I have ...
1
vote
0answers
11 views
CNN: quantitatively evaluate the activation of a filter
When an image is fed into a CNN, it would pass through different layer of different filters. The visualization of these filters look like:
Here we can safely claim that the 1st filter is more ...
0
votes
0answers
4 views
Entropy of multivariate Pearson Type II
I was checking the entropy of multivariate Pearson Type II for maximizing Renyi entropy.
It would be great if someone came up with this question or advise some reliable sources.
0
votes
1answer
21 views
Minimising KL divergence between two distributions
say we want to approximate a distribution $p(x)$ with $q(x|\theta)$. We do not know the distribution $p(x)$ but we can draw samples from $p(x)$. The KL divergence between the two distributions is
$$
\...
3
votes
1answer
67 views
Why doesn't the entropy decrease with an increasing number of observations?
I'm trying to think more about entropy.
I have the following toy example:
Consider a coin flip.
Case 1:
I think p_h = 0.5
The entropy of this is 0.5 ln(0.5) x 2 = ln(0.5)
Case 2:
I don't know what p_h ...
3
votes
0answers
31 views
Statistically, is it effective to design exams to get harder as you answer correctly?
I think we have all taken exams where as you answer more questions correctly, the exam gets harder. Intuitively, this is obvious why...If an IQ test had questions for 3rd graders, everyone would get ...
0
votes
0answers
14 views
Information gain for unequally sized time series
I have some time series data, $\mathbf{x}_{1:T} = \{ x_1, \dots, x_T \}$ where the observation at time $t$, $X_t$, is a continuous random variable. Let $Y_t$ denote a discrete random variable at time $...
1
vote
0answers
24 views
Is there a metric to identify whether samples are uniformly distributed when number of samples is small?
Suppose I have a small number of samples drawn from an unknown distribution $\{X_1,X_2,...,X_n\}$, where $0\le X_i \le L$, and $3\le n \le10$. I want to identify a metric to understand how far these ...
1
vote
0answers
11 views
entropy regularization in generative model
I am wondering if it is possible to use entropy as a regularization in a generative model. For example, in the conjugate model where $x_i \in X$ is observed data and generated from a Normal ...
2
votes
1answer
70 views
Log probabilities versus squared probabilities (entropy vs Gini)
The advantage of log probabilities over direct probabilities, as discussed here and here, is that they make numerical values close to $0$ more easy to work with. (my question, instead of the links, ...
1
vote
0answers
34 views
When would you use purity as a measure of external validity over entropy? [closed]
This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
0
votes
0answers
14 views
How does Information Gain work?
I know the algorithm upon which the information gain is based. But how are we sure that after a split, its entropy will always decrease i.e. on dividing the data into smaller chunks, the entropy will ...
1
vote
1answer
58 views
Maximum Entropy Discrete Distribution
In Pattern Recognition and Machine Learning the author uses Lagrange multipliers to find the discrete distribution with maximum entropy.
Entropy is defined by;
$$H=-\sum_i p(x_i)\ln(p(x_i))$$
and the ...
0
votes
0answers
11 views
What is the effect of number of points on calculation of mutual information between two variables?
I was trying to calculate mutual information between two variables as follows:
mutual_info_score(x,y)
This essentially creates a 2d histogram and evaluates the ...
0
votes
0answers
24 views
If entropy is the underlying measure for KL-divergence, what is the underlying measure for the Wasserstein distance?
If entropy is the basis measure underlying KL-divergence (aka relative entropy), what is the basis measure underlying the Wasserstein distance?
0
votes
0answers
14 views
If a zero entropy distribution implies high information a priori, what does it mean ex posteriori?
The following counteracts the statements made for the maximum entropy principle case in order to posit a pseudo "minimum entropy principle" case that is simply the polar opposite of the ...
0
votes
1answer
28 views
Truncated entropy
If we bound a continuous random variable's probability distribution from below with $a$ and above with $b$, would the differential entropy of this truncated portion of the pdf just be
$$h(X)_{trunc} = ...
0
votes
1answer
47 views
Which has minimum concentration: the uniform distribution or the maximum entropy distribution?
For a continuous random variable, the uniform distribution has high entropy because it demonstrates the greatest level uncertainty.
However, this conflicts with the maximum entropy principle, which ...
2
votes
1answer
16 views
Entropy of RV that is function of coin flips
Suppose that random variables $X_1,\dots,X_n$ correspond to independent and fair coin flips, i.e., it holds for their entropy that $H[ X_i ] = 1$, for all $i$.
Let $Y = f(X_1,\dots,X_n)$ be some ...
1
vote
0answers
13 views
General implications of low entropy for a dataset
I am fairly familiar with entropy, which quantifies uncertainty/surprisal of a random variable. In my case, I have a corpus where I can use empirical word frequencies to estimate entropy of the entire ...
0
votes
0answers
13 views
Survival entropy
If continuous random variable $X$ has a probability distribution $f(x)$, the entropy of $X$ in its entirety (all of $f(x)$) can be calculated as $H(X)$.
If I am more interested in measuring the ...
1
vote
0answers
24 views
What is the entropy of multivariate data multiplied by a vector?
It is a general rule that for multivariate data $\boldsymbol{X}$ and a matrix $\boldsymbol{A}$, their entropy is
$$h(\boldsymbol{A} \boldsymbol{X}) = h(\boldsymbol{X}) + \ln |\det \boldsymbol{A}|$$
(...
1
vote
0answers
15 views
Entropy of large sample $X$ and small sample $Y$ comparison
Distribution $X$ of a real continuous random variable is unimodal and slightly non-normal with $1,000$ observations.
Distribution $Y$ has the same characteristics (not perfectly identical in moments ...
0
votes
1answer
23 views
Is entropy the same for a shifted-mean distribution? [duplicate]
The image below shows two identically shaped (Normal) distributions with the second only different by its mean. If I calculate the differential entropy of both separately, would the entropies of the ...
0
votes
0answers
15 views
Can $f$-divergences narrow the discrepancy between train and test fits in machine learning?
Machine learning models whose task is to predict unseen test data would work best if the test data's distribution turns out to be the same as the training data's distribution. Real data seldom works ...
2
votes
1answer
38 views
What is the entropy of a riskless random variable?
Variance and standard deviation are often used as proxies for risk and volatility. I make the analogy to information theory as follows, correct if it's wrong: a random variable $x\in \mathbb{R}$ that ...
2
votes
1answer
102 views
How come differential entropy can be negative but mutual information can't?
Differential entropy,
$$H(X) = -\int_{-\infty}^{\infty} p(x) \ln p(x) dx,$$
ordinarily is positive (the negative sign in front actually makes the entire expression positive). However, it can be ...
1
vote
0answers
71 views
For real variables, variance is to entropy, what the mean is to -?
If $X$ is a real random variable with a pdf, variance/standard deviation is a measure of $X$'s dispersion about the pdf's central tendency, which in turn is referred to as the mean of $X$. For many, ...
0
votes
0answers
28 views
Is conditional entropy noise entropy?
Can the conditional entropy, $H(X|Y)$, of two random variables $X$ and $Y$ be thought of as noise entropy in the formula for mutual information,
$$I(X,Y) = H(X) - H(X|Y)$$
making $I(X,Y)$ the full ...
-1
votes
1answer
28 views
Do identical entropies imply perfect correlation?
If two random variables $X$ and $Y$ have the same Shannon entropy, $$H(X) = H(Y)$$
can it be said that their Pearson correlation $\rho(X,Y) = 1$?
Is it true always or just in some cases, which cases
0
votes
1answer
52 views
Can two variables' entropies be equal, as well as their joint entropy?
If two random variables $X$ and $Y$ have the same Shannon entropy, $$H(X) = H(Y)$$
can their joint entropy ever be equal to both? $$H(X,Y) = H(X) = H(Y)$$
2
votes
0answers
34 views
Mutual Information and the informational coefficient of correlation
Linfoot (1957) introduced the informational coefficient of correlation, $\mathit{IC}$:
$$IC=\sqrt{1-e^{-2\cdot{I(X;Y)}}}$$
Where $\mathit{I(X;Y)}$ is the mutual information, the measure of the shared ...
1
vote
0answers
36 views
Entropy regularization versus L2 norm regularization?
In multiple regression problems, the decision variable, coefficients $\beta$, can be regularized by its L2 (Euclidean) norm, shown below (in the second term) for least squares regression. This type of ...
2
votes
2answers
74 views
How does a distribution's differential entropy correspond to its moments?
The Gaussian distribution maximizes entropy for the following functional constraints
$$E(x) = \mu$$
and
$$E((x-\mu)^2) = \sigma^2$$
which are just its first and second statistical moments (true ...
1
vote
0answers
32 views
Is KL-divergence just the multiplication rule for independent events, reformulated in terms of entropy?
We know KL-divergence is sometimes expressed like this:
which shows it's capturing the deviation between the joint distribution of X and Y, and the product of marginals for X and Y. This suggests KL-...
4
votes
2answers
160 views
How to measure whether a discrete distribution is uniform or not?
Say I have two vectors [1,2,1,2,2] and [1,2,1,1,1]. The number at each dimension is the frequency of one element. How do I measure whether these two vectors are close to the uniform distribution? I ...
