Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
666
questions
0
votes
0
answers
15
views
Measuring entropy of a binary matrix with biased probability
I tried to measure entropy of binary matrix like below using code at : https://github.com/cosmoharrigan/matrix-entropy
(I already saw the question : Measuring entropy/ information/ patterns of a 2d ...
- 1
0
votes
0
answers
62
views
How can the mutual information be equal to minus conditional entropy?
I am reading the following paper:
https://arxiv.org/abs/2301.06941
The authors in Eq.(8) have obtained a relation which has the mutual information, $i$, in the exponent of the exponential on the RHS ...
user381076
0
votes
0
answers
24
views
Perplexity and cross-entropy relationship
According to wikipedia Perplexity-
A perplexity of discrete distribution p equals to $2^{H(p)}$ where H(p) is the entropy of p.
A perplexity of a probability model M, with unknown probabiltiy p - ${\...
- 31
0
votes
0
answers
45
views
Using zlib to estimate entropy rate of text data
I'm trying to estimate the entropy rate of a text file (about 2 million characters, including spacing and punctuation). Calculating the entropy rate directly is not an option for me, for lack of ...
- 65
0
votes
0
answers
153
views
Marginal contribution of each observation to posterior moments and (differential) entropy are decreasing in the number of observations?
In the Beta-Binomial and Gaussian settings, indeed, the marginal contribution of each observation to the posterior moments (I could check first and second) and (differential) entropy are decreasing in ...
- 93
0
votes
1
answer
40
views
Why the maximum of Rényi entropy is not achieved for the uniform distribution? [closed]
Given a discrete random variable $X$, which takes values in the alphabet $\mathcal {X}$ and is distributed according to $p:{\mathcal {X}}\to [0,1]$ the Shannon entropy is defined as:
$$\mathrm {H} (X):...
- 297
1
vote
0
answers
26
views
How to calculate a path entropy in the context of information theory?
Consider a path where the system goes from states $i$ to $j$ and then from $j$ to $k$. Given that the probability of finding the system in state $i$ is $p_i$ and the transition probabilities are $p_{...
- 61
1
vote
0
answers
29
views
Understanding properties of conditional entropy
I was trying to understand concept of cross entropy. Wikipedia says:
$H(Y|X)\leq H(Y)$
Although the specific conditional entropy $H(X|Y=y)$ can be either less or greater than $H(X)$ for a given ...
- 215
0
votes
0
answers
50
views
Entropy of a Continuous and Uniform Random Variable
The distribution of a uniform r.v. X is given as follows:
The entropy is therefore:
This means that as $∆$ approaches infinity, so does the entropy.
This also means that as $∆$ approaches 0, the ...
- 5
1
vote
1
answer
63
views
Relationship between entropy and predictability
The entropy of a random variable $X$ is defined as $\mathbb{E}(-\log(f(X)))$ (where $f$ is the pdf of X, https://en.wikipedia.org/wiki/Entropy_(information_theory)).
Is there any general relationship ...
- 385
0
votes
1
answer
14
views
Quantifying + comparing the effects of conditioning on Shannon entropy
I'm working on a project that compares how predictable/unpredictable individuals' actions are in terms of how they transition between actions. We consider their actions to part of a first-order Markov ...
- 1
0
votes
0
answers
17
views
How can I calculate the entropy of a binned multivariate normal distribution?
I know that I can calculate the entropy of a multivariate distribution by using
$${N(\mu,{\boldsymbol {\Sigma }}) \\ \mathcal H(N) = \displaystyle {\frac {1}{2}}\ln \det \left(2\pi \mathrm {e} {\...
0
votes
0
answers
35
views
Measure of clumpiness or cluster of 1D continuous data over a given range
I'm looking for a measure of how clustered (or converse empty) data is over a range (e.g. [0,1] in this case). The first obvious metric to use would be entropy (or ...
- 1
0
votes
0
answers
11
views
Probability and Entropy of a Multichannel Image
I'm currently trying to implement the method covered in the paper "Hyperspectral Band Selection via Optimal Combination Strategy"
I found that point 1) of this questions was close to what I ...
- 1
0
votes
0
answers
84
views
PCA evaluation with Shannon Entropy
I'm working on a fairly large dataset (5e5 samples in a 22 dimensional space); I'm reducing the dimensionality of this dataset with PCA.
Since a lot of variables are cross-correlated, I group the ...
- 11
0
votes
0
answers
10
views
How to show whether the entropy of a uniform distribution with infinite image space converges or diverges?
Assuming a random variable $X$ with an infinite image space (i.e. the natural numbers $\mathbf{N}$) and the probability mass function (PMF) being the uniform distribution with PMF $P(X=n) = \frac{1}{n}...
- 303
0
votes
0
answers
8
views
How do entropy-based estimators relate to more conventional ML, least square, and GMM estimators
Over the years i have done a lot of analysis, mostly of parameters of linear approximations to the data or a forecast, and I have used linear and nonlinear least squares, maximum likelihood, and GMM ...
- 2,757
0
votes
0
answers
26
views
How do I calculate the entropy of a uniform continuous variable?
Let $x$ be any random rational number where $x \in (0, 1/8)$ and $f(x) = 8x$.
Therefore, $f(x) \in (0, 1)$. How would I mathematically calculate the
entropy of $x$ and $f(x)$?
Applying the ...
- 101
1
vote
0
answers
33
views
Differential entropy / variance
So I‘m basically more or less trying to prove what is stated in the answer by syeh_106 here:
Is differential entropy always less than infinity?
if the variance is finite, then the differential ...
- 83
0
votes
0
answers
17
views
Why does entropy of Y increase with increase in number of samples making up Y?
Suppose, a sample of $Y=(1/\sqrt{N}) ∑_{i=1,\dotsc,N} X_i$, where $X\sim \mathcal{Uniform Distribution}(-3, 3)$. We have let's say 10,000 such samples of Y.
Here, when we increase the value of N, why ...
- 401
1
vote
0
answers
16
views
Relate cross-entropy formal definition to the cross-entropy loss [duplicate]
Cross entropy for a random variable $x \sim p$ and a distribution $q$ is defined as:
$$H(p,q) = -\sum_{x\in\mathcal{X}} p(x)\log q(x) = \mathbb{E}(\log q(x))$$
$\mathcal{X}$ is all possible values ...
- 303
1
vote
1
answer
68
views
Maximize Mutual Information between multiple ("overlapping") RVs
I'd like to maximize the sum of Mutual Information between a RV $X$ and $K$ out of $N$ possible RVs $Z_i$.
$$ \max \sum_{i \in K} \text{MI}(X, Z_i) $$
However, when I unfold the sum I get
$$ \sum_{i \...
- 57
1
vote
2
answers
58
views
Are there better measures of entropy
Related question here
I am trying to measure the uniformity of multimodal distributions and am looking into using entropy.
I would like a measure of entropy that is higher for the first distribution ...
- 121
0
votes
0
answers
44
views
How to sample distributions far from uniform?
I need to obtain an isotropic sample of discrete distributions over $d$ outcomes which are far from uniform in some measure.
For instance, using linear entropy as the complexity measure, we can ...
- 5,859
1
vote
0
answers
49
views
Is maximal spectral entropy of residuals a poor loss function because phase information is lost?
Suppose I define a custom loss function SpectralEntropy as follows:
...
- 6,135
2
votes
2
answers
93
views
Surprisal in rankings
I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race.
Intuitively, in a race with 100 people, ...
- 1,083
2
votes
0
answers
30
views
How much do we learn about a random subset? [closed]
Suppose we sample the following two random variables, for some large integer $n$:
Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
- 21
1
vote
0
answers
43
views
conditional entropy of finite mixture model
I am trying to understand the conditional entropy of the finite mixture model given in this paper about regularized EM algorithm.
#################
On page 3 of the paper: in the finite mixture model, ...
- 173
2
votes
0
answers
37
views
How can entropy help in machine learning classification? [closed]
I cannot fully grasp why entropy is so important in machine learning classification. I understand the usage in decision trees but I don't see the importance elsewhere. I mean entropy somehow comes up ...
- 61
0
votes
0
answers
79
views
How can Mutual Information be lower than Adjusted Mutual Information?
What would be the statistical reasoning that MI < AMI in cluster evaluation? Wouldn't you expect AMI to always be slightly (or a lot) lower than MI?
0
votes
0
answers
34
views
differential entropy rate for stationary processes
the differential entropy rate of a stochastic process $X = (X_i)_{i \in \mathbb{N}}$ is defined as $$h(X) = \lim_{n \to \infty} \frac{1}{n} h(X_1,\ldots,X_n)$$,
where $h(X_1,\ldots,X_n)$ is the joint ...
- 1
0
votes
0
answers
19
views
Estimeate Entropy of Sequence
Suppose I have a sequence of N symbols. There are 100 symbols. The probability of each symbol is quite small, though they're not necessarily uniform.
s1 - s2 - s5 - s8 - s3 - sk ....
Each symbol is an ...
- 1
1
vote
0
answers
33
views
Intuition of multivariate conditional entropy
This might be a dumb question, but I'm serious.
Per the wiki page about conditional entropy: quantifies the amount of information needed to describe the outcome of a random variable $Y$ given that the ...
- 11
0
votes
0
answers
21
views
How to interpret the Adjusted Rand Index
Given a pair-confusion matrix where:
...
2
votes
1
answer
72
views
Confidence Interval of entropy for a discrete distribution
The following table is a set of ordinal data from a survey I have conducted (one of many).
$$\begin{array}{c|c|c|}
\text{Grading}& \text{Count} & \text{Frequency} \\ \hline
\text{1} & 5 &...
- 679
2
votes
1
answer
68
views
Why is it called the cross-entropy of q relative to p, not p relative to q?
I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy
Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
- 411
1
vote
0
answers
24
views
In the DT algorithm, why not search through all possible splits instead of first choosing the feature required for splitting?
As far as I understand the Decision Trees (DT - CART), the first step is to choose a feature that maximizes IG (information gain), meaning the entropy in the current node - the entropy after splitting ...
- 45
2
votes
1
answer
126
views
Cross entropy of a random variable or a probability distribution function? [duplicate]
I'm looking into the wikipedia page of cross entropy. https://en.wikipedia.org/wiki/Cross_entropy
$$H(p,q)=-\sum_{x\in \mathcal{X}} p(x)\log q(x)$$
It can be written as $$H(p,q) = H(p) + D_{KL} (p||q)$...
- 411
0
votes
0
answers
18
views
Using entropy balancing to achieve general population representativeness in a survey with multiple conditions
We are currently planning a survey in which we ask people about their attitudes towards four different groups of people. Each participant is only asked about their attitudes towards one of the four ...
0
votes
0
answers
23
views
When to consider classification error and entropy in LCA models?
I have good fit statistics in terms of G^2 (bootstrap) and bivariate residuals. However, all the models with good absolute fit statistics have poor classification error (.4) and entropy (.4-.5). If ...
0
votes
0
answers
91
views
What does "tensorization" mean in the entropy tensorization inequality?
I am reading high dimensional statistics written by Wainwright. In chapter 3.1.4 tensorization of entropy is used to extend the entropy bound for univariate functions to multivariate cases. As far as ...
- 1
1
vote
1
answer
48
views
Covering transformation for Lipschitz functions: can we prove that $\mathcal{H}(r,\mathcal {G'},||.||_n)\asymp \mathcal{H}(r/C_n,\mathcal {A'},\rho)$?
Let assume that $\mathcal{A'}\subset \mathcal {A}$ (is a vector space) and $\mathcal {G'}:=\{g_\theta : \ \theta\in \mathcal{A'}\}$ (space of functions parameterized by $\mathcal{A'}$) and a metric $\...
- 31
0
votes
0
answers
60
views
Cluster evaluation - evaluation separation, stability and agreement - best indices?
In the literature many indices exist that aim to quantify the level of separation of the clustering outcome: Silhouette, Davis-Bouldin, Calinski-Harabasz, etc.
To evaluate the stability of a ...
0
votes
0
answers
40
views
Calculating entropy-based indices based on a frequency cross-tabulation of objects in Python
I was wondering whether there exist a Python solution for calculating Mutual Information, Completeness, V-Measure, etc., based on a pair-confusion matrix.
MI defined as H(C_0) + H(C_1) - H(C_O&C_1)...
- 61
1
vote
4
answers
75
views
Mutual Information larger than smaller one of both entropies?
Lot's of questions and good answers on mutual information e.g. here out there and I think I get the concept which is also nicely explained on wikipedia. But I'm nervous that R's infotheo package is ...
- 21
2
votes
1
answer
89
views
ERGM: How come `ttriple` is degenerate but `transitiveties` is not?
This title is an effort to elicit a simplified explanation as to how come the transitiveties term causes less numerical instability than ...
- 165
0
votes
0
answers
37
views
Can transfer entropy be calculated after PCA?
In a problem involving multivariate time series with many dimensions, does it make sense to apply principal component analysis to perform feature reduction before applying transfer entropy? That is, ...
- 11
0
votes
0
answers
39
views
How can I adjust cumulative entropy of moose observations?
I have downloaded the dates of every moose (Alces alces) observation worldwide on iNaturalist. This amounts to about $2 \cdot 10^4$ observations at present. I excluded one observation that was either ...
- 6,135
1
vote
1
answer
61
views
Is "Information" somehow Related to "Variance"?
Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
- 7,028
2
votes
1
answer
66
views
Do Mixture Models "Defy" Entropy?
Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
- 7,028