Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
653
questions
1
vote
2
answers
47
views
Are there better measures of entropy
Related question here
I am trying to measure the uniformity of multimodal distributions and am looking into using entropy.
I would like a measure of entropy that is higher for the first distribution ...
- 121
0
votes
0
answers
43
views
How to sample distributions far from uniform?
I need to obtain an isotropic sample of discrete distributions over $d$ outcomes which are far from uniform in some measure.
For instance, using linear entropy as the complexity measure, we can ...
- 5,699
0
votes
0
answers
16
views
Is maximal spectral entropy of residuals a poor loss function because phase information is lost?
Suppose I define a custom loss function SpectralEntropy as follows:
...
- 4,590
0
votes
0
answers
17
views
Surprisal in rankings
I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race.
Intuitively, in a race with 100 people, ...
- 1,063
2
votes
0
answers
27
views
How much do we learn about a random subset? [closed]
Suppose we sample the following two random variables, for some large integer $n$:
Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
- 21
1
vote
0
answers
34
views
conditional entropy of finite mixture model
I am trying to understand the conditional entropy of the finite mixture model given in this paper about regularized EM algorithm.
#################
On page 3 of the paper: in the finite mixture model, ...
- 63
2
votes
0
answers
24
views
How can entropy help in machine learning classification? [closed]
I cannot fully grasp why entropy is so important in machine learning classification. I understand the usage in decision trees but I don't see the importance elsewhere. I mean entropy somehow comes up ...
- 61
0
votes
0
answers
16
views
How can Mutual Information be lower than Adjusted Mutual Information?
What would be the statistical reasoning that MI < AMI in cluster evaluation? Wouldn't you expect AMI to always be slightly (or a lot) lower than MI?
0
votes
0
answers
24
views
differential entropy rate for stationary processes
the differential entropy rate of a stochastic process $X = (X_i)_{i \in \mathbb{N}}$ is defined as $$h(X) = \lim_{n \to \infty} \frac{1}{n} h(X_1,\ldots,X_n)$$,
where $h(X_1,\ldots,X_n)$ is the joint ...
- 1
0
votes
0
answers
11
views
Estimeate Entropy of Sequence
Suppose I have a sequence of N symbols. There are 100 symbols. The probability of each symbol is quite small, though they're not necessarily uniform.
s1 - s2 - s5 - s8 - s3 - sk ....
Each symbol is an ...
- 1
1
vote
0
answers
19
views
Intuition of multivariate conditional entropy
This might be a dumb question, but I'm serious.
Per the wiki page about conditional entropy: quantifies the amount of information needed to describe the outcome of a random variable $Y$ given that the ...
- 11
0
votes
0
answers
11
views
How to interpret the Adjusted Rand Index
Given a pair-confusion matrix where:
...
1
vote
1
answer
40
views
Confidence Interval of entropy for a discrete distribution
The following table is a set of ordinal data from a survey I have conducted (one of many).
$$\begin{array}{c|c|c|}
\text{Grading}& \text{Count} & \text{Frequency} \\ \hline
\text{1} & 5 &...
- 485
1
vote
1
answer
40
views
Why is it called the cross-entropy of q relative to p, not p relative to q?
I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy
Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
- 381
0
votes
0
answers
21
views
In the DT algorithm, why not search through all possible splits instead of first choosing the feature required for splitting?
As far as I understand the Decision Trees (DT - CART), the first step is to choose a feature that maximizes IG (information gain), meaning the entropy in the current node - the entropy after splitting ...
- 35
2
votes
1
answer
44
views
Cross entropy of a random variable or a probability distribution function? [duplicate]
I'm looking into the wikipedia page of cross entropy. https://en.wikipedia.org/wiki/Cross_entropy
$$H(p,q)=-\sum_{x\in \mathcal{X}} p(x)\log q(x)$$
It can be written as $$H(p,q) = H(p) + D_{KL} (p||q)$...
- 381
0
votes
0
answers
12
views
Using entropy balancing to achieve general population representativeness in a survey with multiple conditions
We are currently planning a survey in which we ask people about their attitudes towards four different groups of people. Each participant is only asked about their attitudes towards one of the four ...
0
votes
0
answers
11
views
When to consider classification error and entropy in LCA models?
I have good fit statistics in terms of G^2 (bootstrap) and bivariate residuals. However, all the models with good absolute fit statistics have poor classification error (.4) and entropy (.4-.5). If ...
0
votes
0
answers
24
views
What does "tensorization" mean in the entropy tensorization inequality?
I am reading high dimensional statistics written by Wainwright. In chapter 3.1.4 tensorization of entropy is used to extend the entropy bound for univariate functions to multivariate cases. As far as ...
- 1
1
vote
1
answer
42
views
Covering transformation for Lipschitz functions: can we prove that $\mathcal{H}(r,\mathcal {G'},||.||_n)\asymp \mathcal{H}(r/C_n,\mathcal {A'},\rho)$?
Let assume that $\mathcal{A'}\subset \mathcal {A}$ (is a vector space) and $\mathcal {G'}:=\{g_\theta : \ \theta\in \mathcal{A'}\}$ (space of functions parameterized by $\mathcal{A'}$) and a metric $\...
- 31
0
votes
0
answers
21
views
Cluster evaluation - evaluation separation, stability and agreement - best indices?
In the literature many indices exist that aim to quantify the level of separation of the clustering outcome: Silhouette, Davis-Bouldin, Calinski-Harabasz, etc.
To evaluate the stability of a ...
0
votes
0
answers
12
views
Calculating entropy-based indices based on a frequency cross-tabulation of objects in Python
I was wondering whether there exist a Python solution for calculating Mutual Information, Completeness, V-Measure, etc., based on a pair-confusion matrix.
MI defined as H(C_0) + H(C_1) - H(C_O&C_1)...
- 61
1
vote
4
answers
52
views
Mutual Information larger than smaller one of both entropies?
Lot's of questions and good answers on mutual information e.g. here out there and I think I get the concept which is also nicely explained on wikipedia. But I'm nervous that R's infotheo package is ...
- 21
1
vote
1
answer
37
views
ERGM: How come `ttriple` is degenerate but `transitiveties` is not?
This title is an effort to elicit a simplified explanation as to how come the transitiveties term causes less numerical instability than ...
- 145
0
votes
0
answers
20
views
Can transfer entropy be calculated after PCA?
In a problem involving multivariate time series with many dimensions, does it make sense to apply principal component analysis to perform feature reduction before applying transfer entropy? That is, ...
- 11
0
votes
0
answers
34
views
How can I adjust cumulative entropy of moose observations?
I have downloaded the dates of every moose (Alces alces) observation worldwide on iNaturalist. This amounts to about $2 \cdot 10^4$ observations at present. I excluded one observation that was either ...
- 4,590
1
vote
1
answer
42
views
Is "Information" somehow Related to "Variance"?
Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
- 6,048
1
vote
0
answers
40
views
Do Mixture Models "Defy" Entropy?
Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
- 6,048
1
vote
0
answers
63
views
Entropy of Gaussian mixture when variance of one component gets larger
I want to prove or disprove the following relation of differential entropies:
Conjecture: $\displaystyle h(f) \le h(g)$
where $f, g$ the density functions of Gaussian mixture models with equal ...
- 1,159
0
votes
0
answers
56
views
Negative transfer entropy
By definition, transfer entropy cannot be negative. However, using the Kraskov estimator, negative values can be obtained. In general, should we take precautions to avoid getting negative values? How ...
- 11
1
vote
1
answer
116
views
Empirical error in Kullback-Leibler KL divergence estimation
In computing the Kullback-Leibler KL divergence $D(P\|Q)$ from an empirical data, it may happen that $Q(x)=0<P(x)$ at some sample point $x$ due to data error and $D(P\|Q)=\infty$. What are some ...
- 905
1
vote
1
answer
60
views
How can we use shannon entropy to discriminate between two similar probability distribution function?
I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
- 11
5
votes
3
answers
742
views
How to apply the formula for Shannon entropy to a 4-sided die?
I have come across calculating entropy, via the formula:
\begin{equation}
Entropy(p) = -\sum_{i=1} ^{N}p_i \log_2(p_i)
\end{equation}
Referring to this formula, how would I calculate the entropy of a ...
1
vote
0
answers
73
views
How to compute transfer entropy between different length time-series?
Is there a way to calculate transfer entropy between two time series having different lengths?
Given two time series:
$x=(x_1,x_2,\dotsc,x_n)$
$y=(y_1,y_2,\dotsc,y_m)$,
where $n \ne m$.
Is there a way ...
- 11
1
vote
0
answers
36
views
Understanding continuous variable entropy
I am struggling to understand continuous variable entropies and mutual informations for 2 or more variables. Consider 2D normal distribution $\rho(x,y)$ defined as follows
$$X\sim\mathcal{N}(0, \...
- 1,609
1
vote
1
answer
138
views
Does increasing the variance increase the value of a function?
Let $V=\sum_{i=1}^{k} a_iX_i$ where $X_i's$ are IID $\sim Bern(q)$ and $V$ with $\sum a_i=k$. Note that $a_i$'s are non-negative integers.
I have a function $f$ as given below :
$$
f= \max_{q} h\...
- 185
0
votes
0
answers
44
views
Multivariate transfer entropy
I have a set of time series to treat as sources and a time series to treat as destination. From the definition of multivariate transfer entropy, it seems to me that it can only be defined on two ...
- 11
0
votes
1
answer
40
views
How to understand the information and entropy?
Let $X$ is a binary variable and $X = \{A, B\}$. The pdf of $X$ is $p(X = A) = .3 \ \ \ p(X=B) = .7$
So it's easy to calculate the ...
- 1
1
vote
0
answers
54
views
How to understand embedding dimension in permutation entropy when checking predictabilty in electric load data?
I am predicting electric load data with different deep learning models and I am trying to define the predictability of the data. So far I came across the permutation entropy (PE) as a measurement for ...
- 121
0
votes
0
answers
29
views
What is the interpretation of negative differential entropy? [duplicate]
I'm currently trying to understand the meaning of negative entropy. With the given equation:
$$
H_{cont}\ (X) = -\int_{-\infty}^{+\infty} p(x) \centerdot ln(p(x)) \ dx
$$
Here, the The_Sympathizer ...
1
vote
1
answer
185
views
Why does mutual information use KL divergence?
Mutual information between a pair of random variables $X,Y$ having joint distribution $P_{(X,Y)}$ and marginal distributions $P_X,P_Y$ respectively is defined as
$$I(X,Y)\equiv D_{\text{KL}}(P_{(X,Y)}\...
- 353
3
votes
1
answer
46
views
What is the proper name for the *other* type of "conditional entropy"?
Suppose we have two random variables $X$ and $Y$ (for simplicity of exposition I will take these to be discrete). If we were to condition our entire analysis on the event $X=x$ and then ask for the ...
- 102k
4
votes
1
answer
240
views
Entropy Conditioned on A Certain Value
Consider this example: I have a box with 10 balls in it, 9 red and 1 blue. I take a ball randomly. Let's call the color $C$. If $C$ is red, I shout the number zero. If $C$ is blue, I roll a fair die ...
- 161
0
votes
0
answers
133
views
How can I quantify the skewness, kurtosis, entropy when all of values of a list is 0?
Background
Let's think, there is a list of values which presents activity of a person for several hours. That person did not have any movement in those hours. Therefore, all the values are 0.
Then, ...
- 294
0
votes
0
answers
23
views
What statistic/algorithm does the decision tree use to partition/cluster the data per depth?
How does a decision tree calculate that a break exists at 1.8 on the root, a break exists at 2.1 and 1.2 on the second depth?
I know Gini and Entropy are used to calculate which feature to partition ...
- 782
2
votes
2
answers
196
views
Minimizing KL-divergence and log-likelihood for generative machine learning models
I am reading a paper on quantum ML: A generative modeling approach for benchmarking and training shallow quantum circuits, where it is claimed that:
Following a standard approach from generative ...
- 133
0
votes
0
answers
16
views
1
vote
0
answers
105
views
Differential entropy of a multivariate log-normal distribution
Let $Y$ be a multivariate log-normal random variable. As such, $\ln(Y)$ follows a multivariate normal distribution, which I denote by $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$. I already ...
- 111
0
votes
0
answers
15
views
How to decompose the autocorrelation function of a variable, resulted from interactions in complex systems?
The output variable (X) of a complex system undergoes numerous interactions between system components. These interactions will impart a distinct signature to the autocorrelation function of the output ...
1
vote
0
answers
26
views
RTransferEntropy - Discrete data
I'm trying to use the RTransferEntropy package to compute TE for my data.
I want to understand how discrete series are handled by functions like the ...
- 195