Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
632
questions
0
votes
0
answers
31
views
How can I adjust cumulative entropy of moose observations?
I have downloaded the dates of every moose (Alces alces) observation worldwide on iNaturalist. This amounts to about $2 \cdot 10^4$ observations at present. I excluded one observation that was either ...
- 3,403
1
vote
1
answer
31
views
Is "Information" somehow Related to "Variance"?
Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
- 5,738
1
vote
0
answers
32
views
Do Mixture Models "Defy" Entropy?
Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
- 5,738
1
vote
0
answers
36
views
Entropy of Gaussian mixture when variance of one component gets larger
I want to prove or disprove the following relation of differential entropies:
Conjecture: $\displaystyle h(f) \le h(g)$
where $f, g$ the density functions of Gaussian mixture models with equal ...
- 1,099
0
votes
0
answers
29
views
Negative transfer entropy
By definition, transfer entropy cannot be negative. However, using the Kraskov estimator, negative values can be obtained. In general, should we take precautions to avoid getting negative values? How ...
- 11
0
votes
1
answer
34
views
Empirical error in Kullback-Leibler KL divergence estimation
In computing the Kullback-Leibler KL divergence $D(P\|Q)$ from an empirical data, it may happen that $Q(x)=0<P(x)$ at some sample point $x$ due to data error and $D(P\|Q)=\infty$. What are some ...
- 875
1
vote
1
answer
35
views
How can we use shannon entropy to discriminate between two similar probability distribution function?
I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
- 11
5
votes
3
answers
662
views
How to apply the formula for Shannon entropy to a 4-sided die?
I have come across calculating entropy, via the formula:
\begin{equation}
Entropy(p) = -\sum_{i=1} ^{N}p_i \log_2(p_i)
\end{equation}
Referring to this formula, how would I calculate the entropy of a ...
1
vote
0
answers
54
views
How to compute transfer entropy between different length time-series?
Is there a way to calculate transfer entropy between two time series having different lengths?
Given two time series:
$x=(x_1,x_2,\dotsc,x_n)$
$y=(y_1,y_2,\dotsc,y_m)$,
where $n \ne m$.
Is there a way ...
- 11
1
vote
0
answers
17
views
Understanding continuous variable entropy
I am struggling to understand continuous variable entropies and mutual informations for 2 or more variables. Consider 2D normal distribution $\rho(x,y)$ defined as follows
$$X\sim\mathcal{N}(0, \...
- 1,509
1
vote
1
answer
127
views
Does increasing the variance increase the value of a function?
Let $V=\sum_{i=1}^{k} a_iX_i$ where $X_i's$ are IID $\sim Bern(q)$ and $V$ with $\sum a_i=k$. Note that $a_i$'s are non-negative integers.
I have a function $f$ as given below :
$$
f= \max_{q} h\...
- 185
0
votes
0
answers
30
views
Multivariate transfer entropy
I have a set of time series to treat as sources and a time series to treat as destination. From the definition of multivariate transfer entropy, it seems to me that it can only be defined on two ...
- 11
0
votes
1
answer
27
views
How to understand the information and entropy?
Let $X$ is a binary variable and $X = \{A, B\}$. The pdf of $X$ is $p(X = A) = .3 \ \ \ p(X=B) = .7$
So it's easy to calculate the ...
- 1
0
votes
0
answers
13
views
How to understand embedding dimension in permutation entropy when checking predictabilty in electric load data?
I am predicting electric load data with different deep learning models and I am trying to define the predictability of the data. So far I came across the permutation entropy (PE) as a measurement for ...
- 111
0
votes
0
answers
27
views
What is the interpretation of negative differential entropy? [duplicate]
I'm currently trying to understand the meaning of negative entropy. With the given equation:
$$
H_{cont}\ (X) = -\int_{-\infty}^{+\infty} p(x) \centerdot ln(p(x)) \ dx
$$
Here, the The_Sympathizer ...
1
vote
1
answer
87
views
Why does mutual information use KL divergence?
Mutual information between a pair of random variables $X,Y$ having joint distribution $P_{(X,Y)}$ and marginal distributions $P_X,P_Y$ respectively is defined as
$$I(X,Y)\equiv D_{\text{KL}}(P_{(X,Y)}\...
- 353
2
votes
1
answer
30
views
What is the proper name for the *other* type of "conditional entropy"?
Suppose we have two random variables $X$ and $Y$ (for simplicity of exposition I will take these to be discrete). If we were to condition our entire analysis on the event $X=x$ and then ask for the ...
- 94.4k
4
votes
1
answer
68
views
Entropy Conditioned on A Certain Value
Consider this example: I have a box with 10 balls in it, 9 red and 1 blue. I take a ball randomly. Let's call the color $C$. If $C$ is red, I shout the number zero. If $C$ is blue, I roll a fair die ...
- 161
0
votes
0
answers
98
views
How can I quantify the skewness, kurtosis, entropy when all of values of a list is 0?
Background
Let's think, there is a list of values which presents activity of a person for several hours. That person did not have any movement in those hours. Therefore, all the values are 0.
Then, ...
- 284
0
votes
0
answers
11
views
What statistic/algorithm does the decision tree use to partition/cluster the data per depth?
How does a decision tree calculate that a break exists at 1.8 on the root, a break exists at 2.1 and 1.2 on the second depth?
I know Gini and Entropy are used to calculate which feature to partition ...
- 224
2
votes
2
answers
107
views
Minimizing KL-divergence and log-likelihood for generative machine learning models
I am reading a paper on quantum ML: A generative modeling approach for benchmarking and training shallow quantum circuits, where it is claimed that:
Following a standard approach from generative ...
- 121
0
votes
0
answers
11
views
1
vote
0
answers
58
views
Differential entropy of a multivariate log-normal distribution
Let $Y$ be a multivariate log-normal random variable. As such, $\ln(Y)$ follows a multivariate normal distribution, which I denote by $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$. I already ...
- 111
0
votes
0
answers
15
views
How to decompose the autocorrelation function of a variable, resulted from interactions in complex systems?
The output variable (X) of a complex system undergoes numerous interactions between system components. These interactions will impart a distinct signature to the autocorrelation function of the output ...
1
vote
0
answers
23
views
RTransferEntropy - Discrete data
I'm trying to use the RTransferEntropy package to compute TE for my data.
I want to understand how discrete series are handled by functions like the ...
- 195
1
vote
0
answers
48
views
Estimating conditional entropy of a series
I have a time series $Y$ (continuous values) and I want to estimate its conditional entropy such that $H(y_{t+1}|y_t)$ represents the conditional entropy of $Y$ at time $t+1$ given its value at time $...
- 195
0
votes
0
answers
31
views
Definition of exponential family and entropy
I'm reading Graphical Models, Exponential Families, and Variational Inference [pdf] by Wainwright and Jordan.
They define (p. 39) an exponential family by its derivative
$$ p_\theta(x) = \exp \{ \...
- 351
0
votes
0
answers
10
views
Relating difference in distribution of dataset to information gain
I wonder if there are ways to relate difference in distribution of dataset to information gain. For example, I train a model on dataset D_1 and obtain a trained model M_1 and train a model on dataset ...
- 223
3
votes
4
answers
135
views
How to measure if a categorical distribution is concentrated in a very few bins
I'm trying to think of a way to measure that a categorical distribution of any size is concentrated in only a few bins, so not uniform. The best way I can think of is checking entropy, but that's kind ...
- 640
0
votes
0
answers
13
views
Expressions for entropy of an observed set of points (continuous variables) [duplicate]
The entropy of a discrete variable can be defined as
$$H(x) = - \sum_{i=1}^n p(x) \log p(x) $$
or for continuous distributions and a density $f(x)$ we could compute
$$h[f] = \int_{\mathbb{X}} f(x) \...
- 46.4k
0
votes
0
answers
39
views
Do we need to normalize all the variables before calculating mutual information for variables using sklearn?
In calculating the mutual information on sklearn using either mutual_info_classif or mutual_info_regression, the underlying algorithm seems to use KNN to derive the mutual information for the variable....
- 135
1
vote
1
answer
80
views
Why KL divergence close to zero when Q close to P?
I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence:
H(P, Q) = H(P) + D_KL(P||Q)
Minimizing Cross-entropy means minimizing ...
0
votes
0
answers
26
views
Relating a scoring mechanism to entropy and homogeneity
Context
I'm working on some clustering algorithms, and my boss wants me to grade them. My go-to method has been to measure homogeneity, completeness, and v-measure.
However, since we are presenting ...
- 309
0
votes
0
answers
18
views
Why is Surprisal the number of bits needed to efficiently encode a class? [duplicate]
In one of the Machine Learning lectures where the topic was Entropy and information, the professor explained that $Surprisal = s_y(c) = -log p(y=c)$ where $p(y=c)$ is the probability of a class given ...
0
votes
1
answer
25
views
Metrics to compare diversity of groups on multiple characteristics
When I search for 'diversity metrics' it seems like most work has focused on ecological diversity, and I most frequently see the Simpson diversity index. This is equivalent to the economics metric, ...
- 1,170
2
votes
1
answer
50
views
Proof of mutual information estimate from Kraskov et al. (2004)
I'm reading the Kraskov et al. (2004) paper about estimating mutual information using the Kozachenko-Leonenko estimate for Shannon entropies. I'm trying to understand the last step in Eq. 17, which ...
- 31
0
votes
0
answers
26
views
Establishing a goodness of fit test based on entropy
The Shannon entropy of a density function $f$ is
$$
\begin{split}
H(f)&=-\int_{-\infty}^{\infty} f(x)\log f(x) dx,\\
&=\int_0^1\log\left(\frac{d}{dp}F^{-1}(p)\right)dp.
\end{split}
$$
...
- 1
1
vote
0
answers
48
views
Why to calculate $\mathbf{weighted}$ average of the leaf node impurities in decision trees? Why not to just add entropies up without weights?
In decision trees why do we calculate weighted average of entropies of each leaf when we calculate the entropy of target variable given some feature? The question is: "Why is it weighted average? ...
- 481
3
votes
1
answer
81
views
Highest Entropy Distribution, on $[0,\infty)$ given Mean, Variance, and goes to $p(0) = 0$?
I am dealing with temperature measurements, and normally we assume the probability of getting a measurement $t_i$ with a certain uncertainty $\sigma_t$ given the model ('true' value) $M(x_i)$ (where $...
- 143
0
votes
0
answers
9
views
Approximating the pdf at a sampled point using k-nearest neighbours?
I'm reading a (more or less classical) paper on Nearest-neighbour approximations to Entropy. At some point in the paper, given $N$ samples $X_1,...X_N$ from a $p$-dimensional random variable $X$ with ...
- 239
1
vote
0
answers
104
views
KL divergence estimates over binary classification data
I have a dataset $D = (x_i, y_i)_{i=1}^n$ where $x_i\in \Bbb R^d$ and $y_i\in\{0, 1\}$. Suppose that $y\sim\mathrm{Bernoulli}(p(x))$ for some probability function $p:\Bbb R^d \to [0,1]$ and I would ...
- 343
2
votes
0
answers
232
views
Are binary splits always better than multi-way splits in decision trees?
I'm trying to devise a decision tree for classification with multi-way split at an attribute but even though calculating the entropy for a multi-way split gives better information gain than a binary ...
2
votes
1
answer
112
views
Binary classification but learning the true probability
I have a dataset $D = (x_i, y_i)_{i=1}^n$ where $x_i\in \Bbb R^d$ and $y_i\in\{0, 1\}$. Suppose that $y\sim\mathrm{Bernoulli}(p(x))$ for some probability function $p:\Bbb R^d \to [0,1]$ and I would ...
- 343
0
votes
0
answers
38
views
Entropy of Poisson random variable via exponential family identity
Exercise 8.1 of Probabilistic Graphical Models asks the reader to use the identity
$$H_{P_{\theta}}(X)= \ln Z(\theta) - \langle E_{P_{\theta}}[\tau(X)], t(\theta)\rangle$$
to compute the entropy $H$ ...
0
votes
0
answers
20
views
Differential entropy of a kernel density estimator
I have a set of (~500k) observations that I use to fit a parametric model (univariate three-parameter lognormal). To have an idea of the goodness of fit, I want to compute the Jensen-Shannon distance ...
- 183
0
votes
0
answers
22
views
Determining history length in Transfer Entropy
Supposing I have a time series observation corresponding to 5 sensors, and I wish to find the causation(Transfer entropy) between all pairs of nodes, how do I go about choosing the history length ...
- 195
2
votes
1
answer
76
views
Measurement of regularity of events
I am dealing with events generated by different processes. My data includes time stamps when the events occur, so I am trying to differentiate the processes in regular or irregular categories or ...
- 121
0
votes
0
answers
7
views
How to calculate the uncertainty of a variable explained by other variables
I'm working on implementing a method from a paper for solving the FRn-k problem for a specific type of biological data.
[FRn−k]: Given an initial set of n features, find the subset with k<n
...
- 111
2
votes
1
answer
49
views
Differential Entropy with negative support
Differential Entropy Estimation
Hello everyone, I am estimating differential entropy from a continuous distribution of a property of materials. The distributions on the picture are Gaussian Mixtures. ...
2
votes
1
answer
137
views
Is there a name for this in statisitics? [closed]
Edit: Ignore this Most of this doesn't make sense and is beyond edits but its too late to delete. If I make too many edits I could be kicked out of this sub like I was in math stack exchange.
I don't ...
- 129