Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
31 views

How can I adjust cumulative entropy of moose observations?

I have downloaded the dates of every moose (Alces alces) observation worldwide on iNaturalist. This amounts to about $2 \cdot 10^4$ observations at present. I excluded one observation that was either ...
user avatar
  • 3,403
1 vote
1 answer
31 views

Is "Information" somehow Related to "Variance"?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
user avatar
  • 5,738
1 vote
0 answers
32 views

Do Mixture Models "Defy" Entropy?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
user avatar
  • 5,738
1 vote
0 answers
36 views

Entropy of Gaussian mixture when variance of one component gets larger

I want to prove or disprove the following relation of differential entropies: Conjecture: $\displaystyle h(f) \le h(g)$ where $f, g$ the density functions of Gaussian mixture models with equal ...
user avatar
  • 1,099
0 votes
0 answers
29 views

Negative transfer entropy

By definition, transfer entropy cannot be negative. However, using the Kraskov estimator, negative values can be obtained. In general, should we take precautions to avoid getting negative values? How ...
user avatar
  • 11
0 votes
1 answer
34 views

Empirical error in Kullback-Leibler KL divergence estimation

In computing the Kullback-Leibler KL divergence $D(P\|Q)$ from an empirical data, it may happen that $Q(x)=0<P(x)$ at some sample point $x$ due to data error and $D(P\|Q)=\infty$. What are some ...
user avatar
  • 875
1 vote
1 answer
35 views

How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
user avatar
  • 11
5 votes
3 answers
662 views

How to apply the formula for Shannon entropy to a 4-sided die?

I have come across calculating entropy, via the formula: \begin{equation} Entropy(p) = -\sum_{i=1} ^{N}p_i \log_2(p_i) \end{equation} Referring to this formula, how would I calculate the entropy of a ...
user avatar
1 vote
0 answers
54 views

How to compute transfer entropy between different length time-series?

Is there a way to calculate transfer entropy between two time series having different lengths? Given two time series: $x=(x_1,x_2,\dotsc,x_n)$ $y=(y_1,y_2,\dotsc,y_m)$, where $n \ne m$. Is there a way ...
user avatar
  • 11
1 vote
0 answers
17 views

Understanding continuous variable entropy

I am struggling to understand continuous variable entropies and mutual informations for 2 or more variables. Consider 2D normal distribution $\rho(x,y)$ defined as follows $$X\sim\mathcal{N}(0, \...
user avatar
1 vote
1 answer
127 views

Does increasing the variance increase the value of a function?

Let $V=\sum_{i=1}^{k} a_iX_i$ where $X_i's$ are IID $\sim Bern(q)$ and $V$ with $\sum a_i=k$. Note that $a_i$'s are non-negative integers. I have a function $f$ as given below : $$ f= \max_{q} h\...
user avatar
  • 185
0 votes
0 answers
30 views

Multivariate transfer entropy

I have a set of time series to treat as sources and a time series to treat as destination. From the definition of multivariate transfer entropy, it seems to me that it can only be defined on two ...
user avatar
  • 11
0 votes
1 answer
27 views

How to understand the information and entropy?

Let $X$ is a binary variable and $X = \{A, B\}$. The pdf of $X$ is $p(X = A) = .3 \ \ \ p(X=B) = .7$ So it's easy to calculate the ...
user avatar
0 votes
0 answers
13 views

How to understand embedding dimension in permutation entropy when checking predictabilty in electric load data?

I am predicting electric load data with different deep learning models and I am trying to define the predictability of the data. So far I came across the permutation entropy (PE) as a measurement for ...
user avatar
  • 111
0 votes
0 answers
27 views

What is the interpretation of negative differential entropy? [duplicate]

I'm currently trying to understand the meaning of negative entropy. With the given equation: $$ H_{cont}\ (X) = -\int_{-\infty}^{+\infty} p(x) \centerdot ln(p(x)) \ dx $$ Here, the The_Sympathizer ...
user avatar
1 vote
1 answer
87 views

Why does mutual information use KL divergence?

Mutual information between a pair of random variables $X,Y$ having joint distribution $P_{(X,Y)}$ and marginal distributions $P_X,P_Y$ respectively is defined as $$I(X,Y)\equiv D_{\text{KL}}(P_{(X,Y)}\...
user avatar
2 votes
1 answer
30 views

What is the proper name for the *other* type of "conditional entropy"?

Suppose we have two random variables $X$ and $Y$ (for simplicity of exposition I will take these to be discrete). If we were to condition our entire analysis on the event $X=x$ and then ask for the ...
user avatar
  • 94.4k
4 votes
1 answer
68 views

Entropy Conditioned on A Certain Value

Consider this example: I have a box with 10 balls in it, 9 red and 1 blue. I take a ball randomly. Let's call the color $C$. If $C$ is red, I shout the number zero. If $C$ is blue, I roll a fair die ...
user avatar
  • 161
0 votes
0 answers
98 views

How can I quantify the skewness, kurtosis, entropy when all of values of a list is 0?

Background Let's think, there is a list of values which presents activity of a person for several hours. That person did not have any movement in those hours. Therefore, all the values are 0. Then, ...
user avatar
0 votes
0 answers
11 views

What statistic/algorithm does the decision tree use to partition/cluster the data per depth?

How does a decision tree calculate that a break exists at 1.8 on the root, a break exists at 2.1 and 1.2 on the second depth? I know Gini and Entropy are used to calculate which feature to partition ...
user avatar
  • 224
2 votes
2 answers
107 views

Minimizing KL-divergence and log-likelihood for generative machine learning models

I am reading a paper on quantum ML: A generative modeling approach for benchmarking and training shallow quantum circuits, where it is claimed that: Following a standard approach from generative ...
user avatar
  • 121
0 votes
0 answers
11 views

Computing Minimum Description Length

I have two integer arrays, x and y: ...
user avatar
  • 474
1 vote
0 answers
58 views

Differential entropy of a multivariate log-normal distribution

Let $Y$ be a multivariate log-normal random variable. As such, $\ln(Y)$ follows a multivariate normal distribution, which I denote by $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$. I already ...
user avatar
  • 111
0 votes
0 answers
15 views

How to decompose the autocorrelation function of a variable, resulted from interactions in complex systems?

The output variable (X) of a complex system undergoes numerous interactions between system components. These interactions will impart a distinct signature to the autocorrelation function of the output ...
user avatar
1 vote
0 answers
23 views

RTransferEntropy - Discrete data

I'm trying to use the RTransferEntropy package to compute TE for my data. I want to understand how discrete series are handled by functions like the ...
user avatar
1 vote
0 answers
48 views

Estimating conditional entropy of a series

I have a time series $Y$ (continuous values) and I want to estimate its conditional entropy such that $H(y_{t+1}|y_t)$ represents the conditional entropy of $Y$ at time $t+1$ given its value at time $...
user avatar
0 votes
0 answers
31 views

Definition of exponential family and entropy

I'm reading Graphical Models, Exponential Families, and Variational Inference [pdf] by Wainwright and Jordan. They define (p. 39) an exponential family by its derivative $$ p_\theta(x) = \exp \{ \...
user avatar
0 votes
0 answers
10 views

Relating difference in distribution of dataset to information gain

I wonder if there are ways to relate difference in distribution of dataset to information gain. For example, I train a model on dataset D_1 and obtain a trained model M_1 and train a model on dataset ...
user avatar
  • 223
3 votes
4 answers
135 views

How to measure if a categorical distribution is concentrated in a very few bins

I'm trying to think of a way to measure that a categorical distribution of any size is concentrated in only a few bins, so not uniform. The best way I can think of is checking entropy, but that's kind ...
user avatar
0 votes
0 answers
13 views

Expressions for entropy of an observed set of points (continuous variables) [duplicate]

The entropy of a discrete variable can be defined as $$H(x) = - \sum_{i=1}^n p(x) \log p(x) $$ or for continuous distributions and a density $f(x)$ we could compute $$h[f] = \int_{\mathbb{X}} f(x) \...
user avatar
0 votes
0 answers
39 views

Do we need to normalize all the variables before calculating mutual information for variables using sklearn?

In calculating the mutual information on sklearn using either mutual_info_classif or mutual_info_regression, the underlying algorithm seems to use KNN to derive the mutual information for the variable....
user avatar
  • 135
1 vote
1 answer
80 views

Why KL divergence close to zero when Q close to P?

I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence: H(P, Q) = H(P) + D_KL(P||Q) Minimizing Cross-entropy means minimizing ...
user avatar
0 votes
0 answers
26 views

Relating a scoring mechanism to entropy and homogeneity

Context I'm working on some clustering algorithms, and my boss wants me to grade them. My go-to method has been to measure homogeneity, completeness, and v-measure. However, since we are presenting ...
user avatar
0 votes
0 answers
18 views

Why is Surprisal the number of bits needed to efficiently encode a class? [duplicate]

In one of the Machine Learning lectures where the topic was Entropy and information, the professor explained that $Surprisal = s_y(c) = -log p(y=c)$ where $p(y=c)$ is the probability of a class given ...
user avatar
0 votes
1 answer
25 views

Metrics to compare diversity of groups on multiple characteristics

When I search for 'diversity metrics' it seems like most work has focused on ecological diversity, and I most frequently see the Simpson diversity index. This is equivalent to the economics metric, ...
user avatar
2 votes
1 answer
50 views

Proof of mutual information estimate from Kraskov et al. (2004)

I'm reading the Kraskov et al. (2004) paper about estimating mutual information using the Kozachenko-Leonenko estimate for Shannon entropies. I'm trying to understand the last step in Eq. 17, which ...
user avatar
0 votes
0 answers
26 views

Establishing a goodness of fit test based on entropy

The Shannon entropy of a density function $f$ is $$ \begin{split} H(f)&=-\int_{-\infty}^{\infty} f(x)\log f(x) dx,\\ &=\int_0^1\log\left(\frac{d}{dp}F^{-1}(p)\right)dp. \end{split} $$ ...
user avatar
  • 1
1 vote
0 answers
48 views

Why to calculate $\mathbf{weighted}$ average of the leaf node impurities in decision trees? Why not to just add entropies up without weights?

In decision trees why do we calculate weighted average of entropies of each leaf when we calculate the entropy of target variable given some feature? The question is: "Why is it weighted average? ...
user avatar
  • 481
3 votes
1 answer
81 views

Highest Entropy Distribution, on $[0,\infty)$ given Mean, Variance, and goes to $p(0) = 0$?

I am dealing with temperature measurements, and normally we assume the probability of getting a measurement $t_i$ with a certain uncertainty $\sigma_t$ given the model ('true' value) $M(x_i)$ (where $...
user avatar
  • 143
0 votes
0 answers
9 views

Approximating the pdf at a sampled point using k-nearest neighbours?

I'm reading a (more or less classical) paper on Nearest-neighbour approximations to Entropy. At some point in the paper, given $N$ samples $X_1,...X_N$ from a $p$-dimensional random variable $X$ with ...
user avatar
  • 239
1 vote
0 answers
104 views

KL divergence estimates over binary classification data

I have a dataset $D = (x_i, y_i)_{i=1}^n$ where $x_i\in \Bbb R^d$ and $y_i\in\{0, 1\}$. Suppose that $y\sim\mathrm{Bernoulli}(p(x))$ for some probability function $p:\Bbb R^d \to [0,1]$ and I would ...
user avatar
  • 343
2 votes
0 answers
232 views

Are binary splits always better than multi-way splits in decision trees?

I'm trying to devise a decision tree for classification with multi-way split at an attribute but even though calculating the entropy for a multi-way split gives better information gain than a binary ...
user avatar
2 votes
1 answer
112 views

Binary classification but learning the true probability

I have a dataset $D = (x_i, y_i)_{i=1}^n$ where $x_i\in \Bbb R^d$ and $y_i\in\{0, 1\}$. Suppose that $y\sim\mathrm{Bernoulli}(p(x))$ for some probability function $p:\Bbb R^d \to [0,1]$ and I would ...
user avatar
  • 343
0 votes
0 answers
38 views

Entropy of Poisson random variable via exponential family identity

Exercise 8.1 of Probabilistic Graphical Models asks the reader to use the identity $$H_{P_{\theta}}(X)= \ln Z(\theta) - \langle E_{P_{\theta}}[\tau(X)], t(\theta)\rangle$$ to compute the entropy $H$ ...
user avatar
0 votes
0 answers
20 views

Differential entropy of a kernel density estimator

I have a set of (~500k) observations that I use to fit a parametric model (univariate three-parameter lognormal). To have an idea of the goodness of fit, I want to compute the Jensen-Shannon distance ...
user avatar
  • 183
0 votes
0 answers
22 views

Determining history length in Transfer Entropy

Supposing I have a time series observation corresponding to 5 sensors, and I wish to find the causation(Transfer entropy) between all pairs of nodes, how do I go about choosing the history length ...
user avatar
2 votes
1 answer
76 views

Measurement of regularity of events

I am dealing with events generated by different processes. My data includes time stamps when the events occur, so I am trying to differentiate the processes in regular or irregular categories or ...
user avatar
  • 121
0 votes
0 answers
7 views

How to calculate the uncertainty of a variable explained by other variables

I'm working on implementing a method from a paper for solving the FRn-k problem for a specific type of biological data. [FRn−k]: Given an initial set of n features, find the subset with k<n ...
user avatar
  • 111
2 votes
1 answer
49 views

Differential Entropy with negative support

Differential Entropy Estimation Hello everyone, I am estimating differential entropy from a continuous distribution of a property of materials. The distributions on the picture are Gaussian Mixtures. ...
user avatar
2 votes
1 answer
137 views

Is there a name for this in statisitics? [closed]

Edit: Ignore this Most of this doesn't make sense and is beyond edits but its too late to delete. If I make too many edits I could be kicked out of this sub like I was in math stack exchange. I don't ...
user avatar
  • 129

1
2 3 4 5
13