# Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

632 questions
Filter by
Sorted by
Tagged with
31 views

### How can I adjust cumulative entropy of moose observations?

I have downloaded the dates of every moose (Alces alces) observation worldwide on iNaturalist. This amounts to about $2 \cdot 10^4$ observations at present. I excluded one observation that was either ...
• 3,403
1 vote
31 views

### Is "Information" somehow Related to "Variance"?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
• 5,738
1 vote
32 views

### Do Mixture Models "Defy" Entropy?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
• 5,738
1 vote
36 views

### Entropy of Gaussian mixture when variance of one component gets larger

I want to prove or disprove the following relation of differential entropies: Conjecture: $\displaystyle h(f) \le h(g)$ where $f, g$ the density functions of Gaussian mixture models with equal ...
• 1,099
29 views

### Negative transfer entropy

By definition, transfer entropy cannot be negative. However, using the Kraskov estimator, negative values can be obtained. In general, should we take precautions to avoid getting negative values? How ...
• 11
34 views

### Empirical error in Kullback-Leibler KL divergence estimation

In computing the Kullback-Leibler KL divergence $D(P\|Q)$ from an empirical data, it may happen that $Q(x)=0<P(x)$ at some sample point $x$ due to data error and $D(P\|Q)=\infty$. What are some ...
• 875
1 vote
35 views

### How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
• 11
662 views

### How to apply the formula for Shannon entropy to a 4-sided die?

I have come across calculating entropy, via the formula: $$Entropy(p) = -\sum_{i=1} ^{N}p_i \log_2(p_i)$$ Referring to this formula, how would I calculate the entropy of a ...
1 vote
54 views

### How to compute transfer entropy between different length time-series?

Is there a way to calculate transfer entropy between two time series having different lengths? Given two time series: $x=(x_1,x_2,\dotsc,x_n)$ $y=(y_1,y_2,\dotsc,y_m)$, where $n \ne m$. Is there a way ...
• 11
1 vote
17 views

• 185
30 views

### Multivariate transfer entropy

I have a set of time series to treat as sources and a time series to treat as destination. From the definition of multivariate transfer entropy, it seems to me that it can only be defined on two ...
• 11
27 views

### How to understand the information and entropy?

Let $X$ is a binary variable and $X = \{A, B\}$. The pdf of $X$ is $p(X = A) = .3 \ \ \ p(X=B) = .7$ So it's easy to calculate the ...
13 views

### How to understand embedding dimension in permutation entropy when checking predictabilty in electric load data?

I am predicting electric load data with different deep learning models and I am trying to define the predictability of the data. So far I came across the permutation entropy (PE) as a measurement for ...
• 111
27 views

### What is the interpretation of negative differential entropy? [duplicate]

I'm currently trying to understand the meaning of negative entropy. With the given equation: $$H_{cont}\ (X) = -\int_{-\infty}^{+\infty} p(x) \centerdot ln(p(x)) \ dx$$ Here, the The_Sympathizer ...
1 vote
87 views

• 351
10 views

### Relating difference in distribution of dataset to information gain

I wonder if there are ways to relate difference in distribution of dataset to information gain. For example, I train a model on dataset D_1 and obtain a trained model M_1 and train a model on dataset ...
• 223
135 views

### How to measure if a categorical distribution is concentrated in a very few bins

I'm trying to think of a way to measure that a categorical distribution of any size is concentrated in only a few bins, so not uniform. The best way I can think of is checking entropy, but that's kind ...
• 640
13 views

20 views

### Differential entropy of a kernel density estimator

I have a set of (~500k) observations that I use to fit a parametric model (univariate three-parameter lognormal). To have an idea of the goodness of fit, I want to compute the Jensen-Shannon distance ...
• 183
22 views

### Determining history length in Transfer Entropy

Supposing I have a time series observation corresponding to 5 sensors, and I wish to find the causation(Transfer entropy) between all pairs of nodes, how do I go about choosing the history length ...
• 195
76 views

### Measurement of regularity of events

I am dealing with events generated by different processes. My data includes time stamps when the events occur, so I am trying to differentiate the processes in regular or irregular categories or ...
• 121
7 views

### How to calculate the uncertainty of a variable explained by other variables

I'm working on implementing a method from a paper for solving the FRn-k problem for a specific type of biological data. [FRn−k]: Given an initial set of n features, find the subset with k<n ...
• 111