# Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

549 questions
Filter by
Sorted by
Tagged with
17 views

### Shannon entropy and Gini impurity are interchangeable in practice yet different?

Gini impurity, not to be confused with the Gini coefficient, is also an information theoretic measure and corresponds to Tsallis Entropy with deformation coefficient $q=2$, which in physics is ...
14 views

### When would you use purity as a measure of external validity over entropy?

This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
11 views

### How does Information Gain work?

I know the algorithm upon which the information gain is based. But how are we sure that after a split, its entropy will always decrease i.e. on dividing the data into smaller chunks, the entropy will ...
42 views

### Maximum Entropy Discrete Distribution

In Pattern Recognition and Machine Learning the author uses Lagrange multipliers to find the discrete distribution with maximum entropy. Entropy is defined by; $$H=-\sum_i p(x_i)\ln(p(x_i))$$ and the ...
8 views

### What is the effect of number of points on calculation of mutual information between two variables?

I was trying to calculate mutual information between two variables as follows: mutual_info_score(x,y) This essentially creates a 2d histogram and evaluates the ...
11 views

### How would the entropy of a random walk change over time? [closed]

Generate a stochastic process random walk (with drift) with $T$ observations. We can further split the random walk into: expanding windows ($10$ observations, for example, are added to the window as ...
17 views

### If entropy is the underlying measure for KL-divergence, what is the underlying measure for the Wasserstein distance?

If entropy is the basis measure underlying KL-divergence (aka relative entropy), what is the basis measure underlying the Wasserstein distance?
11 views

### If a zero entropy distribution implies high information a priori, what does it mean ex posteriori?

The following counteracts the statements made for the maximum entropy principle case in order to posit a pseudo "minimum entropy principle" case that is simply the polar opposite of the ...
20 views

14 views

### How to decompose entropy into linear and non-linear components?

Mutual information is the entropy between two random variables, $X$ and $Y$, based on their probabilities. It captures both the linear and non-linear interactions between $X$ and $Y$, whereas ...
6 views

### Is transfer entropy the same as conditional mutual information?

This encyclopedia article says that transfer entropy is conditional mutual information. But the first measure makes no mention of a third variable being required unlike the second measure, and is the ...
49 views

### What's an intuitive proof for why Shannon's entropy poses a lower bound on the expected number of bits encoding a distribution?

As I understand it, Shannon's entropy is $-\sum p\log p$ which represents an expectation of the self information $-\log p$ over a distribution. The log function was chosen to fit 3 properties: A ...
15 views

9 views

### Are entropy and differential entropy good risk measures?

Standard deviation is often used to measure volatility or risk. But not all continuous random variables have a known statistical distribution and therefore might not even have a mean, variance or ...