# Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

666 questions
Filter by
Sorted by
Tagged with
15 views

### Measuring entropy of a binary matrix with biased probability

I tried to measure entropy of binary matrix like below using code at : https://github.com/cosmoharrigan/matrix-entropy (I already saw the question : Measuring entropy/ information/ patterns of a 2d ...
• 1
62 views

### How can the mutual information be equal to minus conditional entropy?

I am reading the following paper: https://arxiv.org/abs/2301.06941 The authors in Eq.(8) have obtained a relation which has the mutual information, $i$, in the exponent of the exponential on the RHS ... 24 views

8 views

### How do entropy-based estimators relate to more conventional ML, least square, and GMM estimators

Over the years i have done a lot of analysis, mostly of parameters of linear approximations to the data or a forecast, and I have used linear and nonlinear least squares, maximum likelihood, and GMM ...
• 2,757
26 views

### How do I calculate the entropy of a uniform continuous variable?

Let $x$ be any random rational number where $x \in (0, 1/8)$ and $f(x) = 8x$. Therefore, $f(x) \in (0, 1)$. How would I mathematically calculate the entropy of $x$ and $f(x)$? Applying the ...
• 101
1 vote
33 views

### Differential entropy / variance

So I‘m basically more or less trying to prove what is stated in the answer by syeh_106 here: Is differential entropy always less than infinity? if the variance is finite, then the differential ...
• 83
17 views

### Why does entropy of Y increase with increase in number of samples making up Y?

Suppose, a sample of $Y=(1/\sqrt{N}) ∑_{i=1,\dotsc,N} X_i$, where $X\sim \mathcal{Uniform Distribution}(-3, 3)$. We have let's say 10,000 such samples of Y. Here, when we increase the value of N, why ...
• 401
1 vote
16 views

### Relate cross-entropy formal definition to the cross-entropy loss [duplicate]

Cross entropy for a random variable $x \sim p$ and a distribution $q$ is defined as: $$H(p,q) = -\sum_{x\in\mathcal{X}} p(x)\log q(x) = \mathbb{E}(\log q(x))$$ $\mathcal{X}$ is all possible values ...
• 303
1 vote
68 views

• 679
68 views

### Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
• 411
1 vote
24 views

### In the DT algorithm, why not search through all possible splits instead of first choosing the feature required for splitting?

As far as I understand the Decision Trees (DT - CART), the first step is to choose a feature that maximizes IG (information gain), meaning the entropy in the current node - the entropy after splitting ...
• 45
126 views

• 31
60 views

### Cluster evaluation - evaluation separation, stability and agreement - best indices?

In the literature many indices exist that aim to quantify the level of separation of the clustering outcome: Silhouette, Davis-Bouldin, Calinski-Harabasz, etc. To evaluate the stability of a ...
40 views

### Calculating entropy-based indices based on a frequency cross-tabulation of objects in Python

I was wondering whether there exist a Python solution for calculating Mutual Information, Completeness, V-Measure, etc., based on a pair-confusion matrix. MI defined as H(C_0) + H(C_1) - H(C_O&C_1)...
• 61
1 vote
75 views

### Mutual Information larger than smaller one of both entropies?

Lot's of questions and good answers on mutual information e.g. here out there and I think I get the concept which is also nicely explained on wikipedia. But I'm nervous that R's infotheo package is ...
• 21
89 views

### ERGM: How come ttriple is degenerate but transitiveties is not?

This title is an effort to elicit a simplified explanation as to how come the transitiveties term causes less numerical instability than ...
37 views

### Can transfer entropy be calculated after PCA?

In a problem involving multivariate time series with many dimensions, does it make sense to apply principal component analysis to perform feature reduction before applying transfer entropy? That is, ...
• 11
39 views

### How can I adjust cumulative entropy of moose observations?

I have downloaded the dates of every moose (Alces alces) observation worldwide on iNaturalist. This amounts to about $2 \cdot 10^4$ observations at present. I excluded one observation that was either ...
• 6,135
1 vote
61 views

### Is "Information" somehow Related to "Variance"?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
• 7,028