Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Measuring entropy of a binary matrix with biased probability

I tried to measure entropy of binary matrix like below using code at : https://github.com/cosmoharrigan/matrix-entropy (I already saw the question : Measuring entropy/ information/ patterns of a 2d ...
  • 1
0 votes
0 answers
62 views

How can the mutual information be equal to minus conditional entropy?

I am reading the following paper: https://arxiv.org/abs/2301.06941 The authors in Eq.(8) have obtained a relation which has the mutual information, $i$, in the exponent of the exponential on the RHS ...
user avatar
0 votes
0 answers
24 views

Perplexity and cross-entropy relationship

According to wikipedia Perplexity- A perplexity of discrete distribution p equals to $2^{H(p)}$ where H(p) is the entropy of p. A perplexity of a probability model M, with unknown probabiltiy p - ${\...
  • 31
0 votes
0 answers
45 views

Using zlib to estimate entropy rate of text data

I'm trying to estimate the entropy rate of a text file (about 2 million characters, including spacing and punctuation). Calculating the entropy rate directly is not an option for me, for lack of ...
0 votes
0 answers
153 views

Marginal contribution of each observation to posterior moments and (differential) entropy are decreasing in the number of observations?

In the Beta-Binomial and Gaussian settings, indeed, the marginal contribution of each observation to the posterior moments (I could check first and second) and (differential) entropy are decreasing in ...
  • 93
0 votes
1 answer
40 views

Why the maximum of Rényi entropy is not achieved for the uniform distribution? [closed]

Given a discrete random variable $X$, which takes values in the alphabet $\mathcal {X}$ and is distributed according to $p:{\mathcal {X}}\to [0,1]$ the Shannon entropy is defined as: $$\mathrm {H} (X):...
  • 297
1 vote
0 answers
26 views

How to calculate a path entropy in the context of information theory?

Consider a path where the system goes from states $i$ to $j$ and then from $j$ to $k$. Given that the probability of finding the system in state $i$ is $p_i$ and the transition probabilities are $p_{...
  • 61
1 vote
0 answers
29 views

Understanding properties of conditional entropy

I was trying to understand concept of cross entropy. Wikipedia says: $H(Y|X)\leq H(Y)$ Although the specific conditional entropy $H(X|Y=y)$ can be either less or greater than $H(X)$ for a given ...
0 votes
0 answers
50 views

Entropy of a Continuous and Uniform Random Variable

The distribution of a uniform r.v. X is given as follows: The entropy is therefore: This means that as $∆$ approaches infinity, so does the entropy. This also means that as $∆$ approaches 0, the ...
  • 5
1 vote
1 answer
63 views

Relationship between entropy and predictability

The entropy of a random variable $X$ is defined as $\mathbb{E}(-\log(f(X)))$ (where $f$ is the pdf of X, https://en.wikipedia.org/wiki/Entropy_(information_theory)). Is there any general relationship ...
  • 385
0 votes
1 answer
14 views

Quantifying + comparing the effects of conditioning on Shannon entropy

I'm working on a project that compares how predictable/unpredictable individuals' actions are in terms of how they transition between actions. We consider their actions to part of a first-order Markov ...
  • 1
0 votes
0 answers
17 views

How can I calculate the entropy of a binned multivariate normal distribution?

I know that I can calculate the entropy of a multivariate distribution by using $${N(\mu,{\boldsymbol {\Sigma }}) \\ \mathcal H(N) = \displaystyle {\frac {1}{2}}\ln \det \left(2\pi \mathrm {e} {\...
0 votes
0 answers
35 views

Measure of clumpiness or cluster of 1D continuous data over a given range

I'm looking for a measure of how clustered (or converse empty) data is over a range (e.g. [0,1] in this case). The first obvious metric to use would be entropy (or ...
0 votes
0 answers
11 views

Probability and Entropy of a Multichannel Image

I'm currently trying to implement the method covered in the paper "Hyperspectral Band Selection via Optimal Combination Strategy" I found that point 1) of this questions was close to what I ...
0 votes
0 answers
84 views

PCA evaluation with Shannon Entropy

I'm working on a fairly large dataset (5e5 samples in a 22 dimensional space); I'm reducing the dimensionality of this dataset with PCA. Since a lot of variables are cross-correlated, I group the ...
0 votes
0 answers
10 views

How to show whether the entropy of a uniform distribution with infinite image space converges or diverges?

Assuming a random variable $X$ with an infinite image space (i.e. the natural numbers $\mathbf{N}$) and the probability mass function (PMF) being the uniform distribution with PMF $P(X=n) = \frac{1}{n}...
0 votes
0 answers
8 views

How do entropy-based estimators relate to more conventional ML, least square, and GMM estimators

Over the years i have done a lot of analysis, mostly of parameters of linear approximations to the data or a forecast, and I have used linear and nonlinear least squares, maximum likelihood, and GMM ...
  • 2,757
0 votes
0 answers
26 views

How do I calculate the entropy of a uniform continuous variable?

Let $x$ be any random rational number where $x \in (0, 1/8)$ and $f(x) = 8x$. Therefore, $f(x) \in (0, 1)$. How would I mathematically calculate the entropy of $x$ and $f(x)$? Applying the ...
  • 101
1 vote
0 answers
33 views

Differential entropy / variance

So I‘m basically more or less trying to prove what is stated in the answer by syeh_106 here: Is differential entropy always less than infinity? if the variance is finite, then the differential ...
  • 83
0 votes
0 answers
17 views

Why does entropy of Y increase with increase in number of samples making up Y?

Suppose, a sample of $Y=(1/\sqrt{N}) ∑_{i=1,\dotsc,N} X_i$, where $X\sim \mathcal{Uniform Distribution}(-3, 3)$. We have let's say 10,000 such samples of Y. Here, when we increase the value of N, why ...
  • 401
1 vote
0 answers
16 views

Relate cross-entropy formal definition to the cross-entropy loss [duplicate]

Cross entropy for a random variable $x \sim p$ and a distribution $q$ is defined as: $$H(p,q) = -\sum_{x\in\mathcal{X}} p(x)\log q(x) = \mathbb{E}(\log q(x))$$ $\mathcal{X}$ is all possible values ...
  • 303
1 vote
1 answer
68 views

Maximize Mutual Information between multiple ("overlapping") RVs

I'd like to maximize the sum of Mutual Information between a RV $X$ and $K$ out of $N$ possible RVs $Z_i$. $$ \max \sum_{i \in K} \text{MI}(X, Z_i) $$ However, when I unfold the sum I get $$ \sum_{i \...
1 vote
2 answers
58 views

Are there better measures of entropy

Related question here I am trying to measure the uniformity of multimodal distributions and am looking into using entropy. I would like a measure of entropy that is higher for the first distribution ...
0 votes
0 answers
44 views

How to sample distributions far from uniform?

I need to obtain an isotropic sample of discrete distributions over $d$ outcomes which are far from uniform in some measure. For instance, using linear entropy as the complexity measure, we can ...
1 vote
0 answers
49 views

Is maximal spectral entropy of residuals a poor loss function because phase information is lost?

Suppose I define a custom loss function SpectralEntropy as follows: ...
  • 6,135
2 votes
2 answers
93 views

Surprisal in rankings

I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race. Intuitively, in a race with 100 people, ...
  • 1,083
2 votes
0 answers
30 views

How much do we learn about a random subset? [closed]

Suppose we sample the following two random variables, for some large integer $n$: Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
1 vote
0 answers
43 views

conditional entropy of finite mixture model

I am trying to understand the conditional entropy of the finite mixture model given in this paper about regularized EM algorithm. ################# On page 3 of the paper: in the finite mixture model, ...
  • 173
2 votes
0 answers
37 views

How can entropy help in machine learning classification? [closed]

I cannot fully grasp why entropy is so important in machine learning classification. I understand the usage in decision trees but I don't see the importance elsewhere. I mean entropy somehow comes up ...
  • 61
0 votes
0 answers
79 views

How can Mutual Information be lower than Adjusted Mutual Information?

What would be the statistical reasoning that MI < AMI in cluster evaluation? Wouldn't you expect AMI to always be slightly (or a lot) lower than MI?
0 votes
0 answers
34 views

differential entropy rate for stationary processes

the differential entropy rate of a stochastic process $X = (X_i)_{i \in \mathbb{N}}$ is defined as $$h(X) = \lim_{n \to \infty} \frac{1}{n} h(X_1,\ldots,X_n)$$, where $h(X_1,\ldots,X_n)$ is the joint ...
  • 1
0 votes
0 answers
19 views

Estimeate Entropy of Sequence

Suppose I have a sequence of N symbols. There are 100 symbols. The probability of each symbol is quite small, though they're not necessarily uniform. s1 - s2 - s5 - s8 - s3 - sk .... Each symbol is an ...
1 vote
0 answers
33 views

Intuition of multivariate conditional entropy

This might be a dumb question, but I'm serious. Per the wiki page about conditional entropy: quantifies the amount of information needed to describe the outcome of a random variable $Y$ given that the ...
  • 11
0 votes
0 answers
21 views

How to interpret the Adjusted Rand Index

Given a pair-confusion matrix where: ...
2 votes
1 answer
72 views

Confidence Interval of entropy for a discrete distribution

The following table is a set of ordinal data from a survey I have conducted (one of many). $$\begin{array}{c|c|c|} \text{Grading}& \text{Count} & \text{Frequency} \\ \hline \text{1} & 5 &...
  • 679
2 votes
1 answer
68 views

Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
1 vote
0 answers
24 views

In the DT algorithm, why not search through all possible splits instead of first choosing the feature required for splitting?

As far as I understand the Decision Trees (DT - CART), the first step is to choose a feature that maximizes IG (information gain), meaning the entropy in the current node - the entropy after splitting ...
2 votes
1 answer
126 views

Cross entropy of a random variable or a probability distribution function? [duplicate]

I'm looking into the wikipedia page of cross entropy. https://en.wikipedia.org/wiki/Cross_entropy $$H(p,q)=-\sum_{x\in \mathcal{X}} p(x)\log q(x)$$ It can be written as $$H(p,q) = H(p) + D_{KL} (p||q)$...
0 votes
0 answers
18 views

Using entropy balancing to achieve general population representativeness in a survey with multiple conditions

We are currently planning a survey in which we ask people about their attitudes towards four different groups of people. Each participant is only asked about their attitudes towards one of the four ...
0 votes
0 answers
23 views

When to consider classification error and entropy in LCA models?

I have good fit statistics in terms of G^2 (bootstrap) and bivariate residuals. However, all the models with good absolute fit statistics have poor classification error (.4) and entropy (.4-.5). If ...
0 votes
0 answers
91 views

What does "tensorization" mean in the entropy tensorization inequality?

I am reading high dimensional statistics written by Wainwright. In chapter 3.1.4 tensorization of entropy is used to extend the entropy bound for univariate functions to multivariate cases. As far as ...
1 vote
1 answer
48 views

Covering transformation for Lipschitz functions: can we prove that $\mathcal{H}(r,\mathcal {G'},||.||_n)\asymp \mathcal{H}(r/C_n,\mathcal {A'},\rho)$?

Let assume that $\mathcal{A'}\subset \mathcal {A}$ (is a vector space) and $\mathcal {G'}:=\{g_\theta : \ \theta\in \mathcal{A'}\}$ (space of functions parameterized by $\mathcal{A'}$) and a metric $\...
  • 31
0 votes
0 answers
60 views

Cluster evaluation - evaluation separation, stability and agreement - best indices?

In the literature many indices exist that aim to quantify the level of separation of the clustering outcome: Silhouette, Davis-Bouldin, Calinski-Harabasz, etc. To evaluate the stability of a ...
0 votes
0 answers
40 views

Calculating entropy-based indices based on a frequency cross-tabulation of objects in Python

I was wondering whether there exist a Python solution for calculating Mutual Information, Completeness, V-Measure, etc., based on a pair-confusion matrix. MI defined as H(C_0) + H(C_1) - H(C_O&C_1)...
  • 61
1 vote
4 answers
75 views

Mutual Information larger than smaller one of both entropies?

Lot's of questions and good answers on mutual information e.g. here out there and I think I get the concept which is also nicely explained on wikipedia. But I'm nervous that R's infotheo package is ...
  • 21
2 votes
1 answer
89 views

ERGM: How come `ttriple` is degenerate but `transitiveties` is not?

This title is an effort to elicit a simplified explanation as to how come the transitiveties term causes less numerical instability than ...
0 votes
0 answers
37 views

Can transfer entropy be calculated after PCA?

In a problem involving multivariate time series with many dimensions, does it make sense to apply principal component analysis to perform feature reduction before applying transfer entropy? That is, ...
  • 11
0 votes
0 answers
39 views

How can I adjust cumulative entropy of moose observations?

I have downloaded the dates of every moose (Alces alces) observation worldwide on iNaturalist. This amounts to about $2 \cdot 10^4$ observations at present. I excluded one observation that was either ...
  • 6,135
1 vote
1 answer
61 views

Is "Information" somehow Related to "Variance"?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
  • 7,028
2 votes
1 answer
66 views

Do Mixture Models "Defy" Entropy?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
  • 7,028

1
2 3 4 5
14