Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

Filter by
Sorted by
Tagged with
1 vote
2 answers
47 views

Are there better measures of entropy

Related question here I am trying to measure the uniformity of multimodal distributions and am looking into using entropy. I would like a measure of entropy that is higher for the first distribution ...
0 votes
0 answers
43 views

How to sample distributions far from uniform?

I need to obtain an isotropic sample of discrete distributions over $d$ outcomes which are far from uniform in some measure. For instance, using linear entropy as the complexity measure, we can ...
0 votes
0 answers
16 views

Is maximal spectral entropy of residuals a poor loss function because phase information is lost?

Suppose I define a custom loss function SpectralEntropy as follows: ...
  • 4,590
0 votes
0 answers
17 views

Surprisal in rankings

I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race. Intuitively, in a race with 100 people, ...
  • 1,063
2 votes
0 answers
27 views

How much do we learn about a random subset? [closed]

Suppose we sample the following two random variables, for some large integer $n$: Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
1 vote
0 answers
34 views

conditional entropy of finite mixture model

I am trying to understand the conditional entropy of the finite mixture model given in this paper about regularized EM algorithm. ################# On page 3 of the paper: in the finite mixture model, ...
  • 63
2 votes
0 answers
24 views

How can entropy help in machine learning classification? [closed]

I cannot fully grasp why entropy is so important in machine learning classification. I understand the usage in decision trees but I don't see the importance elsewhere. I mean entropy somehow comes up ...
  • 61
0 votes
0 answers
16 views

How can Mutual Information be lower than Adjusted Mutual Information?

What would be the statistical reasoning that MI < AMI in cluster evaluation? Wouldn't you expect AMI to always be slightly (or a lot) lower than MI?
0 votes
0 answers
24 views

differential entropy rate for stationary processes

the differential entropy rate of a stochastic process $X = (X_i)_{i \in \mathbb{N}}$ is defined as $$h(X) = \lim_{n \to \infty} \frac{1}{n} h(X_1,\ldots,X_n)$$, where $h(X_1,\ldots,X_n)$ is the joint ...
  • 1
0 votes
0 answers
11 views

Estimeate Entropy of Sequence

Suppose I have a sequence of N symbols. There are 100 symbols. The probability of each symbol is quite small, though they're not necessarily uniform. s1 - s2 - s5 - s8 - s3 - sk .... Each symbol is an ...
1 vote
0 answers
19 views

Intuition of multivariate conditional entropy

This might be a dumb question, but I'm serious. Per the wiki page about conditional entropy: quantifies the amount of information needed to describe the outcome of a random variable $Y$ given that the ...
  • 11
0 votes
0 answers
11 views

How to interpret the Adjusted Rand Index

Given a pair-confusion matrix where: ...
1 vote
1 answer
40 views

Confidence Interval of entropy for a discrete distribution

The following table is a set of ordinal data from a survey I have conducted (one of many). $$\begin{array}{c|c|c|} \text{Grading}& \text{Count} & \text{Frequency} \\ \hline \text{1} & 5 &...
  • 485
1 vote
1 answer
40 views

Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
0 votes
0 answers
21 views

In the DT algorithm, why not search through all possible splits instead of first choosing the feature required for splitting?

As far as I understand the Decision Trees (DT - CART), the first step is to choose a feature that maximizes IG (information gain), meaning the entropy in the current node - the entropy after splitting ...
2 votes
1 answer
44 views

Cross entropy of a random variable or a probability distribution function? [duplicate]

I'm looking into the wikipedia page of cross entropy. https://en.wikipedia.org/wiki/Cross_entropy $$H(p,q)=-\sum_{x\in \mathcal{X}} p(x)\log q(x)$$ It can be written as $$H(p,q) = H(p) + D_{KL} (p||q)$...
0 votes
0 answers
12 views

Using entropy balancing to achieve general population representativeness in a survey with multiple conditions

We are currently planning a survey in which we ask people about their attitudes towards four different groups of people. Each participant is only asked about their attitudes towards one of the four ...
0 votes
0 answers
11 views

When to consider classification error and entropy in LCA models?

I have good fit statistics in terms of G^2 (bootstrap) and bivariate residuals. However, all the models with good absolute fit statistics have poor classification error (.4) and entropy (.4-.5). If ...
0 votes
0 answers
24 views

What does "tensorization" mean in the entropy tensorization inequality?

I am reading high dimensional statistics written by Wainwright. In chapter 3.1.4 tensorization of entropy is used to extend the entropy bound for univariate functions to multivariate cases. As far as ...
1 vote
1 answer
42 views

Covering transformation for Lipschitz functions: can we prove that $\mathcal{H}(r,\mathcal {G'},||.||_n)\asymp \mathcal{H}(r/C_n,\mathcal {A'},\rho)$?

Let assume that $\mathcal{A'}\subset \mathcal {A}$ (is a vector space) and $\mathcal {G'}:=\{g_\theta : \ \theta\in \mathcal{A'}\}$ (space of functions parameterized by $\mathcal{A'}$) and a metric $\...
  • 31
0 votes
0 answers
21 views

Cluster evaluation - evaluation separation, stability and agreement - best indices?

In the literature many indices exist that aim to quantify the level of separation of the clustering outcome: Silhouette, Davis-Bouldin, Calinski-Harabasz, etc. To evaluate the stability of a ...
0 votes
0 answers
12 views

Calculating entropy-based indices based on a frequency cross-tabulation of objects in Python

I was wondering whether there exist a Python solution for calculating Mutual Information, Completeness, V-Measure, etc., based on a pair-confusion matrix. MI defined as H(C_0) + H(C_1) - H(C_O&C_1)...
  • 61
1 vote
4 answers
52 views

Mutual Information larger than smaller one of both entropies?

Lot's of questions and good answers on mutual information e.g. here out there and I think I get the concept which is also nicely explained on wikipedia. But I'm nervous that R's infotheo package is ...
  • 21
1 vote
1 answer
37 views

ERGM: How come `ttriple` is degenerate but `transitiveties` is not?

This title is an effort to elicit a simplified explanation as to how come the transitiveties term causes less numerical instability than ...
0 votes
0 answers
20 views

Can transfer entropy be calculated after PCA?

In a problem involving multivariate time series with many dimensions, does it make sense to apply principal component analysis to perform feature reduction before applying transfer entropy? That is, ...
  • 11
0 votes
0 answers
34 views

How can I adjust cumulative entropy of moose observations?

I have downloaded the dates of every moose (Alces alces) observation worldwide on iNaturalist. This amounts to about $2 \cdot 10^4$ observations at present. I excluded one observation that was either ...
  • 4,590
1 vote
1 answer
42 views

Is "Information" somehow Related to "Variance"?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
  • 6,048
1 vote
0 answers
40 views

Do Mixture Models "Defy" Entropy?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
  • 6,048
1 vote
0 answers
63 views

Entropy of Gaussian mixture when variance of one component gets larger

I want to prove or disprove the following relation of differential entropies: Conjecture: $\displaystyle h(f) \le h(g)$ where $f, g$ the density functions of Gaussian mixture models with equal ...
  • 1,159
0 votes
0 answers
56 views

Negative transfer entropy

By definition, transfer entropy cannot be negative. However, using the Kraskov estimator, negative values can be obtained. In general, should we take precautions to avoid getting negative values? How ...
  • 11
1 vote
1 answer
116 views

Empirical error in Kullback-Leibler KL divergence estimation

In computing the Kullback-Leibler KL divergence $D(P\|Q)$ from an empirical data, it may happen that $Q(x)=0<P(x)$ at some sample point $x$ due to data error and $D(P\|Q)=\infty$. What are some ...
  • 905
1 vote
1 answer
60 views

How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
  • 11
5 votes
3 answers
742 views

How to apply the formula for Shannon entropy to a 4-sided die?

I have come across calculating entropy, via the formula: \begin{equation} Entropy(p) = -\sum_{i=1} ^{N}p_i \log_2(p_i) \end{equation} Referring to this formula, how would I calculate the entropy of a ...
1 vote
0 answers
73 views

How to compute transfer entropy between different length time-series?

Is there a way to calculate transfer entropy between two time series having different lengths? Given two time series: $x=(x_1,x_2,\dotsc,x_n)$ $y=(y_1,y_2,\dotsc,y_m)$, where $n \ne m$. Is there a way ...
  • 11
1 vote
0 answers
36 views

Understanding continuous variable entropy

I am struggling to understand continuous variable entropies and mutual informations for 2 or more variables. Consider 2D normal distribution $\rho(x,y)$ defined as follows $$X\sim\mathcal{N}(0, \...
1 vote
1 answer
138 views

Does increasing the variance increase the value of a function?

Let $V=\sum_{i=1}^{k} a_iX_i$ where $X_i's$ are IID $\sim Bern(q)$ and $V$ with $\sum a_i=k$. Note that $a_i$'s are non-negative integers. I have a function $f$ as given below : $$ f= \max_{q} h\...
  • 185
0 votes
0 answers
44 views

Multivariate transfer entropy

I have a set of time series to treat as sources and a time series to treat as destination. From the definition of multivariate transfer entropy, it seems to me that it can only be defined on two ...
  • 11
0 votes
1 answer
40 views

How to understand the information and entropy?

Let $X$ is a binary variable and $X = \{A, B\}$. The pdf of $X$ is $p(X = A) = .3 \ \ \ p(X=B) = .7$ So it's easy to calculate the ...
1 vote
0 answers
54 views

How to understand embedding dimension in permutation entropy when checking predictabilty in electric load data?

I am predicting electric load data with different deep learning models and I am trying to define the predictability of the data. So far I came across the permutation entropy (PE) as a measurement for ...
  • 121
0 votes
0 answers
29 views

What is the interpretation of negative differential entropy? [duplicate]

I'm currently trying to understand the meaning of negative entropy. With the given equation: $$ H_{cont}\ (X) = -\int_{-\infty}^{+\infty} p(x) \centerdot ln(p(x)) \ dx $$ Here, the The_Sympathizer ...
1 vote
1 answer
185 views

Why does mutual information use KL divergence?

Mutual information between a pair of random variables $X,Y$ having joint distribution $P_{(X,Y)}$ and marginal distributions $P_X,P_Y$ respectively is defined as $$I(X,Y)\equiv D_{\text{KL}}(P_{(X,Y)}\...
3 votes
1 answer
46 views

What is the proper name for the *other* type of "conditional entropy"?

Suppose we have two random variables $X$ and $Y$ (for simplicity of exposition I will take these to be discrete). If we were to condition our entire analysis on the event $X=x$ and then ask for the ...
  • 102k
4 votes
1 answer
240 views

Entropy Conditioned on A Certain Value

Consider this example: I have a box with 10 balls in it, 9 red and 1 blue. I take a ball randomly. Let's call the color $C$. If $C$ is red, I shout the number zero. If $C$ is blue, I roll a fair die ...
  • 161
0 votes
0 answers
133 views

How can I quantify the skewness, kurtosis, entropy when all of values of a list is 0?

Background Let's think, there is a list of values which presents activity of a person for several hours. That person did not have any movement in those hours. Therefore, all the values are 0. Then, ...
0 votes
0 answers
23 views

What statistic/algorithm does the decision tree use to partition/cluster the data per depth?

How does a decision tree calculate that a break exists at 1.8 on the root, a break exists at 2.1 and 1.2 on the second depth? I know Gini and Entropy are used to calculate which feature to partition ...
2 votes
2 answers
196 views

Minimizing KL-divergence and log-likelihood for generative machine learning models

I am reading a paper on quantum ML: A generative modeling approach for benchmarking and training shallow quantum circuits, where it is claimed that: Following a standard approach from generative ...
  • 133
0 votes
0 answers
16 views

Computing Minimum Description Length

I have two integer arrays, x and y: ...
  • 484
1 vote
0 answers
105 views

Differential entropy of a multivariate log-normal distribution

Let $Y$ be a multivariate log-normal random variable. As such, $\ln(Y)$ follows a multivariate normal distribution, which I denote by $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$. I already ...
  • 111
0 votes
0 answers
15 views

How to decompose the autocorrelation function of a variable, resulted from interactions in complex systems?

The output variable (X) of a complex system undergoes numerous interactions between system components. These interactions will impart a distinct signature to the autocorrelation function of the output ...
1 vote
0 answers
26 views

RTransferEntropy - Discrete data

I'm trying to use the RTransferEntropy package to compute TE for my data. I want to understand how discrete series are handled by functions like the ...

1
2 3 4 5
14