Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

0
votes
1answer
17 views

Why dose discrete data distribution has differential entropy of negative infinity?

Recently I've been reading a paper. In section 3.1, it says "Since the discrete data distribution has differential entropy of negative infinity, this can lead to arbitrary high likelihoods even on ...
0
votes
0answers
5 views

Ranking criterion vs. entropy criterion

Problem In a classical NLP paper (Natural language processing (almost) from scratch) I am reading now, the authors claim that The entropy criterion lacks dynamical range because its numerical ...
0
votes
0answers
3 views

Modelling probability distribution of a set of sequences (to calculate Entropy)

Let $T$ be a set of trajectories $\tau$, where $\tau=\{\mathbf{x}_1,\mathbf{x}_2,...\mathbf{x}_N\}$ with $\mathbf{x}_i\in\mathbb{R}^k$ being a vector of observations. I am looking for an efficient ...
0
votes
0answers
21 views

Calculating entropy of joint probability distribution observations of three variables in R

I have n observations of three variables, each variable being discrete with a different number of categories. For example, these are the first 10 observations: ...
0
votes
0answers
12 views

Quantifying information loss (KL divergence?) between a multivariate and a univariate discrete distribution

Let's say I have n discrete variables, n1, n2, ... n_n, each with a different scale, and another discrete variable ...
2
votes
0answers
42 views

Name/definition of $\int \log F(x) \cdot g(x)dx$?

We know that: $$-\int \log f(x) \cdot g(x)dx,$$ where $f$ and $g$ are density functions, is known as the cross entropy. Does $$-\int \log F(x) \cdot g(x)dx,$$ where $F$ is the cumulative ...
0
votes
0answers
11 views

Why isn't “mutual information” instead called “mutual entropy”, and “pointwise mutual information” instead called “mutual information”?

Unless I misunderstand something: The entropy of a variable is the average information that you get from it with each trial. The mutual information between two variables is the average amount by ...
1
vote
1answer
19 views

Proving that Shannon entropy is maximised for the uniform distribution

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further ...
0
votes
0answers
26 views

Shannon entropy as expected information content

I'm struggling with a form of viewing Shannon Entropy. Cover & Thomas say that entropy is the expected value of information content. So there is a random variable $X$ with distribution $P(X)$ and ...
0
votes
1answer
32 views

Why is entropy formula defined as what it is? [duplicate]

This is probably a naive question. Why is Entropy formula defined as the way what it is intead of more simpler formula? for example just P(x)*P(y) which I imagine can express uncertainty as well (0.5*...
0
votes
0answers
24 views

Am I using conditional entropy formula correctly?

I want to use the formula for a measure of complexity of a system: $$C(X) = H(X) - \sum_{x \in X}{H(x\mid X-x)}$$ where $x$ is a subpart of the system $X,$ and $H(X)$ is shannon entropy. Then I have ...
2
votes
1answer
45 views

Perplexity formula in the t-SNE paper vs. in the implementation

The perplexity formula in the official paper of t-SNE IS NOT the same as in its implementation. In the implementation (MATLAB): ...
1
vote
1answer
30 views

Is there a probability distribution function (PDF) that maximizes entropy for a given mode value?

I want to have a PDF that maximizes entropy for a given mode value. I searched in Maximum entropy probability distribution but here we have maximization done over a certain moment constraint.
0
votes
0answers
16 views

Information Gain property

Studying about information gain I found in the web (from the presentation of a lecture) that $IG(C|X) = IG(X|C)$ is it true? How I prove it?
0
votes
0answers
19 views

Conditional entropy of an outcome

Given three discrete random variables $X$, $Y$ and $Z$, the conditional entropy $$ H(X|Y,Z) = \sum_{X}\sum_{Y}\sum_{Z} p(x,y,z) \ \text{log}\frac{p(y,z)}{p(x,y,z)} $$ If I want to calculate the ...
0
votes
0answers
51 views

Can information entropy be computed on arbitrary set of non-negative numbers smaller than 1?

I have a set of non-negative numbers smaller than 1 which do not sum to 1. These numbers will be used in some weighted sum. I'm wondering in this case can we still measure the degree of concentration ...
0
votes
0answers
15 views

Is Entropy conserved under invertible (but probabilistic) mappings?

I know that under a one-to-one mapping, the entropy is conserved. This was answered very well here: Is entropy conserved under invertible mappings? I am having a doubt about this being applicable to ...
2
votes
1answer
130 views

Interpretation of spectral entropy of a timeseries

The tsfeatures package for R has an entropy() function. The vignette for the package describes it as: The spectral entropy is ...
3
votes
1answer
36 views

Measure of randomness that maps well to “hard to guess”?

It was recently pointed out to me that entropy does not necessarily map well to "hard to guess". E.g.: consider a distribution where there's a very high (say, 90%) chance you select a single, ...
0
votes
0answers
30 views

Entropy/Measure of Knowledge for probability intervals

I have a system which outputs probability interval tuples over the decision set $\{d_1, d_2, d_3\}$, such that a tuple $([\min_1, \max_1], [\min_2, \max_2], [\min_3, \max_3])$ indicates that $\min_i$ ...
0
votes
0answers
29 views

joint entropy and mutual information

If we have joint entropy of 10 bits between distribution A and B, while mutual information is 2 bits. Can we say there is 0.4% of useful communication between two distributions? Would this be proper ...
0
votes
0answers
27 views

What does Joint entropy really tells us?

I have two sequences of data A and B. A = [sunny, sunny, cloudy, rainy, sunny, sunny] with entropy 1.25 bits b = [hot, hot, cold, cold, cold, hot] with entropy 1 bits I have calculated joint ...
2
votes
0answers
36 views

Why is unbiased estimation from a sample only possible for certain properties?

I was thinking about finding some monotonic measure of entropy of a sample from a continuous distribution (2D, btw), but couldn't think of any such without making assumptions. Why is it that one can ...
0
votes
0answers
71 views

Conditional Entropy in the context of Gaussian Processes

I've got a question regarding the conditional entropy of a discrete random variable. According to this paper the conditional entropy of a Gaussian random variable conditioned on a set of variables can ...
0
votes
0answers
21 views

Multivariate Conditional Entropy as a test of correlation between random variables

I use the word columns to mean the data from which a random variable can be estimated. It is a sample of a random variable. I am working with $N$ columns of weakly correlated data. Furthermore, I ...
2
votes
2answers
52 views

Estimating Differential entropy from unbiased samples of a probability distribution

If I can get N unbiased samples $x$ from $p(x)$ how can I approximate the Differential entropy: $$H(X) = -\displaystyle\int_{x} p(x)\log p(x) dx$$ I'm not very knowledgeable in statistics so I'm not ...
0
votes
0answers
24 views

Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
1
vote
1answer
98 views

Multivariate conditional entropy

I would like to take data columns and compute the multivariate conditional entropy. For instance, suppose I have columns $A, B, C D, E$ and I want to compute the conditional entropy $H(E | A,B,C,D)$. ...
0
votes
0answers
18 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
115 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...
1
vote
1answer
98 views

How is the upper bound for Normalized Mutual Information determined?

Mutual information between two clusterings $A$ and $B$ can be calculated as: $$MI(A,B)=H(A)+H(B)-H(A,B)$$ In the 10th page of this paper it is stated that $MI(A,B)$ can vary in the range $[0,\min\{H(...
0
votes
0answers
35 views

Cross - entropy for two variables with different prob. distributions

Let us say that we have given two random variables with different prob. distributions: A = [0.1, 0, 0.5, ...] B = [0.3, 0.1, 0.03, ...] What should I do when I want to compute the reformulated cross-...
0
votes
0answers
110 views

Is there any intuition behind the writing Jensen-Shannon divergence based on the entropy?

I know that Jensen-Shannon divergence between two distribution $P$ and $Q$ can be written as follow: $JS(P||Q) = H(\frac{P+Q}{2}) - \frac{H(P)}{2} - \frac{H(Q)}{2}$ But is there any intuition behind ...
2
votes
1answer
93 views

Estimate Information Entropy from Moments

I hope I'm using the right terminology below. I have access to the moments statistics of a large sample. That is, I have $\sum(x)$, $\sum(x^2)$, ..., $\sum(x^k)$. I also have access to max and min, ...
0
votes
1answer
25 views

Small calculation in Max entropy

I am wondering where the -1 comes from in the derivatives with respect to the probability. For example, these notes. For the basic example with the uniform distribution. We want to minimze $[-\int ...
2
votes
1answer
45 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
0
votes
0answers
9 views

Correct number of grayscale levels when computing entropy of an image

I want to compute and to compare the entropy of images. Right now I am using images with 256 grayscale levels. But should the number of level not depend on the image size? For a really small image, ...
0
votes
0answers
22 views

Confusion about maximising vs minimising (relative) entropy terminology and methods

I'm studying rules of inference for updating from a prior probability distribution to a posterior. One method for doing this is by maximising entropy, subject to constraints. I'm reading papers like ...
1
vote
0answers
36 views

Is it possible to use Information Gain metric for CART?

I've been looking for an example of CART using Information Gain but haven't found one. This made me wonder if it was even possible, so I tried to train one manually (by hand) using the dataset below ...
0
votes
1answer
26 views

Question when using sklearn's DecisionTreeClassifier

sklearn's DecisionTreeClassifier is not behaving as I expected. From the following: ...
3
votes
0answers
111 views

How do we show the exponential distribution has maximal entropy on R+?

I’ve been looking around the internet, and having trouble finding a demonstration that the exponential distribution is maximal entropy on R+. I’d appreciate any points in the right direction.
0
votes
0answers
18 views

is there a measure for the roughness of a contour plot

There has to be a measure for the difference between "instantaneous" change of "energy" along a line in a space compared to averaged changed of energy along a line. I could take a smooth surface in ...
2
votes
1answer
143 views

Difference of notation between cross entropy and joint entropy

Although it is clear to me, how the two concepts differs, it has been difficult for me to find a notation that would make it clear, to which type of entropy we refer. From wikipedia, we can see that ...
0
votes
1answer
78 views

what is the meaning or intuition of entropy (from the point of view of reinforcement learning)

Can someone give an intuition of the concept 'entropy'? I am reading maximum entropy inverse reinforcement learning and I wanted to ask what the meaning intuition of 'entropy' is. I understand ...
0
votes
0answers
22 views

Does perplexity indicate how often my model gets it right?

This answer currently has the most votes and suggests The perplexity of whatever you're evaluating, on the data you're evaluating it on, sort of tells you "this thing is right about as often as an ...
0
votes
1answer
93 views

How to measure how “good” or accurate a probability distribution is? Entropy, variance or what?

How can one measure the accuracy of the probability distribution of, say, a physical magnitude? I know one good candidate is the entropy, which measures the amount of information one has about the ...
1
vote
0answers
131 views

How can I maximise binary cross entropy loss?

I have a multi-task learning model with two binary classification tasks. One part of the model creates a shared feature representation that is fed into two subnets in parallel. The loss function for ...
3
votes
1answer
132 views

Why is the cross-entropy always more than the entropy?

I understand intuitively why cross-entropy is always bigger. However, could someone show that mathematically?
0
votes
0answers
20 views

central limit theorem in term of entropy

The usual central theorem uses iid samples. Are there a generalizing theorem to non-iid samples, using the conditional entropy of each sample given previous ones ?
3
votes
1answer
211 views

Which result for normalized mutual information is correct?

I wanted to find the normalized mutual information to validate a clustering algorithm, but I've encountered two different values depending on the library I use. In Python: ...