Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

Filter by
Sorted by
Tagged with
2
votes
0answers
15 views

Interpretation of variance of negative log probability

Given some discrete random variable $X$ with pmf $P(X)$ the expectation of its negative log probability $H(X) = E_{X}[-log(P(X))]$ is defined as the entropy which is an important statistical quantity. ...
1
vote
0answers
14 views

What is the motivation for using a second order approximation for KL divergence in this example? [closed]

Consider the KL divergence between two discrete distributions P and Q: with probabilities $p_1,...,p_k$, and $q_1,...,q_k$ $I^{KL}(P;Q)= \sum^{k}_{i=1}p_ilog\frac{p_i}{q_i}$. The notes then say ...
6
votes
1answer
533 views

Is my proof that relative entropy is never negative correct?

I wish to prove that relative entropy(Kullback-Liebler divergence) is always non-negative. I.e. that $$I^{KL}(F;G)=E_F\left[\log\frac{f(X)}{g(X)}\right]\geq0$$ where F,G are two different probability ...
2
votes
0answers
26 views

Does the entropy decreases with new observations when doing bayesian probability estimation?

I premise I don't know much information theory, except for some definition. I have this doubt, I'm not sure that makes sense. Suppose I'm estimating a discrete distribution, in a Bayesian setting, ...
0
votes
0answers
14 views

Relationship between entropy of posterior with entropy of prior and likelihood function?

Is there any formula in literature that shows connection between entropy of the posterior $H[ p(x|y)]$ with entropy of likelihood function $H[p(y|x)]$ and entropy of the prior $H[p(x)]$? Here $y$ is ...
0
votes
1answer
29 views

KL-Divergence and Entropy for marginals

I am going through this paper, where the following claim is unproven (page 3, after the first equality): Let $r,c \in \mathbb{R}_+^d$ be discrete probability histograms, and $P \in \mathbb{R}_+^{d,d}$...
0
votes
0answers
14 views

The monotonicity of entropy operator

Define the entropy operator of a distribution as $\mathbb{H}(p) = -\int p \log p$, how does the entropy change for distributions that are proportional to the powers of $p$? For example, define $\...
5
votes
2answers
163 views

Relationship between KL divergence and entropy

In Bishop's Pattern Recognition and Machine Learning, there is a small discussion in section 10.1.2 of the difference between minimizing $D_{KL}(p \:||\: q)$ and $D_{KL}(q \:||\: p)$ with respect to ...
4
votes
1answer
72 views

Why is Kullback Leibler Divergence always positive?

I know there have been mathematical treatments of this question on here. What I'd like help with is my intuitive understanding though. Take the example given on Wikipedia: $$\begin{array}{|c|c|c|c|} \...
3
votes
1answer
40 views

Are Dispersion and Entropy Related?

So for a system, a dispersion is the measure of how the population deviates from the mean. Intuitively the more the dispersion in the system the more the disorder i.e. entropy. A jar of marbles with ...
2
votes
1answer
23 views

Entropy based measure of explained variation

I have an observable $Y$ which is a function of some set of variables $\mathcal{X}=\left\{ X_{n}\right\} _{n=1}^{N}$. Now $Y$ is a deterministic function of the $X_n$s, but conditioned on only a ...
4
votes
2answers
125 views

Posterior entropy behaviour for Poisson model

Suppose $Y \sim Poisson(k \theta)$ is observed (number of events), where $k>0$ is a constant (length of observation time) and $\theta$ is an unknown parameter (event rate) with some prior ...
1
vote
0answers
23 views

Test for random distribution of values in a sequence

I'm looking to measure the degree to which values in a column or list are ordered or pseudo-randomly aligned. In essence I'm trying to distinguish the following scenarios. Say we have a "sex" column ...
5
votes
4answers
85 views

How is the Herfindahl-Hirschman index different from entropy?

The Herfindahl–Hirschman Index (HHI) is a concentration measure defined as $$H = \sum_i p_i^2,$$ where $p_i$ is the market share of firm $i$. It's maximized when one firm has a monopoly and minimized ...
5
votes
1answer
100 views

Informational value of R squared and correlation? [closed]

Taleb has previously undermined the typical interpretation of correlation with regards to the informational value it carries, showing how the uncertainty is reduced in a non-linear fashion. With ...
1
vote
2answers
38 views

Can you compare the entropy between instances even if they have different n_labels

Consider a dataset where each instance was annotated by a number of participants that had to category the instance (a, b, or c). To calculate some form of agreement between these participants, we ...
0
votes
0answers
7 views

Cross validation of sample entropy parameters found using grid search

I have several time-series data spanning multiple weeks and I compute their complexity using sample entropy, which I then, in turn, correlate with a certain numeric value relevant to the particular ...
2
votes
1answer
46 views

Shannon Information | Understanding from a Microstate Perspective

So Shannon's information is a way to quantify "distinct knowledge" by means of combination of microstates. So say 1 bit of information in binary system conveys 2 sets of information due to two ...
3
votes
2answers
99 views

Conflicting Definition of Information in Statistics | Fisher Vs Shannon

The notion of information as per Shannon is that if the probability of RV is close to 1, there is little information in that RV because we are more certain about the outcome of the RV so there is ...
4
votes
0answers
40 views

D-Optimal Criteria Vs Differential Shannon Entropy

I want to understand how minimizing the determinant of the information matrix is equivalent to maximizing the differential Shannon entropy? A similar question was posted in Math SE but hasnt been ...
0
votes
0answers
21 views

Is there a connection between the binomial pmf and the formula for entropy? [duplicate]

Many times when two formulas "look" the same, there is some interesting mathematical result linking them. Both the log binomial likelihood and the entropy formula kind of "look" the same, in that they ...
0
votes
0answers
19 views

How to find the characteristic function of a function related with Shannon entropy?

A random variable X is distributed with a known probability distribution $p(x)$. Suppose that $x$ is sampled in an independent and identically distributed process and with the results $\vec{x}=(x_1,...
0
votes
0answers
21 views

Meaning of entropy for a multivariate data set

In an earlier question, I asked, “Any meaning to the concept of ‘Self Mutual Information?” and got a great answer - thanks. Now, this begs another question/set of questions: What does the concept of ...
0
votes
2answers
127 views

How do I prove conditional entropy is a good measure of information?

This question is a follow-up of Does "expected entropy" make sense?, which you don't have to read as I'll reproduce the relevant parts. Let's begin with the statement of the problem A student has ...
1
vote
1answer
33 views

Understanding Multidimensional Mutual Information

Given random variables $\vec{x}, \vec{y} \in \mathbb{R}^n$, and the mutual information, defined as $I(\vec{x} : \vec{y}) = H(\vec{x}) + H(\vec{y}) - H(\vec{x}, \vec{y})$ is it true that $I(\vec{x}: ...
0
votes
0answers
8 views

Sequential Gaussian Bayesian inference steps and entropy

Consider that I am multiplying a Gaussian prior with a variance of $\sigma^2_{prior}=1$ and a likelihood of $\sigma^2_{l}=1$. Their means are for the moment not interesting. The variance of a product ...
3
votes
1answer
64 views

Intuition for the uniform distribution having the maximum entropy

I saw the following explanation for Entropy in probability: (Entropy). The surprise of learning that an event with probability $p$ happened is defined as $\log_2(1/p)$, measured in a unit called ...
0
votes
1answer
39 views

Do frequentists rely less on entropy-based methods than Bayesian or Machine Learners? [closed]

From what I heard many times, Bayesian and Machine Learning people use entropy based methods. Do frequentists use entropy less?
6
votes
1answer
56 views

Does Shannon Entropy uniquely characterise distribution function $f$?

If I have a distribution $f(x)$ over the real line where the support is the whole line, does the Shannon Entropy uniquely characterise $f$? I.e., do we have $H(f) = H(f^*)$ implies $f = f^*$? (The ...
0
votes
0answers
31 views

Confusion about conditional entropy when conditioning on another event

Suppose $X$ and $Y$ are discrete random variables. I would like to relate the entropy $H[X \mid Y]$ to the same entropy conditioned on the additional event $y>0$. My reasoning is as follows: \...
4
votes
2answers
61 views

Why do typical sequences have probabilities $\sim2^{-nH(p)}$?

I've been reading a bit about typical sequences (in particular from these notes (pdf alert), pages 3 and 4). Let us focus on the case of binary sequences for simplicity. As far as I understand the ...
0
votes
0answers
8 views

Entropy of log normalized data

Is it ok to calculate the entropy of log(2) normalized data? Or should I make to transformation to a linear scale Some more background - I’m dealing with SingleCell data which is by definition is very ...
0
votes
0answers
11 views

Can entropy be used to quantify how ordered a sequence is?

I'm looking at the order that states hold their presidential primaries and want to quantify the degree to which the population -- expressed as a % of the voting eligible national population -- is non-...
0
votes
1answer
20 views

How does the triangle inequality have anything to do with relative entropy?

I am reading Elements of information theory by Thomas M. Cover, and Joy A. Thomas second edition. and on page 19, chapter 2, section 3, it says: I am confused by the random mentioning of triangles. ...
0
votes
0answers
15 views

Maximum entropy correlated prior

This question arose from trying to think about Kaggle's Titanic passenger survival prediction challenge from a probabilistic perspective. Say I'm trying to model a simplified version where the only ...
1
vote
1answer
49 views

What is the difference between Gini index and Gini coefficient?

I am building a decision tree from scratch. I have been using entropy so far (calculated this way): ...
0
votes
0answers
45 views

Jensen–Shannon divergence as a distance measure between nonprobabilistic objects

We are working on an optimization problem. The objective function involves distance between data points. We tried a wide variety of distance measures and found the entropy-based measures, especially ...
1
vote
0answers
20 views

Does Entropy values enough creterion to determine the nature of any probability distribution?

Entropy is the measure of randomness of a process or variable and it can be defined as follows. for a random variable $X \in$ set $A$ :- $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i)) $, Now there are ...
2
votes
0answers
43 views

Decision trees: maximizing information gain vs. minimizing conditional entropy?

Information gain is defined as $$IG(T, a) = H(T) - H(T|a),$$ where $H(T|a)$ is the conditional entropy of $T$ given attribute $a$, and $H(T)$ is the prior entropy of our dataset before we test out the ...
2
votes
1answer
53 views

“Any meaning to the concept of ‘Self Mutual Information?”

** “Any meaning to the concept of ‘Self Mutual Information?” ** A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very ...
3
votes
1answer
64 views

Two meanings of entropy?

In the world of physics, entropy seems to mean something different from stats + information theory world. So, I assumed that there are two definitions to the word entropy. Indeed, I looked up how ...
0
votes
0answers
8 views

State of the art library for estimating differential entropy from real data

I am looking for a state of the art library for estimating differential entropy from finite samples. In an ideal world, it would have the following features: Work with real-valued multi-dimensional ...
0
votes
1answer
21 views

Transfer entropy and the sign of relationship

Transfer entropy (TE) measures the amount of directed transfer of information between two random processes, $X$ and $Y$. In other words, through the TE we can quantify the amount of information that ...
1
vote
0answers
11 views

Entropy in mole fraction distribution

I am studying how mole fractions are distributed in real life fuels (summation of mole fractions for any fuel=1, always). In this regard, I was trying to see if I can correlate my observations to ...
1
vote
1answer
42 views

Entropy of Joint Distribution of two IID Random Variables

I have two identical probability distributions $p_1(x) = p_2(x)\ \forall x$ and wish to construct a joint distribution between them, i.e. $p(x_1,x_2) = p_1(x_1)\cdot p_2(x_2)$. Supposing I know the ...
30
votes
2answers
3k views

Who is Gail Gasram?

Several places (1, 2, 3) quote someone named Gail Gasram as saying "Nothing is random, only uncertain" but a Google search turns up no info, just more places with this quote! Generally, it's in the ...
0
votes
0answers
6 views

Analyzing 1000s of time-stamped tweets. How to automatically identify spikes in terms?

I have about 30,000 tweets from a corporate customer service account, all harvested according to the Twitter TOC. Naturally, each message is time-stamped and not extremely long. I'm trying to ...
1
vote
1answer
55 views

Information Entropy - Ambiguous Notation

I'm constantly confused by the ambiguous notation when discussing Entropy (H) and Mutual Information (I). For example, here's a formula from "Elements of Information Theory": $$ I(X;Y|Z) = H(X|Z) − H(...
0
votes
0answers
46 views

Parameter distribution with less entropy is less likely to overfit

I am following the Tensorflow tutorial titled "Overfit and Underfit". There is written that a simple model is less likely to overfit, where a simple model "is a model where the distribution of ...
2
votes
1answer
36 views

Shannon Index normalization with S distinct species per site

Given S different species and M sites . Where in every site species are unique (not repeated within the site,that is there are no 2 individuals from the same species in the same site , but can be ...

1
2 3 4 5
10