Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
496
questions
2
votes
0answers
15 views
Interpretation of variance of negative log probability
Given some discrete random variable $X$ with pmf $P(X)$ the expectation of its negative log probability $H(X) = E_{X}[-log(P(X))]$ is defined as the entropy which is an important statistical quantity. ...
1
vote
0answers
14 views
What is the motivation for using a second order approximation for KL divergence in this example? [closed]
Consider the KL divergence between two discrete distributions P and Q:
with probabilities $p_1,...,p_k$, and $q_1,...,q_k$
$I^{KL}(P;Q)= \sum^{k}_{i=1}p_ilog\frac{p_i}{q_i}$.
The notes then say ...
6
votes
1answer
533 views
Is my proof that relative entropy is never negative correct?
I wish to prove that relative entropy(Kullback-Liebler divergence) is always non-negative.
I.e. that $$I^{KL}(F;G)=E_F\left[\log\frac{f(X)}{g(X)}\right]\geq0$$
where F,G are two different probability ...
2
votes
0answers
26 views
Does the entropy decreases with new observations when doing bayesian probability estimation?
I premise I don't know much information theory, except for some definition. I have this doubt, I'm not sure that makes sense.
Suppose I'm estimating a discrete distribution, in a Bayesian setting, ...
0
votes
0answers
14 views
Relationship between entropy of posterior with entropy of prior and likelihood function?
Is there any formula in literature that shows connection between entropy of the posterior $H[ p(x|y)]$ with entropy of likelihood function $H[p(y|x)]$ and entropy of the prior $H[p(x)]$?
Here $y$ is ...
0
votes
1answer
29 views
KL-Divergence and Entropy for marginals
I am going through this paper, where the following claim is unproven (page 3, after the first equality):
Let $r,c \in \mathbb{R}_+^d$ be discrete probability histograms, and $P \in \mathbb{R}_+^{d,d}$...
0
votes
0answers
14 views
The monotonicity of entropy operator
Define the entropy operator of a distribution as $\mathbb{H}(p) = -\int p \log p$, how does the entropy change for distributions that are proportional to the powers of $p$?
For example, define $\...
5
votes
2answers
163 views
Relationship between KL divergence and entropy
In Bishop's Pattern Recognition and Machine Learning, there is a small discussion in section 10.1.2 of the difference between minimizing $D_{KL}(p \:||\: q)$ and $D_{KL}(q \:||\: p)$ with respect to ...
4
votes
1answer
72 views
Why is Kullback Leibler Divergence always positive?
I know there have been mathematical treatments of this question on here. What I'd like help with is my intuitive understanding though. Take the example given on Wikipedia:
$$\begin{array}{|c|c|c|c|} \...
3
votes
1answer
40 views
Are Dispersion and Entropy Related?
So for a system, a dispersion is the measure of how the population deviates from the mean. Intuitively the more the dispersion in the system the more the disorder i.e. entropy. A jar of marbles with ...
2
votes
1answer
23 views
Entropy based measure of explained variation
I have an observable $Y$ which is a function of some set of variables $\mathcal{X}=\left\{ X_{n}\right\} _{n=1}^{N}$. Now $Y$ is a deterministic function of the $X_n$s, but conditioned on only a ...
4
votes
2answers
125 views
Posterior entropy behaviour for Poisson model
Suppose $Y \sim Poisson(k \theta)$ is observed (number of events),
where $k>0$ is a constant (length of observation time)
and $\theta$ is an unknown parameter (event rate) with some prior ...
1
vote
0answers
23 views
Test for random distribution of values in a sequence
I'm looking to measure the degree to which values in a column or list are ordered or pseudo-randomly aligned. In essence I'm trying to distinguish the following scenarios.
Say we have a "sex" column ...
5
votes
4answers
85 views
How is the Herfindahl-Hirschman index different from entropy?
The HerfindahlāHirschman Index (HHI) is a concentration measure defined as
$$H = \sum_i p_i^2,$$
where $p_i$ is the market share of firm $i$. It's maximized when one firm has a monopoly and minimized ...
5
votes
1answer
100 views
Informational value of R squared and correlation? [closed]
Taleb has previously undermined the typical interpretation of correlation with regards to the informational value it carries, showing how the uncertainty is reduced in a non-linear fashion.
With ...
1
vote
2answers
38 views
Can you compare the entropy between instances even if they have different n_labels
Consider a dataset where each instance was annotated by a number of participants that had to category the instance (a, b, or c). To calculate some form of agreement between these participants, we ...
0
votes
0answers
7 views
Cross validation of sample entropy parameters found using grid search
I have several time-series data spanning multiple weeks and I compute their complexity using sample entropy, which I then, in turn, correlate with a certain numeric value relevant to the particular ...
2
votes
1answer
46 views
Shannon Information | Understanding from a Microstate Perspective
So Shannon's information is a way to quantify "distinct knowledge" by means of combination of microstates. So say 1 bit of information in binary system conveys 2 sets of information due to two ...
3
votes
2answers
99 views
Conflicting Definition of Information in Statistics | Fisher Vs Shannon
The notion of information as per Shannon is that if the probability of RV is close to 1, there is little information in that RV because we are more certain about the outcome of the RV so there is ...
4
votes
0answers
40 views
D-Optimal Criteria Vs Differential Shannon Entropy
I want to understand how minimizing the determinant of the information matrix is equivalent to maximizing the differential Shannon entropy? A similar question was posted in Math SE but hasnt been ...
0
votes
0answers
21 views
Is there a connection between the binomial pmf and the formula for entropy? [duplicate]
Many times when two formulas "look" the same, there is some interesting mathematical result linking them. Both the log binomial likelihood and the entropy formula kind of "look" the same, in that they ...
0
votes
0answers
19 views
How to find the characteristic function of a function related with Shannon entropy?
A random variable X is distributed with a known probability distribution $p(x)$. Suppose that $x$ is sampled in an independent and identically distributed process and with the results $\vec{x}=(x_1,...
0
votes
0answers
21 views
Meaning of entropy for a multivariate data set
In an earlier question, I asked, āAny meaning to the concept of āSelf Mutual Information?ā and got a great answer - thanks.
Now, this begs another question/set of questions:
What does the concept of ...
0
votes
2answers
127 views
How do I prove conditional entropy is a good measure of information?
This question is a follow-up of Does "expected entropy" make sense?, which you don't have to read as I'll reproduce the relevant parts. Let's begin with the statement of the problem
A student has ...
1
vote
1answer
33 views
Understanding Multidimensional Mutual Information
Given random variables $\vec{x}, \vec{y} \in \mathbb{R}^n$, and the mutual information, defined as
$I(\vec{x} : \vec{y}) = H(\vec{x}) + H(\vec{y}) - H(\vec{x}, \vec{y})$
is it true that
$I(\vec{x}: ...
0
votes
0answers
8 views
Sequential Gaussian Bayesian inference steps and entropy
Consider that I am multiplying a Gaussian prior with a variance of $\sigma^2_{prior}=1$ and a likelihood of $\sigma^2_{l}=1$. Their means are for the moment not interesting.
The variance of a product ...
3
votes
1answer
64 views
Intuition for the uniform distribution having the maximum entropy
I saw the following explanation for Entropy in probability:
(Entropy). The surprise of learning that an event with probability $p$ happened is defined as $\log_2(1/p)$, measured in a unit called ...
0
votes
1answer
39 views
Do frequentists rely less on entropy-based methods than Bayesian or Machine Learners? [closed]
From what I heard many times, Bayesian and Machine Learning people use entropy based methods.
Do frequentists use entropy less?
6
votes
1answer
56 views
Does Shannon Entropy uniquely characterise distribution function $f$?
If I have a distribution $f(x)$ over the real line where the support is the whole line, does the Shannon Entropy uniquely characterise $f$? I.e., do we have $H(f) = H(f^*)$ implies $f = f^*$? (The ...
0
votes
0answers
31 views
Confusion about conditional entropy when conditioning on another event
Suppose $X$ and $Y$ are discrete random variables. I would like to relate the entropy $H[X \mid Y]$ to the same entropy conditioned on the additional event $y>0$.
My reasoning is as follows:
\...
4
votes
2answers
61 views
Why do typical sequences have probabilities $\sim2^{-nH(p)}$?
I've been reading a bit about typical sequences (in particular from these notes (pdf alert), pages 3 and 4). Let us focus on the case of binary sequences for simplicity. As far as I understand the ...
0
votes
0answers
8 views
Entropy of log normalized data
Is it ok to calculate the entropy of log(2) normalized data? Or should I make to transformation to a linear scale
Some more background - Iām dealing with SingleCell data which is by definition is very ...
0
votes
0answers
11 views
Can entropy be used to quantify how ordered a sequence is?
I'm looking at the order that states hold their presidential primaries and want to quantify the degree to which the population -- expressed as a % of the voting eligible national population -- is non-...
0
votes
1answer
20 views
How does the triangle inequality have anything to do with relative entropy?
I am reading Elements of information theory by Thomas M. Cover, and Joy A. Thomas second edition. and on page 19, chapter 2, section 3, it says:
I am confused by the random mentioning of triangles.
...
0
votes
0answers
15 views
Maximum entropy correlated prior
This question arose from trying to think about Kaggle's Titanic passenger survival prediction challenge from a probabilistic perspective.
Say I'm trying to model a simplified version where the only ...
1
vote
1answer
49 views
What is the difference between Gini index and Gini coefficient?
I am building a decision tree from scratch. I have been using entropy so far (calculated this way):
...
0
votes
0answers
45 views
JensenāShannon divergence as a distance measure between nonprobabilistic objects
We are working on an optimization problem. The objective function involves distance between data points. We tried a wide variety of distance measures and found the entropy-based measures, especially ...
1
vote
0answers
20 views
Does Entropy values enough creterion to determine the nature of any probability distribution?
Entropy is the measure of randomness of a process or variable and it can be defined as follows. for a random variable $X \in$ set $A$ :- $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i)) $, Now there are ...
2
votes
0answers
43 views
Decision trees: maximizing information gain vs. minimizing conditional entropy?
Information gain is defined as $$IG(T, a) = H(T) - H(T|a),$$ where $H(T|a)$ is the conditional entropy of $T$ given attribute $a$, and $H(T)$ is the prior entropy of our dataset before we test out the ...
2
votes
1answer
53 views
āAny meaning to the concept of āSelf Mutual Information?ā
**
āAny meaning to the concept of āSelf Mutual Information?ā
**
A blog post entitled, āEntropy in machine learningā dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very ...
3
votes
1answer
64 views
Two meanings of entropy?
In the world of physics, entropy seems to mean something different from stats + information theory world.
So, I assumed that there are two definitions to the word entropy. Indeed, I looked up how ...
0
votes
0answers
8 views
State of the art library for estimating differential entropy from real data
I am looking for a state of the art library for estimating differential entropy from finite samples. In an ideal world, it would have the following features:
Work with real-valued multi-dimensional ...
0
votes
1answer
21 views
Transfer entropy and the sign of relationship
Transfer entropy (TE) measures the amount of directed transfer of information between two random processes, $X$ and $Y$. In other words, through the TE we can quantify the amount of information that ...
1
vote
0answers
11 views
Entropy in mole fraction distribution
I am studying how mole fractions are distributed in real life fuels (summation of mole fractions for any fuel=1, always). In this regard, I was trying to see if I can correlate my observations to ...
1
vote
1answer
42 views
Entropy of Joint Distribution of two IID Random Variables
I have two identical probability distributions $p_1(x) = p_2(x)\ \forall x$ and wish to construct a joint distribution between them, i.e. $p(x_1,x_2) = p_1(x_1)\cdot p_2(x_2)$. Supposing I know the ...
30
votes
2answers
3k views
Who is Gail Gasram?
Several places (1, 2, 3) quote someone named Gail Gasram as saying "Nothing is random, only uncertain" but a Google search turns up no info, just more places with this quote! Generally, it's in the ...
0
votes
0answers
6 views
Analyzing 1000s of time-stamped tweets. How to automatically identify spikes in terms?
I have about 30,000 tweets from a corporate customer service account, all harvested according to the Twitter TOC. Naturally, each message is time-stamped and not extremely long.
I'm trying to ...
1
vote
1answer
55 views
Information Entropy - Ambiguous Notation
I'm constantly confused by the ambiguous notation when discussing Entropy (H) and Mutual Information (I). For example, here's a formula from "Elements of Information Theory":
$$
I(X;Y|Z) = H(X|Z) ā H(...
0
votes
0answers
46 views
Parameter distribution with less entropy is less likely to overfit
I am following the Tensorflow tutorial titled "Overfit and Underfit".
There is written that a simple model is less likely to overfit, where a simple model "is a model where the distribution of ...
2
votes
1answer
36 views
Shannon Index normalization with S distinct species per site
Given S different species and M sites . Where in every site species are unique (not repeated within the site,that is there are no 2 individuals from the same species in the same site , but can be ...
