Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
468
questions
1
vote
0answers
19 views
Does Entropy values enough creterion to determine the nature of any probability distribution?
Entropy is the measure of randomness of a process or variable and it can be defined as follows. for a random variable $X \in$ set $A$ :- $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i)) $, Now there are ...
1
vote
0answers
13 views
Decision trees: maximizing information gain vs. minimizing conditional entropy?
Information gain is defined as $$IG(T, a) = H(T) - H(T|a),$$ where $H(T|a)$ is the conditional entropy of $T$ given attribute $a$, and $H(T)$ is the prior entropy of our dataset before we test out the ...
3
votes
1answer
32 views
“Any meaning to the concept of ‘Self Mutual Information?”
**
“Any meaning to the concept of ‘Self Mutual Information?”
**
A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very ...
3
votes
1answer
50 views
Two meanings of entropy?
In the world of physics, entropy seems to mean something different from stats + information theory world.
So, I assumed that there are two definitions to the word entropy. Indeed, I looked up how ...
0
votes
0answers
6 views
State of the art library for estimating differential entropy from real data
I am looking for a state of the art library for estimating differential entropy from finite samples. In an ideal world, it would have the following features:
Work with real-valued multi-dimensional ...
0
votes
1answer
17 views
Transfer entropy and the sign of relationship
Transfer entropy (TE) measures the amount of directed transfer of information between two random processes, $X$ and $Y$. In other words, through the TE we can quantify the amount of information that ...
1
vote
0answers
7 views
Entropy in mole fraction distribution
I am studying how mole fractions are distributed in real life fuels (summation of mole fractions for any fuel=1, always). In this regard, I was trying to see if I can correlate my observations to ...
1
vote
1answer
29 views
Entropy of Joint Distribution of two IID Random Variables
I have two identical probability distributions $p_1(x) = p_2(x)\ \forall x$ and wish to construct a joint distribution between them, i.e. $p(x_1,x_2) = p_1(x_1)\cdot p_2(x_2)$. Supposing I know the ...
31
votes
2answers
3k views
Who is Gail Gasram?
Several places (1, 2, 3) quote someone named Gail Gasram as saying "Nothing is random, only uncertain" but a Google search turns up no info, just more places with this quote! Generally, it's in the ...
0
votes
0answers
5 views
Analyzing 1000s of time-stamped tweets. How to automatically identify spikes in terms?
I have about 30,000 tweets from a corporate customer service account, all harvested according to the Twitter TOC. Naturally, each message is time-stamped and not extremely long.
I'm trying to ...
1
vote
1answer
38 views
Information Entropy - Ambiguous Notation
I'm constantly confused by the ambiguous notation when discussing Entropy (H) and Mutual Information (I). For example, here's a formula from "Elements of Information Theory":
$$
I(X;Y|Z) = H(X|Z) − H(...
0
votes
0answers
42 views
Parameter distribution with less entropy is less likely to overfit
I am following the Tensorflow tutorial titled "Overfit and Underfit".
There is written that a simple model is less likely to overfit, where a simple model "is a model where the distribution of ...
2
votes
1answer
21 views
Shannon Index normalization with S distinct species per site
Given S different species and M sites . Where in every site species are unique (not repeated within the site,that is there are no 2 individuals from the same species in the same site , but can be ...
1
vote
1answer
20 views
Entropy: Given y=Ax, then H[y] = H[X] + ln |A|
The entropy of H[y] is given by:
$$H[y] = - \int p_y(\mathbf{y})~ \ln p_y(\mathbf{y})~ d\mathbf{y}$$
Now, suppose that I want to make a linear transformation of vector $\mathbf{y}$ to change the ...
0
votes
1answer
39 views
How entropy is calculated when a non categorical feature is available when using Decision tree or random forest algorithms?
How an entropy is calculated on non-categorical feature containing big amount of unique numbers?
Let me give you an example:
When we're having a categorical ...
0
votes
0answers
10 views
Transfer entropy for language domain
Is there a formulation of transfer entropy when the two random variables are instantiated from natural language? I am looking for a method to quantify the directional transfer of information from a ...
4
votes
2answers
63 views
Entropy: Proving information gain formula: h(x) = -log p(x)
We consider a discrete random variable X, and we want to know how much information we receive every time we observe the value of this random variable. We qualify this measure of information transfer ...
11
votes
4answers
174 views
Getting all answers correct by taking the same exam for fewest times
Rain never studies, so she is completely clueless during the midterm even though it consists of Yes/No questions only. Fortunately, Rain's professor allows her to re-take the same midterm as many ...
1
vote
0answers
13 views
How feature is selected to be a decision node after splitting the first root node using entropy in decisions trees?
In this article on how entropy is calculated and how the decision tree is built, the writer chose to split the sunny branch into ...
0
votes
0answers
17 views
How do I calculate the kth order entropy of a list of strings?
I want to measure the $k$th order entropy of a set of strings $s_1, s_2 \dots s_n$. And I am confused between two possible approaches that can be used.
Background
For a string $s\in\Sigma^n$, the ($...
0
votes
0answers
12 views
Proof help for multivariate mutual information as a sum of entropies
I'm following this paper on ICA and I got to equation (1) describing the multivariate mutual information contrast function as a sum of entropies.
$J(Y) = \int p(y_1,...y_D)log(\frac{p(y_1,...y_D)}{p(...
1
vote
0answers
10 views
Are there differentiable estimators for Entropy?
I have recently came across a paper on estimation of Information theoretic measure such as Entropy, Mutual Information and divergence, using a Mean Nearest Neighbor approach. Since, the estimator is ...
0
votes
0answers
46 views
Who to attribute information gain to?
I am writing a paper where I examine information gain specifically with regards to feature selection and am wondering what the proper reference should be. I have looked all over and I can't find a ...
2
votes
1answer
77 views
Joint entropy of multivariate normal distribution less than individual entropy under high correlation
Suppose we are calculating the joint entropy of a multivariate normal distribution with covariance matrix [1,0.99;0.99,1].
From the analytical solution, the joint entropy is 1.27;
However, the ...
0
votes
0answers
45 views
Does the limit of c exist or not
Let say m is dimension
$\exists$ $f(x)$ $f$ is density function
and
\begin{equation*}
f(x) = \frac{c(m,a,b)}{\|x\|^a\left(\log\frac{e}{\|x\|}\right)^{b}}\mathbf{1}_{\|x\|\leq 1} \geq 0
\end{...
4
votes
2answers
342 views
how to know this integral finite or infinite
In here, i want to show this entropy exist or not exist, namely i
should calculate the integral of $\int_0^c\frac{1}{x\log^2\frac{e}{x}}\frac{1}{2} \log\frac{e}{x}\,dx$. If the result is $ <\...
2
votes
1answer
80 views
How to derive this upper bound for the entropy of a bounded random variable? [closed]
A continuous random variable $ Z $ has a density that is $0$ except over the interval $[−A, +A].$ Show that the differential entropy $h(Z)$ is upper bounded by $1+\log_2 A.$
I am stuck in this ...
4
votes
0answers
32 views
How can a probability densitiy be estimated based on the maximum entropy principle, with constraints in the order statistics?
Let's say we are given a sample $\{ z_i \}$, $i=1,2,\ldots,N$, such that each value $z_i$ corresponds to the minimum of $n$ random variables $x$, i.e., $z = \min \{ x_1, x_2,\ldots,x_n \}$.
The ...
0
votes
0answers
26 views
Meaning of the derivative of cross entropy wrt $p(x)$
Lets define the cross entropy between 2 probability distributions $p(x)$ and $q(x)$ as
$$H(p,q) = -\sum p(x) \log{q(x)} $$
What would be the meaning of derivative of $H(p,q)$ when taking the ...
4
votes
1answer
105 views
How to understand units of information [duplicate]
In information theory, specifically in information content, I am struggling to conceptualise what the unit of measurement actually is. I have read quite a few similar questions and worked through the ...
0
votes
0answers
5 views
Statistical uncertainty on mutual overlap/ entropy of data
I have $N$ datapoints, obtained from a monte carlo simulation. Between each pair ${n,m}$ of these datapoints, we can compute an overlap $o(n,m)$ based on some analytic formulas, which is always ...
0
votes
0answers
9 views
Can we use joint entropy to split my nodes in a multi-objective decision tree?
Since for single objective decision tree (CART model), we can use information gain or difference in entropy (other than GINI index, Chi-Square) to split the nodes by choosing the attribute with ...
0
votes
0answers
34 views
entropy coefficient in A3C
I see that my policy entropy is decreasing very fast and converges to zero in no time, causing the policy to sample the same action again and again (which results in a sub-optimal behavior).
I think a ...
1
vote
1answer
40 views
Within sample and between sample categorical variation
I don't really know the correct terminology to ask this question well, so bear with me. I have categorical data with counts and I want a measure of how "diverse" or "spread out" the data is. Variance ...
0
votes
0answers
25 views
What is a reasonable value for KL Divergence
I am using scipy.stats.entropy for KL Divergence. When I create distributions to test I never receive values greater than .5. When I run the function on data I ...
0
votes
1answer
18 views
Joint entropy for estimating conditional mutual sorting information
I'm going through this paper,
and their notation is confusing me a little bit.
In section III (Coupling Measures),
they consider two time series $x_t$ and $y_t$,
as well as their corresponding so-...
0
votes
0answers
13 views
how entropy works in continuous distribution using the K nearest neighbour?
So calculating entropy in continuos distribution was developed by Kozachenko and Leonenko (1987).
I have seen one implementation here. however I have problem understanding the approach. basically, ...
1
vote
0answers
40 views
Entropy evolution while learning?
It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to
$$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows ...
0
votes
0answers
18 views
How to minimise and expectation with respect to a parameter?
Suppose that $X$ is a random variable with distribution $G$. Let $H(X;\theta)$ be a parametric function with $\theta \in \Theta \subset {\mathbb R}^p$. I want to maximize the function
$$\varphi(\...
4
votes
1answer
66 views
Expanding initial sample when the result isn't significant
This inspiring answer describes a variant of hypothesis, and I want to analysis its property further. Basically, it considers a two-sided test and interprets the $p$-value as a measure of how strong ...
0
votes
1answer
48 views
decision tree training: gini vs entropy vs precision vs recall
When training decision trees, the standard algorithms (e.g. ID3, C4.5, C5.0) use either the gini index or entropy to determine which node to add next. Only once the tree is built, and the ROC curve is ...
13
votes
1answer
835 views
How does entropy depend on location and scale?
The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals
$$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\...
0
votes
1answer
47 views
If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?
I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
1
vote
0answers
129 views
Non-negativity of interaction information for special trivariate case
Consider a discrete trivariate distribution $P(X_1, X_2, Y)$, which satisfies
$$
p(x_1, x_2, y) = \min( p(x_1,y), p(x_2,y) ),
$$
for all $x_1$ and $x_2$ for which $p(x_1, x_2) > 0$ and for all ...
0
votes
0answers
7 views
Can the calculation of entropy be generalized to an infinite series?
For a finite problem size $n$, it's straightforward to calculate the entropy of a random variable $X_n$ for the function I'm interested in. With the particular kinds of functions that I'm studying, ...
0
votes
1answer
80 views
Can Latent Class Analysis entropy be infinity? And what does it mean?
I recently ran a latent class analysis with polytomous outcome variables with the poLCA R package.
The model ran fine converged from 2 to 6 classes (the model converged with more classes but fit ...
1
vote
1answer
62 views
Products after sorting
Given two sets $\{a_1, \dots, a_N\}$ and $\{ b_1, \dots, b_N \}$ of positive numbers $0 < a_i, b_i < 1$, $i=1, \dots, N$, is it true that $\sum_i \min(a_i, b_i) \cdot \sum_i \max(a_i, b_i) \le \...
1
vote
1answer
26 views
Shannon's Entropy, manual vs. analytic results differ
Let's say that I have information string A, B, C, D, E.
All letters are equally probable (1/5). So Shannon's formula would give us.
...
1
vote
0answers
27 views
Finding causation in data with Conditional entropy
Conditional entropy is defined here. I was wondering about an algorithm to find causality between random variables. In particular, I want to calculate $H(X|Y)$ and $H(Y|X)$ and make a guess as to ...
1
vote
1answer
80 views
How is the formula for the entropy of the lognormal distribution derived?
Wikipedia gives the entropy of the lognormal distribution in nats as
$$\mu + \frac{1}{2} \ln(2\pi e \sigma^2)$$ Can anyone point me to a derivation of this formula?
