Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
463
questions
1
vote
1answer
13 views
Entropy: Given y=Ax, then H[y] = H[X] + ln |A|
The entropy of H[y] is given by:
$$H[y] = - \int p_y(\mathbf{y})~ \ln p_y(\mathbf{y})~ d\mathbf{y}$$
Now, suppose that I want to make a linear transformation of vector $\mathbf{y}$ to change the ...
0
votes
1answer
14 views
How entropy is calculated when a non categorical feature is available when using Decision tree or random forest algorithms?
How an entropy is calculated on non-categorical feature containing big amount of unique numbers?
Let me give you an example:
When we're having a categorical ...
0
votes
0answers
7 views
Transfer entropy for language domain
Is there a formulation of transfer entropy when the two random variables are instantiated from natural language? I am looking for a method to quantify the directional transfer of information from a ...
4
votes
2answers
55 views
Entropy: Proving information gain formula: h(x) = -log p(x)
We consider a discrete random variable X, and we want to know how much information we receive every time we observe the value of this random variable. We qualify this measure of information transfer ...
9
votes
3answers
122 views
+100
Getting all answers correct by taking the same exam for fewest times
Rain never studies, so she is completely clueless during the midterm even though it consists of Yes/No questions only. Fortunately, Rain's professor allows her to re-take the same midterm as many ...
0
votes
0answers
4 views
How to display categorical values on export tree image of decision tree classifier? [migrated]
I am trying to export the decision tree as an image with the original labels of all categorical fields.
The current data I have is like so:
I transformed the categorical features into numerical:
<...
1
vote
0answers
10 views
How feature is selected to be a decision node after splitting the first root node using entropy in decisions trees?
In this article on how entropy is calculated and how the decision tree is built, the writer chose to split the sunny branch into ...
0
votes
0answers
10 views
How do I calculate the kth order entropy of a list of strings?
I want to measure the $k$th order entropy of a set of strings $s_1, s_2 \dots s_n$. And I am confused between two possible approaches that can be used.
Background
For a string $s\in\Sigma^n$, the ($...
0
votes
0answers
8 views
Proof help for multivariate mutual information as a sum of entropies
I'm following this paper on ICA and I got to equation (1) describing the multivariate mutual information contrast function as a sum of entropies.
$J(Y) = \int p(y_1,...y_D)log(\frac{p(y_1,...y_D)}{p(...
1
vote
0answers
10 views
Are there differentiable estimators for Entropy?
I have recently came across a paper on estimation of Information theoretic measure such as Entropy, Mutual Information and divergence, using a Mean Nearest Neighbor approach. Since, the estimator is ...
-1
votes
0answers
9 views
Neighborhood of the entropy for a given neighborhood of probabilities
Assume $M$ is a real-valued discrete random varibale taking values $m_i,\ i=1,\dots,m$ with uniform distribution.
Given $|p(m_i|y)-\frac{1}{m}|<\epsilon$ is there any $\eta$ such that $|H(M)-H(M|y)...
0
votes
0answers
42 views
Who to attribute information gain to?
I am writing a paper where I examine information gain specifically with regards to feature selection and am wondering what the proper reference should be. I have looked all over and I can't find a ...
2
votes
1answer
35 views
Joint entropy of multivariate normal distribution less than individual entropy under high correlation
Suppose we are calculating the joint entropy of a multivariate normal distribution with covariance matrix [1,0.99;0.99,1].
From the analytical solution, the joint entropy is 1.27;
However, the ...
0
votes
0answers
44 views
Does the limit of c exist or not
Let say m is dimension
$\exists$ $f(x)$ $f$ is density function
and
\begin{equation*}
f(x) = \frac{c(m,a,b)}{\|x\|^a\left(\log\frac{e}{\|x\|}\right)^{b}}\mathbf{1}_{\|x\|\leq 1} \geq 0
\end{...
3
votes
2answers
331 views
how to know this integral finite or infinite
In here, i want to show this entropy exist or not exist, namely i
should calculate the integral of $\int_0^c\frac{1}{x\log^2\frac{e}{x}}\frac{1}{2} \log\frac{e}{x}\,dx$. If the result is $ <\...
2
votes
1answer
40 views
How to derive this upper bound for the entropy of a bounded random variable? [closed]
A continuous random variable $ Z $ has a density that is $0$ except over the interval $[−A, +A].$ Show that the differential entropy $h(Z)$ is upper bounded by $1+\log_2 A.$
I am stuck in this ...
4
votes
0answers
30 views
How can a probability densitiy be estimated based on the maximum entropy principle, with constraints in the order statistics?
Let's say we are given a sample $\{ z_i \}$, $i=1,2,\ldots,N$, such that each value $z_i$ corresponds to the minimum of $n$ random variables $x$, i.e., $z = \min \{ x_1, x_2,\ldots,x_n \}$.
The ...
0
votes
0answers
25 views
Meaning of the derivative of cross entropy wrt $p(x)$
Lets define the cross entropy between 2 probability distributions $p(x)$ and $q(x)$ as
$$H(p,q) = -\sum p(x) \log{q(x)} $$
What would be the meaning of derivative of $H(p,q)$ when taking the ...
4
votes
1answer
100 views
How to understand units of information [duplicate]
In information theory, specifically in information content, I am struggling to conceptualise what the unit of measurement actually is. I have read quite a few similar questions and worked through the ...
0
votes
0answers
5 views
Statistical uncertainty on mutual overlap/ entropy of data
I have $N$ datapoints, obtained from a monte carlo simulation. Between each pair ${n,m}$ of these datapoints, we can compute an overlap $o(n,m)$ based on some analytic formulas, which is always ...
0
votes
0answers
9 views
Can we use joint entropy to split my nodes in a multi-objective decision tree?
Since for single objective decision tree (CART model), we can use information gain or difference in entropy (other than GINI index, Chi-Square) to split the nodes by choosing the attribute with ...
0
votes
0answers
12 views
entropy coefficient in A3C
I see that my policy entropy is decreasing very fast and converges to zero in no time, causing the policy to sample the same action again and again (which results in a sub-optimal behavior).
I think a ...
1
vote
1answer
33 views
Within sample and between sample categorical variation
I don't really know the correct terminology to ask this question well, so bear with me. I have categorical data with counts and I want a measure of how "diverse" or "spread out" the data is. Variance ...
0
votes
0answers
17 views
What is a reasonable value for KL Divergence
I am using scipy.stats.entropy for KL Divergence. When I create distributions to test I never receive values greater than .5. When I run the function on data I ...
0
votes
1answer
16 views
Joint entropy for estimating conditional mutual sorting information
I'm going through this paper,
and their notation is confusing me a little bit.
In section III (Coupling Measures),
they consider two time series $x_t$ and $y_t$,
as well as their corresponding so-...
0
votes
0answers
13 views
how entropy works in continuous distribution using the K nearest neighbour?
So calculating entropy in continuos distribution was developed by Kozachenko and Leonenko (1987).
I have seen one implementation here. however I have problem understanding the approach. basically, ...
1
vote
0answers
35 views
Entropy evolution while learning?
It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to
$$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows ...
0
votes
0answers
16 views
How to minimise and expectation with respect to a parameter?
Suppose that $X$ is a random variable with distribution $G$. Let $H(X;\theta)$ be a parametric function with $\theta \in \Theta \subset {\mathbb R}^p$. I want to maximize the function
$$\varphi(\...
4
votes
1answer
62 views
Expanding initial sample when the result isn't significant
This inspiring answer describes a variant of hypothesis, and I want to analysis its property further. Basically, it considers a two-sided test and interprets the $p$-value as a measure of how strong ...
0
votes
1answer
32 views
decision tree training: gini vs entropy vs precision vs recall
When training decision trees, the standard algorithms (e.g. ID3, C4.5, C5.0) use either the gini index or entropy to determine which node to add next. Only once the tree is built, and the ROC curve is ...
12
votes
1answer
804 views
How does entropy depend on location and scale?
The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals
$$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\...
0
votes
1answer
42 views
If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?
I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
1
vote
0answers
128 views
Non-negativity of interaction information for special trivariate case
Consider a discrete trivariate distribution $P(X_1, X_2, Y)$, which satisfies
$$
p(x_1, x_2, y) = \min( p(x_1,y), p(x_2,y) ),
$$
for all $x_1$ and $x_2$ for which $p(x_1, x_2) > 0$ and for all ...
0
votes
0answers
7 views
Can the calculation of entropy be generalized to an infinite series?
For a finite problem size $n$, it's straightforward to calculate the entropy of a random variable $X_n$ for the function I'm interested in. With the particular kinds of functions that I'm studying, ...
0
votes
1answer
38 views
Can Latent Class Analysis entropy be infinity? And what does it mean?
I recently ran a latent class analysis with polytomous outcome variables with the poLCA R package.
The model ran fine converged from 2 to 6 classes (the model converged with more classes but fit ...
1
vote
1answer
62 views
Products after sorting
Given two sets $\{a_1, \dots, a_N\}$ and $\{ b_1, \dots, b_N \}$ of positive numbers $0 < a_i, b_i < 1$, $i=1, \dots, N$, is it true that $\sum_i \min(a_i, b_i) \cdot \sum_i \max(a_i, b_i) \le \...
1
vote
1answer
26 views
Shannon's Entropy, manual vs. analytic results differ
Let's say that I have information string A, B, C, D, E.
All letters are equally probable (1/5). So Shannon's formula would give us.
...
0
votes
0answers
21 views
Finding causation in data with Conditional entropy
Conditional entropy is defined here. I was wondering about an algorithm to find causality between random variables. In particular, I want to calculate $H(X|Y)$ and $H(Y|X)$ and make a guess as to ...
1
vote
1answer
47 views
How is the formula for the entropy of the lognormal distribution derived?
Wikipedia gives the entropy of the lognormal distribution in nats as
$$\mu + \frac{1}{2} \ln(2\pi e \sigma^2)$$ Can anyone point me to a derivation of this formula?
2
votes
0answers
84 views
Relation Between Wasserstein Distance and Relative Entropy
Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...
1
vote
1answer
25 views
how to calculate entropy on matrix of words, topics
I have been digging in the concept of entropy for a while, now it comes to the implementation part I feel I am confused.
Imagine that we have a matrix 20 * 3 standing for 20 words 3 topics (by 20 ...
0
votes
0answers
44 views
Estimation of NegEntropy
I am trying to evaluate the different ICA algorithms. To do that, one of the measure which I use, is to estimate the non-gaussianity using NegEntropy. I am trying to find a formula/function which can ...
0
votes
1answer
54 views
Why does discrete data distribution has differential entropy of negative infinity?
Recently I've been reading a paper.
In section 3.1, it says "Since the discrete data distribution has differential entropy of negative infinity, this can lead to arbitrary high likelihoods even on ...
0
votes
0answers
5 views
Ranking criterion vs. entropy criterion
Problem
In a classical NLP paper (Natural language processing (almost) from scratch) I am reading now, the authors claim that
The entropy criterion lacks dynamical range because its numerical ...
0
votes
0answers
16 views
Modelling probability distribution of a set of sequences (to calculate Entropy)
Let $T$ be a set of trajectories $\tau$, where $\tau=\{\mathbf{x}_1,\mathbf{x}_2,...\mathbf{x}_N\}$ with $\mathbf{x}_i\in\mathbb{R}^k$ being a vector of observations.
I am looking for an efficient ...
0
votes
0answers
27 views
Calculating entropy of joint probability distribution observations of three variables in R
I have n observations of three variables, each variable being discrete with a different number of categories. For example, these are the first 10 observations:
...
0
votes
0answers
21 views
Quantifying information loss (KL divergence?) between a multivariate and a univariate discrete distribution
Let's say I have n discrete variables, n1, n2, ... n_n, each with a different scale, and another discrete variable ...
2
votes
0answers
43 views
Name/definition of $\int \log F(x) \cdot g(x)dx$?
We know that:
$$-\int \log f(x) \cdot g(x)dx,$$
where $f$ and $g$ are density functions, is known as the cross entropy. Does
$$-\int \log F(x) \cdot g(x)dx,$$
where $F$ is the cumulative ...
0
votes
0answers
37 views
Why isn't “mutual information” instead called “mutual entropy”, and “pointwise mutual information” instead called “mutual information”?
Unless I misunderstand something, the following points are true:
The entropy of a variable is the average information that you get from it with each trial.
The mutual information between two ...
1
vote
1answer
82 views
Proving that Shannon entropy is maximised for the uniform distribution
I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$.
Further ...
