Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

Filter by
Sorted by
Tagged with
1
vote
1answer
13 views

Entropy: Given y=Ax, then H[y] = H[X] + ln |A|

The entropy of H[y] is given by: $$H[y] = - \int p_y(\mathbf{y})~ \ln p_y(\mathbf{y})~ d\mathbf{y}$$ Now, suppose that I want to make a linear transformation of vector $\mathbf{y}$ to change the ...
0
votes
1answer
14 views

How entropy is calculated when a non categorical feature is available when using Decision tree or random forest algorithms?

How an entropy is calculated on non-categorical feature containing big amount of unique numbers? Let me give you an example: When we're having a categorical ...
0
votes
0answers
7 views

Transfer entropy for language domain

Is there a formulation of transfer entropy when the two random variables are instantiated from natural language? I am looking for a method to quantify the directional transfer of information from a ...
4
votes
2answers
55 views

Entropy: Proving information gain formula: h(x) = -log p(x)

We consider a discrete random variable X, and we want to know how much information we receive every time we observe the value of this random variable. We qualify this measure of information transfer ...
9
votes
3answers
122 views
+100

Getting all answers correct by taking the same exam for fewest times

Rain never studies, so she is completely clueless during the midterm even though it consists of Yes/No questions only. Fortunately, Rain's professor allows her to re-take the same midterm as many ...
0
votes
0answers
4 views

How to display categorical values on export tree image of decision tree classifier? [migrated]

I am trying to export the decision tree as an image with the original labels of all categorical fields. The current data I have is like so: I transformed the categorical features into numerical: <...
1
vote
0answers
10 views

How feature is selected to be a decision node after splitting the first root node using entropy in decisions trees?

In this article on how entropy is calculated and how the decision tree is built, the writer chose to split the sunny branch into ...
0
votes
0answers
10 views

How do I calculate the kth order entropy of a list of strings?

I want to measure the $k$th order entropy of a set of strings $s_1, s_2 \dots s_n$. And I am confused between two possible approaches that can be used. Background For a string $s\in\Sigma^n$, the ($...
0
votes
0answers
8 views

Proof help for multivariate mutual information as a sum of entropies

I'm following this paper on ICA and I got to equation (1) describing the multivariate mutual information contrast function as a sum of entropies. $J(Y) = \int p(y_1,...y_D)log(\frac{p(y_1,...y_D)}{p(...
1
vote
0answers
10 views

Are there differentiable estimators for Entropy?

I have recently came across a paper on estimation of Information theoretic measure such as Entropy, Mutual Information and divergence, using a Mean Nearest Neighbor approach. Since, the estimator is ...
-1
votes
0answers
9 views

Neighborhood of the entropy for a given neighborhood of probabilities

Assume $M$ is a real-valued discrete random varibale taking values $m_i,\ i=1,\dots,m$ with uniform distribution. Given $|p(m_i|y)-\frac{1}{m}|<\epsilon$ is there any $\eta$ such that $|H(M)-H(M|y)...
0
votes
0answers
42 views

Who to attribute information gain to?

I am writing a paper where I examine information gain specifically with regards to feature selection and am wondering what the proper reference should be. I have looked all over and I can't find a ...
2
votes
1answer
35 views

Joint entropy of multivariate normal distribution less than individual entropy under high correlation

Suppose we are calculating the joint entropy of a multivariate normal distribution with covariance matrix [1,0.99;0.99,1]. From the analytical solution, the joint entropy is 1.27; However, the ...
0
votes
0answers
44 views

Does the limit of c exist or not

Let say m is dimension $\exists$ $f(x)$ $f$ is density function and \begin{equation*} f(x) = \frac{c(m,a,b)}{\|x\|^a\left(\log\frac{e}{\|x\|}\right)^{b}}\mathbf{1}_{\|x\|\leq 1} \geq 0 \end{...
3
votes
2answers
331 views

how to know this integral finite or infinite

In here, i want to show this entropy exist or not exist, namely i should calculate the integral of $\int_0^c\frac{1}{x\log^2\frac{e}{x}}\frac{1}{2} \log\frac{e}{x}\,dx$. If the result is $ <\...
2
votes
1answer
40 views

How to derive this upper bound for the entropy of a bounded random variable? [closed]

A continuous random variable $ Z $ has a density that is $0$ except over the interval $[−A, +A].$ Show that the differential entropy $h(Z)$ is upper bounded by $1+\log_2 A.$ I am stuck in this ...
4
votes
0answers
30 views

How can a probability densitiy be estimated based on the maximum entropy principle, with constraints in the order statistics?

Let's say we are given a sample $\{ z_i \}$, $i=1,2,\ldots,N$, such that each value $z_i$ corresponds to the minimum of $n$ random variables $x$, i.e., $z = \min \{ x_1, x_2,\ldots,x_n \}$. The ...
0
votes
0answers
25 views

Meaning of the derivative of cross entropy wrt $p(x)$

Lets define the cross entropy between 2 probability distributions $p(x)$ and $q(x)$ as $$H(p,q) = -\sum p(x) \log{q(x)} $$ What would be the meaning of derivative of $H(p,q)$ when taking the ...
4
votes
1answer
100 views

How to understand units of information [duplicate]

In information theory, specifically in information content, I am struggling to conceptualise what the unit of measurement actually is. I have read quite a few similar questions and worked through the ...
0
votes
0answers
5 views

Statistical uncertainty on mutual overlap/ entropy of data

I have $N$ datapoints, obtained from a monte carlo simulation. Between each pair ${n,m}$ of these datapoints, we can compute an overlap $o(n,m)$ based on some analytic formulas, which is always ...
0
votes
0answers
9 views

Can we use joint entropy to split my nodes in a multi-objective decision tree?

Since for single objective decision tree (CART model), we can use information gain or difference in entropy (other than GINI index, Chi-Square) to split the nodes by choosing the attribute with ...
0
votes
0answers
12 views

entropy coefficient in A3C

I see that my policy entropy is decreasing very fast and converges to zero in no time, causing the policy to sample the same action again and again (which results in a sub-optimal behavior). I think a ...
1
vote
1answer
33 views

Within sample and between sample categorical variation

I don't really know the correct terminology to ask this question well, so bear with me. I have categorical data with counts and I want a measure of how "diverse" or "spread out" the data is. Variance ...
0
votes
0answers
17 views

What is a reasonable value for KL Divergence

I am using scipy.stats.entropy for KL Divergence. When I create distributions to test I never receive values greater than .5. When I run the function on data I ...
0
votes
1answer
16 views

Joint entropy for estimating conditional mutual sorting information

I'm going through this paper, and their notation is confusing me a little bit. In section III (Coupling Measures), they consider two time series $x_t$ and $y_t$, as well as their corresponding so-...
0
votes
0answers
13 views

how entropy works in continuous distribution using the K nearest neighbour?

So calculating entropy in continuos distribution was developed by Kozachenko and Leonenko (1987). I have seen one implementation here. however I have problem understanding the approach. basically, ...
1
vote
0answers
35 views

Entropy evolution while learning?

It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows ...
0
votes
0answers
16 views

How to minimise and expectation with respect to a parameter?

Suppose that $X$ is a random variable with distribution $G$. Let $H(X;\theta)$ be a parametric function with $\theta \in \Theta \subset {\mathbb R}^p$. I want to maximize the function $$\varphi(\...
4
votes
1answer
62 views

Expanding initial sample when the result isn't significant

This inspiring answer describes a variant of hypothesis, and I want to analysis its property further. Basically, it considers a two-sided test and interprets the $p$-value as a measure of how strong ...
0
votes
1answer
32 views

decision tree training: gini vs entropy vs precision vs recall

When training decision trees, the standard algorithms (e.g. ID3, C4.5, C5.0) use either the gini index or entropy to determine which node to add next. Only once the tree is built, and the ROC curve is ...
12
votes
1answer
804 views

How does entropy depend on location and scale?

The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals $$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\...
0
votes
1answer
42 views

If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?

I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
1
vote
0answers
128 views

Non-negativity of interaction information for special trivariate case

Consider a discrete trivariate distribution $P(X_1, X_2, Y)$, which satisfies $$ p(x_1, x_2, y) = \min( p(x_1,y), p(x_2,y) ), $$ for all $x_1$ and $x_2$ for which $p(x_1, x_2) > 0$ and for all ...
0
votes
0answers
7 views

Can the calculation of entropy be generalized to an infinite series?

For a finite problem size $n$, it's straightforward to calculate the entropy of a random variable $X_n$ for the function I'm interested in. With the particular kinds of functions that I'm studying, ...
0
votes
1answer
38 views

Can Latent Class Analysis entropy be infinity? And what does it mean?

I recently ran a latent class analysis with polytomous outcome variables with the poLCA R package. The model ran fine converged from 2 to 6 classes (the model converged with more classes but fit ...
1
vote
1answer
62 views

Products after sorting

Given two sets $\{a_1, \dots, a_N\}$ and $\{ b_1, \dots, b_N \}$ of positive numbers $0 < a_i, b_i < 1$, $i=1, \dots, N$, is it true that $\sum_i \min(a_i, b_i) \cdot \sum_i \max(a_i, b_i) \le \...
1
vote
1answer
26 views

Shannon's Entropy, manual vs. analytic results differ

Let's say that I have information string A, B, C, D, E. All letters are equally probable (1/5). So Shannon's formula would give us. ...
0
votes
0answers
21 views

Finding causation in data with Conditional entropy

Conditional entropy is defined here. I was wondering about an algorithm to find causality between random variables. In particular, I want to calculate $H(X|Y)$ and $H(Y|X)$ and make a guess as to ...
1
vote
1answer
47 views

How is the formula for the entropy of the lognormal distribution derived?

Wikipedia gives the entropy of the lognormal distribution in nats as $$\mu + \frac{1}{2} \ln(2\pi e \sigma^2)$$ Can anyone point me to a derivation of this formula?
2
votes
0answers
84 views

Relation Between Wasserstein Distance and Relative Entropy

Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...
1
vote
1answer
25 views

how to calculate entropy on matrix of words, topics

I have been digging in the concept of entropy for a while, now it comes to the implementation part I feel I am confused. Imagine that we have a matrix 20 * 3 standing for 20 words 3 topics (by 20 ...
0
votes
0answers
44 views

Estimation of NegEntropy

I am trying to evaluate the different ICA algorithms. To do that, one of the measure which I use, is to estimate the non-gaussianity using NegEntropy. I am trying to find a formula/function which can ...
0
votes
1answer
54 views

Why does discrete data distribution has differential entropy of negative infinity?

Recently I've been reading a paper. In section 3.1, it says "Since the discrete data distribution has differential entropy of negative infinity, this can lead to arbitrary high likelihoods even on ...
0
votes
0answers
5 views

Ranking criterion vs. entropy criterion

Problem In a classical NLP paper (Natural language processing (almost) from scratch) I am reading now, the authors claim that The entropy criterion lacks dynamical range because its numerical ...
0
votes
0answers
16 views

Modelling probability distribution of a set of sequences (to calculate Entropy)

Let $T$ be a set of trajectories $\tau$, where $\tau=\{\mathbf{x}_1,\mathbf{x}_2,...\mathbf{x}_N\}$ with $\mathbf{x}_i\in\mathbb{R}^k$ being a vector of observations. I am looking for an efficient ...
0
votes
0answers
27 views

Calculating entropy of joint probability distribution observations of three variables in R

I have n observations of three variables, each variable being discrete with a different number of categories. For example, these are the first 10 observations: ...
0
votes
0answers
21 views

Quantifying information loss (KL divergence?) between a multivariate and a univariate discrete distribution

Let's say I have n discrete variables, n1, n2, ... n_n, each with a different scale, and another discrete variable ...
2
votes
0answers
43 views

Name/definition of $\int \log F(x) \cdot g(x)dx$?

We know that: $$-\int \log f(x) \cdot g(x)dx,$$ where $f$ and $g$ are density functions, is known as the cross entropy. Does $$-\int \log F(x) \cdot g(x)dx,$$ where $F$ is the cumulative ...
0
votes
0answers
37 views

Why isn't “mutual information” instead called “mutual entropy”, and “pointwise mutual information” instead called “mutual information”?

Unless I misunderstand something, the following points are true: The entropy of a variable is the average information that you get from it with each trial. The mutual information between two ...
1
vote
1answer
82 views

Proving that Shannon entropy is maximised for the uniform distribution

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further ...