A mathematical quantity designed to measure the amount of randomness of a random variable.

learn more… | top users | synonyms

3
votes
1answer
44 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
3
votes
0answers
38 views

Entropy estimation for a symbol sequence

I am looking for an R-implementation of the Lempel-Ziv data compression algorithm, to estimate the source entropy of a time-series consisting of a sequence of symbols. Rather than simply measuring ...
0
votes
0answers
16 views

Confidence interval on entropy estimation

Over a sample (of about a hundred observations), I have estimated the "empirical" entropy of a sequence of identically independently distributed random variables which have 5 possible outcomes. I ...
0
votes
0answers
29 views

How to compute the total entropy of a system of binary outcomes

This is nowhere near my field of expertise, so if my question is poorly formatted for the context please feel free to edit. My question is at the end of the text below. Let's assume I flip a coin 5 ...
0
votes
0answers
24 views

Entropy of generalized distributions?

What's the entropy of the following generalized probability distributions? $P_1(x) = \delta(x)$ $P_2(x,y) = \delta(x+y)$, for $0\le x\le 1$, and $P_2(x,y)=0$ otherwise. Integrals of the type $-\int ...
4
votes
0answers
88 views

Compute Shannon entropy between every row of a large, sparse matrix

I have a sparse, binary matrix of user (rows) and items (columns). Each element of this matrix is either 0 or 1: ...
1
vote
0answers
29 views

Better markov chain rank aggregation using an entropy-based approach

Background: Dwork et al. in Rank Aggregation Methods for the Web have proposed a few markov chain-based methods to perform rank aggregation (finding an aggregated ranking of items from a set of N ...
0
votes
1answer
32 views

LLR with Positive and Negative Values vs. Dunning method with Entropy-based Calculation

Ted Dunning has a blog post about calculating G2 (aka LLR) using Entropy calculations as components. I found this really intriguing. Ted's original post: ...
19
votes
6answers
2k views

What is the role of the logarithm in Shannon's entropy?

Shannon's entropy is the negative of the sum of the probabilities of each outcome multiplied by the logarithm of probabilities for each outcome. What purpose does the logarithm serve in this equation? ...
3
votes
1answer
42 views

When and why does the “brittleness” of mutual information cause overfitting?

I have frequently heard concern over "brittleness" of entropy and mutual information as performance metrics for a statistical fitting and the fact that it leads to overfitting. You can see an example ...
2
votes
2answers
42 views

Multidimensional Differential Entropy

I am looking for a measure of entropy over multiple random variables, each with values between 0 and 1. Intuitively, it seems possible to talk about the expected value of information of several ...
1
vote
1answer
44 views

Jensen-Shannon divergence for finite samples

I have two finite samples $s_1$ and $s_2$ and two distributions $p_1(s_1)$ and $p_2(s_2)$ that are associated to these samples. I'm essentially interested to measure the distance or similarity between ...
3
votes
1answer
70 views

Difference between different kinds of entropy

The following concepts are baffling and would be obliged for a constructive explanation. (Q1) What is the conceptual difference between (a) Kolmogorov-Sinai entropy, (b) Shannon entropy, (c) Source ...
1
vote
0answers
20 views

Modeling a list with a tunable degree of disorder/shuffling

Imagine we have a list of ordered numbers $L = (1, 2,\dots, N)$. I want to add an arbitrary amount of "disorder" to that list. For instance: Adding a little bit of disorder would permute a few ...
1
vote
1answer
42 views

Renyi divergence identity

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think ...
2
votes
1answer
49 views

Qualitively what is Cross Entropy

This question gives a quantitative definition of cross entropy, in terms of it's formula. I'm looking for a more notional definition, wikipedia says: In information theory, the cross entropy ...
0
votes
0answers
7 views

How to find entropy of vocabulary terms in multilabel document classification problem?

I have 5 million of document s with varying number of labels for each. I intent to find entropy value for selecting discriminative terms to degrade the size of vocab. However, having that multiple ...
0
votes
0answers
36 views

Information entropy of a mixture distribution

I'm working with a high-dimensional mixture distribution, and I'm interested in calculating its entropy. I think I could work it out if there were only two mixture components. Following @Daniel's ...
3
votes
1answer
135 views

Interpreting Shannon entropy

From a computer simulation I have built a histogram of the results and normalized it so that the probability of finding a point $X$ in bin $b_j$ is $\sum_j P(X \in b_j) =1$. From this I have ...
0
votes
0answers
29 views

Are there some risks connected with using k-NN with k=1 by F=0.77?

I am not very expirienced in the field of machine learning. For now I model some binary classifier using k-NN. I experimented a bit with different values of k. In some cases the best one was with ...
1
vote
0answers
105 views

Logistic regression and maximum entropy

I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier. My question is, how does one end up with the formula for logistic regression starting with ...
1
vote
0answers
34 views

Information theoretic substitute for precision and recall?

Is there a substitute for measuring precision, recall of a classifier (binary or multi-class) to evaluate its performance using information-theoretic quantities like entropy, mutual information or ...
0
votes
0answers
129 views

A MATLAB package to calculate Entropy of a discrete non-stationary (hydrologic) time series

I am looking to survey the community to find out any prefered packages researchers, practitioners, or interested modelers are using to calculate entropy (cross-entropy, conditional entropy, etc...) ...
0
votes
1answer
49 views

Entropy and Likelihood Relationship

This is a theoretical question. Suppose that I have a sample s1 coming from distribution K, and a sample s2 coming from distribution M. But I don't know what K or M are. I hypothesize that s1 and s2 ...
2
votes
2answers
91 views

Why does entropy increase with dispersion for continuous but not for discrete distributions?

For a pdf $f(x)$ (i.e. continuous distribution), Entropy (differential entropy) is defined as: $H_C(X) = -\int_\mathbb{X} f(x)\log f(x)\,dx.$ For a discrete distribution with p.m.f $F(x)$, Entropy ...
1
vote
1answer
36 views

When is the differential entropy negative?

The definition of entropy for a continuous signal is: $h[f] = \operatorname{E}[-\ln (f(x))] = -\int\limits_{-\infty}^{\infty} f(x) \ln (f(x))\, dx$ According to Wikipedia, it can be negative. When ...
4
votes
0answers
55 views

Entropy of Inverse-Wishart distribution

What is entropy of Inverse-Wishart distribution? I need just a reference, but derivation (e.g. using inverse property) would be interesting too.
0
votes
0answers
34 views

A distributional test based on entropy and self-information

Say that I have a real-valued discrete distribution $p(x)$ and $N$ samples, $x_1, \ldots, x_N$, and I want to test whether the samples came from the distribution without making any further assumptions ...
2
votes
0answers
25 views

What's the value of information when we decrease the entropy of a probability distribution?

Suppose you have to choose between actions $A_1,\dots,A_n$. You have a probability distribution over each $U(A_i)$, i.e. over the utility of choosing each action. So you should choose the $A_i$ that ...
0
votes
1answer
80 views

Joint entropy of two random variables

The joint entropy is the amount of information we get when we observe X and Y at the same time, but what would happen if we don't observe them at the same time. For example, when i toss a coin, if i ...
1
vote
1answer
37 views

Maximum entropy priors for hypotheses

Suppose I have a random variable X which can take integer values 0 through 99. It can be, for example, numbers written on balls in a large urn. I have three hypotheses about the distribution of these ...
0
votes
0answers
33 views

Entropy as measure of order in data streams

Is it sound/allowed to use entropy as a measure of (non-) uniformity of a data stream? E.g. I calculate the Shannon entropy with the standard formula based on various measures in the data stream. ...
1
vote
0answers
38 views

Random variables for which the distribution of the sum of the RV with a Gaussian RV is known

For a counter example, I am searching for random variables $Y$ such that for a independent normal random variable $X$ the distribution of $Z=Y+X$ is known parametrically. Ideally, the Shannon entropy ...
0
votes
0answers
88 views

Computing Jensen-Shannon Divergence between discrete and continuous distribution

Is it possible to compute the Jensen-Shannon divergence between a discrete and a continuous probability distribution, e.g. between a standard normal distribution and a distribution taking the values ...
3
votes
0answers
33 views

Additional parameter adds entropy to the centre of the density function

When an extra shape parameter is added in the distribution than a statement in favour of this parameter is written "extra parameter can control both tail weights and adding entropy to the centre of ...
2
votes
2answers
104 views

Maximum entropy and non-informative distribution

Is Maximum Entropy rule equivalent to non-informativeness? In other words, when maximizing the entropy of a distribution, given some known stuff, is it equivalent to finding to most non-informative ...
2
votes
0answers
80 views

Regression forest: Leaf node and information gain

Regression forests are basically random forests, however used for regression. They basically use the same framework as decision forests use for classification with a few parts exchanged. Two of these ...
3
votes
0answers
79 views

Statistical interpretation of Maximum Entropy Distribution

I have used the principle of maximum entropy to justify the use of several distributions in various settings; however, I have yet to be able to formulate a statistical, as opposed to ...
-1
votes
1answer
151 views

Why is Entropy maximised when the probability distribution is uniform?

I know that entropy is the measure of randomness of a process/variable and it can be defined as follows. for a random variable $X \in$ set $A$ :- $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i)) $. In ...
2
votes
1answer
168 views

Entropy of (Sum of Gaussians) versus Sum of (Entropy of Gaussians)

Short version: How can the joint entropy of two independent variables be less than the sum of those independent variables? The joint entropy should encode all information that a scalar function can, ...
0
votes
0answers
24 views

Should all quantitative variables analyzed with the same kind of test?

I read an article where people computed Normalized Shannon entropy to evaluate the genetic heterogeneity of viral quasispecies. Quasispecies means the genetic variability observed when a virus ...
0
votes
1answer
71 views

Measures of entropy/information: distinguish clustered configurations that would have the same information entropy

Let us consider the configuration of a 2D system and the standard definition of entropy $H=-\sum_{i=1}^{m}p_{i}\cdot \log(p_{i})$. Let us suppose that I can describe the state of my system by a 2D ...
2
votes
1answer
54 views

hypothetical measure of variability similar to entropy

Variance has the following properties: $Var(cX)=c^2Var(X)$ For independent variables $Var(X+Y)=Var(X)+Var(Y)$. Range of a rv has the following properties: $Range(cX)=|c| Range(X)$ For ...
2
votes
1answer
101 views

Information gain in random trees

When splitting attributes while constructing a random tree, I use information gain in order to determine the best value to split the tree on. I add nodes to the tree until a stopping criterion is met. ...
0
votes
0answers
34 views

Relationship between mutual information and change of variables

On pages 11-12 of this tutorial, equation 28 $$ I(y_1,...,y_n) = \sum_i H(y_i) - H(x) - \log | \det(W)|$$ draws a connection between mutual information and an invertible linear transformation for $y ...
0
votes
0answers
80 views

Overfitting in K-NN and Decision Trees?

To avoid over fitting for K-NN could you increase the value of K to reduce anomalous results etc. However, if the value of K is very large with respect to a sample, would this also incur in over ...
1
vote
0answers
148 views

Alternative to Shannon's entropy when probabilty equal to zero

I have a serie of objects for whom I know the probability of belonging to 10 classes. This probability can be null (see example below with 4 classes). ...
1
vote
0answers
106 views

Entropy calculations and Process ranking

Still working on my financial datasets, I am now trying to observe the effects of the pre-processing on the results of my study. To sum it up: I have thirty time series of prices (the main indices, ...
1
vote
0answers
21 views

Is there any meta-approach for variable selection based of measures of similarity between each two variables?

Is there any meta-approach ( or mayby I should say universal approach which works with different measures ) for variable selection which is based on similarity matrix which every entry ...
1
vote
0answers
69 views

Entropy of noncentral multivariate t distribution

I am looking at calculating the entropy of a multivariate noncentral t-distribution. I found in the central case (mean is zero) the entropy is the same as multivariate normal with a correction for ...