A mathematical quantity designed to measure the amount of randomness of a random variable.

learn more… | top users | synonyms

3
votes
1answer
23 views

Compare skewness of many distributions with few observations

I have a dataset with page view data for about 500,000 users, divided into two groups. Each user can visit up to 5 pages, each as many or as few times as they want. So for each user, I have the ...
0
votes
0answers
15 views

Confusion regarding concept of entropy and information [duplicate]

In general, entropy of a signal or image conveys the uncertainty and it is a measure of impurity. Information on the other hand tells us about how certain we are about the data. It is a measure of ...
2
votes
1answer
33 views

Conceptual question on mutual information and entropy

In respect to parameter estimation using information theoretic concepts, a signal X = S+N where S is the desired signal whose parameters we wish to estimate and N is the noise. The paper ...
1
vote
1answer
27 views

Conceptual questions on Entropy and estimation

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in ...
0
votes
1answer
21 views

do information entropy probabilities have to sum to one?

My understanding of information entropy is that it requires the input probabilities to sum to 1. So, for a sequence a,a,b,b you then have -([1/2 log2 1/2] + [1/2 log2 1/2]) = 1 Are there versions of ...
1
vote
1answer
28 views

Differentiating Shannon's entropy [on hold]

Can somebody please show the steps of how differentiation of Shannon's entropy yields the following result? $H = -\sum_{l=0}^{L-1} p(l)\log_2[p(l)]$ The result of differentiating is $H_m = ...
1
vote
0answers
9 views

What does Class Complexity mean in Weka?

When running Weka on my dataset, in the results printout I get the following rows: ...
1
vote
0answers
44 views

Neuroscience Equations

I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and ...
0
votes
0answers
14 views

General reference to Rényi entropy

Is there a book or a review article serving as a good general reference to Rényi entropy, its applications and related concepts?
1
vote
1answer
21 views

Calculation of relative information gain for multiple splits in decision trees

As it well known, we can calculate Relative Information gain (RIG) as follows: $RIG = \frac{H(x) - H(x|a)}{H(x)}$. In binary decision trees we calculate $H(x|a)$ for univariate split for variable ...
1
vote
1answer
53 views

Feature selection : how to select the Information Gain threshold?

I am trying to use Information Gain to select features when classifying text with a Support Vector Machine. For each word in our training data, we computed its information gain. Then, we should keep ...
0
votes
1answer
32 views

Differential Entropy of Gaussian Process

I have $N$ datapoints that have $d$ features in a GP and their covariance matrix $K$ and I want to calculate the differential entropy of that GP. Is this formula right? $E(I)= \frac{1}{2} ...
0
votes
1answer
43 views

Information gain with numerical data

I'm making a random forest classifier. In every tutorial, there is a very simple example of how to calculate entropy with Boolean attributes. In my problem I have attribute values that are calculated ...
0
votes
1answer
59 views

Relationship between entropy and information gain

Based on papers :1. Deniz Erdogmus, Member, IEEE, and Jose C. Principe, An Error-Entropy Minimization Algorithm for Supervised Training of Nonlinear Adaptive Systems J. Principe, D. Xu, and J. ...
0
votes
1answer
38 views

Entropy and information content

I am curious to know about the relation between Entropy and information content of a signal or trajectory of time series. When a system is at equilibrium, then it has maximum entropy. Does entropy ...
-1
votes
1answer
32 views

Combining values of Shannon Entropy

I have a set of n variables. Each of these variables an be one of a finite number, m, of values. The number of values is different for each variable and independent. From this definition I can ...
4
votes
1answer
120 views

Estimating entropy of multidimensional variable through dimension reduction

Is there anything inherently wrong with trying to estimate the entropy of a multidimensional random variable by first transforming it (by some method) into a single-dimensional variable?
1
vote
1answer
47 views

$I(X:Y|Z=z) \leq I(X:Y|Z)$?

$X,Y$ and $Z$ are discrete random variables. $I$ is the mutual information. The question is in the title. If the inequality is true, how would you show it? Thanks. My intuition: having more ...
1
vote
1answer
40 views

Entropy of Cauchy (Lorentz) Distribution

Entropy is defined as $H$ = -$\int p(x)$ $log$ $p(x)$ $dx$ The Cauchy Distribution is defined as $f(x)$ = $\frac{\gamma}{\pi}$ $\frac{1}{\gamma^2 + x^2} $ I kindly ask to show the steps to ...
2
votes
0answers
78 views

kozachenko-leonenko entropy estimation

I'm trying to implement the entropy estimation based on the closest neighbour from Kozachenko and Leonenko but I'm facing a problem I can't solve. The idea is to work in a new set ...
3
votes
0answers
33 views

Using similarity matrix to measure diversity of a group

I wish to measure the "diversity" of a group of objects. Right now I'm using Euclidean distance to compute the similarity matrix between all the objects in the group. I'm searching for a measure of ...
4
votes
3answers
45 views

$\phi$-divergence?

I am frustrated of looking for a simple explanation of this term $\phi$-divergence, but I cannot find any. Therefore I would be really grateful if somebody could introduce a reference or write a ...
0
votes
2answers
73 views

Why am I getting information entropy greater than 1?

I implemented the following function to calculate entropy: ...
3
votes
1answer
73 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
4
votes
1answer
93 views

Entropy estimation for a symbol sequence

I am looking for an R-implementation of the Lempel-Ziv data compression algorithm, to estimate the source entropy of a time-series consisting of a sequence of symbols. Rather than simply measuring ...
0
votes
0answers
25 views

Confidence interval on entropy estimation

Over a sample (of about a hundred observations), I have estimated the "empirical" entropy of a sequence of identically independently distributed random variables which have 5 possible outcomes. I ...
0
votes
0answers
38 views

How to compute the total entropy of a system of binary outcomes

This is nowhere near my field of expertise, so if my question is poorly formatted for the context please feel free to edit. My question is at the end of the text below. Let's assume I flip a coin 5 ...
1
vote
1answer
39 views

Entropy of generalized distributions?

What's the entropy of the following generalized probability distributions? $P_1(x) = \delta(x)$ $P_2(x,y) = \delta(x+y)$, for $0\le x\le 1$, and $P_2(x,y)=0$ otherwise. Integrals of the type $-\int ...
4
votes
0answers
121 views

Compute Shannon entropy between every row of a large, sparse matrix

I have a sparse, binary matrix of user (rows) and items (columns). Each element of this matrix is either 0 or 1: ...
1
vote
0answers
50 views

Better markov chain rank aggregation using an entropy-based approach

Background: Dwork et al. in Rank Aggregation Methods for the Web have proposed a few markov chain-based methods to perform rank aggregation (finding an aggregated ranking of items from a set of N ...
0
votes
1answer
83 views

LLR with Positive and Negative Values vs. Dunning method with Entropy-based Calculation

Ted Dunning has a blog post about calculating G2 (aka LLR) using Entropy calculations as components. I found this really intriguing. Ted's original post: ...
19
votes
6answers
2k views

What is the role of the logarithm in Shannon's entropy?

Shannon's entropy is the negative of the sum of the probabilities of each outcome multiplied by the logarithm of probabilities for each outcome. What purpose does the logarithm serve in this equation? ...
3
votes
1answer
53 views

When and why does the “brittleness” of mutual information cause overfitting?

I have frequently heard concern over "brittleness" of entropy and mutual information as performance metrics for a statistical fitting and the fact that it leads to overfitting. You can see an example ...
2
votes
2answers
49 views

Multidimensional Differential Entropy

I am looking for a measure of entropy over multiple random variables, each with values between 0 and 1. Intuitively, it seems possible to talk about the expected value of information of several ...
1
vote
1answer
86 views

Jensen-Shannon divergence for finite samples

I have two finite samples $s_1$ and $s_2$ and two distributions $p_1(s_1)$ and $p_2(s_2)$ that are associated to these samples. I'm essentially interested to measure the distance or similarity between ...
3
votes
1answer
75 views

Difference between different kinds of entropy

The following concepts are baffling and would be obliged for a constructive explanation. (Q1) What is the conceptual difference between (a) Kolmogorov-Sinai entropy, (b) Shannon entropy, (c) Source ...
1
vote
0answers
21 views

Modeling a list with a tunable degree of disorder/shuffling

Imagine we have a list of ordered numbers $L = (1, 2,\dots, N)$. I want to add an arbitrary amount of "disorder" to that list. For instance: Adding a little bit of disorder would permute a few ...
1
vote
1answer
52 views

Renyi divergence identity

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think ...
2
votes
1answer
64 views

Qualitively what is Cross Entropy

This question gives a quantitative definition of cross entropy, in terms of it's formula. I'm looking for a more notional definition, wikipedia says: In information theory, the cross entropy ...
0
votes
0answers
12 views

How to find entropy of vocabulary terms in multilabel document classification problem?

I have 5 million of document s with varying number of labels for each. I intent to find entropy value for selecting discriminative terms to degrade the size of vocab. However, having that multiple ...
0
votes
0answers
45 views

Information entropy of a mixture distribution

I'm working with a high-dimensional mixture distribution, and I'm interested in calculating its entropy. I think I could work it out if there were only two mixture components. Following @Daniel's ...
3
votes
1answer
159 views

Interpreting Shannon entropy

From a computer simulation I have built a histogram of the results and normalized it so that the probability of finding a point $X$ in bin $b_j$ is $\sum_j P(X \in b_j) =1$. From this I have ...
0
votes
0answers
29 views

Are there some risks connected with using k-NN with k=1 by F=0.77?

I am not very expirienced in the field of machine learning. For now I model some binary classifier using k-NN. I experimented a bit with different values of k. In some cases the best one was with ...
1
vote
0answers
130 views

Logistic regression and maximum entropy

I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier. My question is, how does one end up with the formula for logistic regression starting with ...
1
vote
0answers
34 views

Information theoretic substitute for precision and recall?

Is there a substitute for measuring precision, recall of a classifier (binary or multi-class) to evaluate its performance using information-theoretic quantities like entropy, mutual information or ...
0
votes
0answers
167 views

A MATLAB package to calculate Entropy of a discrete non-stationary (hydrologic) time series

I am looking to survey the community to find out any prefered packages researchers, practitioners, or interested modelers are using to calculate entropy (cross-entropy, conditional entropy, etc...) ...
0
votes
1answer
53 views

Entropy and Likelihood Relationship

This is a theoretical question. Suppose that I have a sample s1 coming from distribution K, and a sample s2 coming from distribution M. But I don't know what K or M are. I hypothesize that s1 and s2 ...
2
votes
2answers
108 views

Why does entropy increase with dispersion for continuous but not for discrete distributions?

For a pdf $f(x)$ (i.e. continuous distribution), Entropy (differential entropy) is defined as: $H_C(X) = -\int_\mathbb{X} f(x)\log f(x)\,dx.$ For a discrete distribution with p.m.f $F(x)$, Entropy ...
1
vote
1answer
48 views

When is the differential entropy negative?

The definition of entropy for a continuous signal is: $h[f] = \operatorname{E}[-\ln (f(x))] = -\int\limits_{-\infty}^{\infty} f(x) \ln (f(x))\, dx$ According to Wikipedia, it can be negative. When ...
4
votes
0answers
61 views

Entropy of Inverse-Wishart distribution

What is entropy of Inverse-Wishart distribution? I need just a reference, but derivation (e.g. using inverse property) would be interesting too.