A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

learn more… | top users | synonyms

0
votes
3answers
66 views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
0
votes
0answers
7 views

Effect of noise on entropy

Relation of Entropy and SNR : Based on this question and answer, I am curious to know, if somebody can shed some light, on the following situation: $y= desired_{signal} + noise$ is received by the ...
2
votes
1answer
48 views

Conceptual question on mutual information and entropy

What does mutual information (MI) convey? Looking for good reference books on information theory
1
vote
1answer
34 views

Conceptual questions on Entropy and estimation

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in ...
0
votes
1answer
23 views

do information entropy probabilities have to sum to one?

My understanding of information entropy is that it requires the input probabilities to sum to 1. So, for a sequence a,a,b,b you then have -([1/2 log2 1/2] + [1/2 log2 1/2]) = 1 Are there versions of ...
0
votes
0answers
14 views

Upper Bound on Mutual Information

I was advised to post this question here rather than in the math stack exchange. So here it is: I am interested in an upper bound on mutual information that I have been encountering frequently in the ...
1
vote
0answers
46 views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
1
vote
1answer
25 views

Information gain is KL divergence

I am slightly confused by the the statement that Kullback–Leibler divergence is the same as [information gain](Information gain in decision trees). I cannot understand how $D_{KL}(P||Q) = H(P,Q)- ...
0
votes
0answers
32 views

Calculating Perplexity

In the Coursera NLP course , Dan Jurafsky calculates the following perplexity: Operator(1 in 4) Sales(1 in 4) Technical Support(1 in 4) 30,000 names(1 in 120,000 each) He says the Perplexity is 53. ...
0
votes
1answer
67 views

Relationship between entropy and information gain

Based on papers :1. Deniz Erdogmus, Member, IEEE, and Jose C. Principe, An Error-Entropy Minimization Algorithm for Supervised Training of Nonlinear Adaptive Systems J. Principe, D. Xu, and J. ...
0
votes
1answer
42 views

Entropy and information content

I am curious to know about the relation between Entropy and information content of a signal or trajectory of time series. When a system is at equilibrium, then it has maximum entropy. Does entropy ...
1
vote
1answer
38 views

Feature selection based on information gain papers

I want to apply feature selection based on information gain: I have many features many of which are redundant. I am planning on selecting a feature and then iteratively add features that 'add the more ...
7
votes
2answers
315 views

Assessing principal components analysis

What is a good metric for assessing the quality of a pca? I performed this algorithm on a dataset. My objective was to reduce the number of features (the information was very redundant). I know the ...
0
votes
0answers
36 views

information theoretic model selection

I have some questions on the information theoretic approach to linear modelling: Why not just use residuals instead of likelihood? Can same variables with different subsets of data be used to build ...
10
votes
2answers
211 views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
3
votes
1answer
81 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
0
votes
0answers
15 views

Papers on resource prediction over time series with asymmetric information?

I'm looking for resources to help me solve an issue of resource prediction. We can make a number of observations of the resource over time, but the way in which the resource changes is affected by ...
0
votes
0answers
40 views

How to compute the total entropy of a system of binary outcomes

This is nowhere near my field of expertise, so if my question is poorly formatted for the context please feel free to edit. My question is at the end of the text below. Let's assume I flip a coin 5 ...
1
vote
0answers
32 views

Posterior variance reduction

As detailed on its Wikipedia page, Mutual information, $I(X,Y)$, can be bounded by the Jensen inequality to show that it is always positive. Also, one can show that $$ I(X,Y) = H(X) - H(X|Y). $$ ...
0
votes
0answers
16 views

When can one say two data groups (or more) are inseparable?

There are plenty of measures to compute the separability between groups: the bhattacharya distance, the Jeffries-Matusita distance, the (transformed) divergence and information measures... While those ...
2
votes
2answers
415 views

Calculating AIC “by hand” in R

I have tried calculating the AIC of a linear regression in R but without using the AIC function, like this: ...
0
votes
0answers
21 views

Information theoretic approach to model selection: model diagnostics

I've read a few papers that have used the information theoretic to model selection. One of these papers made 36 candidate models. In the methods section of that paper, the authors mentioned nothing ...
1
vote
0answers
29 views

AICc for small sample sizes

I'm selecting models using the information theoretic approach. I've just read that the AICc should be used to rank candidate models where the number of parameters in a model reaches 30% of the sample ...
0
votes
0answers
41 views

AIC, BIC: Multiple dependent variables

I have multiple candidate models, and each needs to be scored on multiple "metrics", where each metric is probability distribution. Essentially the metrics are like dependent variables. The question ...
4
votes
3answers
77 views

Intuition about a joint entropy

I am having trouble building some intuition about joint entropy. $H(X,Y)$ = uncertainty in the joint distribution $p(x,y)$; $H(X)$ = uncertainty in $p_x(x)$; $H(Y)$ = uncertainty in $p_y(y)$. If ...
0
votes
0answers
76 views

Using QAICc with Poisson, or AIC with Poisson lognormal, in information theoretic approach?

I am trying to use an IT approach to analyse some ecological data. I have a mixed model with nested random effects (I'm using glmer in package lme4 in R). I initially fit the model with a Poisson ...
1
vote
1answer
107 views

Jensen-Shannon divergence for finite samples

I have two finite samples $s_1$ and $s_2$ and two distributions $p_1(s_1)$ and $p_2(s_2)$ that are associated to these samples. I'm essentially interested to measure the distance or similarity between ...
1
vote
0answers
251 views

Using mutual information to estimate correlation between a continuous variable and a categorical variable

As for the title, the idea is to use mutual information, here and after MI, to estimate "correlation" (defined as "how much I know about A when I know B") between a continuous variable and a ...
3
votes
1answer
76 views

Difference between different kinds of entropy

The following concepts are baffling and would be obliged for a constructive explanation. (Q1) What is the conceptual difference between (a) Kolmogorov-Sinai entropy, (b) Shannon entropy, (c) Source ...
1
vote
1answer
54 views

Renyi divergence identity

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think ...
2
votes
1answer
70 views

Qualitively what is Cross Entropy

This question gives a quantitative definition of cross entropy, in terms of it's formula. I'm looking for a more notional definition, wikipedia says: In information theory, the cross entropy ...
0
votes
0answers
21 views

how to calculate information value [duplicate]

i have the following tree Outlook( [Sunny] => (Yes, Yes, No, No, No), [Overcast] => (Yes, Yes, Yes, Yes), [Rainy] => (Yes, Yes, Yes, No, No) ) ...
0
votes
0answers
46 views

Information entropy of a mixture distribution

I'm working with a high-dimensional mixture distribution, and I'm interested in calculating its entropy. I think I could work it out if there were only two mixture components. Following @Daniel's ...
2
votes
1answer
43 views

Information theoretic alternative to hypergeometric test

Is there any information-theoretic alternative to the hypergeometric test (or Fisher's exact test)? In other word, is it possible to calculate AIC values for data which one would classically analyse ...
1
vote
0answers
34 views

Information theoretic substitute for precision and recall?

Is there a substitute for measuring precision, recall of a classifier (binary or multi-class) to evaluate its performance using information-theoretic quantities like entropy, mutual information or ...
1
vote
0answers
55 views

Orthogonality of parameter space

I have a basic information theory question. I am fitting a highly parameterized model to some data. In general: $$ y = \sum\limits_{i=1}^{13} \alpha_i X_i $$ Currently I use gradient descent to find ...
1
vote
1answer
52 views

When is the differential entropy negative?

The definition of entropy for a continuous signal is: $h[f] = \operatorname{E}[-\ln (f(x))] = -\int\limits_{-\infty}^{\infty} f(x) \ln (f(x))\, dx$ According to Wikipedia, it can be negative. When ...
0
votes
1answer
179 views

conditional mutual information and how to deal with zero probabilities

the conditional mutual between three sets of mutually exclusive variables, X, Y, and Z, is defined as follows. $I(X,Y|Z) = \sum_{xyz} P(x,y,z) \log \frac{P(z)P(x,y,z)}{P(x,z)P(y,z)}$ my questions ...
3
votes
1answer
164 views

Distance or Similarity metric for 2D frequency data maps

I want to compare the distance/similarity of 2D flood frequency data maps. The maps are square with YxY grid size and in each cell of the map is stored its flood frequency. For example in a 5x5 grid ...
3
votes
1answer
311 views

KL-divergence between two categorical/multinomial distributions gives negative values?

If $$P = [0,0.9,0,0.1]$$ $$Q = [0,1,0,0]$$ Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$ This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?
1
vote
2answers
86 views

Testing whether small Eigenvalues produce a “signal”

My problem is similar to this one but I am looking for a different solution: (so if it should be merged just let me know). Measuring what's 'lost' in PCA dimensionality reduction? I my ...
2
votes
2answers
108 views

Maximum entropy and non-informative distribution

Is Maximum Entropy rule equivalent to non-informativeness? In other words, when maximizing the entropy of a distribution, given some known stuff, is it equivalent to finding to most non-informative ...
2
votes
0answers
111 views

Regression forest: Leaf node and information gain

Regression forests are basically random forests, however used for regression. They basically use the same framework as decision forests use for classification with a few parts exchanged. Two of these ...
2
votes
1answer
256 views

Entropy of (Sum of Gaussians) versus Sum of (Entropy of Gaussians)

Short version: How can the joint entropy of two independent variables be less than the sum of those independent variables? The joint entropy should encode all information that a scalar function can, ...
5
votes
3answers
200 views

Does dimension reduction always lose some information?

Like the title says, does dimension reduction always lose some information? Consider for example PCA. If the data I have is very sparse, I would assume a "better encoding" could be found (is this ...
0
votes
1answer
77 views

Measures of entropy/information: distinguish clustered configurations that would have the same information entropy

Let us consider the configuration of a 2D system and the standard definition of entropy $H=-\sum_{i=1}^{m}p_{i}\cdot \log(p_{i})$. Let us suppose that I can describe the state of my system by a 2D ...
0
votes
0answers
60 views

Identifying argmax result in information theory

I'm reading through a thesis at the moment which uses a "well known result from information theory", namely $\mathrm{argmax}_\mathbf{g}\,\mathbf{f}\,\mathrm{log}\,\mathbf{g} = \mathbf{f}$, for two ...
1
vote
1answer
86 views

how to calculate E[vech(x x')vech(x x')']?

Supposing a vector x follows normal distribution. I want to calculate the expectation of the "fourth moment" in a vector form, meaning $\text{E}[\text{vech}(x x')\text{vech}(x x')']$, given that we ...
1
vote
0answers
57 views

Is it possible to calculate mutual information by moments generating functions?

I went to listen to a workshop and some audience asked the presenter how the moments can improve the mutual information. I am learning the MI(Mutual Information) and moments so don't have enough ...
1
vote
0answers
57 views

MAE/MSE with or without square root

I read some papers about recommender systems and information retrieval, where Mean Absolut Error and Mean Squared Error are mentioned. But I've found some differences between the formal definition of ...