A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

learn more… | top users | synonyms

2
votes
0answers
15 views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
0
votes
0answers
14 views

Measuring entropy of a 2d matrix

In this answer to the question Measuring entropy/ information/ patterns of a 2d binary matrix, the base-2 entropy of the 2-d matrices obtained from a series of moving window sum filters is measured: ...
3
votes
1answer
46 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
0
votes
0answers
15 views

Papers on resource prediction over time series with asymmetric information?

I'm looking for resources to help me solve an issue of resource prediction. We can make a number of observations of the resource over time, but the way in which the resource changes is affected by ...
0
votes
0answers
31 views

How to compute the total entropy of a system of binary outcomes

This is nowhere near my field of expertise, so if my question is poorly formatted for the context please feel free to edit. My question is at the end of the text below. Let's assume I flip a coin 5 ...
1
vote
0answers
20 views

Posterior variance reduction

As detailed on its Wikipedia page, Mutual information, $I(X,Y)$, can be bounded by the Jensen inequality to show that it is always positive. Also, one can show that $$ I(X,Y) = H(X) - H(X|Y). $$ ...
0
votes
0answers
15 views

When can one say two data groups (or more) are inseparable?

There are plenty of measures to compute the separability between groups: the bhattacharya distance, the Jeffries-Matusita distance, the (transformed) divergence and information measures... While those ...
2
votes
2answers
177 views

Calculating AIC “by hand” in R

I have tried calculating the AIC of a linear regression in R but without using the AIC function, like this: ...
0
votes
0answers
14 views

Information theoretic approach to model selection: model diagnostics

I've read a few papers that have used the information theoretic to model selection. One of these papers made 36 candidate models. In the methods section of that paper, the authors mentioned nothing ...
1
vote
0answers
17 views

AICc for small sample sizes

I'm selecting models using the information theoretic approach. I've just read that the AICc should be used to rank candidate models where the number of parameters in a model reaches 30% of the sample ...
0
votes
0answers
28 views

AIC, BIC: Multiple dependent variables

I have multiple candidate models, and each needs to be scored on multiple "metrics", where each metric is probability distribution. Essentially the metrics are like dependent variables. The question ...
4
votes
3answers
61 views

Intuition about a joint entropy

I am having trouble building some intuition about joint entropy. $H(X,Y)$ = uncertainty in the joint distribution $p(x,y)$; $H(X)$ = uncertainty in $p_x(x)$; $H(Y)$ = uncertainty in $p_y(y)$. If ...
0
votes
0answers
41 views

Using QAICc with Poisson, or AIC with Poisson lognormal, in information theoretic approach?

I am trying to use an IT approach to analyse some ecological data. I have a mixed model with nested random effects (I'm using glmer in package lme4 in R). I initially fit the model with a Poisson ...
1
vote
1answer
44 views

Jensen-Shannon divergence for finite samples

I have two finite samples $s_1$ and $s_2$ and two distributions $p_1(s_1)$ and $p_2(s_2)$ that are associated to these samples. I'm essentially interested to measure the distance or similarity between ...
0
votes
0answers
85 views

Using mutual information to estimate correlation between a continuous variable and a categorical variable

As for the title, the idea is to use mutual information, here and after MI, to estimate "correlation" (defined as "how much I know about A when I know B") between a continuous variable and a ...
3
votes
1answer
72 views

Difference between different kinds of entropy

The following concepts are baffling and would be obliged for a constructive explanation. (Q1) What is the conceptual difference between (a) Kolmogorov-Sinai entropy, (b) Shannon entropy, (c) Source ...
1
vote
1answer
43 views

Renyi divergence identity

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think ...
2
votes
1answer
49 views

Qualitively what is Cross Entropy

This question gives a quantitative definition of cross entropy, in terms of it's formula. I'm looking for a more notional definition, wikipedia says: In information theory, the cross entropy ...
0
votes
0answers
21 views

how to calculate information value [duplicate]

i have the following tree Outlook( [Sunny] => (Yes, Yes, No, No, No), [Overcast] => (Yes, Yes, Yes, Yes), [Rainy] => (Yes, Yes, Yes, No, No) ) ...
0
votes
0answers
38 views

Information entropy of a mixture distribution

I'm working with a high-dimensional mixture distribution, and I'm interested in calculating its entropy. I think I could work it out if there were only two mixture components. Following @Daniel's ...
2
votes
1answer
38 views

Information theoretic alternative to hypergeometric test

Is there any information-theoretic alternative to the hypergeometric test (or Fisher's exact test)? In other word, is it possible to calculate AIC values for data which one would classically analyse ...
1
vote
0answers
34 views

Information theoretic substitute for precision and recall?

Is there a substitute for measuring precision, recall of a classifier (binary or multi-class) to evaluate its performance using information-theoretic quantities like entropy, mutual information or ...
1
vote
0answers
49 views

Orthogonality of parameter space

I have a basic information theory question. I am fitting a highly parameterized model to some data. In general: $$ y = \sum\limits_{i=1}^{13} \alpha_i X_i $$ Currently I use gradient descent to find ...
1
vote
1answer
38 views

When is the differential entropy negative?

The definition of entropy for a continuous signal is: $h[f] = \operatorname{E}[-\ln (f(x))] = -\int\limits_{-\infty}^{\infty} f(x) \ln (f(x))\, dx$ According to Wikipedia, it can be negative. When ...
0
votes
1answer
123 views

conditional mutual information and how to deal with zero probabilities

the conditional mutual between three sets of mutually exclusive variables, X, Y, and Z, is defined as follows. $I(X,Y|Z) = \sum_{xyz} P(x,y,z) \log \frac{P(z)P(x,y,z)}{P(x,z)P(y,z)}$ my questions ...
3
votes
1answer
129 views

Distance or Similarity metric for 2D frequency data maps

I want to compare the distance/similarity of 2D flood frequency data maps. The maps are square with YxY grid size and in each cell of the map is stored its flood frequency. For example in a 5x5 grid ...
2
votes
1answer
200 views

KL-divergence between two categorical/multinomial distributions gives negative values?

If $$P = [0,0.9,0,0.1]$$ $$Q = [0,1,0,0]$$ Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$ This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?
1
vote
2answers
78 views

Testing whether small Eigenvalues produce a “signal”

My problem is similar to this one but I am looking for a different solution: (so if it should be merged just let me know). Measuring what's 'lost' in PCA dimensionality reduction? I my ...
2
votes
2answers
104 views

Maximum entropy and non-informative distribution

Is Maximum Entropy rule equivalent to non-informativeness? In other words, when maximizing the entropy of a distribution, given some known stuff, is it equivalent to finding to most non-informative ...
2
votes
0answers
81 views

Regression forest: Leaf node and information gain

Regression forests are basically random forests, however used for regression. They basically use the same framework as decision forests use for classification with a few parts exchanged. Two of these ...
2
votes
1answer
170 views

Entropy of (Sum of Gaussians) versus Sum of (Entropy of Gaussians)

Short version: How can the joint entropy of two independent variables be less than the sum of those independent variables? The joint entropy should encode all information that a scalar function can, ...
5
votes
3answers
172 views

Does dimension reduction always lose some information?

Like the title says, does dimension reduction always lose some information? Consider for example PCA. If the data I have is very sparse, I would assume a "better encoding" could be found (is this ...
0
votes
1answer
72 views

Measures of entropy/information: distinguish clustered configurations that would have the same information entropy

Let us consider the configuration of a 2D system and the standard definition of entropy $H=-\sum_{i=1}^{m}p_{i}\cdot \log(p_{i})$. Let us suppose that I can describe the state of my system by a 2D ...
0
votes
0answers
56 views

Identifying argmax result in information theory

I'm reading through a thesis at the moment which uses a "well known result from information theory", namely $\mathrm{argmax}_\mathbf{g}\,\mathbf{f}\,\mathrm{log}\,\mathbf{g} = \mathbf{f}$, for two ...
1
vote
1answer
81 views

how to calculate E[vech(x x')vech(x x')']?

Supposing a vector x follows normal distribution. I want to calculate the expectation of the "fourth moment" in a vector form, meaning $\text{E}[\text{vech}(x x')\text{vech}(x x')']$, given that we ...
1
vote
0answers
51 views

Is it possible to calculate mutual information by moments generating functions?

I went to listen to a workshop and some audience asked the presenter how the moments can improve the mutual information. I am learning the MI(Mutual Information) and moments so don't have enough ...
1
vote
0answers
51 views

MAE/MSE with or without square root

I read some papers about recommender systems and information retrieval, where Mean Absolut Error and Mean Squared Error are mentioned. But I've found some differences between the formal definition of ...
4
votes
0answers
76 views

Orthogonal intersection in a Riemannian manifold

Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where ...
3
votes
1answer
125 views

Orthogonal intersection of linear family and exponential family

I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question. In statistical maniolds ...
1
vote
0answers
35 views

Bandits without exploitation: finding the best items with incomplete information

I'm trying to analyze a general game. This is probably well-known, in which case pointers to relevant literature would suffice (but explanation would not be declined!). If it's not standard, of course ...
2
votes
1answer
112 views

Pinsker's Inequality for Bayesian hypothesis testing

I am wondering if one can relate the KL divergence to the probability of error in a Bayesian binary hypothesis testing setting. That is, we have to decide between hypotheses $A$ and $B$ given ...
3
votes
0answers
96 views

Comparing term-frequency distributions with unequal sample sizes?

Background I have several datasets of word frequencies where some datasets have much more data than others: from 3000 samples to 20000 samples. I also have large reference corpora with millions of ...
5
votes
1answer
249 views

Differential Entropy

Differential entropy of Gaussian R.V. is $\log_2(\sigma \sqrt{2\pi e})$. This is dependent on $\sigma$, which is the standard deviation. If we normalize the random variable so that it has unit ...
1
vote
1answer
204 views

Mutual Information really invariant to invertible transformations?

"Estimating mutual information" [A Kraskov, H Stögbauer, P Grassberger - Physical Review E, 2004] states that Mutual information is invariant under reparametrization of the marginal variables. ...
1
vote
2answers
215 views

How can I use KL-divergence to weight features?

I have a naive Bayes classifier with two classes (target and non-target) and distributions for a number of features (the same for both classes). I know that some features contribute more, or less to ...
2
votes
1answer
193 views

How, and should I, use KL-divergence to improve a naive Bayes classifier?

How, and should I, use KL-divergence to improve a naive Bayes classifier? I have a Naive Bayes Classifier working on a number of datatypes (real, boolean and categorical). Each variable is weighted ...
2
votes
0answers
62 views

Multi-information of a uniformly distributed random variable on the L1 sphere

I posted this question in the stackexchange mathematics forum without any reponse. Maybe it was the wrong forum, so I try it here. I tried to compute the multi-information (MI) $I[\mathbf U] = ...
3
votes
1answer
284 views

ID3 and C4.5: how does “gain ratio” normalize “gain”?

The ID3 algorithm uses "Information Gain" measure. The C4.5 uses "Gain Ratio" measure which is Information Gain divided by SplitInfo, whereas SplitInfo is high for a split where records split evenly ...
1
vote
0answers
164 views

Conditional entropy of sum of random variables

How can be proven that for random variables $A$ and $B$, and $C = A + B$, $$H(C\mid A) = H(B\mid A).$$ Also, would it be possible to determine if $H(C)$ would be greater than $H(A)$?
4
votes
2answers
217 views

Number of needed samples for entropy estimation

I would like to estimate the entropy of a source generating binary vectors of length M which are very sparse (just a few 1s), using the naive (empirical) estimator ...