A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

learn more… | top users | synonyms

0
votes
0answers
10 views

Shape uncertainty of a 3D point cloud

Given a point cloud of a 3D object, how to calculate the shape uncertainty in this discrete sample set? and what factors maximize or minimize this uncertainty?
2
votes
0answers
9 views

High mutual information = (almost) deterministic relationship?

If two random variables $X, Y$ have high mutual information $I(X;Y)$, intuitively does that mean $X,Y$ have almost deterministic relationships, say $Y=f(X)+\epsilon$ where $\epsilon$ is a noise random ...
0
votes
0answers
18 views

Interpreting the inverse covariance matrix: $S^{-1}x$ and $x^T S^{-1}x$

Let $S$ be the covariance matrix of some data set. $S^{-1}$ is the inverse covariance matrix, also called the precision matrix. Question: In practice, then, what does $S^{-1}x$ mean for a data point ...
0
votes
0answers
14 views

lower bound on square root of KL divergence

$\mu_1$ and $\mu_2$ are two probability measures on $\Omega$. Assume they have probability mass functions $f_1$ and $f_2$. I wonder if the following is a correct usage of Jensen' inequality: $$ ...
2
votes
2answers
58 views

Feature Selection: Information Gain VS Mutual Information

Setting: Multi-class classification problem with discrete nominal features. There are many references mentioning the use of IG(Information Gain) and ...
2
votes
1answer
38 views

Difference between expectations of the same random variable wrt different probability measures

$X$ is a measurable mapping from a discrete sample space $\Omega$ to $\mathbb R$. $\mu_1$ and $\mu_2$ are two probability measures on $\Omega$. Assume they have probability mass functions $f_1$ and ...
0
votes
2answers
27 views

Entropy of a block of characters

I have a question about the following statement about entropy: If a source provides us with a sequence chosen from 4 symbols (say A, C, G, T), then the maximum average information per symbol is 2 ...
0
votes
0answers
18 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
0
votes
0answers
11 views

How to assess information content / mutual information between two sets of variables ?

Am testing data from a clinical device, data was extracted using two complementary methods ...
7
votes
1answer
107 views

Determinant of Fisher information

(I posted a similar question on math.se.) In information geometry, the determinant of the Fisher information matrix is a natural volume form on a statistical manifold, so it has a nice geometrical ...
0
votes
3answers
94 views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
3
votes
1answer
66 views

Conceptual question on mutual information and entropy

What does mutual information (MI) convey? Looking for good reference books on information theory
1
vote
1answer
41 views

Conceptual questions on Entropy and estimation

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in ...
0
votes
1answer
27 views

do information entropy probabilities have to sum to one?

My understanding of information entropy is that it requires the input probabilities to sum to 1. So, for a sequence a,a,b,b you then have -([1/2 log2 1/2] + [1/2 log2 1/2]) = 1 Are there versions of ...
0
votes
0answers
20 views

Upper Bound on Mutual Information

I was advised to post this question here rather than in the math stack exchange. So here it is: I am interested in an upper bound on mutual information that I have been encountering frequently in the ...
1
vote
0answers
82 views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
1
vote
1answer
33 views

Information gain is KL divergence

I am slightly confused by the the statement that Kullback–Leibler divergence is the same as [information gain](Information gain in decision trees). I cannot understand how $D_{KL}(P||Q) = H(P,Q)- ...
1
vote
1answer
60 views

Calculating Perplexity

In the Coursera NLP course , Dan Jurafsky calculates the following perplexity: Operator(1 in 4) Sales(1 in 4) Technical Support(1 in 4) 30,000 names(1 in 120,000 each) He says the Perplexity is 53. ...
0
votes
1answer
86 views

Relationship between entropy and information gain

Based on papers :1. Deniz Erdogmus, Member, IEEE, and Jose C. Principe, An Error-Entropy Minimization Algorithm for Supervised Training of Nonlinear Adaptive Systems J. Principe, D. Xu, and J. ...
0
votes
1answer
47 views

Entropy and information content

I am curious to know about the relation between Entropy and information content of a signal or trajectory of time series. When a system is at equilibrium, then it has maximum entropy. Does entropy ...
1
vote
1answer
44 views

Feature selection based on information gain papers

I want to apply feature selection based on information gain: I have many features many of which are redundant. I am planning on selecting a feature and then iteratively add features that 'add the more ...
7
votes
2answers
327 views

Assessing principal components analysis

What is a good metric for assessing the quality of a pca? I performed this algorithm on a dataset. My objective was to reduce the number of features (the information was very redundant). I know the ...
0
votes
0answers
37 views

information theoretic model selection

I have some questions on the information theoretic approach to linear modelling: Why not just use residuals instead of likelihood? Can same variables with different subsets of data be used to build ...
10
votes
2answers
251 views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
3
votes
1answer
112 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
0
votes
0answers
15 views

Papers on resource prediction over time series with asymmetric information?

I'm looking for resources to help me solve an issue of resource prediction. We can make a number of observations of the resource over time, but the way in which the resource changes is affected by ...
0
votes
0answers
41 views

How to compute the total entropy of a system of binary outcomes

This is nowhere near my field of expertise, so if my question is poorly formatted for the context please feel free to edit. My question is at the end of the text below. Let's assume I flip a coin 5 ...
1
vote
0answers
39 views

Posterior variance reduction

As detailed on its Wikipedia page, Mutual information, $I(X,Y)$, can be bounded by the Jensen inequality to show that it is always positive. Also, one can show that $$ I(X,Y) = H(X) - H(X|Y). $$ ...
0
votes
0answers
17 views

When can one say two data groups (or more) are inseparable?

There are plenty of measures to compute the separability between groups: the bhattacharya distance, the Jeffries-Matusita distance, the (transformed) divergence and information measures... While those ...
2
votes
2answers
732 views

Calculating AIC “by hand” in R

I have tried calculating the AIC of a linear regression in R but without using the AIC function, like this: ...
0
votes
0answers
22 views

Information theoretic approach to model selection: model diagnostics

I've read a few papers that have used the information theoretic to model selection. One of these papers made 36 candidate models. In the methods section of that paper, the authors mentioned nothing ...
1
vote
0answers
31 views

AICc for small sample sizes

I'm selecting models using the information theoretic approach. I've just read that the AICc should be used to rank candidate models where the number of parameters in a model reaches 30% of the sample ...
0
votes
0answers
47 views

AIC, BIC: Multiple dependent variables

I have multiple candidate models, and each needs to be scored on multiple "metrics", where each metric is probability distribution. Essentially the metrics are like dependent variables. The question ...
4
votes
3answers
86 views

Intuition about a joint entropy

I am having trouble building some intuition about joint entropy. $H(X,Y)$ = uncertainty in the joint distribution $p(x,y)$; $H(X)$ = uncertainty in $p_x(x)$; $H(Y)$ = uncertainty in $p_y(y)$. If ...
0
votes
0answers
83 views

Using QAICc with Poisson, or AIC with Poisson lognormal, in information theoretic approach?

I am trying to use an IT approach to analyse some ecological data. I have a mixed model with nested random effects (I'm using glmer in package lme4 in R). I initially fit the model with a Poisson ...
1
vote
1answer
140 views

Jensen-Shannon divergence for finite samples

I have two finite samples $s_1$ and $s_2$ and two distributions $p_1(s_1)$ and $p_2(s_2)$ that are associated to these samples. I'm essentially interested to measure the distance or similarity between ...
1
vote
0answers
353 views

Using mutual information to estimate correlation between a continuous variable and a categorical variable

As for the title, the idea is to use mutual information, here and after MI, to estimate "correlation" (defined as "how much I know about A when I know B") between a continuous variable and a ...
4
votes
1answer
81 views

Difference between different kinds of entropy

The following concepts are baffling and would be obliged for a constructive explanation. (Q1) What is the conceptual difference between (a) Kolmogorov-Sinai entropy, (b) Shannon entropy, (c) Source ...
1
vote
1answer
54 views

Renyi divergence identity

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think ...
2
votes
1answer
78 views

Qualitively what is Cross Entropy

This question gives a quantitative definition of cross entropy, in terms of it's formula. I'm looking for a more notional definition, wikipedia says: In information theory, the cross entropy ...
0
votes
0answers
22 views

how to calculate information value [duplicate]

i have the following tree Outlook( [Sunny] => (Yes, Yes, No, No, No), [Overcast] => (Yes, Yes, Yes, Yes), [Rainy] => (Yes, Yes, Yes, No, No) ) ...
0
votes
0answers
49 views

Information entropy of a mixture distribution

I'm working with a high-dimensional mixture distribution, and I'm interested in calculating its entropy. I think I could work it out if there were only two mixture components. Following @Daniel's ...
2
votes
1answer
44 views

Information theoretic alternative to hypergeometric test

Is there any information-theoretic alternative to the hypergeometric test (or Fisher's exact test)? In other word, is it possible to calculate AIC values for data which one would classically analyse ...
2
votes
0answers
35 views

Information theoretic substitute for precision and recall?

Is there a substitute for measuring precision, recall of a classifier (binary or multi-class) to evaluate its performance using information-theoretic quantities like entropy, mutual information or ...
1
vote
0answers
59 views

Orthogonality of parameter space

I have a basic information theory question. I am fitting a highly parameterized model to some data. In general: $$ y = \sum\limits_{i=1}^{13} \alpha_i X_i $$ Currently I use gradient descent to find ...
1
vote
1answer
56 views

When is the differential entropy negative?

The definition of entropy for a continuous signal is: $h[f] = \operatorname{E}[-\ln (f(x))] = -\int\limits_{-\infty}^{\infty} f(x) \ln (f(x))\, dx$ According to Wikipedia, it can be negative. When ...
0
votes
1answer
211 views

conditional mutual information and how to deal with zero probabilities

the conditional mutual between three sets of mutually exclusive variables, X, Y, and Z, is defined as follows. $I(X,Y|Z) = \sum_{xyz} P(x,y,z) \log \frac{P(z)P(x,y,z)}{P(x,z)P(y,z)}$ my questions ...
3
votes
1answer
181 views

Distance or Similarity metric for 2D frequency data maps

I want to compare the distance/similarity of 2D flood frequency data maps. The maps are square with YxY grid size and in each cell of the map is stored its flood frequency. For example in a 5x5 grid ...
3
votes
1answer
373 views

KL-divergence between two categorical/multinomial distributions gives negative values?

If $$P = [0,0.9,0,0.1]$$ $$Q = [0,1,0,0]$$ Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$ This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?
1
vote
2answers
89 views

Testing whether small Eigenvalues produce a “signal”

My problem is similar to this one but I am looking for a different solution: (so if it should be merged just let me know). Measuring what's 'lost' in PCA dimensionality reduction? I my ...