A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

learn more… | top users | synonyms

2
votes
0answers
19 views

Numerical Approxmation of standard errors for parameter estimation in the EM algorithm

Generally, when you want to compute standard errors for estimated parameters within the ML framework, one uses the diagonal elements of the observed information matrix. In for instance ...
-2
votes
0answers
10 views

What is the relationship between correlation and coherence?

What is the relationship between correlation (autocorrelation, cross-correlation, etc) and coherence?
1
vote
0answers
11 views

lower bound on conditional differential entropy

I was wondering if there exists a lower bound of conditional differential entropy of Gaussian random variables. Formally, let $\mathcal{X}=\{X_1,\ldots,X_m\}$ and $\mathcal{Y}=\{Y_1,\ldots,Y_n\}$ ...
1
vote
0answers
18 views

How to aproach a presentation of information theory without solid background

I study first year of an applied statistics bachelor, in one of my courses i have to do a presentation of information theory and i would like to share with you my ideas about it in order that you ...
2
votes
1answer
16 views

Information content of a set of random variables

Suppose there is some distribution $F$ not known to us. However, we can get information about this distribution by means of samples, i.e. we have a set of random variables from this distribution. ...
0
votes
0answers
24 views

Information gain of flipping coins

I'm studying for an exam on Bishop's Pattern Recognition and Machine Learning (ISBN: 978- 0387310732). One of the questions in the mock exam paper is: Three fair coins are flipped sequentially; you ...
1
vote
0answers
16 views

entropy-calculation in R [closed]

Is there anyone to have used the package "entropy" ? How can I create a vector of counts if my sample is large (2000 points)? I tried creating a vector of counts using table(), is there a more ...
0
votes
0answers
13 views

negative values for frequencies, in package entropy in R [migrated]

Using the package "entropy" in R, I have a vector y with decimal points and positive and negative values. Calculating the frequencies, I get positive and negative values corresponding to the ...
0
votes
0answers
18 views

Variable selection via mutual information

I am building a SVM model and was wondering whether filtering variables based on mutual information between response variable(binary 0/1 variable)is a good approach? Also, would the condinformation ...
2
votes
1answer
48 views

Information criterion for selecting sample size when modeling tails

I want to model the left tail of an unknown distribution with a Generalized Pareto distribution. Somehow I have to select how much of the tail to model. I am wondering if it is possible to create an ...
3
votes
0answers
25 views

Expected ratio of probabilities--is there a term for it?

I recently came across the following quantity when I played around with some information theoretic quantities and Bayesian learning. Given three probability distributions $q(z), p(z)$ and $p(z|x)$. ...
0
votes
1answer
30 views

How much prediction accuracy of SVM (or other ML models) depend on the way features are encoded?

Suppose that for a given ML problem, we have a feature which car the person possesses. We can encode this information in one of the following ways: Assign an id to each of the car. Make a column ...
2
votes
0answers
49 views

Use of information theory in applied data science

Today I ran across the book "Information theory: A tutorial introduction" by James Stone and thought for a moment or two about the extent of use of information theory in applied data science (if ...
0
votes
2answers
32 views

Clean data and noisy data: which one has higher entropy?

I have a question regarding to the information theory. Between clean data and noisy data, which one has higher entropy? I think the noisy data has, am I right? But, noisy data does not have more ...
0
votes
1answer
19 views

Transfer entropy on real, continuous data

Here is the goal and problem: I am trying to calculate a measure of coupling between real-valued, continuous oscillatory data. The data come from two people producing synchronized rhythmic ...
1
vote
1answer
28 views

How does the log(p(x,y)) normalize the pointwise mutual information?

I'm trying to understand the normalized form of pointwise mutual information. $npmi = \frac{pmi(x,y)}{log(p(x,y))}$ Why does the log joint probability normalize the pointwise mutual information to ...
0
votes
0answers
55 views

Calculating the mutual information between two histograms

I've been set a sample exercise by my supervisor, and I'm totally lost as to where I should be heading. What I've been tasked with is to generate two histograms that approximate Gaussian PDFs. Then, ...
1
vote
0answers
23 views

Are wrong standard errors a problem if using information theoretic model selection?

In linear regression, if the assumptions of normally distributed residuals and homogenous residuals are broken, incorrect standard errors can be calculated. This can lead to some predictors appearing ...
5
votes
1answer
81 views

Asymmetric measure of non-linear dependence/correlation?

I am definitely not a statistician/mathematician so feel free to tell me I'm an idiot if I am. As far as I can tell from my Wikipediaing all of the main measures of dependence are symmetric and ...
18
votes
2answers
863 views

Differences between Bhattacharyya distance and KL divergence

I'm looking for an intuitive explanation for the following questions: In statistics and information theory, what's the difference between Bhattacharyya distance and KL divergence, as measures of the ...
1
vote
1answer
69 views

How to find the perplexity of a corpus

The formula of the perplexity measure is: $$p: \left(\frac{1}{\sqrt[n]{p(w_1^n)}}\right)$$ where: $p(w_1^n)$ is: $\prod_{i=1}^n p(w_i)$. If I understand it correctly, this means that I could ...
2
votes
1answer
73 views

Information theory without normalization

I'd like to know if there is a way anyone knows of for doing information theory with unnormalized densities. Specifically, I hav two log likelihoods $\phi(x), \psi(x)$ and so I can write: $p(x) = ...
0
votes
0answers
16 views

Derivation of description length

In class, our professor posted the following: We will discretize $\theta$ (some model) into $1/\sqrt{n}$ distinct values. Intuitive argument: with $N$ data points, our estimation error for $\hat ...
0
votes
0answers
54 views

Mean Average Precision vs Mean Reciprocal Rank

I am trying to understand when it is appropriate to use the MAP and when MRR should be used. I found this presentation that states that MRR is best utilised when the number of relevant results is less ...
0
votes
1answer
34 views

Similarity datasets

Given two (or maybe more) datasets with the same samples/members, but with different variables. Is there a general way to compare the information available in the two datasets without looking into the ...
0
votes
1answer
30 views

Fisher information always 0? What's wrong with this argument?

We exchange derivative and integral while proving that the expected value of the score is 0. Here is the proof which does so: \begin{align*} \int \left( \frac{\partial}{\partial \theta} \log ...
0
votes
0answers
45 views

Shape uncertainty of a 3D point cloud

Given a point cloud of a 3D object, how to calculate the shape uncertainty in this discrete sample set? and what factors maximize or minimize this uncertainty?
2
votes
0answers
14 views

High mutual information = (almost) deterministic relationship?

If two random variables $X, Y$ have high mutual information $I(X;Y)$, intuitively does that mean $X,Y$ have almost deterministic relationships, say $Y=f(X)+\epsilon$ where $\epsilon$ is a noise random ...
0
votes
0answers
26 views

Interpreting the inverse covariance matrix: $S^{-1}x$ and $x^T S^{-1}x$

Let $S$ be the covariance matrix of some data set. $S^{-1}$ is the inverse covariance matrix, also called the precision matrix. Question: In practice, then, what does $S^{-1}x$ mean for a data point ...
0
votes
0answers
29 views

lower bound on square root of KL divergence

$\mu_1$ and $\mu_2$ are two probability measures on $\Omega$. Assume they have probability mass functions $f_1$ and $f_2$. I wonder if the following is a correct usage of Jensen' inequality: $$ ...
2
votes
2answers
1k views

Feature Selection: Information Gain VS Mutual Information

Setting: Multi-class classification problem with discrete nominal features. There are many references mentioning the use of IG(Information Gain) and ...
3
votes
1answer
62 views

Difference between expectations of the same random variable wrt different probability measures

$X$ is a measurable mapping from a discrete sample space $\Omega$ to $\mathbb R$. $\mu_1$ and $\mu_2$ are two probability measures on $\Omega$. Assume they have probability mass functions $f_1$ and ...
0
votes
2answers
33 views

Entropy of a block of characters

I have a question about the following statement about entropy: If a source provides us with a sequence chosen from 4 symbols (say A, C, G, T), then the maximum average information per symbol is 2 ...
0
votes
0answers
39 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
0
votes
0answers
28 views

How to assess information content / mutual information between two sets of variables ?

Am testing data from a clinical device, data was extracted using two complementary methods ...
7
votes
1answer
321 views

Determinant of Fisher information

(I posted a similar question on math.se.) In information geometry, the determinant of the Fisher information matrix is a natural volume form on a statistical manifold, so it has a nice geometrical ...
0
votes
3answers
180 views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
1
vote
1answer
95 views

Conceptual question on mutual information and entropy

What does mutual information (MI) convey? Looking for good reference books on information theory
1
vote
1answer
48 views

Conceptual questions on Entropy and estimation

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in ...
0
votes
1answer
64 views

Do information entropy probabilities have to sum to one?

My understanding of information entropy is that it requires the input probabilities to sum to 1. So, for a sequence a,a,b,b you then have $$- \left(\frac12 \log_2 \frac12 + \frac12 \log_2 ...
0
votes
0answers
39 views

Upper Bound on Mutual Information

I was advised to post this question here rather than in the math stack exchange. So here it is: I am interested in an upper bound on mutual information that I have been encountering frequently in the ...
1
vote
0answers
180 views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
1
vote
1answer
45 views

Information gain is KL divergence

I am slightly confused by the the statement that Kullback–Leibler divergence is the same as [information gain](Information gain in decision trees). I cannot understand how $D_{KL}(P||Q) = H(P,Q)- ...
2
votes
1answer
101 views

Calculating Perplexity

In the Coursera NLP course , Dan Jurafsky calculates the following perplexity: Operator(1 in 4) Sales(1 in 4) Technical Support(1 in 4) 30,000 names(1 in 120,000 each) He says the Perplexity is 53. ...
0
votes
1answer
184 views

Relationship between entropy and information gain

Based on papers :1. Deniz Erdogmus, Member, IEEE, and Jose C. Principe, An Error-Entropy Minimization Algorithm for Supervised Training of Nonlinear Adaptive Systems J. Principe, D. Xu, and J. ...
1
vote
1answer
93 views

Entropy and information content

I am curious to know about the relation between Entropy and information content of a signal or trajectory of time series. When a system is at equilibrium, then it has maximum entropy. Does entropy ...
1
vote
1answer
52 views

Feature selection based on information gain papers

I want to apply feature selection based on information gain: I have many features many of which are redundant. I am planning on selecting a feature and then iteratively add features that 'add the more ...
9
votes
2answers
370 views

What are good metrics to assess the quality of a PCA fit, in order to select the number of components?

What is a good metric for assessing the quality of principal component analysis (PCA)? I performed this algorithm on a dataset. My objective was to reduce the number of features (the information was ...
0
votes
0answers
41 views

information theoretic model selection

I have some questions on the information theoretic approach to linear modelling: Why not just use residuals instead of likelihood? Can same variables with different subsets of data be used to build ...
10
votes
2answers
565 views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...